Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #79440 > unrolled thread

Bug#1039883: linux-image-6.3.0-1-amd64: ext4 corruption with symlinks

Back to article view | Back to linux.debian.kernel

Contents

  Bug#1039883: linux-image-6.3.0-1-amd64: ext4 corruption with symlinks dud225 <dud225@hotmail.com> - 2023-06-29 10:40 +0200
    Processed: Re: Bug#1039883: The issue impacts SSD disks as well "Debian Bug Tracking System" <owner@bugs.debian.org> - 2023-07-24 11:30 +0200
    Bug#1039883: The issue impacts SSD disks as well Salvatore Bonaccorso <carnil@debian.org> - 2023-07-24 11:30 +0200
    Bug#1039883: The issue impacts SSD disks as well Emanuele Rocca <ema@debian.org> - 2024-05-30 14:50 +0200
    Bug#1039883: linux: ext4 corruption with symlinks Luis Henriques <luis.henriques@linux.dev> - 2024-06-14 18:30 +0200
      Bug#1039883: linux: ext4 corruption with symlinks Luis Henriques <luis.henriques@linux.dev> - 2024-06-18 12:10 +0200
        Bug#1039883: linux: ext4 corruption with symlinks Luis Henriques <luis.henriques@linux.dev> - 2024-06-18 15:30 +0200
    Bug#1039883: [PATCH] ext4: don't track ranges in fast_commit if inode has inlined data "Luis Henriques (SUSE)" <luis.henriques@linux.dev> - 2024-06-19 00:40 +0200
    Processed: Re: linux: ext4 corruption with symlinks "Debian Bug Tracking System" <owner@bugs.debian.org> - 2024-07-09 19:40 +0200
    Bug#1039883: linux: ext4 corruption with symlinks Ben Hutchings <ben@decadent.org.uk> - 2024-07-31 21:00 +0200
    Bug#1039883: marked as done (linux: ext4 corruption with symlinks) "Debian Bug Tracking System" <owner@bugs.debian.org> - 2024-08-05 08:10 +0200

#79440 — Bug#1039883: linux-image-6.3.0-1-amd64: ext4 corruption with symlinks

Package: linux-image-6.3.0-1-amd64
Version: linux-image-6.3.0-1-amd64
Severity: important
Tags: upstream
X-Debbugs-Cc: dud225@hotmail.com

Hello

I've stored data on a USB external hard drive using ext4 over LUKS2 and I'm getting the following error:
	kernel: EXT4-fs error (device dm-11): ext4_map_blocks:607: inode #8159552: block 959787320: comm git-annex:w: lblock 0 mapped>

I then stumbled upon that kernel bug [1] which matches my case as git-annex is making heavy use of symlinks. However I've faced this issue on the kernel 6.3.0 (linux-image-6.3.0-1-amd64 version 6.3.7-1) so it doesn't look to be actually addressed.

I've got 3 disks, 2 HDDs and 1 SSD, and oddly the failure only happens on the HDD, the SSD is running fine.
After reformatting my HDD without the inline_data feature, the issue has disappeared.

[1] https://bugzilla.kernel.org/show_bug.cgi?id=216317

[toc] | [next] | [standalone]

#79735 — Processed: Re: Bug#1039883: The issue impacts SSD disks as well

Processing control commands:

> tags -1 + moreinfo
Bug #1039883 [src:linux] linux-image-6.3.0-1-amd64: ext4 corruption with symlinks
Ignoring request to alter tags of bug #1039883 to the same tags previously set

-- 
1039883: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1039883
Debian Bug Tracking System
Contact owner@bugs.debian.org with problems

[toc] | [prev] | [next] | [standalone]

#79736 — Bug#1039883: The issue impacts SSD disks as well

Control: tags -1 + moreinfo

Hi,

[Ted, Andreas, context in https://bugs.debian.org/1039883]

On Sun, Jul 02, 2023 at 09:14:50PM +0000, Hervé Werner wrote:
> I've just faced this issue on the SSD disk as well, so it seems that
> the probability is just lower on a speedier disk.

Are you able to reliably preoeduce the issue and can bisect it to the
introducing commit? 

Can you retest please with recent kernel, 6.3.11-1 and ideally 6.4.4-1
as recently uploaded to unstable?

Regards,
Salvatore

[toc] | [prev] | [next] | [standalone]

#82626 — Bug#1039883: The issue impacts SSD disks as well

Hey,

On 2023-11-05 04:12, Hervé Werner wrote:
> I faced this issue on real data but I struggled to find a reliable scenario to reproduce it. Here is what I just came up with:
>   sudo mkfs -t ext4 -O fast_commit,inline_data /dev/sdb
>   sudo mount /dev/sdb /mnt/
>   sudo install -d -o myuser /mnt/annex
>   cd /mnt/annex
>   git init && git annex init
>   for i in {1..2}; do
>     for i in {1..10000}; do
>       dd if=/dev/urandom of=file-${i} bs=1K count=1 2>/dev/null
>     done
>     git annex add -J cpus . >/dev/null && git annex sync -J cpus && git annex fsck -J cpus >/dev/null
>     git rm * && git annex sync  && git annex dropunused all
>   done
> 
> Then at some point the following error appears:
>   EXT4-fs error (device sdb): ext4_map_blocks:577: inode #3942343: block 4: comm git-annex:w: lblock 1 mapped to illegal pblock 4 (length 1)

Just a quick note to confirm that I can reliably reproduce the issue
using a USB stick and the above script on Bookworm. After running the
reproducer for a few minutes I start getting the following in dmesg:

kernel: EXT4-fs error (device sdf): ext4_map_blocks:607: inode #9971675: block 4: comm git-annex:w: lblock 1 mapped to illegal pblock 4 (length 1)

uname -a is:
Linux ariel 6.1.0-21-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.90-1 (2024-05-03) x86_64 GNU/Linux

[toc] | [prev] | [next] | [standalone]

#82724 — Bug#1039883: linux: ext4 corruption with symlinks

On Mon 10 Jun 2024 06:03:58 PM +02, Ben Hutchings wrote;

> On Sun, 5 Nov 2023 16:12:41 +0000 Hervé Werner <dud225@hotmail.com>
> wrote:
>> Hello
>> 
>> I'm sorry for the delay.
>> 
>> > Are you able to reliably preoeduce the issue and can bisect it to
>> > the introducing commit?
>> I faced this issue on real data but I struggled to find a reliable
>> scenario to reproduce it. Here is what I just came up with:
>>   sudo mkfs -t ext4 -O fast_commit,inline_data /dev/sdb
>>   sudo mount /dev/sdb /mnt/
>>   sudo install -d -o myuser /mnt/annex
>>   cd /mnt/annex
>>   git init && git annex init
>>   for i in {1..2}; do
>>     for i in {1..10000}; do
>>       dd if=/dev/urandom of=file-${i} bs=1K count=1 2>/dev/null
>>     done
>>     git annex add -J cpus . >/dev/null && git annex sync -J cpus && git annex fsck -J cpus >/dev/null
>>     git rm * && git annex sync  && git annex dropunused all
>>   done
>> 
>> Then at some point the following error appears:
>>   EXT4-fs error (device sdb): ext4_map_blocks:577: inode #3942343: block 4: comm git-annex:w: lblock 1 mapped to illegal pblock 4 (length 1)
> [...]
>
> I can also reproduce this error message using the above script and:
>
> - Linux 6.10-rc2
> - A 2 GiB loopback devic instead of /dev/sdb
>
> I bisected this back to:
>
> commit 9725958bb75cdfa10f2ec11526fdb23e7485e8e4
> Author: Xin Yin <yinxin.x@bytedance.com>
> Date:   Thu Dec 23 11:23:37 2021 +0800
>  
>     ext4: fast commit may miss tracking unwritten range during ftruncate
>
> It is still possible to cleanly revert that commit from 6.10-rc2, and
> doing so removes the error message.

Because I recently fixed an issue in the fast commit code[1] I was hoping
that you were hitting the same bug.  I've executed the reproducer with the
fix (which hasn't been merged yet) and realised it's definitely a
different problem.

Debugged the issue a bit, it seems to be related with the fact that
ext4_fc_write_inode_data() isn't able to cope with the fact that
'ei->i_fc_lblk_len' is set to EXT_MAX_BLOCKS.

I'm CC'ing Harshad, maybe he has some idea.

[1] https://lore.kernel.org/all/20240529092030.9557-2-luis.henriques@linux.dev

Cheers,
-- 
Luís

[toc] | [prev] | [next] | [standalone]

#82744 — Bug#1039883: linux: ext4 corruption with symlinks

On Fri 14 Jun 2024 05:18:45 PM +01, Luis Henriques wrote;
[...}
>>
>> I can also reproduce this error message using the above script and:
>>
>> - Linux 6.10-rc2
>> - A 2 GiB loopback devic instead of /dev/sdb
>>
>> I bisected this back to:
>>
>> commit 9725958bb75cdfa10f2ec11526fdb23e7485e8e4
>> Author: Xin Yin <yinxin.x@bytedance.com>
>> Date:   Thu Dec 23 11:23:37 2021 +0800
>>  
>>     ext4: fast commit may miss tracking unwritten range during ftruncate
>>
>> It is still possible to cleanly revert that commit from 6.10-rc2, and
>> doing so removes the error message.
>
> Because I recently fixed an issue in the fast commit code[1] I was hoping
> that you were hitting the same bug.  I've executed the reproducer with the
> fix (which hasn't been merged yet) and realised it's definitely a
> different problem.
>
> Debugged the issue a bit, it seems to be related with the fact that
> ext4_fc_write_inode_data() isn't able to cope with the fact that
> 'ei->i_fc_lblk_len' is set to EXT_MAX_BLOCKS.

OK, I've looked into this again.  And something I didn't pay attention
before was that the filesystem was created with both fast_commit *and*
inline_data features.  And after some more debugging, I _think_ the patch
bellow should be the fix for this bug.

If I understand it correctly, when an inode has inlined data it means that
there's no inode data to be written and this case should be handled as if
the inode length was zero.

I'll send out a patch later after running a few more tests just to make
sure it doesn't break something else.  But it would awesome if you could
test it too.

Cheers,
-- 
Luís

diff --git a/fs/ext4/fast_commit.c b/fs/ext4/fast_commit.c
index 87c009e0c59a..c56b39a51865 100644
--- a/fs/ext4/fast_commit.c
+++ b/fs/ext4/fast_commit.c
@@ -897,7 +897,7 @@ static int ext4_fc_write_inode_data(struct inode *inode, u32 *crc)
 	int ret;
 
 	mutex_lock(&ei->i_fc_lock);
-	if (ei->i_fc_lblk_len == 0) {
+	if ((ei->i_fc_lblk_len == 0) || (ext4_has_inline_data(inode))) {
 		mutex_unlock(&ei->i_fc_lock);
 		return 0;
 	}

[toc] | [prev] | [next] | [standalone]

#82746 — Bug#1039883: linux: ext4 corruption with symlinks

On Tue 18 Jun 2024 10:52:55 AM +01, Luis Henriques wrote;

> On Fri 14 Jun 2024 05:18:45 PM +01, Luis Henriques wrote;
> [...}
>>>
>>> I can also reproduce this error message using the above script and:
>>>
>>> - Linux 6.10-rc2
>>> - A 2 GiB loopback devic instead of /dev/sdb
>>>
>>> I bisected this back to:
>>>
>>> commit 9725958bb75cdfa10f2ec11526fdb23e7485e8e4
>>> Author: Xin Yin <yinxin.x@bytedance.com>
>>> Date:   Thu Dec 23 11:23:37 2021 +0800
>>>  
>>>     ext4: fast commit may miss tracking unwritten range during ftruncate
>>>
>>> It is still possible to cleanly revert that commit from 6.10-rc2, and
>>> doing so removes the error message.
>>
>> Because I recently fixed an issue in the fast commit code[1] I was hoping
>> that you were hitting the same bug.  I've executed the reproducer with the
>> fix (which hasn't been merged yet) and realised it's definitely a
>> different problem.
>>
>> Debugged the issue a bit, it seems to be related with the fact that
>> ext4_fc_write_inode_data() isn't able to cope with the fact that
>> 'ei->i_fc_lblk_len' is set to EXT_MAX_BLOCKS.
>
> OK, I've looked into this again.  And something I didn't pay attention
> before was that the filesystem was created with both fast_commit *and*
> inline_data features.  And after some more debugging, I _think_ the patch
> bellow should be the fix for this bug.
>
> If I understand it correctly, when an inode has inlined data it means that
> there's no inode data to be written and this case should be handled as if
> the inode length was zero.
>
> I'll send out a patch later after running a few more tests just to make
> sure it doesn't break something else.  But it would awesome if you could
> test it too.

Hmm... looking closer, this patch seems to work with this specific test
script, but only because file data is probably small enough to fit in
inode->i_block.  However, it may actually truncate files that have inlined
data if the file data is also stored in the extended attribute space
(i.e. > 60 bytes).

So, the correct fix is probably something like the below patch (which I'll
send out soon).

Cheers,
-- 
Luís

diff --git a/fs/ext4/fast_commit.c b/fs/ext4/fast_commit.c
index 87c009e0c59a..d3a67bc06d10 100644
--- a/fs/ext4/fast_commit.c
+++ b/fs/ext4/fast_commit.c
@@ -649,6 +649,12 @@ void ext4_fc_track_range(handle_t *handle, struct inode *inode, ext4_lblk_t star
 	if (ext4_test_mount_flag(inode->i_sb, EXT4_MF_FC_INELIGIBLE))
 		return;
 
+	if (ext4_has_inline_data(inode)) {
+		ext4_fc_mark_ineligible(inode->i_sb, EXT4_FC_REASON_XATTR,
+					handle);
+		return;
+	}
+
 	args.start = start;
 	args.end = end;
 

[toc] | [prev] | [next] | [standalone]

#82752 — Bug#1039883: [PATCH] ext4: don't track ranges in fast_commit if inode has inlined data

When fast-commit needs to track ranges, it has to handle inodes that have
inlined data in a different way because ext4_fc_write_inode_data(), in the
actual commit path, will attempt to map the required blocks for the range.
However, inodes that have inlined data will have it's data stored in
inode->i_block and, eventually, in the extended attribute space.

Unfortunately, because fast commit doesn't currently support extended
attributes, the solution is to mark this commit as ineligible.

Link: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1039883
Signed-off-by: Luis Henriques (SUSE) <luis.henriques@linux.dev>
---
 fs/ext4/fast_commit.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/fs/ext4/fast_commit.c b/fs/ext4/fast_commit.c
index 87c009e0c59a..d3a67bc06d10 100644
--- a/fs/ext4/fast_commit.c
+++ b/fs/ext4/fast_commit.c
@@ -649,6 +649,12 @@ void ext4_fc_track_range(handle_t *handle, struct inode *inode, ext4_lblk_t star
 	if (ext4_test_mount_flag(inode->i_sb, EXT4_MF_FC_INELIGIBLE))
 		return;
 
+	if (ext4_has_inline_data(inode)) {
+		ext4_fc_mark_ineligible(inode->i_sb, EXT4_FC_REASON_XATTR,
+					handle);
+		return;
+	}
+
 	args.start = start;
 	args.end = end;
 

[toc] | [prev] | [next] | [standalone]

#82945 — Processed: Re: linux: ext4 corruption with symlinks

Processing control commands:

> tag -1 patch fixed-upstream
Bug #1039883 [src:linux] linux: ext4 corruption with symlinks
Added tag(s) patch and fixed-upstream.

-- 
1039883: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1039883
Debian Bug Tracking System
Contact owner@bugs.debian.org with problems

[toc] | [prev] | [next] | [standalone]

#83298 — Bug#1039883: linux: ext4 corruption with symlinks

[Multipart message — attachments visible in raw view] — view raw

On Tue, 2024-07-09 at 19:32 +0200, Ben Hutchings wrote:
> Control: tag -1 patch fixed-upstream
> 
> A fix was applied to the ext4 "dev" branch earlier today:
> <https://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4.git/commit/?h=dev&id=7882b0187bbeb647967a7b5998ce4ad26ef68a9a>
> 
> I would still want to wait at least a week or two before cherry-picking
> it, in case of regressions.

This was included in 6.11-rc1 and is queued for the 5.15, 6.1, 6.6, and
6.10 stable branches.

It is *not* queued for 5.10 and some work may be needed to backport it.

Ben.

-- 
Ben Hutchings
Any sufficiently advanced bug is indistinguishable from a feature.

[toc] | [prev] | [next] | [standalone]

#83321 — Bug#1039883: marked as done (linux: ext4 corruption with symlinks)

[Multipart message — attachments visible in raw view] — view raw

Your message dated Mon, 05 Aug 2024 06:00:10 +0000
with message-id <E1saqlO-000Drh-QB@fasolo.debian.org>
and subject line Bug#1039883: fixed in linux 6.10.3-1
has caused the Debian Bug report #1039883,
regarding linux: ext4 corruption with symlinks
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact owner@bugs.debian.org
immediately.)


-- 
1039883: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1039883
Debian Bug Tracking System
Contact owner@bugs.debian.org with problems

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.kernel

csiph-web