Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #273226 > unrolled thread

ext4 file system corruption - again

Started byJesper Dybdal <jd-debian-user@dybdal.dk>
First post2024-09-23 11:30 +0200
Last post2024-09-26 15:30 +0200
Articles 5 — 3 participants

Back to article view | Back to linux.debian.user

Contents

  ext4 file system corruption - again Jesper Dybdal <jd-debian-user@dybdal.dk> - 2024-09-23 11:30 +0200
    Re: ext4 file system corruption - again Arno Lehmann <al@its-lehmann.de> - 2024-09-23 12:00 +0200
      Re: ext4 file system corruption - again Jesper Dybdal <jd-debian-user@dybdal.dk> - 2024-09-24 11:10 +0200
    Re: ext4 file system corruption - again Stefan Monnier <monnier@iro.umontreal.ca> - 2024-09-23 16:20 +0200
      Re: ext4 file system corruption - again Jesper Dybdal <jd-debian-user@dybdal.dk> - 2024-09-26 15:30 +0200

#273226 — ext4 file system corruption - again

FromJesper Dybdal <jd-debian-user@dybdal.dk>
Date2024-09-23 11:30 +0200
Subjectext4 file system corruption - again
Message-ID<JpNyh-efLc-3@gated-at.bofh.it>
Some time ago I had a problem with the i_blocks field of a few inodes 
being corrupted (replaced by extremely large numbers).

It now happened again, and the strange thing is that one of the two 
files affected was also affected the earlier time (or more precisely, 
the file that now has the same name as the old one had). That file was 
/etc/postfix/master.cf - a simple text configuration file that is 
modified only manually using emacs.

I wrote about it here when it first happened - that thread began with:
    Message-ID: <bf46ee0b-8af1-43ce-a48f-e304ef850b3d@dybdal.dk>
    Date: Tue, 19 Mar 2024 15:43:30 +0100
    Subject: Filsystemkorruption i ext4?

I don't believe that it is a disk error - the file system is on a RAID1 
partition and the RAID consistency is checked regularly.
I also find it hard to believe that it is a RAM error - the mashine has 
run memtest86+ overnight without finding anything.
There was a power outage some time ago, but surely ext4 should be able 
to handle that without introducing errors.

Fsck fixes it.
The system is an up-to-date Bookworm.

Any ideas as to how this can happen - twice, and effecting (among 
others) the same file?

---------------- fsck log:
Log of fsck -C -a -T -t ext4 /dev/md0
Sun Sep 22 20:20:13 2024

root has been mounted 13 times without being checked, check forced.
root: Inode 10748715, i_blocks is 281474976710631, should be 5. FIXED.
root: Inode 10751288, i_blocks is 281474976710647, should be 3. FIXED.
root: 223986/32759808 files (6.5% non-contiguous), 6827061/131038976 blocks
fsck exited with status code 1

---------------- stat(1) for the two files before fsck:
   File: main.cf
   Size: 16959         Blocks: 2251799813685048 IO Block: 4096 regular file
Device: 9,0    Inode: 10748715    Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/ root)
Access: 2024-09-22 20:31:58.081768853 +0200
Modify: 2024-08-03 19:33:20.665350446 +0200
Change: 2024-09-22 22:09:54.053359071 +0200
  Birth: 2024-08-03 11:20:14.671832520 +0200

   File: master.cf
   Size: 10782         Blocks: 2251799813685176 IO Block: 4096 regular file
Device: 9,0    Inode: 10751288    Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/ root)
Access: 2024-09-22 19:20:07.216191493 +0200
Modify: 2024-06-05 14:37:24.205001097 +0200
Change: 2024-09-22 22:09:54.053359071 +0200
  Birth: 2024-03-19 13:36:38.971618859 +0100
----------------

Thanks,
Jesper

-- 
Jesper Dybdal
https://www.dybdal.dk

[toc] | [next] | [standalone]

#273227

FromArno Lehmann <al@its-lehmann.de>
Date2024-09-23 12:00 +0200
Message-ID<JpO1j-efVT-1@gated-at.bofh.it>
In reply to#273226
Hi Jesper,

Am 23.09.2024 um 11:20 schrieb Jesper Dybdal:
...
 > ---------------- fsck log:
 > Log of fsck -C -a -T -t ext4 /dev/md0
 > Sun Sep 22 20:20:13 2024
 >
 > root has been mounted 13 times without being checked, check forced.
 > root: Inode 10748715, i_blocks is 281474976710631, should be 5. FIXED.
 > root: Inode 10751288, i_blocks is 281474976710647, should be 3. FIXED.
 > root: 223986/32759808 files (6.5% non-contiguous), 6827061/131038976 
blocks


I do not think it is likely to be a hardware problem. The bit patterns 
for the number of blocks you quote actually have all the high bits of 
the potentially used 48 bits of the block count set, for example 
111111111111111111111111111111111111111111100111 and given that the 16 
higher bits are not stored next to the lower 32 bits, there should be 
many other values and flags set to all ones, which should result in 
other things to notice -- for example, the file should be considered to 
be part of the file system structure and compressed, which fsck should 
loudly complain about (not verified).

The large values with peculiar bit patterns do look like some flag 
values to me. It might be worth asking among the ext4 developers if 
those values could be introduced by some particular condition, I think.

Cheers,

Arno

-- 
Arno Lehmann

IT-Service Lehmann
Sandstr. 6, 49080 Osnabrück

[toc] | [prev] | [next] | [standalone]

#273248

FromJesper Dybdal <jd-debian-user@dybdal.dk>
Date2024-09-24 11:10 +0200
Message-ID<Jq9It-etjJ-3@gated-at.bofh.it>
In reply to#273227

[Multipart message — attachments visible in raw view] — view raw

On 2024-09-23 11:55, Arno Lehmann wrote:
> Am 23.09.2024 um 11:20 schrieb Jesper Dybdal:
> ...
> > root: Inode 10748715, i_blocks is 281474976710631, should be 5. FIXED.
> > root: Inode 10751288, i_blocks is 281474976710647, should be 3. FIXED.
> > root: 223986/32759808 files (6.5% non-contiguous), 6827061/131038976 
> blocks
>
> I do not think it is likely to be a hardware problem. The bit patterns 
> for the number of blocks you quote actually have all the high bits of 
> the potentially used 48 bits of the block count set 
...
> The large values with peculiar bit patterns do look like some flag 
> values to me. It might be worth asking among the ext4 developers if 
> those values could be introduced by some particular condition, I think.

I think I'll do that.


On 2024-09-23 16:10, Stefan Monnier wrote:
> root has been mounted 13 times without being checked, check forced.
>> root: Inode 10748715, i_blocks is 281474976710631, should be 5. FIXED.
>                                      ^^^^^^^^^^^^^^^
> AKA -25
>
>> root: Inode 10751288, i_blocks is 281474976710647, should be 3. FIXED.
>                                     ^^^^^^^^^^^^^^^
> AKA -9
>
> It's odd that it looks like -N².  Maybe it's just happenstance, but I'd
> be curious to see if you have other "data points".
Unfortunately, I have no other "data points".  The data from the first 
occurrence of this are lost.

Many thanks to Arno and Stefan - I probably had not noticed the 
interesting values without their help.

Jesper

-- 
Jesper Dybdal
https://www.dybdal.dk

[toc] | [prev] | [next] | [standalone]

#273233

FromStefan Monnier <monnier@iro.umontreal.ca>
Date2024-09-23 16:20 +0200
Message-ID<JpS4V-eivZ-1@gated-at.bofh.it>
In reply to#273226
> root has been mounted 13 times without being checked, check forced.
> root: Inode 10748715, i_blocks is 281474976710631, should be 5. FIXED.
                                    ^^^^^^^^^^^^^^^
AKA -25

> root: Inode 10751288, i_blocks is 281474976710647, should be 3. FIXED.
                                    ^^^^^^^^^^^^^^^
AKA -9

It's odd that it looks like -N².  Maybe it's just happenstance, but I'd
be curious to see if you have other "data points".


        Stefan

[toc] | [prev] | [next] | [standalone]

#273298

FromJesper Dybdal <jd-debian-user@dybdal.dk>
Date2024-09-26 15:30 +0200
Message-ID<JqWJb-eYe1-1@gated-at.bofh.it>
In reply to#273233

[Multipart message — attachments visible in raw view] — view raw

On 2024-09-23 16:10, Stefan Monnier wrote:
>> root has been mounted 13 times without being checked, check forced.
>> root: Inode 10748715, i_blocks is 281474976710631, should be 5. FIXED.
>                                      ^^^^^^^^^^^^^^^
> AKA -25
>
>> root: Inode 10751288, i_blocks is 281474976710647, should be 3. FIXED.
>                                      ^^^^^^^^^^^^^^^
> AKA -9
>
> It's odd that it looks like -N².  Maybe it's just happenstance, but I'd
> be curious to see if you have other "data points".
>
One more "data point" just appeared today.  I had edited /etc/fstab and 
rebooted, and then fstab suddenly had the problem.:

root: Inode 10748542, i_blocks is 281474976710653, should be 1. FIXED.
AKA -3, so not of the -N² form.

 From a stat() (after the fsck that repaired the file):
   File: fstab
   Size: 1855          Blocks: 8          IO Block: 4096   regular file

I haven't yet had time to describe the problem on the ext4 mailing 
list.  I have remembered that there is one thing I do differently from 
the default: I use the ext4 option "nodelalloc" (because several years 
ago, there was a discussion about "delalloc or not" from which It seemed 
that nodelalloc was probably slightly safer - if the associated 
performance reduction is not a problem, which it is not for me).

I've now turned nodelalloc off, just in case that changes something.

-- 
Jesper Dybdal
https://www.dybdal.dk

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web