Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #275415 > unrolled thread

Structure needs cleaning / Directory block does not have space for checksum

Back to article view | Back to linux.debian.user

Contents

  Structure needs cleaning / Directory block does not have space for  checksum Gregor Zattler <telegraph@gmx.net> - 2024-12-09 02:00 +0100
    Re: Structure needs cleaning / Directory block does not have space  for checksum Gregor Zattler <telegraph@gmx.net> - 2024-12-11 12:10 +0100
      Re: Structure needs cleaning / Directory block does not have space  for checksum Frank Guthausen <fg.debian@shimps.de> - 2024-12-11 12:30 +0100
        Re: Structure needs cleaning / Directory block does not have space  for checksum Gregor Zattler <telegraph@gmx.net> - 2024-12-11 19:20 +0100
      Re: Structure needs cleaning / Directory block does not have space  for checksum Arno Lehmann <al@its-lehmann.de> - 2024-12-11 13:00 +0100
        Re: Structure needs cleaning / Directory block does not have space  for checksum Gregor Zattler <telegraph@gmx.net> - 2024-12-11 19:20 +0100
      Re: Structure needs cleaning / Directory block does not have space  for checksum Tim Woodall <debianuser@woodall.me.uk> - 2024-12-11 21:10 +0100

#275415 — Structure needs cleaning / Directory block does not have space for checksum

Dear debian enthusiasts, I use
rdiff-backup, which now is not able to
work with my most precious backup,
instead throws a python backtrace which
contains:

OSError: [Errno 117] Structure needs cleaning: b'/mnt/mic-backup/rdiff-backup/durable/rdiff-backup-data/increments/home/grfz/.procmail/backup-post-mailmunge/new'

While a fsck.ext4 -vvvtfDfy on that file
system gives

Failed to optimize directory /rdiff-backup/durable/rdiff-backup-data/increments/home/grfz/.procmail/backup-post-mailmunge/new (498074110): Directory block does not have space for checksum

in Pass 3A: Optimizing directories.


Because of
https://blogs.oracle.com/linux/post/space-management-with-large-directories-in-ext4
I tried to copy said directory:

cp -a new neu

this too does not work:

cp: cannot access 'new': Structure needs cleaning
cp: preserving times for 'neu': Read-only file system

Any ideas how to repair said directory,
clean the structure, or another work
around to at least get rdiff-backup get
to use the backup again for restoring?

Or where to ask?   The ext3-users
mailing list does not exist any more?

Or how to avoid such a problem next time?



Thanks in advance for any hints, Gregor

[toc] | [next] | [standalone]

#275498 — Re: Structure needs cleaning / Directory block does not have space for checksum

Hi,
* Gregor Zattler <telegraph@gmx.net> [2024-12-09; 01:54 +01]:
> Dear debian enthusiasts, I use
> rdiff-backup, which now is not able to
> work with my most precious backup,
> instead throws a python backtrace which
> contains:
>
> OSError: [Errno 117] Structure needs cleaning: b'/mnt/mic-backup/rdiff-backup/durable/rdiff-backup-data/increments/home/grfz/.procmail/backup-post-mailmunge/new'
>
> While a fsck.ext4 -vvvtfDfy on that file
> system gives
>
> Failed to optimize directory /rdiff-backup/durable/rdiff-backup-data/increments/home/grfz/.procmail/backup-post-mailmunge/new (498074110): Directory block does not have space for checksum
>
> in Pass 3A: Optimizing directories.
>
>
> Because of
> https://blogs.oracle.com/linux/post/space-management-with-large-directories-in-ext4
> I tried to copy said directory:
>
> cp -a new neu
>
> this too does not work:
>
> cp: cannot access 'new': Structure needs cleaning
> cp: preserving times for 'neu': Read-only file system
>
> Any ideas how to repair said directory,
> clean the structure, or another work
> around to at least get rdiff-backup get
> to use the backup again for restoring?
>
> Or where to ask?   The ext3-users
> mailing list does not exist any more?
>
> Or how to avoid such a problem next time?


this directory in question is really
huge:

> ls -Altr
[...]
-rwx-----x 1 grfz grfz          0 Nov 21 01:26  new.2024-11-21T01:32:48+01:00.dir
-rwx------ 1 grfz grfz          0 Nov 21 16:37  tmp.2024-11-21T16:42:36+01:00.dir
-rwx-----x 1 grfz grfz          0 Nov 21 16:37  new.2024-11-21T16:42:36+01:00.dir
-rwx------ 1 grfz grfz          0 Nov 21 23:12  tmp.2024-11-21T23:36:57+01:00.dir
-rwx-----x 1 grfz grfz          0 Nov 21 23:12  new.2024-11-21T23:36:57+01:00.dir
-rwx------ 1 grfz grfz          0 Nov 23 01:01  tmp.2024-11-23T01:25:22+01:00.dir
-rwx-----x 1 grfz grfz          0 Nov 23 01:01  new.2024-11-23T01:25:22+01:00.dir
-rwx------ 1 grfz grfz          0 Nov 25 00:41  tmp.2024-11-25T00:44:07+01:00.dir
-rwx-----x 1 grfz grfz          0 Nov 25 00:41  new.2024-11-25T00:44:07+01:00.dir
-rwx------ 1 grfz grfz          0 Nov 25 15:54  tmp.2024-11-25T15:56:34+01:00.dir
-rwx-----x 1 grfz grfz          0 Nov 25 15:54  new.2024-11-25T15:56:34+01:00.dir
-rwx------ 1 grfz grfz          0 Nov 26 00:27  tmp.2024-11-26T00:50:56+01:00.dir
-rwx-----x 1 grfz grfz          0 Nov 26 00:27  new.2024-11-26T00:50:56+01:00.dir
-rwx-----x 1 grfz grfz          0 Nov 28 01:06  new.2024-11-28T01:34:32+01:00.dir
drwx------ 2 root root 2139340800 Nov 28 01:37  new

I tried to move the files in the
directory to another one, but this gives

mv: cannot stat FILENAME: Bad message

So I cannot stat, mv, cp, cat these
files or at least some of them.


dmesg shows 35 lines like this
one:

ext4_dirblock_csum_verify:405: inode #498074110: comm ls: No space for directory leaf checksum. Please run e2fsck -D

and this one:

[76268.580904] EXT4-fs error (device dm-6): ext4_readdir:218: inode #498074110: comm ls: path (unknown): directory fails checksum at offset 0

But fsck -fDy does not help (as stated
in quoted part, above).

After fsck.ext4 -vvvtfDfy a fsck without
any options tells me the fs is clean.



Any ideas, pointers?

Ciao; Gregor
--
 -... --- .-. . -.. ..--.. ...-.-

[toc] | [prev] | [next] | [standalone]

#275500 — Re: Structure needs cleaning / Directory block does not have space for checksum

On Wed, 11 Dec 2024 12:01:30 +0100
Gregor Zattler <telegraph@gmx.net> wrote:
> 
> Any ideas, pointers?

Please check free space and available inodes:

df -h
df -hi

Check mount state:

mount

You might try to mount the file system read-only and copy or rsync the
data from this state. The target file system should provide enough free
space for the data, too.

kind regards
Frank

[toc] | [prev] | [next] | [standalone]

#275513 — Re: Structure needs cleaning / Directory block does not have space for checksum

Hi Frank,
* Frank Guthausen <fg.debian@shimps.de> [2024-12-11; 12:21 +01]:
> On Wed, 11 Dec 2024 12:01:30 +0100
> Gregor Zattler <telegraph@gmx.net> wrote:
>>
>> Any ideas, pointers?
>
> Please check free space and available inodes:
>
> df -h
> df -hi

not via df, but this information:

0 (master *) grfz@no:~$ sudo tune2fs -l /dev/mapper/mic-backup
tune2fs 1.47.0 (5-Feb-2023)
Filesystem volume name:   mic-backupfs
Last mounted on:          /mnt/mic-backup
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index sparse_super2 filetype needs_recovery extent 64bit mmp flex_bg large_dir sparse_super large_file huge_file dir_nlink extra_isize metadata_csum
Filesystem flags:         signed_directory_hash
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              960430080
Block count:              960427008
Reserved block count:     48021350
Overhead clusters:        60352093
Free blocks:              256205203
Free inodes:              871021955


So the problem is not on the file system
level but with one directory.


> Check mount state:
>
> mount
>
> You might try to mount the file system read-only and copy or rsync the
> data from this state. The target file system should provide enough free
> space for the data, too.

Sadly not possible, does not stat the
files in the directory.

Thanks.


Ciao; Gregor
--
 -... --- .-. . -.. ..--.. ...-.-

[toc] | [prev] | [next] | [standalone]

#275501 — Re: Structure needs cleaning / Directory block does not have space for checksum

Hi Gregor,

Am 11.12.2024 um 12:01 schrieb Gregor Zattler:
> Hi,
> * Gregor Zattler <telegraph@gmx.net> [2024-12-09; 01:54 +01]:
>> Dear debian enthusiasts, I use
>> rdiff-backup, which now is not able to
>> work with my most precious backup,
>> instead throws a python backtrace which
>> contains:

rdiff-backup, I guess, creates and manages its backup history by using 
hard links between different "generations" of backup directories.

>> OSError: [Errno 117] Structure needs cleaning: b'/mnt/mic-backup/rdiff-backup/durable/rdiff-backup-data/increments/home/grfz/.procmail/backup-post-mailmunge/new'
>>
>> While a fsck.ext4 -vvvtfDfy on that file
>> system gives
>>
>> Failed to optimize directory /rdiff-backup/durable/rdiff-backup-data/increments/home/grfz/.procmail/backup-post-mailmunge/new (498074110): Directory block does not have space for checksum

So the file system is sufficiently well filled to not allow further 
operations, including fixes.

>> in Pass 3A: Optimizing directories.
>>
>>
>> Because of
>> https://blogs.oracle.com/linux/post/space-management-with-large-directories-in-ext4
>> I tried to copy said directory:
>>
>> cp -a new neu
>>
>> this too does not work:
>>
>> cp: cannot access 'new': Structure needs cleaning
>> cp: preserving times for 'neu': Read-only file system
>>
>> Any ideas how to repair said directory,

This does not look like there will be an easy solution.

>> clean the structure, or another work
>> around to at least get rdiff-backup get
>> to use the backup again for restoring?

I know nothing about rdiff-backup. Also, if fsck can not do its work any 
more due to limitations or constraints, you will probably have to make a 
tough decision:
- Ask some *real* experts. In this case, probably ext file system 
developers, or a professional data recovery company.
- try to salvage what you can by manually copying data out
- give up, wipe the complete file system and take a note to avoid such 
situations in the future.

>> Or where to ask?   The ext3-users
>> mailing list does not exist any more?

Linux kernel mailing lists might help, at least to ask where to find the 
ext family developers.

>> Or how to avoid such a problem next time?

Avoid excessive amounts of hard links.

The problem is to know what the practical limits are.

Unfortunately, you will usually only know *after* hitting some wall.

> 
> this directory in question is really
> huge:
> 
>> ls -Altr
> [...]
> -rwx-----x 1 grfz grfz          0 Nov 21 01:26  new.2024-11-21T01:32:48+01:00.dir
> -rwx------ 1 grfz grfz          0 Nov 21 16:37  tmp.2024-11-21T16:42:36+01:00.dir
> -rwx-----x 1 grfz grfz          0 Nov 21 16:37  new.2024-11-21T16:42:36+01:00.dir
> -rwx------ 1 grfz grfz          0 Nov 21 23:12  tmp.2024-11-21T23:36:57+01:00.dir
> -rwx-----x 1 grfz grfz          0 Nov 21 23:12  new.2024-11-21T23:36:57+01:00.dir
> -rwx------ 1 grfz grfz          0 Nov 23 01:01  tmp.2024-11-23T01:25:22+01:00.dir
> -rwx-----x 1 grfz grfz          0 Nov 23 01:01  new.2024-11-23T01:25:22+01:00.dir
> -rwx------ 1 grfz grfz          0 Nov 25 00:41  tmp.2024-11-25T00:44:07+01:00.dir
> -rwx-----x 1 grfz grfz          0 Nov 25 00:41  new.2024-11-25T00:44:07+01:00.dir
> -rwx------ 1 grfz grfz          0 Nov 25 15:54  tmp.2024-11-25T15:56:34+01:00.dir
> -rwx-----x 1 grfz grfz          0 Nov 25 15:54  new.2024-11-25T15:56:34+01:00.dir
> -rwx------ 1 grfz grfz          0 Nov 26 00:27  tmp.2024-11-26T00:50:56+01:00.dir
> -rwx-----x 1 grfz grfz          0 Nov 26 00:27  new.2024-11-26T00:50:56+01:00.dir
> -rwx-----x 1 grfz grfz          0 Nov 28 01:06  new.2024-11-28T01:34:32+01:00.dir
> drwx------ 2 root root 2139340800 Nov 28 01:37  new

It would be very interesting to understand the inode allocation and use 
through hardlinks, but I suspect you will not find user friendly tools 
such as ls, du, df, stat or fsck able to help here. The problem is not 
with those tools, but the file system code itself. Which usually also 
implies that you have no way to run any automatic recovery at all any more.

You might be able to get somewhere using debugfs, but I have no 
experience with it, so can not even give you a starting point.

> I tried to move the files in the
> directory to another one, but this gives
> 
> mv: cannot stat FILENAME: Bad message
> 
> So I cannot stat, mv, cp, cat these
> files or at least some of them.

As stated above, I'm not surprised.

> 
> dmesg shows 35 lines like this
> one:
> 
> ext4_dirblock_csum_verify:405: inode #498074110: comm ls: No space for directory leaf checksum. Please run e2fsck -D
> 
> and this one:
> 
> [76268.580904] EXT4-fs error (device dm-6): ext4_readdir:218: inode #498074110: comm ls: path (unknown): directory fails checksum at offset 0

So pretty clearly some file system structure have run out of space and 
more space can not be allocated. Given that the first message is about 
lacking space for a checksum, I would try disabling all checksumming and 
see if that allows you to recover some files.

> But fsck -fDy does not help (as stated
> in quoted part, above).
> 
> After fsck.ext4 -vvvtfDfy a fsck without
> any options tells me the fs is clean.
> 
> 
> 
> Any ideas, pointers?

What I would do is the following.

1. Find storage to store a full dump of the file system (or, actually, 
its underlying block storage). Create such a dump, because it's quite 
likely that further problems arise. Dump in this context means dd, not 
the file system dump tool.

2. Familiarize yourself with ext file system options and debugfs tool. 
Eventually, see if you can access the file system without any features 
that would add extra storage capacity or inode needs. Like without 
journal, without checksumming, without any redundancy and in particular 
read only.

3. See if you can at least read some files. If you can, copy them 
somewhere else.

4. Try to delete copied files -- either by mounting RW and using regular 
rm, or using debugging tools. At this point, you'll not expect that a 
deleted file will actually free up visible space (it might just be a 
hard link...) but you may have a chance to free up some of the directory 
or other data structures.

5. Repeat until you have noticeable free space, both bulk capacity and 
inodes. You can probably not check with regular tools, so this is more 
about debugging the file system or trusting your gut feeling.

6. Eventually, try an fsck run. If it fails, go back to step 3.

This will be a time consuming and nerve wrecking exercise.


If that works, you'll lose most of the structure and metadata 
represented as file system contents, in particular everything that is in 
the hard link relationships of the directory tree.

Afterwards, it's time to investigate how you should have tuned the file 
system for its use case, or how you could at least have monitored for 
its reaching operational boundaries. Perhaps it's also useful to use a 
different file system for such purposes.

Personally, I would recommend against using such hardlink based backup 
schemes in general, because they are too fragile and can not be fixed. 
Which you just found out.


Oh, and only after you have exhausted your options, implemented and 
monitored an improved or different backup solution, and proved that you 
can restore is it time to delete the last copy of your broken file system.


Good luck.

Arno

> Ciao; Gregor
> --
>   -... --- .-. . -.. ..--.. ...-.-
> 

-- 
Arno Lehmann

IT-Service Lehmann
Sandstr. 6, 49080 Osnabrück

[toc] | [prev] | [next] | [standalone]

#275514 — Re: Structure needs cleaning / Directory block does not have space for checksum

Hi Arno,
* Arno Lehmann <al@its-lehmann.de> [2024-12-11; 12:59 +01]:
> Am 11.12.2024 um 12:01 schrieb Gregor Zattler:
>>> Or where to ask?   The ext3-users
>>> mailing list does not exist any more?
>
> Linux kernel mailing lists might help, at least to ask where to find the
> ext family developers.

thanks, will do.
[...]
> What I would do is the following.
>
> 1. Find storage to store a full dump of the file system (or, actually,
> its underlying block storage). Create such a dump, because it's quite
> likely that further problems arise. Dump in this context means dd, not
> the file system dump tool.

Too expensive...

> 2. Familiarize yourself with ext file system options and debugfs tool.
> Eventually, see if you can access the file system without any features
> that would add extra storage capacity or inode needs. Like without
> journal, without checksumming, without any redundancy and in particular
> read only.

The man page ext4 (5) does not show any
way to mount without checksumming (if
enabled as a feature).



I probably will throw away the backup.

Thanks for your thoughts.

Ciao; Gregor
--
 -... --- .-. . -.. ..--.. ...-.-

[toc] | [prev] | [next] | [standalone]

#275518 — Re: Structure needs cleaning / Directory block does not have space for checksum

you can see if anything in debugfs will help. That's part of e2fstools I
think.

In particular, you could use freei to free the inode for this directory
(although I suspect another fsck will then attach it back to lost and
found but I'm not sure)

N.B. Actions like this are last resort and you really do need a backup
before you start.

You might also be able to clear the checksum feature, delete this
directory and then turn it back on.

You can also try rmdir from debugfs.

FWIW, here's a command I use in a testcase to ensure that the first
block of a file on disk is not all zeros:

debugfs -w -R "zap_block -f uninit -p 0x5a 0" "$SRC_LOOPDEV"

That isn't any use to you but gives an example of how debugfs can be
used to modify a filesystem bypassing normal

inode_dump might also help you.

I'm assuming you just want to delete this directory, not save whatever
was in it - if you do want to save whatever was in it then dump here;
https://sourceforge.net/projects/dump/ might do it for you - but use the
latest version from sourceforge, not the version in debian stable.
(I've recently taken over as the maintainer but unfortunately the code
had rotted badly and the versions in debian stable do not understand
some common ext4 features)

(I can provide a deb if you don't want to build from source - if you
want a random deb from a random person on the internet...)

Tim.




On Wed, 11 Dec 2024, Gregor Zattler wrote:

> Hi,
> * Gregor Zattler <telegraph@gmx.net> [2024-12-09; 01:54 +01]:
>> Dear debian enthusiasts, I use
>> rdiff-backup, which now is not able to
>> work with my most precious backup,
>> instead throws a python backtrace which
>> contains:
>>
>> OSError: [Errno 117] Structure needs cleaning: b'/mnt/mic-backup/rdiff-backup/durable/rdiff-backup-data/increments/home/grfz/.procmail/backup-post-mailmunge/new'
>>
>> While a fsck.ext4 -vvvtfDfy on that file
>> system gives
>>
>> Failed to optimize directory /rdiff-backup/durable/rdiff-backup-data/increments/home/grfz/.procmail/backup-post-mailmunge/new (498074110): Directory block does not have space for checksum
>>
>> in Pass 3A: Optimizing directories.
>>
>>
>> Because of
>> https://blogs.oracle.com/linux/post/space-management-with-large-directories-in-ext4
>> I tried to copy said directory:
>>
>> cp -a new neu
>>
>> this too does not work:
>>
>> cp: cannot access 'new': Structure needs cleaning
>> cp: preserving times for 'neu': Read-only file system
>>
>> Any ideas how to repair said directory,
>> clean the structure, or another work
>> around to at least get rdiff-backup get
>> to use the backup again for restoring?
>>
>> Or where to ask?   The ext3-users
>> mailing list does not exist any more?
>>
>> Or how to avoid such a problem next time?
>
>
> this directory in question is really
> huge:
>
>> ls -Altr
> [...]
> -rwx-----x 1 grfz grfz          0 Nov 21 01:26  new.2024-11-21T01:32:48+01:00.dir
> -rwx------ 1 grfz grfz          0 Nov 21 16:37  tmp.2024-11-21T16:42:36+01:00.dir
> -rwx-----x 1 grfz grfz          0 Nov 21 16:37  new.2024-11-21T16:42:36+01:00.dir
> -rwx------ 1 grfz grfz          0 Nov 21 23:12  tmp.2024-11-21T23:36:57+01:00.dir
> -rwx-----x 1 grfz grfz          0 Nov 21 23:12  new.2024-11-21T23:36:57+01:00.dir
> -rwx------ 1 grfz grfz          0 Nov 23 01:01  tmp.2024-11-23T01:25:22+01:00.dir
> -rwx-----x 1 grfz grfz          0 Nov 23 01:01  new.2024-11-23T01:25:22+01:00.dir
> -rwx------ 1 grfz grfz          0 Nov 25 00:41  tmp.2024-11-25T00:44:07+01:00.dir
> -rwx-----x 1 grfz grfz          0 Nov 25 00:41  new.2024-11-25T00:44:07+01:00.dir
> -rwx------ 1 grfz grfz          0 Nov 25 15:54  tmp.2024-11-25T15:56:34+01:00.dir
> -rwx-----x 1 grfz grfz          0 Nov 25 15:54  new.2024-11-25T15:56:34+01:00.dir
> -rwx------ 1 grfz grfz          0 Nov 26 00:27  tmp.2024-11-26T00:50:56+01:00.dir
> -rwx-----x 1 grfz grfz          0 Nov 26 00:27  new.2024-11-26T00:50:56+01:00.dir
> -rwx-----x 1 grfz grfz          0 Nov 28 01:06  new.2024-11-28T01:34:32+01:00.dir
> drwx------ 2 root root 2139340800 Nov 28 01:37  new
>
> I tried to move the files in the
> directory to another one, but this gives
>
> mv: cannot stat FILENAME: Bad message
>
> So I cannot stat, mv, cp, cat these
> files or at least some of them.
>
>
> dmesg shows 35 lines like this
> one:
>
> ext4_dirblock_csum_verify:405: inode #498074110: comm ls: No space for directory leaf checksum. Please run e2fsck -D
>
> and this one:
>
> [76268.580904] EXT4-fs error (device dm-6): ext4_readdir:218: inode #498074110: comm ls: path (unknown): directory fails checksum at offset 0
>
> But fsck -fDy does not help (as stated
> in quoted part, above).
>
> After fsck.ext4 -vvvtfDfy a fsck without
> any options tells me the fs is clean.
>
>
>
> Any ideas, pointers?
>
> Ciao; Gregor
> --
> -... --- .-. . -.. ..--.. ...-.-
>
>
>

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web