Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.debian.kernel > #89368 > unrolled thread
| Started by | Salvatore Bonaccorso <carnil@debian.org> |
|---|---|
| First post | 2025-09-23 21:00 +0200 |
| Last post | 2025-10-15 04:10 +0200 |
| Articles | 5 — 3 participants |
Back to article view | Back to linux.debian.kernel
This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by
below is the oldest one visible, not the original post.
Bug#1116067: linux-image-6.1.0-32-amd64: btrfs compression quietly stopped around 60TB in Salvatore Bonaccorso <carnil@debian.org> - 2025-09-23 21:00 +0200
Processed: Re: Bug#1116067: linux-image-6.1.0-32-amd64: btrfs compression quietly stopped around 60TB in "Debian Bug Tracking System" <owner@bugs.debian.org> - 2025-09-23 21:00 +0200
Bug#1116067: linux-image-6.1.0-32-amd64: btrfs compression quietly stopped around 60TB in james young <pronoiac@gmail.com> - 2025-09-23 21:20 +0200
Bug#1116067: linux-image-6.1.0-32-amd64: btrfs compression quietly stopped around 60TB in james young <pronoiac@gmail.com> - 2025-09-25 20:00 +0200
Bug#1116067: linux-image-6.1.0-32-amd64: btrfs compression quietly stopped around 60TB in james young <pronoiac@gmail.com> - 2025-10-15 04:10 +0200
| From | Salvatore Bonaccorso <carnil@debian.org> |
|---|---|
| Date | 2025-09-23 21:00 +0200 |
| Subject | Bug#1116067: linux-image-6.1.0-32-amd64: btrfs compression quietly stopped around 60TB in |
| Message-ID | <Lygp3-1YM-1@gated-at.bofh.it> |
Control: tags -1 + moreinfo Hi James, On Tue, Sep 23, 2025 at 08:04:25PM +0200, James Young wrote: > Package: src:linux > Version: 6.1.129-1 > Severity: normal > X-Debbugs-Cc: pronoiac@gmail.com > > Dear Maintainer, > > > * What led up to the situation? > We made empty files in a loop, in parallel, under CPU and I/O load. > We had an outer Btrfs image file with compression, which contained a Btrfs image file, which contained billions of empty files. > We wrote around 100TB to the inner image file. > Around 60TB in, compression quietly shut off. > We ran out of space; both mounts presented i/o errors. > > * What exactly did you do (or not do) that was effective (or ineffective)? > * I unmounted the inner and outer images. > I didn't take note of memory usage before this point. > * dump debug info for the outer image - `btrfs inspect-internal dump-tree --dfs ...` > * We started a btrfsck. (twice, actually; breadth-first hit memory limits, I think) > After that, I learned about `btrfs check`, but didn't interrupt the btrfsck, due to Sunk Cost Fallacy. > The btrfsck is still running. It's of extremely dubious value now. > * check the kernel logs > * I grepped for btrfs, the mount points, compress, and zstd. I didn’t find a smoking gun in the right timeframe. > > not done yet: > * mount the outer image > * rebooted > * tried a newer kernel. we're currently on kernel 6.1.129; we could go to newer 6.1 or 6.12 kernels > * redo live file system compression, with e.g. `btrfs filesystem defrag -czstd` > * fstrim the outer image > > goals: > * work out what happened. > How can we help? > * help avoid it happening again, to others > * salvage what we can > > I've run `bugreport` as a non-privileged user. Let me know if root access would give a fuller picture. I believe the best thing you could do here is to contact actually upstream people directly. get_maintainers and the MAINTAINERS file has: BTRFS FILE SYSTEM M: Chris Mason <clm@fb.com> M: Josef Bacik <josef@toxicpanda.com> M: David Sterba <dsterba@suse.com> L: linux-btrfs@vger.kernel.org S: Maintained W: https://btrfs.readthedocs.io Q: https://patchwork.kernel.org/project/linux-btrfs/list/ C: irc://irc.libera.chat/btrfs T: git git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git F: Documentation/filesystems/btrfs.rst F: fs/btrfs/ F: include/linux/btrfs* F: include/trace/events/btrfs.h F: include/uapi/linux/btrfs* So I would suggest you to contact above maintainers including the list. Please keep this downstream bugreport as well in the recipients list. Regards, Salvatore
[toc] | [next] | [standalone]
| From | "Debian Bug Tracking System" <owner@bugs.debian.org> |
|---|---|
| Date | 2025-09-23 21:00 +0200 |
| Subject | Processed: Re: Bug#1116067: linux-image-6.1.0-32-amd64: btrfs compression quietly stopped around 60TB in |
| Message-ID | <Lygp3-1YM-5@gated-at.bofh.it> |
| In reply to | #89368 |
Processing control commands: > tags -1 + moreinfo Bug #1116067 [src:linux] linux-image-6.1.0-32-amd64: btrfs compression quietly stopped around 60TB in Added tag(s) moreinfo. -- 1116067: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1116067 Debian Bug Tracking System Contact owner@bugs.debian.org with problems
[toc] | [prev] | [next] | [standalone]
| From | james young <pronoiac@gmail.com> |
|---|---|
| Date | 2025-09-23 21:20 +0200 |
| Message-ID | <LygIp-2mG-7@gated-at.bofh.it> |
| In reply to | #89368 |
I hit an issue with btrfs compression; I reported it to Debian, which I was using, and they suggested that I take it upstream. Thanks, Salvatore. My apologies to everyone if I misunderstood. -James On Tue, Sep 23, 2025 at 1:50 PM Salvatore Bonaccorso <carnil@debian.org> wrote: > > Control: tags -1 + moreinfo > > Hi James, > > On Tue, Sep 23, 2025 at 08:04:25PM +0200, James Young wrote: > > Package: src:linux > > Version: 6.1.129-1 > > Severity: normal > > X-Debbugs-Cc: pronoiac@gmail.com > > > > Dear Maintainer, > > > > > > * What led up to the situation? > > We made empty files in a loop, in parallel, under CPU and I/O load. > > We had an outer Btrfs image file with compression, which contained a Btrfs image file, which contained billions of empty files. > > We wrote around 100TB to the inner image file. > > Around 60TB in, compression quietly shut off. > > We ran out of space; both mounts presented i/o errors. > > > > * What exactly did you do (or not do) that was effective (or ineffective)? > > * I unmounted the inner and outer images. > > I didn't take note of memory usage before this point. > > * dump debug info for the outer image - `btrfs inspect-internal dump-tree --dfs ...` > > * We started a btrfsck. (twice, actually; breadth-first hit memory limits, I think) > > After that, I learned about `btrfs check`, but didn't interrupt the btrfsck, due to Sunk Cost Fallacy. > > The btrfsck is still running. It's of extremely dubious value now. > > * check the kernel logs > > * I grepped for btrfs, the mount points, compress, and zstd. I didn’t find a smoking gun in the right timeframe. > > > > not done yet: > > * mount the outer image > > * rebooted > > * tried a newer kernel. we're currently on kernel 6.1.129; we could go to newer 6.1 or 6.12 kernels > > * redo live file system compression, with e.g. `btrfs filesystem defrag -czstd` > > * fstrim the outer image > > > > goals: > > * work out what happened. > > How can we help? > > * help avoid it happening again, to others > > * salvage what we can > > > > I've run `bugreport` as a non-privileged user. Let me know if root access would give a fuller picture. > > I believe the best thing you could do here is to contact actually > upstream people directly. get_maintainers and the MAINTAINERS file > has: > > BTRFS FILE SYSTEM > M: Chris Mason <clm@fb.com> > M: Josef Bacik <josef@toxicpanda.com> > M: David Sterba <dsterba@suse.com> > L: linux-btrfs@vger.kernel.org > S: Maintained > W: https://btrfs.readthedocs.io > Q: https://patchwork.kernel.org/project/linux-btrfs/list/ > C: irc://irc.libera.chat/btrfs > T: git git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git > F: Documentation/filesystems/btrfs.rst > F: fs/btrfs/ > F: include/linux/btrfs* > F: include/trace/events/btrfs.h > F: include/uapi/linux/btrfs* > > So I would suggest you to contact above maintainers including the > list. > > Please keep this downstream bugreport as well in the recipients list. > > Regards, > Salvatore
[toc] | [prev] | [next] | [standalone]
| From | james young <pronoiac@gmail.com> |
|---|---|
| Date | 2025-09-25 20:00 +0200 |
| Message-ID | <LyYq5-wDl-1@gated-at.bofh.it> |
| In reply to | #89374 |
I'm not sure what timeline to expect for a response. Would a tarball of the outer image preserve everything needed for diagnostics? -James On Tue, Sep 23, 2025 at 2:10 PM james young <pronoiac@gmail.com> wrote: > > I hit an issue with btrfs compression; I reported it to Debian, which > I was using, and they suggested that I take it upstream. > > Thanks, Salvatore. My apologies to everyone if I misunderstood. > > -James > > On Tue, Sep 23, 2025 at 1:50 PM Salvatore Bonaccorso <carnil@debian.org> wrote: > > > > Control: tags -1 + moreinfo > > > > Hi James, > > > > On Tue, Sep 23, 2025 at 08:04:25PM +0200, James Young wrote: > > > Package: src:linux > > > Version: 6.1.129-1 > > > Severity: normal > > > X-Debbugs-Cc: pronoiac@gmail.com > > > > > > Dear Maintainer, > > > > > > > > > * What led up to the situation? > > > We made empty files in a loop, in parallel, under CPU and I/O load. > > > We had an outer Btrfs image file with compression, which contained a Btrfs image file, which contained billions of empty files. > > > We wrote around 100TB to the inner image file. > > > Around 60TB in, compression quietly shut off. > > > We ran out of space; both mounts presented i/o errors. > > > > > > * What exactly did you do (or not do) that was effective (or ineffective)? > > > * I unmounted the inner and outer images. > > > I didn't take note of memory usage before this point. > > > * dump debug info for the outer image - `btrfs inspect-internal dump-tree --dfs ...` > > > * We started a btrfsck. (twice, actually; breadth-first hit memory limits, I think) > > > After that, I learned about `btrfs check`, but didn't interrupt the btrfsck, due to Sunk Cost Fallacy. > > > The btrfsck is still running. It's of extremely dubious value now. > > > * check the kernel logs > > > * I grepped for btrfs, the mount points, compress, and zstd. I didn’t find a smoking gun in the right timeframe. > > > > > > not done yet: > > > * mount the outer image > > > * rebooted > > > * tried a newer kernel. we're currently on kernel 6.1.129; we could go to newer 6.1 or 6.12 kernels > > > * redo live file system compression, with e.g. `btrfs filesystem defrag -czstd` > > > * fstrim the outer image > > > > > > goals: > > > * work out what happened. > > > How can we help? > > > * help avoid it happening again, to others > > > * salvage what we can > > > > > > I've run `bugreport` as a non-privileged user. Let me know if root access would give a fuller picture. > > > > I believe the best thing you could do here is to contact actually > > upstream people directly. get_maintainers and the MAINTAINERS file > > has: > > > > BTRFS FILE SYSTEM > > M: Chris Mason <clm@fb.com> > > M: Josef Bacik <josef@toxicpanda.com> > > M: David Sterba <dsterba@suse.com> > > L: linux-btrfs@vger.kernel.org > > S: Maintained > > W: https://btrfs.readthedocs.io > > Q: https://patchwork.kernel.org/project/linux-btrfs/list/ > > C: irc://irc.libera.chat/btrfs > > T: git git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git > > F: Documentation/filesystems/btrfs.rst > > F: fs/btrfs/ > > F: include/linux/btrfs* > > F: include/trace/events/btrfs.h > > F: include/uapi/linux/btrfs* > > > > So I would suggest you to contact above maintainers including the > > list. > > > > Please keep this downstream bugreport as well in the recipients list. > > > > Regards, > > Salvatore
[toc] | [prev] | [next] | [standalone]
| From | james young <pronoiac@gmail.com> |
|---|---|
| Date | 2025-10-15 04:10 +0200 |
| Message-ID | <LFZ7H-5h9b-3@gated-at.bofh.it> |
| In reply to | #89409 |
Whoops, I hadn't intended to top-post... I'll do it correctly this time. On Thu, Sep 25, 2025 at 12:57 PM james young <pronoiac@gmail.com> wrote: > > I'm not sure what timeline to expect for a response. > Would a tarball of the outer image preserve everything needed for diagnostics? > > -James > > On Tue, Sep 23, 2025 at 2:10 PM james young <pronoiac@gmail.com> wrote: > > > > I hit an issue with btrfs compression; I reported it to Debian, which > > I was using, and they suggested that I take it upstream. > > > > Thanks, Salvatore. My apologies to everyone if I misunderstood. > > > > -James > > > > On Tue, Sep 23, 2025 at 1:50 PM Salvatore Bonaccorso <carnil@debian.org> wrote: > > > > > > Control: tags -1 + moreinfo > > > > > > Hi James, > > > > > > On Tue, Sep 23, 2025 at 08:04:25PM +0200, James Young wrote: > > > > Package: src:linux > > > > Version: 6.1.129-1 > > > > Severity: normal > > > > X-Debbugs-Cc: pronoiac@gmail.com > > > > > > > > Dear Maintainer, > > > > > > > > > > > > * What led up to the situation? > > > > We made empty files in a loop, in parallel, under CPU and I/O load. > > > > We had an outer Btrfs image file with compression, which contained a Btrfs image file, which contained billions of empty files. > > > > We wrote around 100TB to the inner image file. > > > > Around 60TB in, compression quietly shut off. > > > > We ran out of space; both mounts presented i/o errors. > > > > > > > > * What exactly did you do (or not do) that was effective (or ineffective)? > > > > * I unmounted the inner and outer images. > > > > I didn't take note of memory usage before this point. > > > > * dump debug info for the outer image - `btrfs inspect-internal dump-tree --dfs ...` > > > > * We started a btrfsck. (twice, actually; breadth-first hit memory limits, I think) > > > > After that, I learned about `btrfs check`, but didn't interrupt the btrfsck, due to Sunk Cost Fallacy. > > > > The btrfsck is still running. It's of extremely dubious value now. > > > > * check the kernel logs > > > > * I grepped for btrfs, the mount points, compress, and zstd. I didn’t find a smoking gun in the right timeframe. > > > > > > > > not done yet: > > > > * mount the outer image > > > > * rebooted > > > > * tried a newer kernel. we're currently on kernel 6.1.129; we could go to newer 6.1 or 6.12 kernels > > > > * redo live file system compression, with e.g. `btrfs filesystem defrag -czstd` > > > > * fstrim the outer image > > > > > > > > goals: > > > > * work out what happened. > > > > How can we help? > > > > * help avoid it happening again, to others > > > > * salvage what we can > > > > > > > > I've run `bugreport` as a non-privileged user. Let me know if root access would give a fuller picture. > > > > > > I believe the best thing you could do here is to contact actually > > > upstream people directly. get_maintainers and the MAINTAINERS file > > > has: > > > > > > BTRFS FILE SYSTEM > > > M: Chris Mason <clm@fb.com> > > > M: Josef Bacik <josef@toxicpanda.com> > > > M: David Sterba <dsterba@suse.com> > > > L: linux-btrfs@vger.kernel.org > > > S: Maintained > > > W: https://btrfs.readthedocs.io > > > Q: https://patchwork.kernel.org/project/linux-btrfs/list/ > > > C: irc://irc.libera.chat/btrfs > > > T: git git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git > > > F: Documentation/filesystems/btrfs.rst > > > F: fs/btrfs/ > > > F: include/linux/btrfs* > > > F: include/trace/events/btrfs.h > > > F: include/uapi/linux/btrfs* > > > > > > So I would suggest you to contact above maintainers including the > > > list. > > > > > > Please keep this downstream bugreport as well in the recipients list. > > > > > > Regards, > > > Salvatore I made a tarball of the file system, then mounted and looked at the file systems. I attempted to recompress (with btrfs defrag) and fstrim, with little success in freeing up space. I started btrfs check with the progress option; within two hours, it had gotten to “[2/7] checking extents, 82 items checked”. I confused the extents with the compressed chunk length - 128KiB - so that seemed woefully low on progress. Over a week later, it’s still "82 items checked". It’s still taking CPU (3% right now) and gigs of memory; it’s doing something, though slowly. So, a question: * is this business as usual for a btrfs check? * is this a clue about what happened? * is this a symptom? If this is a useful metric for file system robustness, is this something I could / should experiment with to shorten? * run `sync` * periodically pause writes, to let the buffers empty Any thoughts or suggestions? -James
[toc] | [prev] | [standalone]
Back to top | Article view | linux.debian.kernel
csiph-web