Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.debian.kernel > #79241 > unrolled thread
| Started by | Niels Hendriks <niels@rootnet.nl> |
|---|---|
| First post | 2023-06-08 12:10 +0200 |
| Last post | 2023-06-08 22:00 +0200 |
| Articles | 4 — 3 participants |
Back to article view | Back to linux.debian.kernel
This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by
below is the oldest one visible, not the original post.
Bug#1037223: Possible bug causing I/O hangs Niels Hendriks <niels@rootnet.nl> - 2023-06-08 12:10 +0200
Processed: Re: Bug#1037223: Possible bug causing I/O hangs "Debian Bug Tracking System" <owner@bugs.debian.org> - 2023-06-08 21:30 +0200
Bug#1037223: Possible bug causing I/O hangs Salvatore Bonaccorso <carnil@debian.org> - 2023-06-08 21:30 +0200
Bug#1037223: Possible bug causing I/O hangs Niels Hendriks <niels@rootnet.nl> - 2023-06-08 22:00 +0200
| From | Niels Hendriks <niels@rootnet.nl> |
|---|---|
| Date | 2023-06-08 12:10 +0200 |
| Subject | Bug#1037223: Possible bug causing I/O hangs |
| Message-ID | <GEkKB-eu7e-5@gated-at.bofh.it> |
[Multipart message — attachments visible in raw view] — view raw
Package: linux-image-amd64
Version: 5.10.178-3
Hi all,
I do not usually report kernel bugs so hopefully this is the right place!
We recently updated the kernel of our Debian 11 servers and since then we have encountered a bunch of servers (both VMs and bare metal) that suffer I/O hanging issues.
We can access the server through a console where I cannot copy text, but I have attached a screenshot showing the message we see in dmesg.
We initially thought this was related to the ext4 fast_commit feature flag we have enabled, and we do feel the issue occurs less often with fast_commit disabled, but it does not appear to be solved completely when we disable this feature.
With this error, we've been googling a bit and I ended up on this thread: https://www.spinics.net/lists/linux-ext4/msg86261.html through initially https://github.com/flatcar/Flatcar/issues/847
It mentions this fix: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/fs/ext4?h=linux-5.15.y&id=5bc0b2fda4b47c86278f7c6d30c211f425bf51cf
I believe this fix is currently not present in the 5.10 kernel available for Debian 11.
However, the linked fix also mentions:
> This bug has been around for many years, but it became *much* easier
to hit after commit 65f8b80053a1 ("ext4: fix race when reusing xattr
blocks").
Looking at the changelog: https://metadata.ftp-master.debian.org/changelogs//main/l/linux-signed-amd64/linux-signed-amd64_5.10.178+3_changelog
We do see the "ext4: fix race when reusing xattr blocks" change being added in 5.10.178-1.
This is why we believe we are now hitting this bug.
My question is whether this seems plausible, and if so, whether the fix I linked can also be released for Debian 11?
We could also upgrade to the bullseye-backports kernel, but given that this issue makes the system essentially unusable and we hit it every few days on one of our servers it may be more widespread and worth it to fix it in the regular bullseye kernel as well.
Thank you!
Best regards,
Niels Hendriks
[toc] | [next] | [standalone]
| From | "Debian Bug Tracking System" <owner@bugs.debian.org> |
|---|---|
| Date | 2023-06-08 21:30 +0200 |
| Subject | Processed: Re: Bug#1037223: Possible bug causing I/O hangs |
| Message-ID | <GEtux-ezhs-11@gated-at.bofh.it> |
| In reply to | #79241 |
Processing control commands: > tags -1 + moreinfo Bug #1037223 [src:linux] Possible bug causing I/O hangs Added tag(s) moreinfo. -- 1037223: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1037223 Debian Bug Tracking System Contact owner@bugs.debian.org with problems
[toc] | [prev] | [next] | [standalone]
| From | Salvatore Bonaccorso <carnil@debian.org> |
|---|---|
| Date | 2023-06-08 21:30 +0200 |
| Message-ID | <GEtux-ezhs-9@gated-at.bofh.it> |
| In reply to | #79241 |
Control: tags -1 + moreinfo
Hi Niels,
On Thu, Jun 08, 2023 at 11:33:13AM +0200, Niels Hendriks wrote:
> Package: linux-image-amd64
> Version: 5.10.178-3
From the screenshot I guess you mean 5.10.179-1, or possibly already
in 5.10.178-3?
>
>
> Hi all,
>
> I do not usually report kernel bugs so hopefully this is the right
> place!
>
> We recently updated the kernel of our Debian 11 servers and since
> then we have encountered a bunch of servers (both VMs and bare
> metal) that suffer I/O hanging issues.
> We can access the server through a console where I cannot copy text,
> but I have attached a screenshot showing the message we see in
> dmesg.
>
> We initially thought this was related to the ext4 fast_commit
> feature flag we have enabled, and we do feel the issue occurs less
> often with fast_commit disabled, but it does not appear to be solved
> completely when we disable this feature.
>
> With this error, we've been googling a bit and I ended up on this
> thread: https://www.spinics.net/lists/linux-ext4/msg86261.html
> through initially https://github.com/flatcar/Flatcar/issues/847 It
> mentions this
> fix: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/fs/ext4?h=linux-5.15.y&id=5bc0b2fda4b47c86278f7c6d30c211f425bf51cf
> I believe this fix is currently not present in the 5.10 kernel
> available for Debian 11.
That commit is upstream commit
a44e84a9b7764c72896f7241a0ec9ac7e7ef38dd, which was backported to
various stable series, in particular 5.10.163 with
1be16a0c2f10186df505e28b0cc92d7f3366e2a8 .
>
> However, the linked fix also mentions:
> > This bug has been around for many years, but it became *much* easier
> to hit after commit 65f8b80053a1 ("ext4: fix race when reusing xattr
> blocks").
>
> Looking at the
> changelog: https://metadata.ftp-master.debian.org/changelogs//main/l/linux-signed-amd64/linux-signed-amd64_5.10.178+3_changelog
> We do see the "ext4: fix race when reusing xattr blocks" change
> being added in 5.10.178-1. This is why we believe we are now
> hitting this bug.
>
> My question is whether this seems plausible, and if so, whether the
> fix I linked can also be released for Debian 11?
Right now I do not see that to be the cause, as the above mentioned
commit *is* in the version, unless I'm missunderstanding.
>
> We could also upgrade to the bullseye-backports kernel, but given
> that this issue makes the system essentially unusable and we hit it
> every few days on one of our servers it may be more widespread and
> worth it to fix it in the regular bullseye kernel as well.
Do you had a 5.10.y kernel which was fine, and can you bisect the
changes between that version and 5.10.179 to pin point the first bad
commit causing the issue?
If your infrastructure is prepared to do so, next steps might involve
trying the most recent 5.10.y kernel to see if it still exhibit the
problem, then going up to newer stable series and/or mainline.
Please in particular test the current 5.10.182 upstream as it has
interesting ext4 related changes between 5.10.179 and 5.10.182.
Regards,
Salvatore
[toc] | [prev] | [next] | [standalone]
| From | Niels Hendriks <niels@rootnet.nl> |
|---|---|
| Date | 2023-06-08 22:00 +0200 |
| Message-ID | <GEtXA-ezsg-1@gated-at.bofh.it> |
| In reply to | #79245 |
[Multipart message — attachments visible in raw view] — view raw
Hi Salvatore,
Thanks for your response!
> From the screenshot I guess you mean 5.10.179-1, or possibly already
in 5.10.178-3?
Ah yes, I took the version from https://packages.debian.org/bullseye/linux-image-amd64 which mentions 5.10.178-3 as the current version, however indeed we are running 5.10.179-1:
apt-cache policy linux-image-amd64
linux-image-amd64:
Installed: 5.10.179-1
Candidate: 5.10.179-1
Version table:
6.1.20-2~bpo11+1 100
100 http://ftp.nl.debian.org/debian bullseye-backports/main amd64 Packages
*** 5.10.179-1 500
500 http://security.debian.org/debian-security bullseye-security/updates/main amd64 Packages
100 /var/lib/dpkg/status
5.10.178-3 500
500 http://ftp.nl.debian.org/debian bullseye/main amd64 Packages
I can see that we upgraded from 5.10.158-2, where the issue did not occur. Interestingly this is *older* than the version you mentioned as having received the fix for the issue I thought was the cause (5.10.163).
Thank you for for confirming the possible fix I mentioned is already in the kernel we are running. This is unfortunate for me as it means the cause remains unknown.
We have a VM where we can sort-of reproduce the issue but we haven't been able to reliably reproduce it. I.e. it takes ~18-24 hours of stress-testing the VM before we see the issue occur. This is why bisecting will be difficult, but I understand it is very helpful if we are able to do so.
I will report back if I have any additional information.
Best regards,
Niels Hendriks
From: Salvatore Bonaccorso <carnil@debian.org>
To: Niels Hendriks <niels@rootnet.nl>, <1037223@bugs.debian.org>
Sent: 08/06/2023 9:23 PM
Subject: Re: Bug#1037223: Possible bug causing I/O hangs
Control: tags -1 + moreinfo
Hi Niels,
On Thu, Jun 08, 2023 at 11:33:13AM +0200, Niels Hendriks wrote:
> Package: linux-image-amd64
> Version: 5.10.178-3
From the screenshot I guess you mean 5.10.179-1, or possibly already
in 5.10.178-3?
>
>
> Hi all,
>
> I do not usually report kernel bugs so hopefully this is the right
> place!
>
> We recently updated the kernel of our Debian 11 servers and since
> then we have encountered a bunch of servers (both VMs and bare
> metal) that suffer I/O hanging issues.
> We can access the server through a console where I cannot copy text,
> but I have attached a screenshot showing the message we see in
> dmesg.
>
> We initially thought this was related to the ext4 fast_commit
> feature flag we have enabled, and we do feel the issue occurs less
> often with fast_commit disabled, but it does not appear to be solved
> completely when we disable this feature.
>
> With this error, we've been googling a bit and I ended up on this
> thread: https://www.spinics.net/lists/linux-ext4/msg86261.html
> through initially https://github.com/flatcar/Flatcar/issues/847 It
> mentions this
> fix: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/fs/ext4?h=linux-5.15.y&id=5bc0b2fda4b47c86278f7c6d30c211f425bf51cf
> I believe this fix is currently not present in the 5.10 kernel
> available for Debian 11.
That commit is upstream commit
a44e84a9b7764c72896f7241a0ec9ac7e7ef38dd, which was backported to
various stable series, in particular 5.10.163 with
1be16a0c2f10186df505e28b0cc92d7f3366e2a8 .
>
> However, the linked fix also mentions:
> > This bug has been around for many years, but it became *much* easier
> to hit after commit 65f8b80053a1 ("ext4: fix race when reusing xattr
> blocks").
>
> Looking at the
> changelog: https://metadata.ftp-master.debian.org/changelogs//main/l/linux-signed-amd64/linux-signed-amd64_5.10.178+3_changelog
> We do see the "ext4: fix race when reusing xattr blocks" change
> being added in 5.10.178-1. This is why we believe we are now
> hitting this bug.
>
> My question is whether this seems plausible, and if so, whether the
> fix I linked can also be released for Debian 11?
Right now I do not see that to be the cause, as the above mentioned
commit *is* in the version, unless I'm missunderstanding.
>
> We could also upgrade to the bullseye-backports kernel, but given
> that this issue makes the system essentially unusable and we hit it
> every few days on one of our servers it may be more widespread and
> worth it to fix it in the regular bullseye kernel as well.
Do you had a 5.10.y kernel which was fine, and can you bisect the
changes between that version and 5.10.179 to pin point the first bad
commit causing the issue?
If your infrastructure is prepared to do so, next steps might involve
trying the most recent 5.10.y kernel to see if it still exhibit the
problem, then going up to newer stable series and/or mainline.
Please in particular test the current 5.10.182 upstream as it has
interesting ext4 related changes between 5.10.179 and 5.10.182.
Regards,
Salvatore
[toc] | [prev] | [standalone]
Back to top | Article view | linux.debian.kernel
csiph-web