Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #81352 > unrolled thread

Bug#1059624: linux-image-6.1.0-16-amd64: aacraid abort request / SCSI hang after upgrade from 11.8 -> 12.4

Back to article view | Back to linux.debian.kernel

Contents

  Bug#1059624: linux-image-6.1.0-16-amd64: aacraid abort request / SCSI hang after upgrade from 11.8 -> 12.4 Samuel Wolf <samuelwolf85@googlemail.com> - 2023-12-29 14:50 +0100
    Bug#1059624: linux-image-6.1.0-16-amd64: aacraid abort request / SCSI hang after upgrade from 11.8 -> 12.4 Salvatore Bonaccorso <carnil@debian.org> - 2023-12-29 15:10 +0100
      Bug#1059624: linux-image-6.1.0-16-amd64: aacraid abort request / SCSI hang after upgrade from 11.8 -> 12.4 Samuel Wolf <samuelwolf85@googlemail.com> - 2023-12-29 16:00 +0100
        Bug#1059624: linux-image-6.1.0-16-amd64: aacraid abort request / SCSI hang after upgrade from 11.8 -> 12.4 Salvatore Bonaccorso <carnil@debian.org> - 2023-12-29 17:50 +0100
          Bug#1059624: linux-image-6.1.0-16-amd64: aacraid abort request / SCSI hang after upgrade from 11.8 -> 12.4 Samuel Wolf <samuelwolf85@googlemail.com> - 2023-12-29 20:40 +0100
            Bug#1059624: linux-image-6.1.0-16-amd64: aacraid abort request / SCSI hang after upgrade from 11.8 -> 12.4 Salvatore Bonaccorso <carnil@debian.org> - 2023-12-29 21:10 +0100
              Bug#1059624: linux-image-6.1.0-16-amd64: aacraid abort request / SCSI hang after upgrade from 11.8 -> 12.4 Samuel Wolf <samuelwolf85@googlemail.com> - 2023-12-30 01:10 +0100
                Bug#1059624: linux-image-6.1.0-16-amd64: aacraid abort request / SCSI hang after upgrade from 11.8 -> 12.4 Salvatore Bonaccorso <carnil@debian.org> - 2023-12-30 10:00 +0100
                  Bug#1059624: linux-image-6.1.0-16-amd64: aacraid abort request / SCSI hang after upgrade from 11.8 -> 12.4 Samuel Wolf <samuelwolf85@googlemail.com> - 2023-12-30 10:50 +0100
                    Bug#1059624: linux-image-6.1.0-16-amd64: aacraid abort request / SCSI hang after upgrade from 11.8 -> 12.4 Salvatore Bonaccorso <carnil@debian.org> - 2023-12-30 12:20 +0100
                      Bug#1059624: linux-image-6.1.0-16-amd64: aacraid abort request / SCSI hang after upgrade from 11.8 -> 12.4 Samuel Wolf <samuelwolf85@googlemail.com> - 2023-12-30 16:10 +0100
    Processed: Re: Bug#1059624: linux-image-6.1.0-16-amd64: aacraid  abort request / SCSI hang after upgrade from 11.8 -> 12.4 "Debian Bug Tracking System" <owner@bugs.debian.org> - 2023-12-29 15:10 +0100
    Processed: Re: Bug#1059624: linux-image-6.1.0-16-amd64: aacraid  abort request / SCSI hang after upgrade from 11.8 -> 12.4 "Debian Bug Tracking System" <owner@bugs.debian.org> - 2023-12-30 10:00 +0100
    Bug#1059624: marked as done (linux-image-6.1.0-16-amd64: aacraid  abort request / SCSI hang after upgrade from 11.8 -> 12.4) "Debian Bug Tracking System" <owner@bugs.debian.org> - 2024-01-01 20:10 +0100
    Bug#1059624: marked as done (linux-image-6.1.0-16-amd64: aacraid  abort request / SCSI hang after upgrade from 11.8 -> 12.4) "Debian Bug Tracking System" <owner@bugs.debian.org> - 2024-01-03 08:00 +0100

#81352 — Bug#1059624: linux-image-6.1.0-16-amd64: aacraid abort request / SCSI hang after upgrade from 11.8 -> 12.4

Package: src:linux
Version: 6.1.67-1
Severity: normal
X-Debbugs-Cc: samuelwolf85@googlemail.com

Hello,

we upgraded our server with Adaptec ASR8805 raid controller from Debian 11.8 to 12.4.

After booting into the 6.1x kernel the system boot and works without load,
but as soon the server has some load the system stops/freeze with abort request / SCSI hang.

This is a known issue, is it possible to backport this bugfix into Debian 12?
https://bugzilla.kernel.org/show_bug.cgi?id=217599


-- Package-specific info:
** Version:
Linux version 6.1.0-16-amd64 (debian-kernel@lists.debian.org) (gcc-12 (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC Debian 6.1.67-1 (2023-12-12)

** Command line:
BOOT_IMAGE=/boot/vmlinuz-6.1.0-16-amd64 root=UUID=3f04dcfb-f323-4b62-aefa-219e71ea4f35 ro quiet splash

** Not tainted

[toc] | [next] | [standalone]

#81354

Control: tags -1 + moreinfo

Hi Samuel,

On Fri, Dec 29, 2023 at 02:44:35PM +0100, Samuel Wolf wrote:
> Package: src:linux
> Version: 6.1.67-1
> Severity: normal
> X-Debbugs-Cc: samuelwolf85@googlemail.com
> 
> Hello,
> 
> we upgraded our server with Adaptec ASR8805 raid controller from Debian 11.8 to 12.4.
> 
> After booting into the 6.1x kernel the system boot and works without load,
> but as soon the server has some load the system stops/freeze with abort request / SCSI hang.
> 
> This is a known issue, is it possible to backport this bugfix into Debian 12?
> https://bugzilla.kernel.org/show_bug.cgi?id=217599

And can you confirm that the patch revert fixes your issue? it is most
likely so looking trough the upstream bug log, but an additional
confirmation is welcome. The revert landed in mainline, but has not
been queued for the stable series yet.

Regards,
Salvatore

[toc] | [prev] | [next] | [standalone]

#81357

Hi Salvatore,

> And can you confirm that the patch revert fixes your issue?

unfortunately not, we downgraded to Debian 11 to get the server
working again and
I have not enough knowledge to build and test such an kernel.

> The revert landed in mainline, but has not been queued for the stable series yet.

I guess it's better to wait for the stable series revert (from the
Debian standpoint) and backport it than into Debian 12?

Thanks.

Samuel

[toc] | [prev] | [next] | [standalone]

#81358

Hi Samuel,

On Fri, Dec 29, 2023 at 03:52:58PM +0100, Samuel Wolf wrote:
> Hi Salvatore,
> 
> > And can you confirm that the patch revert fixes your issue?
> 
> unfortunately not, we downgraded to Debian 11 to get the server
> working again and
> I have not enough knowledge to build and test such an kernel.

if you are allowed to deploy unofficial builds: Would you be willing
to test that the packages in
https://people.debian.org/~carnil/tmp/linux/1059624/ fix the problem?
They contain that specific revert on top. I have signed the sha256sum
file with my key found in the Debian keyring.

> > The revert landed in mainline, but has not been queued for the stable series yet.
> 
> I guess it's better to wait for the stable series revert (from the
> Debian standpoint) and backport it than into Debian 12?

It's marked for stable and people on the bug did confirm it fix the
issue, I'm pretty confident that it will be picked up as well for the
stable queues, and quess the whole was delayed.

I cannot really determine right now how many people are affected.
There will be quite soon a new bookworm upload, the next one following
will be likely in february for the point release.

So in case you manage to test the above packages quite soon there is a
chance we can have it in the next upload. Otherwise in the next after.

Regards,
Salvatore

[toc] | [prev] | [next] | [standalone]

#81359

Hi Salvatore,

> if you are allowed to deploy unofficial builds: Would you be willing
> to test that the packages in
> https://people.debian.org/~carnil/tmp/linux/1059624/ fix the problem?
> They contain that specific revert on top. I have signed the sha256sum
> file with my key found in the Debian keyring.

thank you, I can test this kernel on a test server with the same
Adaptec controller.

> I cannot really determine right now how many people are affected.

In theory, anyone who uses aacraid drivers/controllers.

> So in case you manage to test the above packages quite soon there is a
> chance we can have it in the next upload. Otherwise in the next after.

I'll try to test this by saturday evening, is that too late?

Samuel

[toc] | [prev] | [next] | [standalone]

#81362

Hi Samuel,

On Fri, Dec 29, 2023 at 08:30:55PM +0100, Samuel Wolf wrote:
> Hi Salvatore,
> 
> > if you are allowed to deploy unofficial builds: Would you be willing
> > to test that the packages in
> > https://people.debian.org/~carnil/tmp/linux/1059624/ fix the problem?
> > They contain that specific revert on top. I have signed the sha256sum
> > file with my key found in the Debian keyring.
> 
> thank you, I can test this kernel on a test server with the same
> Adaptec controller.
> 
> > I cannot really determine right now how many people are affected.
> 
> In theory, anyone who uses aacraid drivers/controllers.

Right, but apparently not every controller type, but at least we know
some types affected.

> > So in case you manage to test the above packages quite soon there is a
> > chance we can have it in the next upload. Otherwise in the next after.
> 
> I'll try to test this by saturday evening, is that too late?

Well yes the timing is a bit unfortunate. I was working today on
finalizing the upload for bookworm (6.1.69-1) and bullseye
(5.10.205-1), before then your bugreport hit the bugtracker.

But at this point I guess I can delay that finalizing a bit further to
see your testing happening. So it would be welcome if you find time to
make it possible to test it by saturday evening.

Regards,
Salvatore

[toc] | [prev] | [next] | [standalone]

#81365

Hi Salvatore,

>  So it would be welcome if you find time to make it possible to test it by saturday evening.

my test was quicker than expected since i found a way to reproduce the
issue on my test server.

Behind the Adaptec 8805 is a raid6 storage with 54TB and LUKS encrypted.
As soon I open and mount the LUKS drive with kernel 6.1.67-1 the
controller hang:

[  480.888273] aacraid: Host adapter abort request.
               aacraid: Outstanding commands on (0,0,3,0):
[  480.902784] aacraid: Host bus reset request. SCSI hang ?
[  480.902933] aacraid 0000:02:00.0: outstanding cmd: midlevel-0
[  480.902935] aacraid 0000:02:00.0: outstanding cmd: lowlevel-0
[  480.902936] aacraid 0000:02:00.0: outstanding cmd: error handler-0
[  480.902936] aacraid 0000:02:00.0: outstanding cmd: firmware-251
[  480.902937] aacraid 0000:02:00.0: outstanding cmd: kernel-0
[  480.916921] aacraid 0000:02:00.0: Controller reset type is 3
[  480.917076] aacraid 0000:02:00.0: Issuing IOP reset
[  517.004437] aacraid 0000:02:00.0: IOP reset succeeded
[  517.029007] aacraid: Comm Interface type2 enabled
[  529.479247] aacraid 0000:02:00.0: Scheduling bus rescan
[  539.678274] aacraid 0000:02:00.0: DDR cache data recovered successfull

This is reproducible with every luksClose and luksOpen mount.

Now I booting into your test kernel 6.1.67-1a~test and try the same again:

[    9.610151] IPv6: ADDRCONF(NETDEV_CHANGE): enp1s0: link becomes ready
[   81.503552] EXT4-fs (dm-0): mounted filesystem with ordered data
mode. Quota mode: none.
[  119.133460] EXT4-fs (dm-0): unmounting filesystem.
[  138.547366] sd 0:0:3:0: [sda] Very big device. Trying to use READ
CAPACITY(16).
[  139.214205] EXT4-fs (dm-0): mounted filesystem with ordered data
mode. Quota mode: none.
[  162.376044] EXT4-fs (dm-0): unmounting filesystem.
[  182.222397] sd 0:0:3:0: [sda] Very big device. Trying to use READ
CAPACITY(16).
[  182.913977] EXT4-fs (dm-0): mounted filesystem with ordered data
mode. Quota mode: none.
[  217.611072] EXT4-fs (dm-0): unmounting filesystem.
[  230.778060] sd 0:0:3:0: [sda] Very big device. Trying to use READ
CAPACITY(16).
[  231.386349] EXT4-fs (dm-0): mounted filesystem with ordered data
mode. Quota mode: none.

No errors and the LUKS device is opened in ~1 second not like before
in 1 minute.

Since I can not technical overview the patch/revert, is this enough
testing for you?

Thanks for the test kernel.

Samuel

[toc] | [prev] | [next] | [standalone]

#81366

Control: tags -1 + confirmed pending
Control: tags -1 - moreinfo

On Sat, Dec 30, 2023 at 01:06:20AM +0100, Samuel Wolf wrote:
> Hi Salvatore,
> 
> >  So it would be welcome if you find time to make it possible to test it by saturday evening.
> 
> my test was quicker than expected since i found a way to reproduce the
> issue on my test server.
> 
> Behind the Adaptec 8805 is a raid6 storage with 54TB and LUKS encrypted.
> As soon I open and mount the LUKS drive with kernel 6.1.67-1 the
> controller hang:
> 
> [  480.888273] aacraid: Host adapter abort request.
>                aacraid: Outstanding commands on (0,0,3,0):
> [  480.902784] aacraid: Host bus reset request. SCSI hang ?
> [  480.902933] aacraid 0000:02:00.0: outstanding cmd: midlevel-0
> [  480.902935] aacraid 0000:02:00.0: outstanding cmd: lowlevel-0
> [  480.902936] aacraid 0000:02:00.0: outstanding cmd: error handler-0
> [  480.902936] aacraid 0000:02:00.0: outstanding cmd: firmware-251
> [  480.902937] aacraid 0000:02:00.0: outstanding cmd: kernel-0
> [  480.916921] aacraid 0000:02:00.0: Controller reset type is 3
> [  480.917076] aacraid 0000:02:00.0: Issuing IOP reset
> [  517.004437] aacraid 0000:02:00.0: IOP reset succeeded
> [  517.029007] aacraid: Comm Interface type2 enabled
> [  529.479247] aacraid 0000:02:00.0: Scheduling bus rescan
> [  539.678274] aacraid 0000:02:00.0: DDR cache data recovered successfull
> 
> This is reproducible with every luksClose and luksOpen mount.
> 
> Now I booting into your test kernel 6.1.67-1a~test and try the same again:
> 
> [    9.610151] IPv6: ADDRCONF(NETDEV_CHANGE): enp1s0: link becomes ready
> [   81.503552] EXT4-fs (dm-0): mounted filesystem with ordered data
> mode. Quota mode: none.
> [  119.133460] EXT4-fs (dm-0): unmounting filesystem.
> [  138.547366] sd 0:0:3:0: [sda] Very big device. Trying to use READ
> CAPACITY(16).
> [  139.214205] EXT4-fs (dm-0): mounted filesystem with ordered data
> mode. Quota mode: none.
> [  162.376044] EXT4-fs (dm-0): unmounting filesystem.
> [  182.222397] sd 0:0:3:0: [sda] Very big device. Trying to use READ
> CAPACITY(16).
> [  182.913977] EXT4-fs (dm-0): mounted filesystem with ordered data
> mode. Quota mode: none.
> [  217.611072] EXT4-fs (dm-0): unmounting filesystem.
> [  230.778060] sd 0:0:3:0: [sda] Very big device. Trying to use READ
> CAPACITY(16).
> [  231.386349] EXT4-fs (dm-0): mounted filesystem with ordered data
> mode. Quota mode: none.
> 
> No errors and the LUKS device is opened in ~1 second not like before
> in 1 minute.
> 
> Since I can not technical overview the patch/revert, is this enough
> testing for you?
> 
> Thanks for the test kernel.

Thanks for your testing! Yes this is enough from your side, thanks a
lot for taking the time for the explict test rounds!

Regards,
Salvatore

[toc] | [prev] | [next] | [standalone]

#81369

Hi Salvatore,

> Thanks for your testing! Yes this is enough from your side, thanks a
> lot for taking the time for the explict test rounds!

no problem, thanks for all you work on the Debian project!

I hope you get a feedback on your question here:
https://lore.kernel.org/all/ZY8oXge0QKYUkO1b@eldamar.lan/

I've been very careful since the last O_DIRECT issue..
But on the other hand, hanging storage is also not without danger.

Samuel

[toc] | [prev] | [next] | [standalone]

#81371

Hi,

On Sat, Dec 30, 2023 at 10:44:27AM +0100, Samuel Wolf wrote:
> Hi Salvatore,
> 
> > Thanks for your testing! Yes this is enough from your side, thanks a
> > lot for taking the time for the explict test rounds!
> 
> no problem, thanks for all you work on the Debian project!

Very welcome as well. Thanks for this kind and very motivating
feedback.

> I hope you get a feedback on your question here:
> https://lore.kernel.org/all/ZY8oXge0QKYUkO1b@eldamar.lan/

Greg just queued it:
https://lore.kernel.org/all/2023123013-dose-skirmish-27c2@gregkh/

> I've been very careful since the last O_DIRECT issue..
> But on the other hand, hanging storage is also not without danger.

Yes fully understandable. Even though the impact was quite targeted,
as Jan Kara explained https://lwn.net/Articles/954841/, this was a
major hassle to handle right in the mittle of of the point release
pushing.

Regards,
Salvatore

[toc] | [prev] | [next] | [standalone]

#81372

Hi Salvatore,

> Greg just queued it:
> https://lore.kernel.org/all/2023123013-dose-skirmish-27c2@gregkh/

queued sounds good, so happy there is a solution in sight.

The good thing about this bug is that we can use our "faulty" server
again, we thought the server had a hardware issue:
https://bugzilla.kernel.org/show_bug.cgi?id=217599#c55

Samuel

[toc] | [prev] | [next] | [standalone]

#81355 — Processed: Re: Bug#1059624: linux-image-6.1.0-16-amd64: aacraid abort request / SCSI hang after upgrade from 11.8 -> 12.4

Processing control commands:

> tags -1 + moreinfo
Bug #1059624 [src:linux] linux-image-6.1.0-16-amd64: aacraid abort request / SCSI hang after upgrade from 11.8 -> 12.4
Added tag(s) moreinfo.

-- 
1059624: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1059624
Debian Bug Tracking System
Contact owner@bugs.debian.org with problems

[toc] | [prev] | [next] | [standalone]

#81367 — Processed: Re: Bug#1059624: linux-image-6.1.0-16-amd64: aacraid abort request / SCSI hang after upgrade from 11.8 -> 12.4

Processing control commands:

> tags -1 + confirmed pending
Bug #1059624 [src:linux] linux-image-6.1.0-16-amd64: aacraid abort request / SCSI hang after upgrade from 11.8 -> 12.4
Added tag(s) confirmed and pending.
> tags -1 - moreinfo
Bug #1059624 [src:linux] linux-image-6.1.0-16-amd64: aacraid abort request / SCSI hang after upgrade from 11.8 -> 12.4
Removed tag(s) moreinfo.

-- 
1059624: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1059624
Debian Bug Tracking System
Contact owner@bugs.debian.org with problems

[toc] | [prev] | [next] | [standalone]

#81394 — Bug#1059624: marked as done (linux-image-6.1.0-16-amd64: aacraid abort request / SCSI hang after upgrade from 11.8 -> 12.4)

[Multipart message — attachments visible in raw view] — view raw

Your message dated Mon, 01 Jan 2024 19:02:08 +0000
with message-id <E1rKNY8-00Cm3b-Q5@fasolo.debian.org>
and subject line Bug#1059624: fixed in linux 6.6.9-1
has caused the Debian Bug report #1059624,
regarding linux-image-6.1.0-16-amd64: aacraid abort request / SCSI hang after upgrade from 11.8 -> 12.4
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact owner@bugs.debian.org
immediately.)


-- 
1059624: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1059624
Debian Bug Tracking System
Contact owner@bugs.debian.org with problems

[toc] | [prev] | [next] | [standalone]

#81417 — Bug#1059624: marked as done (linux-image-6.1.0-16-amd64: aacraid abort request / SCSI hang after upgrade from 11.8 -> 12.4)

[Multipart message — attachments visible in raw view] — view raw

Your message dated Wed, 03 Jan 2024 06:47:08 +0000
with message-id <E1rKv1w-00429O-Uz@fasolo.debian.org>
and subject line Bug#1059624: fixed in linux 6.1.69-1
has caused the Debian Bug report #1059624,
regarding linux-image-6.1.0-16-amd64: aacraid abort request / SCSI hang after upgrade from 11.8 -> 12.4
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact owner@bugs.debian.org
immediately.)


-- 
1059624: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1059624
Debian Bug Tracking System
Contact owner@bugs.debian.org with problems

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.kernel

csiph-web