Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #85204 > unrolled thread

Bug#1088159: linux-image-6.1.0-27-amd64 fails mpt3sas_cm0 _scsih_probe

Started bySalvatore Bonaccorso <carnil@debian.org>
First post2025-01-19 16:00 +0100
Last post2025-02-08 11:00 +0100
Articles 5 — 2 participants

Back to article view | Back to linux.debian.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Bug#1088159: linux-image-6.1.0-27-amd64 fails mpt3sas_cm0 _scsih_probe Salvatore Bonaccorso <carnil@debian.org> - 2025-01-19 16:00 +0100
    Processed: Re: Bug#1088159: linux-image-6.1.0-27-amd64 fails  mpt3sas_cm0 _scsih_probe "Debian Bug Tracking System" <owner@bugs.debian.org> - 2025-01-19 16:00 +0100
    Bug#1088159: linux-image-6.1.0-27-amd64 fails mpt3sas_cm0 _scsih_probe Salvatore Bonaccorso <carnil@debian.org> - 2025-01-22 21:00 +0100
    Bug#1088159: linux-image-6.1.0-27-amd64 fails mpt3sas_cm0 _scsih_probe Salvatore Bonaccorso <carnil@debian.org> - 2025-02-07 15:00 +0100
      Bug#1088159: linux-image-6.1.0-27-amd64 fails mpt3sas_cm0 _scsih_probe Salvatore Bonaccorso <carnil@debian.org> - 2025-02-08 11:00 +0100

#85204 — Bug#1088159: linux-image-6.1.0-27-amd64 fails mpt3sas_cm0 _scsih_probe

FromSalvatore Bonaccorso <carnil@debian.org>
Date2025-01-19 16:00 +0100
SubjectBug#1088159: linux-image-6.1.0-27-amd64 fails mpt3sas_cm0 _scsih_probe
Message-ID<K6EWl-acyA-5@gated-at.bofh.it>
Control: tags -1 + moreinfo

Hi,

On Sun, Nov 24, 2024 at 12:23:37PM +0100, Radoslav Bodó wrote:
> Package: linux-image-6.1.0-27-amd64
> Version: 6.1.115-1
> 
> Hello,
> 
> after standard patch management, the new kernel does not see mpt3sas device
> for HBA, syslong ends with
> 
> ```
> mpt3sas_cm0: failure at
> drivers/scsi/mpt3sas/mpt3sas_scsih.c:12348/_scsih_probe()!
> ```
> 
> Older release linux-image-6.1.0-26-amd64 works as expected
> The server
> 
> * Dell PowerEdge R440
> * HBA for SAS ME4012 disk array
> * running XEN xen-hypervisor-4.17-amd64

The logs were slightly harder to read due to wrapping so I extracted
them go easier diff.

6.1.0-27 (6.1.115-1)

mpt3sas version 43.100.00.00 loaded
mpt3sas_cm0: 63 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (8086116 kB)
mpt3sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k
mpt3sas_cm0: MSI-X vectors supported: 96
mpt3sas_cm0:  0 40 40
mpt3sas_cm0: High IOPs queues : disabled
mpt3sas0-msix0: PCI-MSI-X enabled: IRQ 447
mpt3sas0-msix1: PCI-MSI-X enabled: IRQ 448
mpt3sas0-msix2: PCI-MSI-X enabled: IRQ 449
mpt3sas0-msix3: PCI-MSI-X enabled: IRQ 450
mpt3sas0-msix4: PCI-MSI-X enabled: IRQ 451
mpt3sas0-msix5: PCI-MSI-X enabled: IRQ 452
mpt3sas0-msix6: PCI-MSI-X enabled: IRQ 453
mpt3sas0-msix7: PCI-MSI-X enabled: IRQ 454
mpt3sas0-msix8: PCI-MSI-X enabled: IRQ 455
mpt3sas0-msix9: PCI-MSI-X enabled: IRQ 456
mpt3sas0-msix10: PCI-MSI-X enabled: IRQ 457
mpt3sas0-msix11: PCI-MSI-X enabled: IRQ 458
mpt3sas0-msix12: PCI-MSI-X enabled: IRQ 459
mpt3sas0-msix13: PCI-MSI-X enabled: IRQ 460
mpt3sas0-msix14: PCI-MSI-X enabled: IRQ 461
mpt3sas0-msix15: PCI-MSI-X enabled: IRQ 462
mpt3sas0-msix16: PCI-MSI-X enabled: IRQ 463
mpt3sas0-msix17: PCI-MSI-X enabled: IRQ 464
mpt3sas0-msix18: PCI-MSI-X enabled: IRQ 465
mpt3sas0-msix19: PCI-MSI-X enabled: IRQ 466
mpt3sas0-msix20: PCI-MSI-X enabled: IRQ 467
mpt3sas0-msix21: PCI-MSI-X enabled: IRQ 468
mpt3sas0-msix22: PCI-MSI-X enabled: IRQ 469
mpt3sas0-msix23: PCI-MSI-X enabled: IRQ 470
mpt3sas0-msix24: PCI-MSI-X enabled: IRQ 471
mpt3sas0-msix25: PCI-MSI-X enabled: IRQ 472
mpt3sas0-msix26: PCI-MSI-X enabled: IRQ 473
mpt3sas0-msix27: PCI-MSI-X enabled: IRQ 474
mpt3sas0-msix28: PCI-MSI-X enabled: IRQ 475
mpt3sas0-msix29: PCI-MSI-X enabled: IRQ 476
mpt3sas0-msix30: PCI-MSI-X enabled: IRQ 477
mpt3sas0-msix31: PCI-MSI-X enabled: IRQ 478
mpt3sas0-msix32: PCI-MSI-X enabled: IRQ 479
mpt3sas0-msix33: PCI-MSI-X enabled: IRQ 480
mpt3sas0-msix34: PCI-MSI-X enabled: IRQ 481
mpt3sas0-msix35: PCI-MSI-X enabled: IRQ 482
mpt3sas0-msix36: PCI-MSI-X enabled: IRQ 483
mpt3sas0-msix37: PCI-MSI-X enabled: IRQ 484
mpt3sas0-msix38: PCI-MSI-X enabled: IRQ 485
mpt3sas0-msix39: PCI-MSI-X enabled: IRQ 486
mpt3sas_cm0: iomem(0x00000000ac400000), mapped(0x00000000d9f45f61), size(65536)
mpt3sas_cm0: ioport(0x0000000000006000), size(256)
mpt3sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k
mpt3sas_cm0: scatter gather: sge_in_main_msg(1), sge_per_chain(7), sge_per_io(128), chains_per_io(19)
mpt3sas_cm0: failure at drivers/scsi/mpt3sas/mpt3sas_scsih.c:12348/_scsih_probe()!

6.1.0-26 (6.1.112-1)

mpt3sas version 43.100.00.00 loaded
mpt3sas_cm0: 63 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (8086116 kB)
mpt3sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k
mpt3sas_cm0: MSI-X vectors supported: 96
mpt3sas_cm0:  0 40 40
mpt3sas_cm0: High IOPs queues : disabled
mpt3sas0-msix0: PCI-MSI-X enabled: IRQ 447
mpt3sas0-msix1: PCI-MSI-X enabled: IRQ 448
mpt3sas0-msix2: PCI-MSI-X enabled: IRQ 449
mpt3sas0-msix3: PCI-MSI-X enabled: IRQ 450
mpt3sas0-msix4: PCI-MSI-X enabled: IRQ 451
mpt3sas0-msix5: PCI-MSI-X enabled: IRQ 452
mpt3sas0-msix6: PCI-MSI-X enabled: IRQ 453
mpt3sas0-msix7: PCI-MSI-X enabled: IRQ 454
mpt3sas0-msix8: PCI-MSI-X enabled: IRQ 455
mpt3sas0-msix9: PCI-MSI-X enabled: IRQ 456
mpt3sas0-msix10: PCI-MSI-X enabled: IRQ 457
mpt3sas0-msix11: PCI-MSI-X enabled: IRQ 458
mpt3sas0-msix12: PCI-MSI-X enabled: IRQ 459
mpt3sas0-msix13: PCI-MSI-X enabled: IRQ 460
mpt3sas0-msix14: PCI-MSI-X enabled: IRQ 461
mpt3sas0-msix15: PCI-MSI-X enabled: IRQ 462
mpt3sas0-msix16: PCI-MSI-X enabled: IRQ 463
mpt3sas0-msix17: PCI-MSI-X enabled: IRQ 464
mpt3sas0-msix18: PCI-MSI-X enabled: IRQ 465
mpt3sas0-msix19: PCI-MSI-X enabled: IRQ 466
mpt3sas0-msix20: PCI-MSI-X enabled: IRQ 467
mpt3sas0-msix21: PCI-MSI-X enabled: IRQ 468
mpt3sas0-msix22: PCI-MSI-X enabled: IRQ 469
mpt3sas0-msix23: PCI-MSI-X enabled: IRQ 470
mpt3sas0-msix24: PCI-MSI-X enabled: IRQ 471
mpt3sas0-msix25: PCI-MSI-X enabled: IRQ 472
mpt3sas0-msix26: PCI-MSI-X enabled: IRQ 473
mpt3sas0-msix27: PCI-MSI-X enabled: IRQ 474
mpt3sas0-msix28: PCI-MSI-X enabled: IRQ 475
mpt3sas0-msix29: PCI-MSI-X enabled: IRQ 476
mpt3sas0-msix30: PCI-MSI-X enabled: IRQ 477
mpt3sas0-msix31: PCI-MSI-X enabled: IRQ 478
mpt3sas0-msix32: PCI-MSI-X enabled: IRQ 479
mpt3sas0-msix33: PCI-MSI-X enabled: IRQ 480
mpt3sas0-msix34: PCI-MSI-X enabled: IRQ 481
mpt3sas0-msix35: PCI-MSI-X enabled: IRQ 482
mpt3sas0-msix36: PCI-MSI-X enabled: IRQ 483
mpt3sas0-msix37: PCI-MSI-X enabled: IRQ 484
mpt3sas0-msix38: PCI-MSI-X enabled: IRQ 485
mpt3sas0-msix39: PCI-MSI-X enabled: IRQ 486
mpt3sas_cm0: iomem(0x00000000ac400000), mapped(0x00000000d9f45f61), size(65536)
mpt3sas_cm0: ioport(0x0000000000006000), size(256)
mpt3sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k
mpt3sas_cm0: scatter gather: sge_in_main_msg(1), sge_per_chain(7), sge_per_io(128), chains_per_io(19)
mpt3sas_cm0: failure at drivers/scsi/mpt3sas/mpt3sas_scsih.c:12348/_scsih_probe()!

(can you maybe post full boot logs?)

I do not see anything related to the mp3sas driver between 6.1.112 and
6.1.115 upstream.

You explictly mention the host in running Xen, so I assume the log is
from booting Dom0 and there are a few xen-related changes as well
between 6.1.112 and 6.119.

Can you confirm if the issue happens as well if you do not boot into
Xen and dom0? Can you for the first broken kernel provide both full
boot logs?

Does the issue happens with a kernel from backports on ideally 6.12.10
from unstable as well?

Would you be able to bisect the upstream changes between 6.112 and
6.115 to identify the breaking commit (but let's first answer the
above questions)?

Regards,
Salvatore

[toc] | [next] | [standalone]

#85205 — Processed: Re: Bug#1088159: linux-image-6.1.0-27-amd64 fails mpt3sas_cm0 _scsih_probe

From"Debian Bug Tracking System" <owner@bugs.debian.org>
Date2025-01-19 16:00 +0100
SubjectProcessed: Re: Bug#1088159: linux-image-6.1.0-27-amd64 fails mpt3sas_cm0 _scsih_probe
Message-ID<K6EWl-acyA-9@gated-at.bofh.it>
In reply to#85204
Processing control commands:

> tags -1 + moreinfo
Bug #1088159 [src:linux] linux-image-6.1.0-27-amd64 fails mpt3sas_cm0 _scsih_probe
Added tag(s) moreinfo.

-- 
1088159: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1088159
Debian Bug Tracking System
Contact owner@bugs.debian.org with problems

[toc] | [prev] | [next] | [standalone]

#85256

FromSalvatore Bonaccorso <carnil@debian.org>
Date2025-01-22 21:00 +0100
Message-ID<K7P3j-aZot-9@gated-at.bofh.it>
In reply to#85204
Hi

[adding back the bugreport address]

On Wed, Jan 22, 2025 at 01:59:10PM +0100, Dröszler Gábor (datatrans) wrote:
> Hi,
> 
> 
> Without Xen, the kernel (linux-image-6.1.0-30-amd64 ) starts normally, but
> under Xen, dom0 cannot load the mpt3sas module.
> 
> 
> I use a Dell R630, and Dell R640 with "Broadcom / LSI SAS3008 PCI-Express
> Fusion-MPT SAS-3 (rev 02)" controllers.

Thanks for the confirmation, this now makes me think that your report
and #1093371 will be linkend or are the same.

Can you please provide full boot logs in both cases, once not booting
into Xen and once with booting into Xen?

In my original reply I had as well:

> Does the issue happens with a kernel from backports on ideally 6.12.10
> from unstable as well?

Any chance you could test that? 

Additionally:

> Would you be able to bisect the upstream changes between 6.112 and
> 6.115 to identify the breaking commit (but let's first answer the
> above questions)?

I understand this might be possible on production systems.

Regards,
Salvatore

[toc] | [prev] | [next] | [standalone]

#85415

FromSalvatore Bonaccorso <carnil@debian.org>
Date2025-02-07 15:00 +0100
Message-ID<Kdx3I-f5B1-3@gated-at.bofh.it>
In reply to#85204
Hi 

On Fri, Feb 07, 2025 at 08:40:57AM +0100, Radoslav Bodó wrote:
> Hello,
> 
> > On Sun, Jan 26, 2025 at 10:45:33AM +0100, Radoslav Bodó wrote:
> > > > Can you confirm if the issue happens as well if you do not boot into
> > > > Xen and dom0? Can you for the first broken kernel provide both full
> > > > boot logs?
> > > > 
> > > > Does the issue happens with a kernel from backports on ideally 6.12.10
> > > > from unstable as well?
> 
> I've been able to do experimentation to answer above questions with somewhat
> expected results:
> 
> * any linux kernel on it's own works fine
> * last kernel working with xen4.17 is 6.1.0-26

Thanks for doing that and provding the results.

I suspect this, #1088159 and #1093371 have the same root cause.

Candidates for the breaking behaviour could be (just adding this now
for documentation purposes, so that we might not loose track):

6434af166441 ("xen: use correct end address of kernel for conflict checking") 
b1e6e80a1b42 ("xen/swiotlb: add alignment check for dma buffers")
1bd4e5a74da9 ("xen/swiotlb: fix allocated size")

(all of those did land in 6.1.113).

Regards,
Salvatore

[toc] | [prev] | [next] | [standalone]

#85420

FromSalvatore Bonaccorso <carnil@debian.org>
Date2025-02-08 11:00 +0100
Message-ID<KdPMZ-fhhn-9@gated-at.bofh.it>
In reply to#85415
Hi 

On Fri, Feb 07, 2025 at 10:02:34PM +0100, Radoslav Bodó wrote:
> Hi again ;)
> 
> > Candidates for the breaking behaviour could be (just adding this now
> > for documentation purposes, so that we might not loose track):
> > 
> > 6434af166441 ("xen: use correct end address of kernel for conflict checking")
> > b1e6e80a1b42 ("xen/swiotlb: add alignment check for dma buffers")
> > 1bd4e5a74da9 ("xen/swiotlb: fix allocated size")
> > 
> > (all of those did land in 6.1.113).
> 
> thank you for excelent pointers. So I wen't to GPT and find out that
> building basic .debs from vanila tree is really easy nowadays ;)
> 
> I've compiled and tested 3 kernels each with one reverted commit and find
> that breaking commit is actually
> 
> b1e6e80a1b42 ("xen/swiotlb: add alignment check for dma buffers")
> 
> ```
> boot_xen4.17-amd54_linux-6.1.113-custom_fail.log (plain 6.1.113)
>   kernel: [    4.477998] mpt3sas_cm0: failure ...
> boot_xen4.17-amd54_linux-6.1.113-revert1bd4e5a74da9_fail.log
>   kernel: [    4.406562] mpt3sas_cm0: failure
> boot_xen4.17-amd54_linux-6.1.113-revert6434af166441_fail.log
>   kernel: [    4.740379] mpt3sas_cm0: failure
> boot_xen4.17-amd54_linux-6.1.113-revertb1e6e80a1b42_ok.log
>   kernel: [   11.795343] mpt3sas_cm0: port enable: SUCCESS
> ```
> 
> 
> here just notes for anyone else who might be willing to confirm the finding
> 
> ## how to build kernel .deb from modified git tree
> 
> ```
> git clone https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
> cd linux/
> git checkout v6.1.113
> 
> cp /boot/config-$(uname -r) .config
> make oldconfig
> scripts/config --set-str SYSTEM_REVISION "$(git rev-parse --short HEAD)"
> scripts/config --set-str SYSTEM_SERIAL "$(date +%s)"
> 
> git revert COMMITID
> export CONCURRENCY_LEVEL=$(nproc)
> fakeroot make -j$(nproc) deb-pkg LOCALVERSION=-revertCOMMITID
> git reset HEAD^ --hard
> ```

Awesome, thanks for goin this extra step. With the breaking commit
identified we can now report this to upstream, will do shortly and
keep (you) in the loop as the reporter, as upstream might ask some
addional things to do.

Thanks for your contirbution identifying the problem!

Regards,
Salvatore

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.kernel

csiph-web