Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.debian.kernel > #85204 > unrolled thread
| Started by | Salvatore Bonaccorso <carnil@debian.org> |
|---|---|
| First post | 2025-01-19 16:00 +0100 |
| Last post | 2025-02-08 11:00 +0100 |
| Articles | 5 — 2 participants |
Back to article view | Back to linux.debian.kernel
This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by
below is the oldest one visible, not the original post.
Bug#1088159: linux-image-6.1.0-27-amd64 fails mpt3sas_cm0 _scsih_probe Salvatore Bonaccorso <carnil@debian.org> - 2025-01-19 16:00 +0100
Processed: Re: Bug#1088159: linux-image-6.1.0-27-amd64 fails mpt3sas_cm0 _scsih_probe "Debian Bug Tracking System" <owner@bugs.debian.org> - 2025-01-19 16:00 +0100
Bug#1088159: linux-image-6.1.0-27-amd64 fails mpt3sas_cm0 _scsih_probe Salvatore Bonaccorso <carnil@debian.org> - 2025-01-22 21:00 +0100
Bug#1088159: linux-image-6.1.0-27-amd64 fails mpt3sas_cm0 _scsih_probe Salvatore Bonaccorso <carnil@debian.org> - 2025-02-07 15:00 +0100
Bug#1088159: linux-image-6.1.0-27-amd64 fails mpt3sas_cm0 _scsih_probe Salvatore Bonaccorso <carnil@debian.org> - 2025-02-08 11:00 +0100
| From | Salvatore Bonaccorso <carnil@debian.org> |
|---|---|
| Date | 2025-01-19 16:00 +0100 |
| Subject | Bug#1088159: linux-image-6.1.0-27-amd64 fails mpt3sas_cm0 _scsih_probe |
| Message-ID | <K6EWl-acyA-5@gated-at.bofh.it> |
Control: tags -1 + moreinfo Hi, On Sun, Nov 24, 2024 at 12:23:37PM +0100, Radoslav Bodó wrote: > Package: linux-image-6.1.0-27-amd64 > Version: 6.1.115-1 > > Hello, > > after standard patch management, the new kernel does not see mpt3sas device > for HBA, syslong ends with > > ``` > mpt3sas_cm0: failure at > drivers/scsi/mpt3sas/mpt3sas_scsih.c:12348/_scsih_probe()! > ``` > > Older release linux-image-6.1.0-26-amd64 works as expected > The server > > * Dell PowerEdge R440 > * HBA for SAS ME4012 disk array > * running XEN xen-hypervisor-4.17-amd64 The logs were slightly harder to read due to wrapping so I extracted them go easier diff. 6.1.0-27 (6.1.115-1) mpt3sas version 43.100.00.00 loaded mpt3sas_cm0: 63 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (8086116 kB) mpt3sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k mpt3sas_cm0: MSI-X vectors supported: 96 mpt3sas_cm0: 0 40 40 mpt3sas_cm0: High IOPs queues : disabled mpt3sas0-msix0: PCI-MSI-X enabled: IRQ 447 mpt3sas0-msix1: PCI-MSI-X enabled: IRQ 448 mpt3sas0-msix2: PCI-MSI-X enabled: IRQ 449 mpt3sas0-msix3: PCI-MSI-X enabled: IRQ 450 mpt3sas0-msix4: PCI-MSI-X enabled: IRQ 451 mpt3sas0-msix5: PCI-MSI-X enabled: IRQ 452 mpt3sas0-msix6: PCI-MSI-X enabled: IRQ 453 mpt3sas0-msix7: PCI-MSI-X enabled: IRQ 454 mpt3sas0-msix8: PCI-MSI-X enabled: IRQ 455 mpt3sas0-msix9: PCI-MSI-X enabled: IRQ 456 mpt3sas0-msix10: PCI-MSI-X enabled: IRQ 457 mpt3sas0-msix11: PCI-MSI-X enabled: IRQ 458 mpt3sas0-msix12: PCI-MSI-X enabled: IRQ 459 mpt3sas0-msix13: PCI-MSI-X enabled: IRQ 460 mpt3sas0-msix14: PCI-MSI-X enabled: IRQ 461 mpt3sas0-msix15: PCI-MSI-X enabled: IRQ 462 mpt3sas0-msix16: PCI-MSI-X enabled: IRQ 463 mpt3sas0-msix17: PCI-MSI-X enabled: IRQ 464 mpt3sas0-msix18: PCI-MSI-X enabled: IRQ 465 mpt3sas0-msix19: PCI-MSI-X enabled: IRQ 466 mpt3sas0-msix20: PCI-MSI-X enabled: IRQ 467 mpt3sas0-msix21: PCI-MSI-X enabled: IRQ 468 mpt3sas0-msix22: PCI-MSI-X enabled: IRQ 469 mpt3sas0-msix23: PCI-MSI-X enabled: IRQ 470 mpt3sas0-msix24: PCI-MSI-X enabled: IRQ 471 mpt3sas0-msix25: PCI-MSI-X enabled: IRQ 472 mpt3sas0-msix26: PCI-MSI-X enabled: IRQ 473 mpt3sas0-msix27: PCI-MSI-X enabled: IRQ 474 mpt3sas0-msix28: PCI-MSI-X enabled: IRQ 475 mpt3sas0-msix29: PCI-MSI-X enabled: IRQ 476 mpt3sas0-msix30: PCI-MSI-X enabled: IRQ 477 mpt3sas0-msix31: PCI-MSI-X enabled: IRQ 478 mpt3sas0-msix32: PCI-MSI-X enabled: IRQ 479 mpt3sas0-msix33: PCI-MSI-X enabled: IRQ 480 mpt3sas0-msix34: PCI-MSI-X enabled: IRQ 481 mpt3sas0-msix35: PCI-MSI-X enabled: IRQ 482 mpt3sas0-msix36: PCI-MSI-X enabled: IRQ 483 mpt3sas0-msix37: PCI-MSI-X enabled: IRQ 484 mpt3sas0-msix38: PCI-MSI-X enabled: IRQ 485 mpt3sas0-msix39: PCI-MSI-X enabled: IRQ 486 mpt3sas_cm0: iomem(0x00000000ac400000), mapped(0x00000000d9f45f61), size(65536) mpt3sas_cm0: ioport(0x0000000000006000), size(256) mpt3sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k mpt3sas_cm0: scatter gather: sge_in_main_msg(1), sge_per_chain(7), sge_per_io(128), chains_per_io(19) mpt3sas_cm0: failure at drivers/scsi/mpt3sas/mpt3sas_scsih.c:12348/_scsih_probe()! 6.1.0-26 (6.1.112-1) mpt3sas version 43.100.00.00 loaded mpt3sas_cm0: 63 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (8086116 kB) mpt3sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k mpt3sas_cm0: MSI-X vectors supported: 96 mpt3sas_cm0: 0 40 40 mpt3sas_cm0: High IOPs queues : disabled mpt3sas0-msix0: PCI-MSI-X enabled: IRQ 447 mpt3sas0-msix1: PCI-MSI-X enabled: IRQ 448 mpt3sas0-msix2: PCI-MSI-X enabled: IRQ 449 mpt3sas0-msix3: PCI-MSI-X enabled: IRQ 450 mpt3sas0-msix4: PCI-MSI-X enabled: IRQ 451 mpt3sas0-msix5: PCI-MSI-X enabled: IRQ 452 mpt3sas0-msix6: PCI-MSI-X enabled: IRQ 453 mpt3sas0-msix7: PCI-MSI-X enabled: IRQ 454 mpt3sas0-msix8: PCI-MSI-X enabled: IRQ 455 mpt3sas0-msix9: PCI-MSI-X enabled: IRQ 456 mpt3sas0-msix10: PCI-MSI-X enabled: IRQ 457 mpt3sas0-msix11: PCI-MSI-X enabled: IRQ 458 mpt3sas0-msix12: PCI-MSI-X enabled: IRQ 459 mpt3sas0-msix13: PCI-MSI-X enabled: IRQ 460 mpt3sas0-msix14: PCI-MSI-X enabled: IRQ 461 mpt3sas0-msix15: PCI-MSI-X enabled: IRQ 462 mpt3sas0-msix16: PCI-MSI-X enabled: IRQ 463 mpt3sas0-msix17: PCI-MSI-X enabled: IRQ 464 mpt3sas0-msix18: PCI-MSI-X enabled: IRQ 465 mpt3sas0-msix19: PCI-MSI-X enabled: IRQ 466 mpt3sas0-msix20: PCI-MSI-X enabled: IRQ 467 mpt3sas0-msix21: PCI-MSI-X enabled: IRQ 468 mpt3sas0-msix22: PCI-MSI-X enabled: IRQ 469 mpt3sas0-msix23: PCI-MSI-X enabled: IRQ 470 mpt3sas0-msix24: PCI-MSI-X enabled: IRQ 471 mpt3sas0-msix25: PCI-MSI-X enabled: IRQ 472 mpt3sas0-msix26: PCI-MSI-X enabled: IRQ 473 mpt3sas0-msix27: PCI-MSI-X enabled: IRQ 474 mpt3sas0-msix28: PCI-MSI-X enabled: IRQ 475 mpt3sas0-msix29: PCI-MSI-X enabled: IRQ 476 mpt3sas0-msix30: PCI-MSI-X enabled: IRQ 477 mpt3sas0-msix31: PCI-MSI-X enabled: IRQ 478 mpt3sas0-msix32: PCI-MSI-X enabled: IRQ 479 mpt3sas0-msix33: PCI-MSI-X enabled: IRQ 480 mpt3sas0-msix34: PCI-MSI-X enabled: IRQ 481 mpt3sas0-msix35: PCI-MSI-X enabled: IRQ 482 mpt3sas0-msix36: PCI-MSI-X enabled: IRQ 483 mpt3sas0-msix37: PCI-MSI-X enabled: IRQ 484 mpt3sas0-msix38: PCI-MSI-X enabled: IRQ 485 mpt3sas0-msix39: PCI-MSI-X enabled: IRQ 486 mpt3sas_cm0: iomem(0x00000000ac400000), mapped(0x00000000d9f45f61), size(65536) mpt3sas_cm0: ioport(0x0000000000006000), size(256) mpt3sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k mpt3sas_cm0: scatter gather: sge_in_main_msg(1), sge_per_chain(7), sge_per_io(128), chains_per_io(19) mpt3sas_cm0: failure at drivers/scsi/mpt3sas/mpt3sas_scsih.c:12348/_scsih_probe()! (can you maybe post full boot logs?) I do not see anything related to the mp3sas driver between 6.1.112 and 6.1.115 upstream. You explictly mention the host in running Xen, so I assume the log is from booting Dom0 and there are a few xen-related changes as well between 6.1.112 and 6.119. Can you confirm if the issue happens as well if you do not boot into Xen and dom0? Can you for the first broken kernel provide both full boot logs? Does the issue happens with a kernel from backports on ideally 6.12.10 from unstable as well? Would you be able to bisect the upstream changes between 6.112 and 6.115 to identify the breaking commit (but let's first answer the above questions)? Regards, Salvatore
[toc] | [next] | [standalone]
| From | "Debian Bug Tracking System" <owner@bugs.debian.org> |
|---|---|
| Date | 2025-01-19 16:00 +0100 |
| Subject | Processed: Re: Bug#1088159: linux-image-6.1.0-27-amd64 fails mpt3sas_cm0 _scsih_probe |
| Message-ID | <K6EWl-acyA-9@gated-at.bofh.it> |
| In reply to | #85204 |
Processing control commands: > tags -1 + moreinfo Bug #1088159 [src:linux] linux-image-6.1.0-27-amd64 fails mpt3sas_cm0 _scsih_probe Added tag(s) moreinfo. -- 1088159: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1088159 Debian Bug Tracking System Contact owner@bugs.debian.org with problems
[toc] | [prev] | [next] | [standalone]
| From | Salvatore Bonaccorso <carnil@debian.org> |
|---|---|
| Date | 2025-01-22 21:00 +0100 |
| Message-ID | <K7P3j-aZot-9@gated-at.bofh.it> |
| In reply to | #85204 |
Hi [adding back the bugreport address] On Wed, Jan 22, 2025 at 01:59:10PM +0100, Dröszler Gábor (datatrans) wrote: > Hi, > > > Without Xen, the kernel (linux-image-6.1.0-30-amd64 ) starts normally, but > under Xen, dom0 cannot load the mpt3sas module. > > > I use a Dell R630, and Dell R640 with "Broadcom / LSI SAS3008 PCI-Express > Fusion-MPT SAS-3 (rev 02)" controllers. Thanks for the confirmation, this now makes me think that your report and #1093371 will be linkend or are the same. Can you please provide full boot logs in both cases, once not booting into Xen and once with booting into Xen? In my original reply I had as well: > Does the issue happens with a kernel from backports on ideally 6.12.10 > from unstable as well? Any chance you could test that? Additionally: > Would you be able to bisect the upstream changes between 6.112 and > 6.115 to identify the breaking commit (but let's first answer the > above questions)? I understand this might be possible on production systems. Regards, Salvatore
[toc] | [prev] | [next] | [standalone]
| From | Salvatore Bonaccorso <carnil@debian.org> |
|---|---|
| Date | 2025-02-07 15:00 +0100 |
| Message-ID | <Kdx3I-f5B1-3@gated-at.bofh.it> |
| In reply to | #85204 |
Hi
On Fri, Feb 07, 2025 at 08:40:57AM +0100, Radoslav Bodó wrote:
> Hello,
>
> > On Sun, Jan 26, 2025 at 10:45:33AM +0100, Radoslav Bodó wrote:
> > > > Can you confirm if the issue happens as well if you do not boot into
> > > > Xen and dom0? Can you for the first broken kernel provide both full
> > > > boot logs?
> > > >
> > > > Does the issue happens with a kernel from backports on ideally 6.12.10
> > > > from unstable as well?
>
> I've been able to do experimentation to answer above questions with somewhat
> expected results:
>
> * any linux kernel on it's own works fine
> * last kernel working with xen4.17 is 6.1.0-26
Thanks for doing that and provding the results.
I suspect this, #1088159 and #1093371 have the same root cause.
Candidates for the breaking behaviour could be (just adding this now
for documentation purposes, so that we might not loose track):
6434af166441 ("xen: use correct end address of kernel for conflict checking")
b1e6e80a1b42 ("xen/swiotlb: add alignment check for dma buffers")
1bd4e5a74da9 ("xen/swiotlb: fix allocated size")
(all of those did land in 6.1.113).
Regards,
Salvatore
[toc] | [prev] | [next] | [standalone]
| From | Salvatore Bonaccorso <carnil@debian.org> |
|---|---|
| Date | 2025-02-08 11:00 +0100 |
| Message-ID | <KdPMZ-fhhn-9@gated-at.bofh.it> |
| In reply to | #85415 |
Hi
On Fri, Feb 07, 2025 at 10:02:34PM +0100, Radoslav Bodó wrote:
> Hi again ;)
>
> > Candidates for the breaking behaviour could be (just adding this now
> > for documentation purposes, so that we might not loose track):
> >
> > 6434af166441 ("xen: use correct end address of kernel for conflict checking")
> > b1e6e80a1b42 ("xen/swiotlb: add alignment check for dma buffers")
> > 1bd4e5a74da9 ("xen/swiotlb: fix allocated size")
> >
> > (all of those did land in 6.1.113).
>
> thank you for excelent pointers. So I wen't to GPT and find out that
> building basic .debs from vanila tree is really easy nowadays ;)
>
> I've compiled and tested 3 kernels each with one reverted commit and find
> that breaking commit is actually
>
> b1e6e80a1b42 ("xen/swiotlb: add alignment check for dma buffers")
>
> ```
> boot_xen4.17-amd54_linux-6.1.113-custom_fail.log (plain 6.1.113)
> kernel: [ 4.477998] mpt3sas_cm0: failure ...
> boot_xen4.17-amd54_linux-6.1.113-revert1bd4e5a74da9_fail.log
> kernel: [ 4.406562] mpt3sas_cm0: failure
> boot_xen4.17-amd54_linux-6.1.113-revert6434af166441_fail.log
> kernel: [ 4.740379] mpt3sas_cm0: failure
> boot_xen4.17-amd54_linux-6.1.113-revertb1e6e80a1b42_ok.log
> kernel: [ 11.795343] mpt3sas_cm0: port enable: SUCCESS
> ```
>
>
> here just notes for anyone else who might be willing to confirm the finding
>
> ## how to build kernel .deb from modified git tree
>
> ```
> git clone https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
> cd linux/
> git checkout v6.1.113
>
> cp /boot/config-$(uname -r) .config
> make oldconfig
> scripts/config --set-str SYSTEM_REVISION "$(git rev-parse --short HEAD)"
> scripts/config --set-str SYSTEM_SERIAL "$(date +%s)"
>
> git revert COMMITID
> export CONCURRENCY_LEVEL=$(nproc)
> fakeroot make -j$(nproc) deb-pkg LOCALVERSION=-revertCOMMITID
> git reset HEAD^ --hard
> ```
Awesome, thanks for goin this extra step. With the breaking commit
identified we can now report this to upstream, will do shortly and
keep (you) in the loop as the reporter, as upstream might ask some
addional things to do.
Thanks for your contirbution identifying the problem!
Regards,
Salvatore
[toc] | [prev] | [standalone]
Back to top | Article view | linux.debian.kernel
csiph-web