Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.debian.kernel > #85540 > unrolled thread
| Started by | Noah Meyerhans <noahm@debian.org> |
|---|---|
| First post | 2025-02-18 14:40 +0100 |
| Last post | 2025-02-19 15:00 +0100 |
| Articles | 5 — 2 participants |
Back to article view | Back to linux.debian.kernel
Bug#1098261: linux: kernel panic on boot with certain large NVME configurations Noah Meyerhans <noahm@debian.org> - 2025-02-18 14:40 +0100
Bug#1098261: linux: kernel panic on boot with certain large NVME configurations Salvatore Bonaccorso <carnil@debian.org> - 2025-02-18 15:20 +0100
Bug#1098261: linux: kernel panic on boot with certain large NVME configurations Noah Meyerhans <noahm@debian.org> - 2025-02-18 16:10 +0100
Bug#1098261: linux: kernel panic on boot with certain large NVME configurations Salvatore Bonaccorso <carnil@debian.org> - 2025-02-18 17:30 +0100
Bug#1098261: linux: kernel panic on boot with certain large NVME configurations Noah Meyerhans <noahm@debian.org> - 2025-02-19 15:00 +0100
| From | Noah Meyerhans <noahm@debian.org> |
|---|---|
| Date | 2025-02-18 14:40 +0100 |
| Subject | Bug#1098261: linux: kernel panic on boot with certain large NVME configurations |
| Message-ID | <KhvZn-8YF-3@gated-at.bofh.it> |
Source: linux
Version: 5.10.223-1
Severity: important
Tags: upstream patch
X-Debbugs-Cc: debian-cloud@lists.debian.org, jaboutboul@microsoft.com
Microsoft has observed that the 5.10.y kernels in bullseye are susceptible
to crashes due to race conditions in the NVME/PCI subsystem. See below for
a representative kernel log. The problem appears most frequently in larger
systems, e.g. with 4 or more NVME devices and >= 64 CPUs, but it could
potentially occur on smaller systems as well.
The issue was fixed with the 5.14 kernel upstream in e4b9852a0 ("nvme-pci:
fix multiple races in nvme_setup_io_queues"), so this only impacts
oldstable. I have provided a backport of this commit upstream in
https://lore.kernel.org/stable/E1tj8vO-00471h-2H@lore/
I'm requesting that this commit be included in a bullseye kernel update.
Thanks
noah
SCSI subsystem initialized
[ 1.183291] pci c05b:00:00.0: 32.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x16 link at c05b:00:00.0 (capable of 504.112 Gb/s with 32.0 GT/s PCIe x16 link)
[ 1.191131] hv_vmbus: registering driver hv_storvsc
[ 1.191420] pci c05b:00:00.0: BAR 0: assigned [mem 0xfc0000000-0xfc007ffff 64bit]
[ 1.194446] scsi host0: storvsc_host_t
[ 1.200395] nvme nvme0: pci function c05b:00:00.0
[ 1.215047] scsi 0:0:0:2: CD-ROM Msft Virtual DVD-ROM 1.0 PQ: 0 ANSI: 5
[ 1.223119] hv_pci b8fc58bf-fbf1-4a6a-99af-6dbd17b6c699: PCI VMBus probing: Using version 0x10003
[ 1.229528] hv_pci b8fc58bf-fbf1-4a6a-99af-6dbd17b6c699: PCI host bridge to bus fbf1:00
[ 1.233573] pci_bus fbf1:00: root bus resource [mem 0xfc0100000-0xfc01fffff window]
[ 1.237857] pci fbf1:00:00.0: [1414:b111] type 00 class 0x010802
[ 1.243124] pci fbf1:00:00.0: reg 0x10: [mem 0xfc0100000-0xfc01fffff 64bit]
[ 1.256727] pci fbf1:00:00.0: BAR 0: assigned [mem 0xfc0100000-0xfc01fffff 64bit]
[ 1.262177] nvme nvme1: pci function fbf1:00:00.0
[ 1.286577] hv_pci a4a2ee22-e6bb-487b-8db8-0af54d215545: PCI VMBus probing: Using version 0x10003
[ 1.288043] BUG: unable to handle page fault for address: ffffa93540967800
[ 1.293817] hv_pci a4a2ee22-e6bb-487b-8db8-0af54d215545: PCI host bridge to bus e6bb:00
[ 1.295878] #PF: supervisor write access in kernel mode
[ 1.295879] #PF: error_code(0x0002) - not-present page
[ 1.295880] PGD 100000067 P4D 100000067 PUD 10019e067 PMD 1011c5067 PTE 0
[ 1.295884] Oops: 0002 [#1] SMP NOPTI
[ 1.295887] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 5.10.0-25-cloud-amd64 #1 Debian 5.10.191-1
[ 1.299931] pci_bus e6bb:00: root bus resource [mem 0xfc0200000-0xfc02fffff window]
[ 1.302446] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.1 09/28/2024
[ 1.302451] RIP: 0010:nvme_process_cq+0x209/0x220
[ 1.302454] Code: ae f8 8b 30 89 10 0f ae f0 8b 3f 89 c8 29 f1 29 f8 83 e8 01 66 39 c8 73 ba 48 8b 03 8b 88 98 02 00 00 48 8b 43 60 48 8d 04 88 <89> 10 eb a5 45 31 ed eb a0 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f
[ 1.306200] pci e6bb:00:00.0: [1414:b111] type 00 class 0x010802
[ 1.309039] RSP: 0018:ffffa9354022cf28 EFLAGS: 00010046
[ 1.315751] pci e6bb:00:00.0: reg 0x10: [mem 0xfc0200000-0xfc02fffff 64bit]
[ 1.318046] RAX: ffffa93540967800 RBX: ffff9461001e0000 RCX: 0000000000000200
[ 1.318047] RDX: 0000000000000002 RSI: ffffa93540017000 RDI: 0000000000000010
[ 1.318048] RBP: ffff94610d499dc0 R08: 0000000000010000 R09: 0000000000000000
[ 1.318048] R10: 0000000000000000 R11: 0000000000000000 R12: ffff947112b08010
[ 1.318049] R13: 0000000000000001 R14: 0000000000001011 R15: 0000000000000011
[ 1.318053] FS: 0000000000000000(0000) GS:ffff94807f6c0000(0000) knlGS:0000000000000000
[ 1.344250] pci e6bb:00:00.0: BAR 0: assigned [mem 0xfc0200000-0xfc02fffff 64bit]
[ 1.352082] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1.352083] CR2: ffffa93540967800 CR3: 0000000106c8c002 CR4: 0000000000370ea0
[ 1.352085] Call Trace:
[ 1.352088] <IRQ>
[ 1.352092] ? __die_body.cold+0x1a/0x1f
[ 1.358415] nvme nvme2: pci function e6bb:00:00.0
[ 1.359605] ? no_context+0x1a6/0x3c0
[ 1.380401] hv_pci 57af36b7-784f-4841-869a-cbd3bfdba787: PCI VMBus probing: Using version 0x10003
[ 1.381687] ? exc_page_fault+0xd9/0x160
[ 1.386902] hv_pci 57af36b7-784f-4841-869a-cbd3bfdba787: PCI host bridge to bus 784f:00
[ 1.389633] ? asm_exc_page_fault+0x1e/0x30
[ 1.392335] pci_bus 784f:00: root bus resource [mem 0xfc0300000-0xfc03fffff window]
[ 1.395560] ? nvme_process_cq+0x209/0x220
[ 1.395563] ? nvme_process_cq+0x13d/0x220
[ 1.397733] pci 784f:00:00.0: [1414:b111] type 00 class 0x010802
[ 1.398457] nvme_irq+0x10/0x20
[ 1.398462] __handle_irq_event_percpu+0x3d/0x160
[ 1.398464] handle_irq_event+0x57/0xb0
[ 1.404076] pci 784f:00:00.0: reg 0x10: [mem 0xfc0300000-0xfc03fffff 64bit]
[ 1.404747] handle_edge_irq+0x87/0x220
[ 1.417428] pci 784f:00:00.0: BAR 0: assigned [mem 0xfc0300000-0xfc03fffff 64bit]
[ 1.418921] asm_call_irq_on_stack+0x12/0x20
[ 1.418923] </IRQ>
[ 1.418925] common_interrupt+0xb0/0x130
[ 1.418927] asm_common_interrupt+0x1e/0x40
[ 1.454390] RIP: 0010:native_safe_halt+0xe/0x20
[ 1.456577] Code: 00 f0 80 48 02 20 48 8b 00 a8 08 75 c0 e9 77 ff ff ff cc cc cc cc cc cc cc cc cc cc e9 07 00 00 00 0f 00 2d f6 94 55 00 fb f4 <e9> 8d 2c 35 00 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 e9 07 00 00
[ 1.464747] RSP: 0018:ffffa935400c3ef0 EFLAGS: 00000296
[ 1.467219] RAX: ffffffff958af2e0 RBX: 0000000000000003 RCX: ffff94807f6f4180
[ 1.467382] nvme nvme3: pci function 784f:00:00.0
[ 1.470434] RDX: 000000000000ef22 RSI: ffffa935400c3e88 RDI: 000000004add0a00
[ 1.470435] RBP: ffff946100878000 R08: 0000000000000001 R09: 000000000001a400
[ 1.470436] R10: 000000000001a400 R11: 0000000000000000 R12: 0000000000000000
[ 1.470436] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 1.470441] ? __sched_text_end+0x6/0x6
[ 1.470443] default_idle+0xa/0x20
[ 1.470445] default_idle_call+0x3c/0xd0
[ 1.491031] hv_pci 04f5eb1d-1fee-484f-b878-5de54af0fcc8: PCI VMBus probing: Using version 0x10003
[ 1.492420] do_idle+0x20c/0x2b0
[ 1.498091] cpu_startup_entry+0x19/0x20
[ 1.499988] secondary_startup_64_no_verify+0xb0/0xbb
[ 1.502318] Modules linked in: hv_storvsc scsi_transport_fc scsi_mod hv_netvsc pci_hyperv(+) pci_hyperv_intf hv_vmbus crc32c_intel
[ 1.507799] CR2: ffffa93540967800
[ 1.509486] ---[ end trace 62ab8a0f0ae3d1b0 ]---
[ 1.511683] RIP: 0010:nvme_process_cq+0x209/0x220
[ 1.513899] Code: ae f8 8b 30 89 10 0f ae f0 8b 3f 89 c8 29 f1 29 f8 83 e8 01 66 39 c8 73 ba 48 8b 03 8b 88 98 02 00 00 48 8b 43 60 48 8d 04 88 <89> 10 eb a5 45 31 ed eb a0 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f
[ 1.521857] RSP: 0018:ffffa9354022cf28 EFLAGS: 00010046
[ 1.525054] RAX: ffffa93540967800 RBX: ffff9461001e0000 RCX: 0000000000000200
[ 1.528223] RDX: 0000000000000002 RSI: ffffa93540017000 RDI: 0000000000000010
[ 1.531363] RBP: ffff94610d499dc0 R08: 0000000000010000 R09: 0000000000000000
[ 1.534517] R10: 0000000000000000 R11: 0000000000000000 R12: ffff947112b08010
[ 1.537682] R13: 0000000000000001 R14: 0000000000001011 R15: 0000000000000011
[ 1.541051] FS: 0000000000000000(0000) GS:ffff94807f6c0000(0000) knlGS:0000000000000000
[ 1.544563] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1.547224] CR2: ffffa93540967800 CR3: 0000000106c8c002 CR4: 0000000000370ea0
[ 1.550391] Kernel panic - not syncing: Fatal exception in interrupt
[ 1.562507] Kernel Offset: 0x14000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 1.588697] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---
[toc] | [next] | [standalone]
| From | Salvatore Bonaccorso <carnil@debian.org> |
|---|---|
| Date | 2025-02-18 15:20 +0100 |
| Message-ID | <KhwC5-9sK-15@gated-at.bofh.it> |
| In reply to | #85540 |
Hi Noah,
On Tue, Feb 18, 2025 at 08:32:25AM -0500, Noah Meyerhans wrote:
> Source: linux
> Version: 5.10.223-1
> Severity: important
> Tags: upstream patch
> X-Debbugs-Cc: debian-cloud@lists.debian.org, jaboutboul@microsoft.com
>
> Microsoft has observed that the 5.10.y kernels in bullseye are susceptible
> to crashes due to race conditions in the NVME/PCI subsystem. See below for
> a representative kernel log. The problem appears most frequently in larger
> systems, e.g. with 4 or more NVME devices and >= 64 CPUs, but it could
> potentially occur on smaller systems as well.
>
> The issue was fixed with the 5.14 kernel upstream in e4b9852a0 ("nvme-pci:
> fix multiple races in nvme_setup_io_queues"), so this only impacts
> oldstable. I have provided a backport of this commit upstream in
> https://lore.kernel.org/stable/E1tj8vO-00471h-2H@lore/
>
> I'm requesting that this commit be included in a bullseye kernel update.
AFAICS, this backport has not been accepted back then for 5.10.y. Can
you re-ping upstream to make sure it get included in the 5.10.y
series? Once this has happened as we follow the 5.10.y series it will
be included (or can be included in advance once it has been queued).
Regards,
Salvatore
[toc] | [prev] | [next] | [standalone]
| From | Noah Meyerhans <noahm@debian.org> |
|---|---|
| Date | 2025-02-18 16:10 +0100 |
| Message-ID | <Khxot-9ZG-7@gated-at.bofh.it> |
| In reply to | #85545 |
On Tue, Feb 18, 2025 at 03:11:08PM +0100, Salvatore Bonaccorso wrote:
> > Microsoft has observed that the 5.10.y kernels in bullseye are susceptible
> > to crashes due to race conditions in the NVME/PCI subsystem. See below for
> > a representative kernel log. The problem appears most frequently in larger
> > systems, e.g. with 4 or more NVME devices and >= 64 CPUs, but it could
> > potentially occur on smaller systems as well.
> >
> > The issue was fixed with the 5.14 kernel upstream in e4b9852a0 ("nvme-pci:
> > fix multiple races in nvme_setup_io_queues"), so this only impacts
> > oldstable. I have provided a backport of this commit upstream in
> > https://lore.kernel.org/stable/E1tj8vO-00471h-2H@lore/
> >
> > I'm requesting that this commit be included in a bullseye kernel update.
>
> AFAICS, this backport has not been accepted back then for 5.10.y. Can
> you re-ping upstream to make sure it get included in the 5.10.y
> series? Once this has happened as we follow the 5.10.y series it will
> be included (or can be included in advance once it has been queued).
Yes, I forgot to reset the date on the commit that I sent upstream,
which is why it looks like it's been around since 2021. I requested
that upstream apply the fix to 5.10.y last week, and will ping them in
another week or two if it hasn't been acknowledged either way...
noah
[toc] | [prev] | [next] | [standalone]
| From | Salvatore Bonaccorso <carnil@debian.org> |
|---|---|
| Date | 2025-02-18 17:30 +0100 |
| Message-ID | <KhyDT-aIt-7@gated-at.bofh.it> |
| In reply to | #85548 |
Hi,
On Tue, Feb 18, 2025 at 09:56:53AM -0500, Noah Meyerhans wrote:
> On Tue, Feb 18, 2025 at 03:11:08PM +0100, Salvatore Bonaccorso wrote:
> > > Microsoft has observed that the 5.10.y kernels in bullseye are susceptible
> > > to crashes due to race conditions in the NVME/PCI subsystem. See below for
> > > a representative kernel log. The problem appears most frequently in larger
> > > systems, e.g. with 4 or more NVME devices and >= 64 CPUs, but it could
> > > potentially occur on smaller systems as well.
> > >
> > > The issue was fixed with the 5.14 kernel upstream in e4b9852a0 ("nvme-pci:
> > > fix multiple races in nvme_setup_io_queues"), so this only impacts
> > > oldstable. I have provided a backport of this commit upstream in
> > > https://lore.kernel.org/stable/E1tj8vO-00471h-2H@lore/
> > >
> > > I'm requesting that this commit be included in a bullseye kernel update.
> >
> > AFAICS, this backport has not been accepted back then for 5.10.y. Can
> > you re-ping upstream to make sure it get included in the 5.10.y
> > series? Once this has happened as we follow the 5.10.y series it will
> > be included (or can be included in advance once it has been queued).
>
> Yes, I forgot to reset the date on the commit that I sent upstream,
> which is why it looks like it's been around since 2021. I requested
> that upstream apply the fix to 5.10.y last week, and will ping them in
> another week or two if it hasn't been acknowledged either way...
Noah, thanks!
Regards,
Salvatore
[toc] | [prev] | [next] | [standalone]
| From | Noah Meyerhans <noahm@debian.org> |
|---|---|
| Date | 2025-02-19 15:00 +0100 |
| Message-ID | <KhSMh-nBH-5@gated-at.bofh.it> |
| In reply to | #85559 |
On Tue, Feb 18, 2025 at 05:23:52PM +0100, Salvatore Bonaccorso wrote:
> > > > Microsoft has observed that the 5.10.y kernels in bullseye are susceptible
> > > > to crashes due to race conditions in the NVME/PCI subsystem. See below for
> > > > a representative kernel log. The problem appears most frequently in larger
> > > > systems, e.g. with 4 or more NVME devices and >= 64 CPUs, but it could
> > > > potentially occur on smaller systems as well.
> > > >
> > > > The issue was fixed with the 5.14 kernel upstream in e4b9852a0 ("nvme-pci:
> > > > fix multiple races in nvme_setup_io_queues"), so this only impacts
> > > > oldstable. I have provided a backport of this commit upstream in
> > > > https://lore.kernel.org/stable/E1tj8vO-00471h-2H@lore/
> > > >
> > > > I'm requesting that this commit be included in a bullseye kernel update.
> > >
> > > AFAICS, this backport has not been accepted back then for 5.10.y. Can
> > > you re-ping upstream to make sure it get included in the 5.10.y
> > > series? Once this has happened as we follow the 5.10.y series it will
> > > be included (or can be included in advance once it has been queued).
> >
> > Yes, I forgot to reset the date on the commit that I sent upstream,
> > which is why it looks like it's been around since 2021. I requested
> > that upstream apply the fix to 5.10.y last week, and will ping them in
> > another week or two if it hasn't been acknowledged either way...
>
> Noah, thanks!
This has now been queued for acceptance upstream.
noah
[toc] | [prev] | [standalone]
Back to top | Article view | linux.debian.kernel
csiph-web