Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.debian.kernel > #90005 > unrolled thread
| Started by | Peter Morrow <pdmorrow@gmail.com> |
|---|---|
| First post | 2025-11-13 00:00 +0100 |
| Last post | 2025-11-26 07:20 +0100 |
| Articles | 12 — 4 participants |
Back to article view | Back to linux.debian.kernel
Bug#1120602: hyper-v: BUG: kernel NULL pointer dereference, address: 00000000000000a0 Peter Morrow <pdmorrow@gmail.com> - 2025-11-13 00:00 +0100
Processed: Re: Bug#1120602: hyper-v: BUG: kernel NULL pointer dereference, address: 00000000000000a0 "Debian Bug Tracking System" <owner@bugs.debian.org> - 2025-11-13 07:30 +0100
Bug#1120602: hyper-v: BUG: kernel NULL pointer dereference, address: 00000000000000a0 Salvatore Bonaccorso <carnil@debian.org> - 2025-11-13 07:30 +0100
Bug#1120602: hyper-v: BUG: kernel NULL pointer dereference, address: 00000000000000a0 Peter Morrow <pdmorrow@gmail.com> - 2025-11-13 14:40 +0100
Processed: Re: Bug#1120602: hyper-v: BUG: kernel NULL pointer dereference, address: 00000000000000a0 "Debian Bug Tracking System" <owner@bugs.debian.org> - 2025-11-13 15:00 +0100
Bug#1120602: hyper-v: BUG: kernel NULL pointer dereference, address: 00000000000000a0 Salvatore Bonaccorso <carnil@debian.org> - 2025-11-13 15:00 +0100
Bug#1120602: hyper-v: BUG: kernel NULL pointer dereference, address: 00000000000000a0 Peter Morrow <pdmorrow@gmail.com> - 2025-11-13 15:40 +0100
Bug#1120602: [REGRESSION 6.12.y] hyper-v: BUG: kernel NULL pointer dereference, address: 00000000000000a0: RIP: 0010:hv_uio_channel_cb+0xd/0x20 [uio_hv_generic] Naman Jain <namjain@linux.microsoft.com> - 2025-11-14 07:10 +0100
Bug#1120602: [REGRESSION 6.12.y] hyper-v: BUG: kernel NULL pointer dereference, address: 00000000000000a0: RIP: 0010:hv_uio_channel_cb+0xd/0x20 [uio_hv_generic] Naman Jain <namjain@linux.microsoft.com> - 2025-11-14 15:40 +0100
Bug#1120602: [REGRESSION 6.12.y] hyper-v: BUG: kernel NULL pointer dereference, address: 00000000000000a0: RIP: 0010:hv_uio_channel_cb+0xd/0x20 [uio_hv_generic] Salvatore Bonaccorso <carnil@debian.org> - 2025-11-14 22:50 +0100
Bug#1120602: [REGRESSION 6.12.y] hyper-v: BUG: kernel NULL pointer dereference, address: 00000000000000a0: RIP: 0010:hv_uio_channel_cb+0xd/0x20 [uio_hv_generic] Naman Jain <namjain@linux.microsoft.com> - 2025-11-15 10:10 +0100
Bug#1120602: [REGRESSION 6.12.y] hyper-v: BUG: kernel NULL pointer dereference, address: 00000000000000a0: RIP: 0010:hv_uio_channel_cb+0xd/0x20 [uio_hv_generic] Naman Jain <namjain@linux.microsoft.com> - 2025-11-26 07:20 +0100
| From | Peter Morrow <pdmorrow@gmail.com> |
|---|---|
| Date | 2025-11-13 00:00 +0100 |
| Subject | Bug#1120602: hyper-v: BUG: kernel NULL pointer dereference, address: 00000000000000a0 |
| Message-ID | <LQrYJ-cvPu-1@gated-at.bofh.it> |
Package: src:linux
Version: 6.12.57-1
Severity: important
X-Debbugs-Cc: pdmorrow@gmail.com
Dear Maintainer,
I'm seeing a kernel crash quite soon after boot on a debian trixie based
system running 6.12.57+deb13-amd64, unfortunately the kernel panics before
I can access the system to gather more information. Thus I'll provide details
of the system using a previously known good version. The panic is happening
100% of the time unfortunately. I have access to the serial console however
so can enable any required verbose logging during boot if necessary.
Crucially the crash is not seen with kernel version 6.12.41+deb13-amd64 with the
same userspace. We had pinned to that version until very recently to in order
to work around https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1109676
I'm running a dpdk application here (VPP) on Azure, VM form factor is a
"Standard DS3 v2 (4 vcpus, 14 GiB memory)".
The only relevant upstream commit in this area (as far as I can see) is:
https://lore.kernel.org/linux-hyperv/1bb599ee-fe28-409d-b430-2fc086268936@linux.microsoft.com/
The comment regarding avoiding races at start adds a bit more weight behind this
hunch, though it's only a hunch as I am most definitely nowhere near an expert
in this area.
-- Package-specific info:
[ 19.625535] BUG: kernel NULL pointer dereference, address: 00000000000000a0
[ 19.628874] #PF: supervisor read access in kernel mode
[ 19.630841] #PF: error_code(0x0000) - not-present page
[ 19.632788] PGD 0 P4D 0
[ 19.633905] Oops: Oops: 0000 [#1] PREEMPT SMP PTI
[ 19.635586] CPU: 3 UID: 0 PID: 0 Comm: swapper/3 Not tainted 6.12.57+deb13-amd64 #1 Debian 6.12.57-1
[ 19.640216] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.1 09/28/2024
[ 19.644514] RIP: 0010:hv_uio_channel_cb+0xd/0x20 [uio_hv_generic]
[ 19.646994] Code: 02 00 00 5b 5d e9 53 98 69 e9 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 48 8b 47 10 <48> 8b b8 a0 00 00 00 f0 83 44 24 fc 00 e9 51 6f fa ff 90 90 90 90
[ 19.654377] RSP: 0018:ffffb15ac01a4fa8 EFLAGS: 00010046
[ 19.656385] RAX: 0000000000000000 RBX: 0000000000000015 RCX: 0000000000000015
[ 19.659240] RDX: 0000000000000001 RSI: ffffffffffffffff RDI: ffff8ff69c759400
[ 19.662168] RBP: ffff8ff548790200 R08: ffff8ff548790200 R09: 00fca75150b080e9
[ 19.665239] R10: 0000000000000000 R11: ffffb15ac01a4ff8 R12: ffff8ff871dc1480
[ 19.668193] R13: ffff8ff69c759400 R14: ffff8ff69c7596a0 R15: ffffffffc106e160
[ 19.671106] FS: 0000000000000000(0000) GS:ffff8ff871d80000(0000) knlGS:0000000000000000
[ 19.674281] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 19.676533] CR2: 00000000000000a0 CR3: 0000000100ba6003 CR4: 00000000003706f0
[ 19.679385] Call Trace:
[ 19.680361] <IRQ>
[ 19.681181] vmbus_isr+0x1a5/0x210 [hv_vmbus]
[ 19.682916] __sysvec_hyperv_callback+0x32/0x60
[ 19.684991] sysvec_hyperv_callback+0x6c/0x90
[ 19.686665] </IRQ>
[ 19.687509] <TASK>
[ 19.688366] asm_sysvec_hyperv_callback+0x1a/0x20
[ 19.690262] RIP: 0010:pv_native_safe_halt+0xf/0x20
[ 19.692067] Code: 09 e9 c5 08 01 00 0f 1f 44 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 66 90 0f 00 2d e5 3b 31 00 fb f4 <c3> cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90
[ 19.699119] RSP: 0018:ffffb15ac0103ed8 EFLAGS: 00000246
[ 19.701412] RAX: 0000000000000003 RBX: ffff8ff5403b1fc0 RCX: ffff8ff54c64ce30
[ 19.704328] RDX: 0000000000000000 RSI: 0000000000000003 RDI: 000000000001f894
[ 19.706910] RBP: 0000000000000003 R08: 000000000bb760d9 R09: 00fca75150b080e9
[ 19.709762] R10: 0000000000000003 R11: 0000000000000001 R12: 0000000000000000
[ 19.712510] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 19.715173] default_idle+0x9/0x20
[ 19.716846] default_idle_call+0x29/0x100
[ 19.718623] do_idle+0x1fe/0x240
[ 19.720045] cpu_startup_entry+0x29/0x30
[ 19.721595] start_secondary+0x11e/0x140
[ 19.723080] common_startup_64+0x13e/0x141
[ 19.725222] </TASK>
[ 19.726387] Modules linked in: isofs cdrom uio_hv_generic uio binfmt_misc intel_rapl_msr intel_rapl_common intel_uncore_frequency_common isst_if_mbox_msr isst_if_common rpcrdma skx_edac_common nfit sunrpc libnvdimm crct10dif_pclmul ghash_clmulni_intel sha512_ssse3 sha256_ssse3 rdma_ucm ib_iser sha1_ssse3 rdma_cm aesni_intel iw_cm gf128mul crypto_simd libiscsi cryptd ib_umad ib_ipoib scsi_transport_iscsi ib_cm rapl sg hv_utils hv_balloon evdev pcspkr joydev mpls_router ip_tunnel ramoops configfs pstore_blk efi_pstore pstore_zone nfnetlink vsock_loopback vmw_vsock_virtio_transport_common hv_sock vmw_vsock_vmci_transport vsock vmw_vmci efivarfs ip_tables x_tables autofs4 overlay squashfs dm_verity dm_bufio reed_solomon dm_mod loop ext4 crc16 mbcache jbd2 crc32c_generic mlx5_ib ib_uverbs ib_core mlx5_core mlxfw pci_hyperv pci_hyperv_intf hyperv_drm drm_shmem_helper sd_mod drm_kms_helper hv_storvsc scsi_transport_fc drm scsi_mod hid_generic hid_hyperv hid serio_raw hv_netvsc hyperv_keyboard scsi_common hv_vmbus
[ 19.726466] crc32_pclmul crc32c_intel
[ 19.765771] CR2: 00000000000000a0
[ 19.767524] ---[ end trace 0000000000000000 ]---
[ 19.800433] RIP: 0010:hv_uio_channel_cb+0xd/0x20 [uio_hv_generic]
[ 19.803170] Code: 02 00 00 5b 5d e9 53 98 69 e9 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 48 8b 47 10 <48> 8b b8 a0 00 00 00 f0 83 44 24 fc 00 e9 51 6f fa ff 90 90 90 90
[ 19.811041] RSP: 0018:ffffb15ac01a4fa8 EFLAGS: 00010046
[ 19.813466] RAX: 0000000000000000 RBX: 0000000000000015 RCX: 0000000000000015
[ 19.816504] RDX: 0000000000000001 RSI: ffffffffffffffff RDI: ffff8ff69c759400
[ 19.819484] RBP: ffff8ff548790200 R08: ffff8ff548790200 R09: 00fca75150b080e9
[ 19.822625] R10: 0000000000000000 R11: ffffb15ac01a4ff8 R12: ffff8ff871dc1480
[ 19.825569] R13: ffff8ff69c759400 R14: ffff8ff69c7596a0 R15: ffffffffc106e160
[ 19.828804] FS: 0000000000000000(0000) GS:ffff8ff871d80000(0000) knlGS:0000000000000000
[ 19.832214] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 19.834709] CR2: 00000000000000a0 CR3: 0000000100ba6003 CR4: 00000000003706f0
[ 19.837976] Kernel panic - not syncing: Fatal exception in interrupt
[ 19.841825] Kernel Offset: 0x28a00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 19.896620] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---
lspci output:
Collected from a system that is not crashing (6.12.41+deb13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.41-1 (2025-08-12) x86_64 GNU/Linux)...
$ sudo lspci -v
2f22:00:02.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] (rev 80)
Subsystem: Mellanox Technologies Device 0190
Physical Slot: 3
Flags: bus master, fast devsel, latency 0, NUMA node 0
Memory at fe0100000 (64-bit, prefetchable) [size=1M]
Capabilities: [60] Express Endpoint, IntMsgNum 0
Capabilities: [9c] MSI-X: Enable+ Count=8 Masked-
Kernel driver in use: mlx5_core
Kernel modules: mlx5_core
52f7:00:02.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] (rev 80)
Subsystem: Mellanox Technologies Device 0190
Physical Slot: 2
Flags: bus master, fast devsel, latency 0, NUMA node 0
Memory at fe0000000 (64-bit, prefetchable) [size=1M]
Capabilities: [60] Express Endpoint, IntMsgNum 0
Capabilities: [9c] MSI-X: Enable+ Count=8 Masked-
Kernel driver in use: mlx5_core
Kernel modules: mlx5_core
7852:00:02.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] (rev 80)
Subsystem: Mellanox Technologies Device 0190
Physical Slot: 4
Flags: bus master, fast devsel, latency 0, NUMA node 0
Memory at fe0200000 (64-bit, prefetchable) [size=1M]
Capabilities: [60] Express Endpoint, IntMsgNum 0
Capabilities: [9c] MSI-X: Enable+ Count=8 Masked-
Kernel driver in use: mlx5_core
Kernel modules: mlx5_core
dmidecode output:
$ sudo dmidecode
# dmidecode 3.6
Getting SMBIOS data from sysfs.
SMBIOS 3.1.0 present.
Table at 0x3FF82000.
Handle 0x0000, DMI type 0, 26 bytes
BIOS Information
Vendor: Microsoft Corporation
Version: Hyper-V UEFI Release v4.1
Release Date: 05/13/2024
ROM Size: 64 kB
Characteristics:
BIOS characteristics not supported
ACPI is supported
Targeted content distribution is supported
UEFI is supported
System is a virtual machine
BIOS Revision: 4.1
Handle 0x0001, DMI type 1, 27 bytes
System Information
Manufacturer: Microsoft Corporation
Product Name: Virtual Machine
Version: Hyper-V UEFI Release v4.1
Serial Number: 0000-0010-5437-9499-5225-4477-46
UUID: 925315af-4af4-4d42-915a-1516b5a1fe5c
Wake-up Type: Power Switch
SKU Number: None
Family: Virtual Machine
Handle 0x0002, DMI type 3, 24 bytes
Chassis Information
Manufacturer: Microsoft Corporation
Type: Desktop
Lock: Not Present
Version: Hyper-V UEFI Release v4.1
Serial Number: 2466-9316-1078-9783-6078-7718-80
Asset Tag: 7783-7084-3265-9085-8269-3286-77
Boot-up State: Safe
Power Supply State: Safe
Thermal State: Safe
Security Status: Unknown
OEM Information: 0x00000000
Height: Unspecified
Number Of Power Cords: Unspecified
Contained Elements: 0
SKU Number: Virtual Machine
Handle 0x0003, DMI type 2, 17 bytes
Base Board Information
Manufacturer: Microsoft Corporation
Product Name: Virtual Machine
Version: Hyper-V UEFI Release v4.1
Serial Number: 0000-0010-4737-0707-0684-2660-76
Asset Tag: None
Features:
Board is a hosting board
Location In Chassis: Virtual Machine
Chassis Handle: 0x0002
Type: Motherboard
Contained Object Handles: 0
Handle 0x0004, DMI type 4, 48 bytes
Processor Information
Socket Designation: None
Type: Central Processor
Family: Unknown
Manufacturer: None
ID: 00 00 00 00 00 00 00 00
Version: None
Voltage: Unknown
External Clock: Unknown
Max Speed: Unknown
Current Speed: Unknown
Status: Unpopulated
Upgrade: Other
L1 Cache Handle: Not Provided
L2 Cache Handle: Not Provided
L3 Cache Handle: Not Provided
Serial Number: None
Asset Tag: None
Part Number: None
Core Count: 4
Core Enabled: 4
Thread Count: 1
Characteristics: None
Handle 0x0005, DMI type 11, 5 bytes
OEM Strings
String 1: [MS_VM_CERT/SHA1/9b80ca0d5dd061ec9da4e494f4c3fd1196270c22]
String 2: None
String 3: To be filled by OEM
Handle 0x0006, DMI type 16, 23 bytes
Physical Memory Array
Location: System Board Or Motherboard
Use: System Memory
Error Correction Type: None
Maximum Capacity: 0 bytes
Error Information Handle: Not Provided
Number Of Devices: 3
Handle 0x0007, DMI type 17, 92 bytes
Memory Device
Array Handle: 0x0006
Error Information Handle: Not Provided
Total Width: Unknown
Data Width: Unknown
Size: 26 MB
Form Factor: Unknown
Set: None
Locator: M0001
Bank Locator: None
Type: Unknown
Type Detail: Unknown
Speed: Unknown
Manufacturer: Microsoft Corporation
Serial Number: None
Asset Tag: None
Part Number: None
Rank: Unknown
Configured Memory Speed: Unknown
Minimum Voltage: Unknown
Maximum Voltage: Unknown
Configured Voltage: Unknown
Memory Technology: <OUT OF SPEC>
Memory Operating Mode Capability: None
Firmware Version: Not Specified
Module Manufacturer ID: Unknown
Module Product ID: Unknown
Memory Subsystem Controller Manufacturer ID: Unknown
Memory Subsystem Controller Product ID: Unknown
Non-Volatile Size: None
Volatile Size: None
Cache Size: None
Logical Size: None
Handle 0x0008, DMI type 19, 31 bytes
Memory Array Mapped Address
Starting Address: 0x00000000000
Ending Address: 0x00001A003FF
Range Size: 26625 kB
Physical Array Handle: 0x0006
Partition Width: 0
Handle 0x0009, DMI type 20, 35 bytes
Memory Device Mapped Address
Starting Address: 0x00000000000
Ending Address: 0x00001A003FF
Range Size: 26625 kB
Physical Device Handle: 0x0007
Memory Array Mapped Address Handle: 0x0008
Partition Row Position: Unknown
Handle 0x000A, DMI type 17, 92 bytes
Memory Device
Array Handle: 0x0006
Error Information Handle: Not Provided
Total Width: Unknown
Data Width: Unknown
Size: 948 MB
Form Factor: Unknown
Set: None
Locator: M0002
Bank Locator: None
Type: Unknown
Type Detail: Unknown
Speed: Unknown
Manufacturer: Microsoft Corporation
Serial Number: None
Asset Tag: None
Part Number: None
Rank: Unknown
Configured Memory Speed: Unknown
Minimum Voltage: Unknown
Maximum Voltage: Unknown
Configured Voltage: Unknown
Memory Technology: <OUT OF SPEC>
Memory Operating Mode Capability: None
Firmware Version: Not Specified
Module Manufacturer ID: Unknown
Module Product ID: Unknown
Memory Subsystem Controller Manufacturer ID: Unknown
Memory Subsystem Controller Product ID: Unknown
Non-Volatile Size: None
Volatile Size: None
Cache Size: None
Logical Size: None
Handle 0x000B, DMI type 19, 31 bytes
Memory Array Mapped Address
Starting Address: 0x00004C00000
Ending Address: 0x000400003FF
Range Size: 970753 kB
Physical Array Handle: 0x0006
Partition Width: 0
Handle 0x000C, DMI type 20, 35 bytes
Memory Device Mapped Address
Starting Address: 0x00004C00000
Ending Address: 0x000400003FF
Range Size: 970753 kB
Physical Device Handle: 0x000A
Memory Array Mapped Address Handle: 0x000B
Partition Row Position: Unknown
Handle 0x000D, DMI type 17, 92 bytes
Memory Device
Array Handle: 0x0006
Error Information Handle: Not Provided
Total Width: Unknown
Data Width: Unknown
Size: 13 GB
Form Factor: Unknown
Set: None
Locator: M0003
Bank Locator: None
Type: Unknown
Type Detail: Unknown
Speed: Unknown
Manufacturer: Microsoft Corporation
Serial Number: None
Asset Tag: None
Part Number: None
Rank: Unknown
Configured Memory Speed: Unknown
Minimum Voltage: Unknown
Maximum Voltage: Unknown
Configured Voltage: Unknown
Memory Technology: <OUT OF SPEC>
Memory Operating Mode Capability: None
Firmware Version: Not Specified
Module Manufacturer ID: Unknown
Module Product ID: Unknown
Memory Subsystem Controller Manufacturer ID: Unknown
Memory Subsystem Controller Product ID: Unknown
Non-Volatile Size: None
Volatile Size: None
Cache Size: None
Logical Size: None
Handle 0x000E, DMI type 19, 31 bytes
Memory Array Mapped Address
Starting Address: 0x00100000000
Ending Address: 0x004400003FF
Range Size: 13 GB
Physical Array Handle: 0x0006
Partition Width: 0
Handle 0x000F, DMI type 20, 35 bytes
Memory Device Mapped Address
Starting Address: 0x00100000000
Ending Address: 0x004400003FF
Range Size: 13 GB
Physical Device Handle: 0x000D
Memory Array Mapped Address Handle: 0x000E
Partition Row Position: Unknown
Handle 0x0010, DMI type 32, 11 bytes
System Boot Information
Status: No errors detected
Handle 0xFEFF, DMI type 127, 4 bytes
End Of Table
[toc] | [next] | [standalone]
| From | "Debian Bug Tracking System" <owner@bugs.debian.org> |
|---|---|
| Date | 2025-11-13 07:30 +0100 |
| Subject | Processed: Re: Bug#1120602: hyper-v: BUG: kernel NULL pointer dereference, address: 00000000000000a0 |
| Message-ID | <LQz0d-cB5Q-7@gated-at.bofh.it> |
| In reply to | #90005 |
Processing control commands: > tags -1 + moreinfo Bug #1120602 [src:linux] hyper-v: BUG: kernel NULL pointer dereference, address: 00000000000000a0 Added tag(s) moreinfo. -- 1120602: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1120602 Debian Bug Tracking System Contact owner@bugs.debian.org with problems
[toc] | [prev] | [next] | [standalone]
| From | Salvatore Bonaccorso <carnil@debian.org> |
|---|---|
| Date | 2025-11-13 07:30 +0100 |
| Message-ID | <LQz0d-cB5Q-5@gated-at.bofh.it> |
| In reply to | #90005 |
Control: tags -1 + moreinfo
Hi Peter,
Thanks a lot for the report.
On Wed, Nov 12, 2025 at 10:56:38PM +0000, Peter Morrow wrote:
> Package: src:linux
> Version: 6.12.57-1
> Severity: important
> X-Debbugs-Cc: pdmorrow@gmail.com
>
> Dear Maintainer,
>
> I'm seeing a kernel crash quite soon after boot on a debian trixie based
> system running 6.12.57+deb13-amd64, unfortunately the kernel panics before
> I can access the system to gather more information. Thus I'll provide details
> of the system using a previously known good version. The panic is happening
> 100% of the time unfortunately. I have access to the serial console however
> so can enable any required verbose logging during boot if necessary.
>
> Crucially the crash is not seen with kernel version 6.12.41+deb13-amd64 with the
> same userspace. We had pinned to that version until very recently to in order
> to work around https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1109676
>
> I'm running a dpdk application here (VPP) on Azure, VM form factor is a
> "Standard DS3 v2 (4 vcpus, 14 GiB memory)".
>
> The only relevant upstream commit in this area (as far as I can see) is:
>
> https://lore.kernel.org/linux-hyperv/1bb599ee-fe28-409d-b430-2fc086268936@linux.microsoft.com/
>
> The comment regarding avoiding races at start adds a bit more weight behind this
> hunch, though it's only a hunch as I am most definitely nowhere near an expert
> in this area.
I have a couple of questions. As you pinned 6.12.41+deb13-amd64, but
this was not the most recent kernel for trixie, can you confirm you
are seeing or not seeing the issue as well with 6.12.48-1 (this was
released via security update).
This would help narrowing the range further.
The commit b15b7d2a1b09 ("uio_hv_generic: Let userspace take care of
interrupt mask") was backported to 6.12.53, so if you do not see the
problem with 6.12.48-1 then this further weight for the offending
commit.
Secondly: Would you have either the possibility to bisect the changes
between the given range of good/bad kernel to isolate the offending
commit with a proof? Alternatively could you build 6.12.57-1 with the
commit reverted only? (you can use the debian/bin/test-patches script
for that, instructions in
https://kernel-team.pages.debian.net/kernel-handbook/ch-common-tasks.html#id-1.6.6.4,
let me know though if you need help to make the patch to be used).
Could you additionally (just to test, then revert back to the regular
trixie kernel series) test the 6.17.7-2 kernel from unstable? This one
would include as well a backport of b15b7d2a1b09.
Regards,
Salvatore
[toc] | [prev] | [next] | [standalone]
| From | Peter Morrow <pdmorrow@gmail.com> |
|---|---|
| Date | 2025-11-13 14:40 +0100 |
| Message-ID | <LQFIm-cFss-3@gated-at.bofh.it> |
| In reply to | #90005 |
+ bug-tracking alias.
On Thu, 13 Nov 2025 at 13:14, Peter Morrow <pdmorrow@gmail.com> wrote:
>
> Hi Salvatore,
>
> Thank you for your reply and maintainership!
>
> On Thu, 13 Nov 2025 at 06:20, Salvatore Bonaccorso <carnil@debian.org> wrote:
> >
> > Control: tags -1 + moreinfo
> >
> > Hi Peter,
> >
> > Thanks a lot for the report.
> >
> > On Wed, Nov 12, 2025 at 10:56:38PM +0000, Peter Morrow wrote:
> > > Package: src:linux
> > > Version: 6.12.57-1
> > > Severity: important
> > > X-Debbugs-Cc: pdmorrow@gmail.com
> > >
> > > Dear Maintainer,
> > >
> > > I'm seeing a kernel crash quite soon after boot on a debian trixie based
> > > system running 6.12.57+deb13-amd64, unfortunately the kernel panics before
> > > I can access the system to gather more information. Thus I'll provide details
> > > of the system using a previously known good version. The panic is happening
> > > 100% of the time unfortunately. I have access to the serial console however
> > > so can enable any required verbose logging during boot if necessary.
> > >
> > > Crucially the crash is not seen with kernel version 6.12.41+deb13-amd64 with the
> > > same userspace. We had pinned to that version until very recently to in order
> > > to work around https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1109676
> > >
> > > I'm running a dpdk application here (VPP) on Azure, VM form factor is a
> > > "Standard DS3 v2 (4 vcpus, 14 GiB memory)".
> > >
> > > The only relevant upstream commit in this area (as far as I can see) is:
> > >
> > > https://lore.kernel.org/linux-hyperv/1bb599ee-fe28-409d-b430-2fc086268936@linux.microsoft.com/
> > >
> > > The comment regarding avoiding races at start adds a bit more weight behind this
> > > hunch, though it's only a hunch as I am most definitely nowhere near an expert
> > > in this area.
> >
> > I have a couple of questions. As you pinned 6.12.41+deb13-amd64, but
> > this was not the most recent kernel for trixie, can you confirm you
> > are seeing or not seeing the issue as well with 6.12.48-1 (this was
> > released via security update).
>
> I installed linux-image-6.12.48+deb13-amd64 and can confirm it's working fine,
> no crashes on boot.
>
> >
> > This would help narrowing the range further.
> >
> > The commit b15b7d2a1b09 ("uio_hv_generic: Let userspace take care of
> > interrupt mask") was backported to 6.12.53, so if you do not see the
> > problem with 6.12.48-1 then this further weight for the offending
> > commit.
> >
> > Secondly: Would you have either the possibility to bisect the changes
> > between the given range of good/bad kernel to isolate the offending
> > commit with a proof? Alternatively could you build 6.12.57-1 with the
> > commit reverted only? (you can use the debian/bin/test-patches script
> > for that, instructions in
> > https://kernel-team.pages.debian.net/kernel-handbook/ch-common-tasks.html#id-1.6.6.4,
> > let me know though if you need help to make the patch to be used).
>
> I was able to build 6.12.57-1 with b15b7d2a1b09 reverted using test-patches and
> confirm that the kernel does not panic on boot. So it looks fairly
> conclusive to me that
> b15b7d2a1b09 is the culprit for me.
>
> >
> > Could you additionally (just to test, then revert back to the regular
> > trixie kernel series) test the 6.17.7-2 kernel from unstable? This one
> > would include as well a backport of b15b7d2a1b09.
>
> Would you still like this test done? I suspect it's not required now with the
> data we have on b15b7d2a1b09.
>
> Thanks,
> Peter.
>
> >
> > Regards,
> > Salvatore
[toc] | [prev] | [next] | [standalone]
| From | "Debian Bug Tracking System" <owner@bugs.debian.org> |
|---|---|
| Date | 2025-11-13 15:00 +0100 |
| Subject | Processed: Re: Bug#1120602: hyper-v: BUG: kernel NULL pointer dereference, address: 00000000000000a0 |
| Message-ID | <LQG1H-cFBi-1@gated-at.bofh.it> |
| In reply to | #90005 |
Processing control commands: > tags -1 + upstream Bug #1120602 [src:linux] hyper-v: BUG: kernel NULL pointer dereference, address: 00000000000000a0 Added tag(s) upstream. > tags -1 - moreinfo Bug #1120602 [src:linux] hyper-v: BUG: kernel NULL pointer dereference, address: 00000000000000a0 Removed tag(s) moreinfo. -- 1120602: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1120602 Debian Bug Tracking System Contact owner@bugs.debian.org with problems
[toc] | [prev] | [next] | [standalone]
| From | Salvatore Bonaccorso <carnil@debian.org> |
|---|---|
| Date | 2025-11-13 15:00 +0100 |
| Message-ID | <LQG1H-cFBi-3@gated-at.bofh.it> |
| In reply to | #90005 |
Control: tags -1 + upstream
Control: tags -1 - moreinfo
Hi Peter,
On Thu, Nov 13, 2025 at 01:14:37PM +0000, Peter Morrow wrote:
> Hi Salvatore,
>
> Thank you for your reply and maintainership!
Welcome :)
> On Thu, 13 Nov 2025 at 06:20, Salvatore Bonaccorso <carnil@debian.org> wrote:
> >
> > Control: tags -1 + moreinfo
> >
> > Hi Peter,
> >
> > Thanks a lot for the report.
> >
> > On Wed, Nov 12, 2025 at 10:56:38PM +0000, Peter Morrow wrote:
> > > Package: src:linux
> > > Version: 6.12.57-1
> > > Severity: important
> > > X-Debbugs-Cc: pdmorrow@gmail.com
> > >
> > > Dear Maintainer,
> > >
> > > I'm seeing a kernel crash quite soon after boot on a debian trixie based
> > > system running 6.12.57+deb13-amd64, unfortunately the kernel panics before
> > > I can access the system to gather more information. Thus I'll provide details
> > > of the system using a previously known good version. The panic is happening
> > > 100% of the time unfortunately. I have access to the serial console however
> > > so can enable any required verbose logging during boot if necessary.
> > >
> > > Crucially the crash is not seen with kernel version 6.12.41+deb13-amd64 with the
> > > same userspace. We had pinned to that version until very recently to in order
> > > to work around https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1109676
> > >
> > > I'm running a dpdk application here (VPP) on Azure, VM form factor is a
> > > "Standard DS3 v2 (4 vcpus, 14 GiB memory)".
> > >
> > > The only relevant upstream commit in this area (as far as I can see) is:
> > >
> > > https://lore.kernel.org/linux-hyperv/1bb599ee-fe28-409d-b430-2fc086268936@linux.microsoft.com/
> > >
> > > The comment regarding avoiding races at start adds a bit more weight behind this
> > > hunch, though it's only a hunch as I am most definitely nowhere near an expert
> > > in this area.
> >
> > I have a couple of questions. As you pinned 6.12.41+deb13-amd64, but
> > this was not the most recent kernel for trixie, can you confirm you
> > are seeing or not seeing the issue as well with 6.12.48-1 (this was
> > released via security update).
>
> I installed linux-image-6.12.48+deb13-amd64 and can confirm it's working fine,
> no crashes on boot.
Thanks for doing so.
> >
> > This would help narrowing the range further.
> >
> > The commit b15b7d2a1b09 ("uio_hv_generic: Let userspace take care of
> > interrupt mask") was backported to 6.12.53, so if you do not see the
> > problem with 6.12.48-1 then this further weight for the offending
> > commit.
> >
> > Secondly: Would you have either the possibility to bisect the changes
> > between the given range of good/bad kernel to isolate the offending
> > commit with a proof? Alternatively could you build 6.12.57-1 with the
> > commit reverted only? (you can use the debian/bin/test-patches script
> > for that, instructions in
> > https://kernel-team.pages.debian.net/kernel-handbook/ch-common-tasks.html#id-1.6.6.4,
> > let me know though if you need help to make the patch to be used).
>
> I was able to build 6.12.57-1 with b15b7d2a1b09 reverted using test-patches and
> confirm that the kernel does not panic on boot. So it looks fairly
> conclusive to me that
> b15b7d2a1b09 is the culprit for me.
Ok that is already a good result thank you. The next step is to
forward it upstream, will do shortly, still the following point:
> >
> > Could you additionally (just to test, then revert back to the regular
> > trixie kernel series) test the 6.17.7-2 kernel from unstable? This one
> > would include as well a backport of b15b7d2a1b09.
>
> Would you still like this test done? I suspect it's not required now with the
> data we have on b15b7d2a1b09.
Not necessarily if it is a problem for deploying it for your
enviornment. If you can, then we have the benefit to confirm to
upstream that indeed it is still affecting more recent versions, and
unlikely that some other changes inbetween make the issue disapear
(even if there is no commit with fixes tag to the orginal commit).
So not necessary, but still helpful to confirm that even newer
branches are affected.
I will preapre the report upstream, and upstream might have questions
back to you.
Regards,
Salvatore
[toc] | [prev] | [next] | [standalone]
| From | Peter Morrow <pdmorrow@gmail.com> |
|---|---|
| Date | 2025-11-13 15:40 +0100 |
| Message-ID | <LQGEp-cG8M-7@gated-at.bofh.it> |
| In reply to | #90012 |
Hi Salvatore,
On Thu, 13 Nov 2025 at 13:57, Salvatore Bonaccorso <carnil@debian.org> wrote:
>
> Control: tags -1 + upstream
> Control: tags -1 - moreinfo
>
> Hi Peter,
>
> On Thu, Nov 13, 2025 at 01:14:37PM +0000, Peter Morrow wrote:
> > Hi Salvatore,
> >
> > Thank you for your reply and maintainership!
>
> Welcome :)
>
> > On Thu, 13 Nov 2025 at 06:20, Salvatore Bonaccorso <carnil@debian.org> wrote:
> > >
> > > Control: tags -1 + moreinfo
> > >
> > > Hi Peter,
> > >
> > > Thanks a lot for the report.
> > >
> > > On Wed, Nov 12, 2025 at 10:56:38PM +0000, Peter Morrow wrote:
> > > > Package: src:linux
> > > > Version: 6.12.57-1
> > > > Severity: important
> > > > X-Debbugs-Cc: pdmorrow@gmail.com
> > > >
> > > > Dear Maintainer,
> > > >
> > > > I'm seeing a kernel crash quite soon after boot on a debian trixie based
> > > > system running 6.12.57+deb13-amd64, unfortunately the kernel panics before
> > > > I can access the system to gather more information. Thus I'll provide details
> > > > of the system using a previously known good version. The panic is happening
> > > > 100% of the time unfortunately. I have access to the serial console however
> > > > so can enable any required verbose logging during boot if necessary.
> > > >
> > > > Crucially the crash is not seen with kernel version 6.12.41+deb13-amd64 with the
> > > > same userspace. We had pinned to that version until very recently to in order
> > > > to work around https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1109676
> > > >
> > > > I'm running a dpdk application here (VPP) on Azure, VM form factor is a
> > > > "Standard DS3 v2 (4 vcpus, 14 GiB memory)".
> > > >
> > > > The only relevant upstream commit in this area (as far as I can see) is:
> > > >
> > > > https://lore.kernel.org/linux-hyperv/1bb599ee-fe28-409d-b430-2fc086268936@linux.microsoft.com/
> > > >
> > > > The comment regarding avoiding races at start adds a bit more weight behind this
> > > > hunch, though it's only a hunch as I am most definitely nowhere near an expert
> > > > in this area.
> > >
> > > I have a couple of questions. As you pinned 6.12.41+deb13-amd64, but
> > > this was not the most recent kernel for trixie, can you confirm you
> > > are seeing or not seeing the issue as well with 6.12.48-1 (this was
> > > released via security update).
> >
> > I installed linux-image-6.12.48+deb13-amd64 and can confirm it's working fine,
> > no crashes on boot.
>
> Thanks for doing so.
>
> > >
> > > This would help narrowing the range further.
> > >
> > > The commit b15b7d2a1b09 ("uio_hv_generic: Let userspace take care of
> > > interrupt mask") was backported to 6.12.53, so if you do not see the
> > > problem with 6.12.48-1 then this further weight for the offending
> > > commit.
> > >
> > > Secondly: Would you have either the possibility to bisect the changes
> > > between the given range of good/bad kernel to isolate the offending
> > > commit with a proof? Alternatively could you build 6.12.57-1 with the
> > > commit reverted only? (you can use the debian/bin/test-patches script
> > > for that, instructions in
> > > https://kernel-team.pages.debian.net/kernel-handbook/ch-common-tasks.html#id-1.6.6.4,
> > > let me know though if you need help to make the patch to be used).
> >
> > I was able to build 6.12.57-1 with b15b7d2a1b09 reverted using test-patches and
> > confirm that the kernel does not panic on boot. So it looks fairly
> > conclusive to me that
> > b15b7d2a1b09 is the culprit for me.
>
> Ok that is already a good result thank you. The next step is to
> forward it upstream, will do shortly, still the following point:
>
> > >
> > > Could you additionally (just to test, then revert back to the regular
> > > trixie kernel series) test the 6.17.7-2 kernel from unstable? This one
> > > would include as well a backport of b15b7d2a1b09.
> >
> > Would you still like this test done? I suspect it's not required now with the
> > data we have on b15b7d2a1b09.
>
> Not necessarily if it is a problem for deploying it for your
> enviornment. If you can, then we have the benefit to confirm to
> upstream that indeed it is still affecting more recent versions, and
> unlikely that some other changes inbetween make the issue disapear
> (even if there is no commit with fixes tag to the orginal commit).
>
> So not necessary, but still helpful to confirm that even newer
> branches are affected.
It was straight forward to try out, Interestingly I am not seeing the
panic on 6.17.7-2:
gnos@vEdgeOlder:~$ uname -a
Linux vEdgeOlder 6.17.7+deb14+1-amd64 #1 SMP PREEMPT_DYNAMIC Debian
6.17.7-2 (2025-11-06) x86_64 GNU/Linux
gnos@vEdgeOlder:~$
I also then retried 6.12.57-1 to make sure I am not hallucinating
today, and it indeed still panic'd on boot:
vEdgeOlder login: [ 30.339054] BUG: kernel NULL pointer dereference,
address: 00000000000000a0
[ 30.340151] #PF: supervisor read access in kernel mode
[ 30.341032] #PF: error_code(0x0000) - not-present page
[ 30.341739] PGD 80000003075ef067 P4D 80000003075ef067 PUD 0
[ 30.342525] Oops: Oops: 0000 [#1] PREEMPT SMP PTI
[ 30.343223] CPU: 1 UID: 0 PID: 1468 Comm: vpp_wk_0 Not tainted
6.12.57+deb13-amd64 #1 Debian 6.12.57-1
[ 30.344492] Hardware name: Microsoft Corporation Virtual
Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.1 05/13/2024
[ 30.346032] RIP: 0010:hv_uio_channel_cb+0xd/0x20 [uio_hv_generic]
[ 30.346898] Code: 02 00 00 5b 5d e9 53 d8 ef d9 0f 1f 00 90 90 90
90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 48
8b 47 10 <48> 8b b8 a0 00 00 00 f0 83 44 24 fc 00 e9 51 3f ff ff 90 90
90 90
[ 30.349433] RSP: 0000:ffffa92213647ef0 EFLAGS: 00010046
[ 30.350469] RAX: 0000000000000000 RBX: 0000000000000018 RCX: 0000000000000018
[ 30.351752] RDX: 0000000000000001 RSI: ffffffffffffffff RDI: ffff998fdbdfb000
[ 30.352880] RBP: ffff998fc1b2f200 R08: ffff998fc1b2f200 R09: 0000000000000000
[ 30.354030] R10: 0000000000000000 R11: 0000000000000000 R12: ffff9992f1cc1460
[ 30.355004] R13: ffff998fdbdfb000 R14: ffff998fdbdfb2a0 R15: ffffffffc100a160
[ 30.356005] FS: 00007fcea0f2a6c0(0000) GS:ffff9992f1c80000(0000)
knlGS:0000000000000000
[ 30.357069] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 30.357907] CR2: 00000000000000a0 CR3: 0000000150a4e001 CR4: 00000000003706f0
[ 30.358848] Call Trace:
[ 30.359209] <TASK>
[ 30.359568] vmbus_isr+0x1a5/0x210 [hv_vmbus]
[ 30.360162] __sysvec_hyperv_callback+0x32/0x60
[ 30.360911] sysvec_hyperv_callback+0x38/0x90
[ 30.363023] asm_sysvec_hyperv_callback+0x1a/0x20
[ 30.363819] RIP: 0033:0x7fcfedf3cbf6
[ 30.364362] Code: 54 24 18 90 49 8b 04 24 4a 8b 44 e8 40 49 8b 0e
48 85 c9 74 0f 8b 49 f8 eb 0c 66 2e 0f 1f 84 00 00 00 00 00 31 c9 31
d2 f7 f1 <39> c5 75 86 f3 90 eb d2 0f 31 41 8b 76 70 48 8b 4c 24 10 41
3b 76
[ 30.366863] RSP: 002b:00007fcea0f29980 EFLAGS: 00000246
[ 30.367542] RAX: 0000000000000000 RBX: 00007fcf1e01c9c0 RCX: 0000000000000004
[ 30.368579] RDX: 0000000000000003 RSI: 00007fcfee0d1ea0 RDI: 00007fcf1e01c9c0
[ 30.369522] RBP: 0000000000000000 R08: 0000000000000002 R09: 00007fcf1dfdc700
[ 30.370457] R10: 0000000000000000 R11: 00007fcf1da0db90 R12: 00007fcfee0d3f28
[ 30.371407] R13: 0000000000000003 R14: 00007fcfee0d1000 R15: 00007fcfee0ce240
[ 30.372438] </TASK>
[ 30.372768] Modules linked in: uio_hv_generic uio binfmt_misc
dm_crypt intel_rapl_msr intel_rapl_common rpcrdma
intel_uncore_frequency_common isst_if_mbox_msr sunrpc isst_if_common
rdma_ucm ib_iser rdma_cm skx_edac_common nfit ib_umad iw_cm libnvdimm
ib_ipoib libiscsi crct10dif_pclmul ghash_clmulni_intel sha512_ssse3
sha256_ssse3 sha1_ssse3 scsi_transport_iscsi ib_cm aesni_intel
gf128mul crypto_simd cryptd rapl pcspkr hv_utils hv_balloon sg evdev
joydev mpls_router ip_tunnel ramoops pstore_blk pstore_zone efi_pstore
configfs nfnetlink vsock_loopback vmw_vsock_virtio_transport_common
hv_sock vmw_vsock_vmci_transport vsock vmw_vmci efivarfs ip_tables
x_tables autofs4 overlay squashfs dm_verity dm_bufio reed_solomon
dm_mod loop ext4 crc16 mbcache jbd2 crc32c_generic mlx5_ib ib_uverbs
ib_core mlx5_core mlxfw pci_hyperv pci_hyperv_intf hyperv_drm sr_mod
drm_shmem_helper drm_kms_helper cdrom sd_mod hv_storvsc
scsi_transport_fc drm scsi_mod hid_generic hid_hyperv serio_raw hid
hv_netvsc hyperv_keyboard scsi_common hv_vmbus
[ 30.372857] crc32_pclmul crc32c_intel
[ 30.385686] CR2: 00000000000000a0
[ 30.386562] ---[ end trace 0000000000000000 ]---
[ 31.285814] BUG: kernel NULL pointer dereference, address: 00000000000000a0
[ 31.287264] #PF: supervisor read access in kernel mode
[ 31.288392] #PF: error_code(0x0000) - not-present page
[ 31.289550] PGD 80000003075ef067 P4D 80000003075ef067 PUD 0
[ 31.290704] Oops: Oops: 0000 [#2] PREEMPT SMP PTI
[ 31.291711] CPU: 2 UID: 0 PID: 1469 Comm: vpp_wk_1 Tainted: G
D 6.12.57+deb13-amd64 #1 Debian 6.12.57-1
[ 31.293647] Tainted: [D]=DIE
[ 31.294427] Hardware name: Microsoft Corporation Virtual
Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.1 05/13/2024
[ 31.296231] RIP: 0010:hv_uio_channel_cb+0xd/0x20 [uio_hv_generic]
[ 31.297428] Code: 02 00 00 5b 5d e9 53 d8 ef d9 0f 1f 00 90 90 90
90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 48
8b 47 10 <48> 8b b8 a0 00 00 00 f0 83 44 24 fc 00 e9 51 3f ff ff 90 90
90 90
[ 31.300539] RSP: 0000:ffffa9221364fef0 EFLAGS: 00010046
[ 31.301656] RAX: 0000000000000000 RBX: 0000000000000014 RCX: 0000000000000014
[ 31.302999] RDX: 0000000000000001 RSI: ffffffffffffffff RDI: ffff998fdbe1c400
[ 31.304360] RBP: ffff998fc1b2d200 R08: ffff998fc1b2d200 R09: 0000000000000000
[ 31.305658] R10: 0000000000000000 R11: 0000000000000000 R12: ffff9992f1d41460
[ 31.306980] R13: ffff998fdbe1c400 R14: ffff998fdbe1c6a0 R15: ffffffffc100a160
[ 31.308293] FS: 00007fcea0d296c0(0000) GS:ffff9992f1d00000(0000)
knlGS:0000000000000000
[ 31.309787] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 31.310955] CR2: 00000000000000a0 CR3: 0000000150a4e005 CR4: 00000000003706f0
[ 31.312277] Call Trace:
[ 31.313024] <TASK>
[ 31.313711] vmbus_isr+0x1a5/0x210 [hv_vmbus]
[ 31.314712] __sysvec_hyperv_callback+0x32/0x60
[ 31.315725] sysvec_hyperv_callback+0x38/0x90
[ 31.316715] asm_sysvec_hyperv_callback+0x1a/0x20
[ 31.317739] RIP: 0033:0x7fcfedf3cbf6
[ 31.318728] Code: 54 24 18 90 49 8b 04 24 4a 8b 44 e8 40 49 8b 0e
48 85 c9 74 0f 8b 49 f8 eb 0c 66 2e 0f 1f 84 00 00 00 00 00 31 c9 31
d2 f7 f1 <39> c5 75 86 f3 90 eb d2 0f 31 41 8b 76 70 48 8b 4c 24 10 41
3b 76
[ 31.321967] RSP: 002b:00007fcea0d28980 EFLAGS: 00000246
[ 31.323101] RAX: 0000000000000000 RBX: 00007fcf1e079800 RCX: 0000000000000004
[ 31.324480] RDX: 0000000000000002 RSI: 0000000000000137 RDI: 00007fcf13400740
[ 31.325810] RBP: 0000000000000000 R08: 00007fcf144df828 R09: 00007fcf1e0cdc10
[ 31.327196] R10: 0000000000000100 R11: 00007fcf13400030 R12: 00007fcfee0d3f28
[ 31.328618] R13: 0000000000000004 R14: 00007fcfee0d1000 R15: 00007fcfee0ce240
[ 31.330010] </TASK>
[ 31.330721] Modules linked in: uio_hv_generic uio binfmt_misc
dm_crypt intel_rapl_msr intel_rapl_common rpcrdma
intel_uncore_frequency_common isst_if_mbox_msr sunrpc isst_if_common
rdma_ucm ib_iser rdma_cm skx_edac_common nfit ib_umad iw_cm libnvdimm
ib_ipoib libiscsi crct10dif_pclmul ghash_clmulni_intel sha512_ssse3
sha256_ssse3 sha1_ssse3 scsi_transport_iscsi ib_cm aesni_intel
gf128mul crypto_simd cryptd rapl pcspkr hv_utils hv_balloon sg evdev
joydev mpls_router ip_tunnel ramoops pstore_blk pstore_zone efi_pstore
configfs nfnetlink vsock_loopback vmw_vsock_virtio_transport_common
hv_sock vmw_vsock_vmci_transport vsock vmw_vmci efivarfs ip_tables
x_tables autofs4 overlay squashfs dm_verity dm_bufio reed_solomon
dm_mod loop ext4 crc16 mbcache jbd2 crc32c_generic mlx5_ib ib_uverbs
ib_core mlx5_core mlxfw pci_hyperv pci_hyperv_intf hyperv_drm sr_mod
drm_shmem_helper drm_kms_helper cdrom sd_mod hv_storvsc
scsi_transport_fc drm scsi_mod hid_generic hid_hyperv serio_raw hid
hv_netvsc hyperv_keyboard scsi_common hv_vmbus
[ 31.330809] crc32_pclmul crc32c_intel
[ 31.347331] CR2: 00000000000000a0
[ 31.348178] ---[ end trace 0000000000000000 ]---
[ 35.274759] BUG: kernel NULL pointer dereference, address: 00000000000000a0
[ 35.276406] #PF: supervisor read access in kernel mode
[ 35.277688] #PF: error_code(0x0000) - not-present page
[ 35.278974] PGD 80000003075ef067 P4D 80000003075ef067 PUD 0
[ 35.280288] Oops: Oops: 0000 [#3] PREEMPT SMP PTI
[ 35.281501] CPU: 0 UID: 0 PID: 1008 Comm: vpp_main Tainted: G
D 6.12.57+deb13-amd64 #1 Debian 6.12.57-1
[ 35.283706] Tainted: [D]=DIE
[ 35.284643] Hardware name: Microsoft Corporation Virtual
Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.1 05/13/2024
[ 35.286771] RIP: 0010:hv_uio_channel_cb+0xd/0x20 [uio_hv_generic]
[ 35.288201] Code: 02 00 00 5b 5d e9 53 d8 ef d9 0f 1f 00 90 90 90
90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 48
8b 47 10 <48> 8b b8 a0 00 00 00 f0 83 44 24 fc 00 e9 51 3f ff ff 90 90
90 90
[ 35.291920] RSP: 0000:ffffa922071cfef0 EFLAGS: 00010046
[ 35.293179] RAX: 0000000000000000 RBX: 0000000000000017 RCX: 0000000000000017
[ 35.294753] RDX: 0000000000000001 RSI: ffffffffffffffff RDI: ffff998fdbdf8000
[ 35.296292] RBP: ffff998fc1b31200 R08: ffff998fc1b31200 R09: 0000000000000000
[ 35.297830] R10: 0000000000000000 R11: 0000000000000000 R12: ffff9992f1c41460
[ 35.299480] R13: ffff998fdbdf8000 R14: ffff998fdbdf82a0 R15: ffffffffc100a160
[ 35.301039] FS: 00007fcfedc5ab00(0000) GS:ffff9992f1c00000(0000)
knlGS:0000000000000000
[ 35.302758] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 35.304064] CR2: 00000000000000a0 CR3: 0000000150a4e004 CR4: 00000000003706f0
[ 35.305642] Call Trace:
[ 35.306470] <TASK>
[ 35.308091] vmbus_isr+0x1a5/0x210 [hv_vmbus]
[ 35.309235] __sysvec_hyperv_callback+0x32/0x60
[ 35.310384] sysvec_hyperv_callback+0x38/0x90
[ 35.311556] asm_sysvec_hyperv_callback+0x1a/0x20
[ 35.312689] RIP: 0033:0x7fcfedf426fc
[ 35.313702] Code: 49 2b 56 30 41 0f b6 4e 44 48 d3 ea 48 85 d2 75
68 66 49 0f 6e d4 66 0f 62 d3 66 0f 5c d4 66 0f 28 ca 66 0f 15 ca f2
0f 58 ca <f2> 0f 59 e9 f2 41 0f 58 6e 50 66 0f 2e e8 76 94 48 8b 05 6d
c7 18
[ 35.317465] RSP: 002b:00007fcf1285fc80 EFLAGS: 00000246
[ 35.318710] RAX: 00000000477f033e RBX: 0000000000000003 RCX: 0000000000000022
[ 35.320334] RDX: 0000000000000000 RSI: 0000000000000004 RDI: 00007fcf13400740
[ 35.321894] RBP: 0000000000000004 R08: 0000000000000007 R09: 00007fcf1413ea28
[ 35.323385] R10: 0000000000001965 R11: 0000000000000000 R12: 00000006969940b8
[ 35.324878] R13: 0000000000000004 R14: 00007fcf13400740 R15: 00007fcfee0d3f28
[ 35.326423] </TASK>
[ 35.327186] Modules linked in: uio_hv_generic uio binfmt_misc
dm_crypt intel_rapl_msr intel_rapl_common rpcrdma
intel_uncore_frequency_common isst_if_mbox_msr sunrpc isst_if_common
rdma_ucm ib_iser rdma_cm skx_edac_common nfit ib_umad iw_cm libnvdimm
ib_ipoib libiscsi crct10dif_pclmul ghash_clmulni_intel sha512_ssse3
sha256_ssse3 sha1_ssse3 scsi_transport_iscsi ib_cm aesni_intel
gf128mul crypto_simd cryptd rapl pcspkr hv_utils hv_balloon sg evdev
joydev mpls_router ip_tunnel ramoops pstore_blk pstore_zone efi_pstore
configfs nfnetlink vsock_loopback vmw_vsock_virtio_transport_common
hv_sock vmw_vsock_vmci_transport vsock vmw_vmci efivarfs ip_tables
x_tables autofs4 overlay squashfs dm_verity dm_bufio reed_solomon
dm_mod loop ext4 crc16 mbcache jbd2 crc32c_generic mlx5_ib ib_uverbs
ib_core mlx5_core mlxfw pci_hyperv pci_hyperv_intf hyperv_drm sr_mod
drm_shmem_helper drm_kms_helper cdrom sd_mod hv_storvsc
scsi_transport_fc drm scsi_mod hid_generic hid_hyperv serio_raw hid
hv_netvsc hyperv_keyboard scsi_common hv_vmbus
[ 35.327278] crc32_pclmul crc32c_intel
[ 35.347826] CR2: 00000000000000a0
[ 35.348819] ---[ end trace 0000000000000000 ]---
[ 45.629814] RIP: 0010:hv_uio_channel_cb+0xd/0x20 [uio_hv_generic]
[ 45.631166] Code: 02 00 00 5b 5d e9 53 d8 ef d9 0f 1f 00 90 90 90
90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 48
8b 47 10 <48> 8b b8 a0 00 00 00 f0 83 44 24 fc 00 e9 51 3f ff ff 90 90
90 90
[ 45.634659] RSP: 0000:ffffa92213647ef0 EFLAGS: 00010046
[ 45.635762] RAX: 0000000000000000 RBX: 0000000000000018 RCX: 0000000000000018
[ 45.637168] RDX: 0000000000000001 RSI: ffffffffffffffff RDI: ffff998fdbdfb000
[ 45.638559] RBP: ffff998fc1b2f200 R08: ffff998fc1b2f200 R09: 0000000000000000
[ 45.639911] R10: 0000000000000000 R11: 0000000000000000 R12: ffff9992f1cc1460
[ 45.641269] R13: ffff998fdbdfb000 R14: ffff998fdbdfb2a0 R15: ffffffffc100a160
[ 45.642633] FS: 00007fcea0f2a6c0(0000) GS:ffff9992f1c80000(0000)
knlGS:0000000000000000
[ 45.644064] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 45.645346] CR2: 00000000000000a0 CR3: 0000000150a4e001 CR4: 00000000003706f0
[ 45.646697] Kernel panic - not syncing: Fatal exception in interrupt
[ 46.640112] Kernel Offset: 0x19200000 from 0xffffffff81000000
(relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 46.672487] pstore: dump skipped in Panic path because of concurrent dump
[ 46.674016] ---[ end Kernel panic - not syncing: Fatal exception in
interrupt ]---
Thanks,
Peter.
>
> I will preapre the report upstream, and upstream might have questions
> back to you.
>
> Regards,
> Salvatore
[toc] | [prev] | [next] | [standalone]
| From | Naman Jain <namjain@linux.microsoft.com> |
|---|---|
| Date | 2025-11-14 07:10 +0100 |
| Subject | Bug#1120602: [REGRESSION 6.12.y] hyper-v: BUG: kernel NULL pointer dereference, address: 00000000000000a0: RIP: 0010:hv_uio_channel_cb+0xd/0x20 [uio_hv_generic] |
| Message-ID | <LQVap-cQkf-3@gated-at.bofh.it> |
| In reply to | #90005 |
On 11/13/2025 11:59 PM, Salvatore Bonaccorso wrote:
> Peter Morrow reported in Debian a regression, reported in
> https://bugs.debian.org/1120602 . The regression was seen after
> updating, to 6.12.57-1 in Debian, but details on the offending commit
> follows.
>
> His report was as follows:
>
>> Dear Maintainer,
>>
>> I'm seeing a kernel crash quite soon after boot on a debian trixie based
>> system running 6.12.57+deb13-amd64, unfortunately the kernel panics before
>> I can access the system to gather more information. Thus I'll provide details
>> of the system using a previously known good version. The panic is happening
>> 100% of the time unfortunately. I have access to the serial console however
>> so can enable any required verbose logging during boot if necessary.
>>
>> Crucially the crash is not seen with kernel version 6.12.41+deb13-amd64 with the
>> same userspace. We had pinned to that version until very recently to in order
>> to work around https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1109676
>>
>> I'm running a dpdk application here (VPP) on Azure, VM form factor is a
>> "Standard DS3 v2 (4 vcpus, 14 GiB memory)".
>>
>> The only relevant upstream commit in this area (as far as I can see) is:
>>
>> https://lore.kernel.org/linux-hyperv/1bb599ee-fe28-409d-b430-2fc086268936@linux.microsoft.com/
>>
>> The comment regarding avoiding races at start adds a bit more weight behind this
>> hunch, though it's only a hunch as I am most definitely nowhere near an expert
>> in this area.
>>
>> -- Package-specific info:
>>
>> [ 19.625535] BUG: kernel NULL pointer dereference, address: 00000000000000a0
>> [ 19.628874] #PF: supervisor read access in kernel mode
>> [ 19.630841] #PF: error_code(0x0000) - not-present page
>> [ 19.632788] PGD 0 P4D 0
>> [ 19.633905] Oops: Oops: 0000 [#1] PREEMPT SMP PTI
>> [ 19.635586] CPU: 3 UID: 0 PID: 0 Comm: swapper/3 Not tainted 6.12.57+deb13-amd64 #1 Debian 6.12.57-1
>> [ 19.640216] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.1 09/28/2024
>> [ 19.644514] RIP: 0010:hv_uio_channel_cb+0xd/0x20 [uio_hv_generic]
>> [ 19.646994] Code: 02 00 00 5b 5d e9 53 98 69 e9 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 48 8b 47 10 <48> 8b b8 a0 00 00 00 f0 83 44 24 fc 00 e9 51 6f fa ff 90 90 90 90
>> [ 19.654377] RSP: 0018:ffffb15ac01a4fa8 EFLAGS: 00010046
>> [ 19.656385] RAX: 0000000000000000 RBX: 0000000000000015 RCX: 0000000000000015
>> [ 19.659240] RDX: 0000000000000001 RSI: ffffffffffffffff RDI: ffff8ff69c759400
>> [ 19.662168] RBP: ffff8ff548790200 R08: ffff8ff548790200 R09: 00fca75150b080e9
>> [ 19.665239] R10: 0000000000000000 R11: ffffb15ac01a4ff8 R12: ffff8ff871dc1480
>> [ 19.668193] R13: ffff8ff69c759400 R14: ffff8ff69c7596a0 R15: ffffffffc106e160
>> [ 19.671106] FS: 0000000000000000(0000) GS:ffff8ff871d80000(0000) knlGS:0000000000000000
>> [ 19.674281] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [ 19.676533] CR2: 00000000000000a0 CR3: 0000000100ba6003 CR4: 00000000003706f0
>> [ 19.679385] Call Trace:
>> [ 19.680361] <IRQ>
>> [ 19.681181] vmbus_isr+0x1a5/0x210 [hv_vmbus]
>> [ 19.682916] __sysvec_hyperv_callback+0x32/0x60
>> [ 19.684991] sysvec_hyperv_callback+0x6c/0x90
>> [ 19.686665] </IRQ>
>> [ 19.687509] <TASK>
>> [ 19.688366] asm_sysvec_hyperv_callback+0x1a/0x20
>> [ 19.690262] RIP: 0010:pv_native_safe_halt+0xf/0x20
>> [ 19.692067] Code: 09 e9 c5 08 01 00 0f 1f 44 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 66 90 0f 00 2d e5 3b 31 00 fb f4 <c3> cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90
>> [ 19.699119] RSP: 0018:ffffb15ac0103ed8 EFLAGS: 00000246
>> [ 19.701412] RAX: 0000000000000003 RBX: ffff8ff5403b1fc0 RCX: ffff8ff54c64ce30
>> [ 19.704328] RDX: 0000000000000000 RSI: 0000000000000003 RDI: 000000000001f894
>> [ 19.706910] RBP: 0000000000000003 R08: 000000000bb760d9 R09: 00fca75150b080e9
>> [ 19.709762] R10: 0000000000000003 R11: 0000000000000001 R12: 0000000000000000
>> [ 19.712510] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
>> [ 19.715173] default_idle+0x9/0x20
>> [ 19.716846] default_idle_call+0x29/0x100
>> [ 19.718623] do_idle+0x1fe/0x240
>> [ 19.720045] cpu_startup_entry+0x29/0x30
>> [ 19.721595] start_secondary+0x11e/0x140
>> [ 19.723080] common_startup_64+0x13e/0x141
>> [ 19.725222] </TASK>
>> [ 19.726387] Modules linked in: isofs cdrom uio_hv_generic uio binfmt_misc intel_rapl_msr intel_rapl_common intel_uncore_frequency_common isst_if_mbox_msr isst_if_common rpcrdma skx_edac_common nfit sunrpc libnvdimm crct10dif_pclmul ghash_clmulni_intel sha512_ssse3 sha256_ssse3 rdma_ucm ib_iser sha1_ssse3 rdma_cm aesni_intel iw_cm gf128mul crypto_simd libiscsi cryptd ib_umad ib_ipoib scsi_transport_iscsi ib_cm rapl sg hv_utils hv_balloon evdev pcspkr joydev mpls_router ip_tunnel ramoops configfs pstore_blk efi_pstore pstore_zone nfnetlink vsock_loopback vmw_vsock_virtio_transport_common hv_sock vmw_vsock_vmci_transport vsock vmw_vmci efivarfs ip_tables x_tables autofs4 overlay squashfs dm_verity dm_bufio reed_solomon dm_mod loop ext4 crc16 mbcache jbd2 crc32c_generic mlx5_ib ib_uverbs ib_core mlx5_core mlxfw pci_hyperv pci_hyperv_intf hyperv_drm drm_shmem_helper sd_mod drm_kms_helper hv_storvsc scsi_transport_fc drm scsi_mod hid_generic hid_hyperv hid serio_raw hv_netvsc hyperv_keyboard scsi_common hv_vmbus
>> [ 19.726466] crc32_pclmul crc32c_intel
>> [ 19.765771] CR2: 00000000000000a0
>> [ 19.767524] ---[ end trace 0000000000000000 ]---
>> [ 19.800433] RIP: 0010:hv_uio_channel_cb+0xd/0x20 [uio_hv_generic]
>> [ 19.803170] Code: 02 00 00 5b 5d e9 53 98 69 e9 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 48 8b 47 10 <48> 8b b8 a0 00 00 00 f0 83 44 24 fc 00 e9 51 6f fa ff 90 90 90 90
>> [ 19.811041] RSP: 0018:ffffb15ac01a4fa8 EFLAGS: 00010046
>> [ 19.813466] RAX: 0000000000000000 RBX: 0000000000000015 RCX: 0000000000000015
>> [ 19.816504] RDX: 0000000000000001 RSI: ffffffffffffffff RDI: ffff8ff69c759400
>> [ 19.819484] RBP: ffff8ff548790200 R08: ffff8ff548790200 R09: 00fca75150b080e9
>> [ 19.822625] R10: 0000000000000000 R11: ffffb15ac01a4ff8 R12: ffff8ff871dc1480
>> [ 19.825569] R13: ffff8ff69c759400 R14: ffff8ff69c7596a0 R15: ffffffffc106e160
>> [ 19.828804] FS: 0000000000000000(0000) GS:ffff8ff871d80000(0000) knlGS:0000000000000000
>> [ 19.832214] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [ 19.834709] CR2: 00000000000000a0 CR3: 0000000100ba6003 CR4: 00000000003706f0
>> [ 19.837976] Kernel panic - not syncing: Fatal exception in interrupt
>> [ 19.841825] Kernel Offset: 0x28a00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
>> [ 19.896620] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---
>>
<snip>
> The offending commit appers to be the backport of b15b7d2a1b09
> ("uio_hv_generic: Let userspace take care of interrupt mask") for
> 6.12.y.
>
> Peter confirmed that reverting this commit on top of 6.12.57-1 as
> packaged in Debian resolves indeed the issue. Interestingly the issue
> is *not* seen with 6.17.7 based kernel in Debian.
>
> #regzbot introduced: 37bd91f22794dc05436130d6983302cb90ecfe7e
> #regzbot monitor: https://bugs.debian.org/1120602
>
> Thank you already!
>
> Regards,
> Salvatore
Hi Peter, Salvatore,
Thanks for reporting this crash, and sorry for the trouble. Here is my
analysis.
On 6.17.7, where commit d062463edf17 ("uio_hv_generic: Set event for all
channels on the device") is present, hv_uio_irqcontrol() supports
setting of interrupt mask from userspace for sub-channels as well.
This aligns with commit e29587c07537 ("uio_hv_generic: Let userspace
take care of interrupt mask") which relies on userspace to manage
interrupt mask, so it safely removes the interrupt mask management logic
in the driver.
However, in 6.12.57, the first commit is not present, but the second one
is, so there is no way to disable interrupt mask for sub-channels and
interrupt_mask stays 0, which means interrupts are not masked. So we may
be having an interrupt callback being handled for a sub-channel, where
we do not expect it to come. This may be causing this issue.
This would have led to a crash in hv_uio_channel_cb() for sub-channels:
struct hv_device *hv_dev = chan->device_obj;
I have ported commit d062463edf17 ("uio_hv_generic: Set event for all
channels on the device") on 6.12.57, and resolved some merge conflicts.
Could you please help with testing this, if it works for you.
Hi Long,
If this works, do you see any concerns if I back-port your patch on
older kernels (6.12 and prior)?
Regards,
Naman
--------------
Patch:
From 2f14d48d2bde3f86b153b9f756a9cd688cda3453 Mon Sep 17 00:00:00 2001
From: Long Li <longli@microsoft.com>
Date: Mon, 10 Mar 2025 15:12:01 -0700
Subject: [PATCH] uio_hv_generic: Set event for all channels on the device
Hyper-V may offer a non latency sensitive device with subchannels without
monitor bit enabled. The decision is entirely on the Hyper-V host not
configurable within guest.
When a device has subchannels, also signal events for the subchannel
if its monitor bit is disabled.
This patch also removes the memory barrier when monitor bit is enabled
as it is not necessary. The memory barrier is only needed between
setting up interrupt mask and calling vmbus_set_event() when monitor
bit is disabled.
Signed-off-by: Long Li <longli@microsoft.com>
Reviewed-by: Michael Kelley <mhklinux@outlook.com>
Reviewed-by: Saurabh Sengar <ssengar@linux.microsoft.com>
Signed-off-by: Naman Jain <namjain@linux.microsoft.com>
---
drivers/uio/uio_hv_generic.c | 32 ++++++++++++++++++++++++++------
1 file changed, 26 insertions(+), 6 deletions(-)
diff --git a/drivers/uio/uio_hv_generic.c b/drivers/uio/uio_hv_generic.c
index 0b414d1168dd..9f3b124a5e09 100644
--- a/drivers/uio/uio_hv_generic.c
+++ b/drivers/uio/uio_hv_generic.c
@@ -65,6 +65,16 @@ struct hv_uio_private_data {
char send_name[32];
};
+static void set_event(struct vmbus_channel *channel, s32 irq_state)
+{
+ channel->inbound.ring_buffer->interrupt_mask = !irq_state;
+ if (!channel->offermsg.monitor_allocated && irq_state) {
+ /* MB is needed for host to see the interrupt mask first */
+ virt_mb();
+ vmbus_set_event(channel);
+ }
+}
+
/*
* This is the irqcontrol callback to be registered to uio_info.
* It can be used to disable/enable interrupt from user space processes.
@@ -79,12 +89,15 @@ hv_uio_irqcontrol(struct uio_info *info, s32 irq_state)
{
struct hv_uio_private_data *pdata = info->priv;
struct hv_device *dev = pdata->device;
+ struct vmbus_channel *primary, *sc;
- dev->channel->inbound.ring_buffer->interrupt_mask = !irq_state;
- virt_mb();
+ primary = dev->channel;
+ set_event(primary, irq_state);
- if (!dev->channel->offermsg.monitor_allocated && irq_state)
- vmbus_setevent(dev->channel);
+ mutex_lock(&vmbus_connection.channel_mutex);
+ list_for_each_entry(sc, &primary->sc_list, sc_list)
+ set_event(sc, irq_state);
+ mutex_unlock(&vmbus_connection.channel_mutex);
return 0;
}
@@ -95,11 +108,18 @@ hv_uio_irqcontrol(struct uio_info *info, s32 irq_state)
static void hv_uio_channel_cb(void *context)
{
struct vmbus_channel *chan = context;
- struct hv_device *hv_dev = chan->device_obj;
- struct hv_uio_private_data *pdata = hv_get_drvdata(hv_dev);
+ struct hv_device *hv_dev;
+ struct hv_uio_private_data *pdata;
virt_mb();
+ /*
+ * The callback may come from a subchannel, in which case look
+ * for the hv device in the primary channel
+ */
+ hv_dev = chan->primary_channel ?
+ chan->primary_channel->device_obj : chan->device_obj;
+ pdata = hv_get_drvdata(hv_dev);
uio_event_notify(&pdata->info);
}
--
2.43.0
[toc] | [prev] | [next] | [standalone]
| From | Naman Jain <namjain@linux.microsoft.com> |
|---|---|
| Date | 2025-11-14 15:40 +0100 |
| Subject | Bug#1120602: [REGRESSION 6.12.y] hyper-v: BUG: kernel NULL pointer dereference, address: 00000000000000a0: RIP: 0010:hv_uio_channel_cb+0xd/0x20 [uio_hv_generic] |
| Message-ID | <LR37Y-cVxj-27@gated-at.bofh.it> |
| In reply to | #90005 |
On 11/14/2025 5:19 PM, Peter Morrow wrote:
> Hi Naman,
>
> On Fri, 14 Nov 2025 at 06:03, Naman Jain <namjain@linux.microsoft.com> wrote:
>>
>>
>>
>> On 11/13/2025 11:59 PM, Salvatore Bonaccorso wrote:
>>> Peter Morrow reported in Debian a regression, reported in
>>> https://bugs.debian.org/1120602 . The regression was seen after
>>> updating, to 6.12.57-1 in Debian, but details on the offending commit
>>> follows.
>>>
>>> His report was as follows:
>>>
>>>> Dear Maintainer,
>>>>
>>>> I'm seeing a kernel crash quite soon after boot on a debian trixie based
>>>> system running 6.12.57+deb13-amd64, unfortunately the kernel panics before
>>>> I can access the system to gather more information. Thus I'll provide details
>>>> of the system using a previously known good version. The panic is happening
>>>> 100% of the time unfortunately. I have access to the serial console however
>>>> so can enable any required verbose logging during boot if necessary.
>>>>
>>>> Crucially the crash is not seen with kernel version 6.12.41+deb13-amd64 with the
>>>> same userspace. We had pinned to that version until very recently to in order
>>>> to work around https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1109676
>>>>
>>>> I'm running a dpdk application here (VPP) on Azure, VM form factor is a
>>>> "Standard DS3 v2 (4 vcpus, 14 GiB memory)".
>>>>
>>>> The only relevant upstream commit in this area (as far as I can see) is:
>>>>
>>>> https://lore.kernel.org/linux-hyperv/1bb599ee-fe28-409d-b430-2fc086268936@linux.microsoft.com/
>>>>
>>>> The comment regarding avoiding races at start adds a bit more weight behind this
>>>> hunch, though it's only a hunch as I am most definitely nowhere near an expert
>>>> in this area.
>>>>
>>>> -- Package-specific info:
>>>>
>>>> [ 19.625535] BUG: kernel NULL pointer dereference, address: 00000000000000a0
>>>> [ 19.628874] #PF: supervisor read access in kernel mode
>>>> [ 19.630841] #PF: error_code(0x0000) - not-present page
>>>> [ 19.632788] PGD 0 P4D 0
>>>> [ 19.633905] Oops: Oops: 0000 [#1] PREEMPT SMP PTI
>>>> [ 19.635586] CPU: 3 UID: 0 PID: 0 Comm: swapper/3 Not tainted 6.12.57+deb13-amd64 #1 Debian 6.12.57-1
>>>> [ 19.640216] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.1 09/28/2024
>>>> [ 19.644514] RIP: 0010:hv_uio_channel_cb+0xd/0x20 [uio_hv_generic]
>>>> [ 19.646994] Code: 02 00 00 5b 5d e9 53 98 69 e9 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 48 8b 47 10 <48> 8b b8 a0 00 00 00 f0 83 44 24 fc 00 e9 51 6f fa ff 90 90 90 90
>>>> [ 19.654377] RSP: 0018:ffffb15ac01a4fa8 EFLAGS: 00010046
>>>> [ 19.656385] RAX: 0000000000000000 RBX: 0000000000000015 RCX: 0000000000000015
>>>> [ 19.659240] RDX: 0000000000000001 RSI: ffffffffffffffff RDI: ffff8ff69c759400
>>>> [ 19.662168] RBP: ffff8ff548790200 R08: ffff8ff548790200 R09: 00fca75150b080e9
>>>> [ 19.665239] R10: 0000000000000000 R11: ffffb15ac01a4ff8 R12: ffff8ff871dc1480
>>>> [ 19.668193] R13: ffff8ff69c759400 R14: ffff8ff69c7596a0 R15: ffffffffc106e160
>>>> [ 19.671106] FS: 0000000000000000(0000) GS:ffff8ff871d80000(0000) knlGS:0000000000000000
>>>> [ 19.674281] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>> [ 19.676533] CR2: 00000000000000a0 CR3: 0000000100ba6003 CR4: 00000000003706f0
>>>> [ 19.679385] Call Trace:
>>>> [ 19.680361] <IRQ>
>>>> [ 19.681181] vmbus_isr+0x1a5/0x210 [hv_vmbus]
>>>> [ 19.682916] __sysvec_hyperv_callback+0x32/0x60
>>>> [ 19.684991] sysvec_hyperv_callback+0x6c/0x90
>>>> [ 19.686665] </IRQ>
>>>> [ 19.687509] <TASK>
>>>> [ 19.688366] asm_sysvec_hyperv_callback+0x1a/0x20
>>>> [ 19.690262] RIP: 0010:pv_native_safe_halt+0xf/0x20
>>>> [ 19.692067] Code: 09 e9 c5 08 01 00 0f 1f 44 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 66 90 0f 00 2d e5 3b 31 00 fb f4 <c3> cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90
>>>> [ 19.699119] RSP: 0018:ffffb15ac0103ed8 EFLAGS: 00000246
>>>> [ 19.701412] RAX: 0000000000000003 RBX: ffff8ff5403b1fc0 RCX: ffff8ff54c64ce30
>>>> [ 19.704328] RDX: 0000000000000000 RSI: 0000000000000003 RDI: 000000000001f894
>>>> [ 19.706910] RBP: 0000000000000003 R08: 000000000bb760d9 R09: 00fca75150b080e9
>>>> [ 19.709762] R10: 0000000000000003 R11: 0000000000000001 R12: 0000000000000000
>>>> [ 19.712510] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
>>>> [ 19.715173] default_idle+0x9/0x20
>>>> [ 19.716846] default_idle_call+0x29/0x100
>>>> [ 19.718623] do_idle+0x1fe/0x240
>>>> [ 19.720045] cpu_startup_entry+0x29/0x30
>>>> [ 19.721595] start_secondary+0x11e/0x140
>>>> [ 19.723080] common_startup_64+0x13e/0x141
>>>> [ 19.725222] </TASK>
>>>> [ 19.726387] Modules linked in: isofs cdrom uio_hv_generic uio binfmt_misc intel_rapl_msr intel_rapl_common intel_uncore_frequency_common isst_if_mbox_msr isst_if_common rpcrdma skx_edac_common nfit sunrpc libnvdimm crct10dif_pclmul ghash_clmulni_intel sha512_ssse3 sha256_ssse3 rdma_ucm ib_iser sha1_ssse3 rdma_cm aesni_intel iw_cm gf128mul crypto_simd libiscsi cryptd ib_umad ib_ipoib scsi_transport_iscsi ib_cm rapl sg hv_utils hv_balloon evdev pcspkr joydev mpls_router ip_tunnel ramoops configfs pstore_blk efi_pstore pstore_zone nfnetlink vsock_loopback vmw_vsock_virtio_transport_common hv_sock vmw_vsock_vmci_transport vsock vmw_vmci efivarfs ip_tables x_tables autofs4 overlay squashfs dm_verity dm_bufio reed_solomon dm_mod loop ext4 crc16 mbcache jbd2 crc32c_generic mlx5_ib ib_uverbs ib_core mlx5_core mlxfw pci_hyperv pci_hyperv_intf hyperv_drm drm_shmem_helper sd_mod drm_kms_helper hv_storvsc scsi_transport_fc drm scsi_mod hid_generic hid_hyperv hid serio_raw hv_netvsc hyperv_keyboard scsi_common hv_vmbus
>>>> [ 19.726466] crc32_pclmul crc32c_intel
>>>> [ 19.765771] CR2: 00000000000000a0
>>>> [ 19.767524] ---[ end trace 0000000000000000 ]---
>>>> [ 19.800433] RIP: 0010:hv_uio_channel_cb+0xd/0x20 [uio_hv_generic]
>>>> [ 19.803170] Code: 02 00 00 5b 5d e9 53 98 69 e9 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 48 8b 47 10 <48> 8b b8 a0 00 00 00 f0 83 44 24 fc 00 e9 51 6f fa ff 90 90 90 90
>>>> [ 19.811041] RSP: 0018:ffffb15ac01a4fa8 EFLAGS: 00010046
>>>> [ 19.813466] RAX: 0000000000000000 RBX: 0000000000000015 RCX: 0000000000000015
>>>> [ 19.816504] RDX: 0000000000000001 RSI: ffffffffffffffff RDI: ffff8ff69c759400
>>>> [ 19.819484] RBP: ffff8ff548790200 R08: ffff8ff548790200 R09: 00fca75150b080e9
>>>> [ 19.822625] R10: 0000000000000000 R11: ffffb15ac01a4ff8 R12: ffff8ff871dc1480
>>>> [ 19.825569] R13: ffff8ff69c759400 R14: ffff8ff69c7596a0 R15: ffffffffc106e160
>>>> [ 19.828804] FS: 0000000000000000(0000) GS:ffff8ff871d80000(0000) knlGS:0000000000000000
>>>> [ 19.832214] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>> [ 19.834709] CR2: 00000000000000a0 CR3: 0000000100ba6003 CR4: 00000000003706f0
>>>> [ 19.837976] Kernel panic - not syncing: Fatal exception in interrupt
>>>> [ 19.841825] Kernel Offset: 0x28a00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
>>>> [ 19.896620] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---
>>>>
>>
>> <snip>
>>
>>> The offending commit appers to be the backport of b15b7d2a1b09
>>> ("uio_hv_generic: Let userspace take care of interrupt mask") for
>>> 6.12.y.
>>>
>>> Peter confirmed that reverting this commit on top of 6.12.57-1 as
>>> packaged in Debian resolves indeed the issue. Interestingly the issue
>>> is *not* seen with 6.17.7 based kernel in Debian.
>>>
>>> #regzbot introduced: 37bd91f22794dc05436130d6983302cb90ecfe7e
>>> #regzbot monitor: https://bugs.debian.org/1120602
>>>
>>> Thank you already!
>>>
>>> Regards,
>>> Salvatore
>>
>> Hi Peter, Salvatore,
>> Thanks for reporting this crash, and sorry for the trouble. Here is my
>> analysis.
>>
>> On 6.17.7, where commit d062463edf17 ("uio_hv_generic: Set event for all
>> channels on the device") is present, hv_uio_irqcontrol() supports
>> setting of interrupt mask from userspace for sub-channels as well.
>>
>> This aligns with commit e29587c07537 ("uio_hv_generic: Let userspace
>> take care of interrupt mask") which relies on userspace to manage
>> interrupt mask, so it safely removes the interrupt mask management logic
>> in the driver.
>>
>> However, in 6.12.57, the first commit is not present, but the second one
>> is, so there is no way to disable interrupt mask for sub-channels and
>> interrupt_mask stays 0, which means interrupts are not masked. So we may
>> be having an interrupt callback being handled for a sub-channel, where
>> we do not expect it to come. This may be causing this issue.
>>
>> This would have led to a crash in hv_uio_channel_cb() for sub-channels:
>> struct hv_device *hv_dev = chan->device_obj;
>>
>>
>> I have ported commit d062463edf17 ("uio_hv_generic: Set event for all
>> channels on the device") on 6.12.57, and resolved some merge conflicts.
>> Could you please help with testing this, if it works for you.
>
> Applying the patch against the debian 6.12.57 kernel worked, I am no
> longer seeing that panic on boot:
>
> gnos@vEdge:~$ uname -a
> Linux vEdge 6.12+unreleased-amd64 #1 SMP PREEMPT_DYNAMIC Debian
> 6.12.57-1a~test (2025-11-14) x86_64 GNU/Linux
> gnos@vEdge:~$ uptime
> 11:46:33 up 4 min, 1 user, load average: 3.31, 2.07, 0.89
> gnos@vEdge:~$ sudo dmidecode -t system
> # dmidecode 3.6
> Getting SMBIOS data from sysfs.
> SMBIOS 3.1.0 present.
>
> Handle 0x0001, DMI type 1, 27 bytes
> System Information
> Manufacturer: Microsoft Corporation
> Product Name: Virtual Machine
> Version: Hyper-V UEFI Release v4.1
> Serial Number: 0000-0002-8036-1108-7588-3134-50
> UUID: 26e86d6e-140c-496a-862c-a3b3bbcd16ad
> Wake-up Type: Power Switch
> SKU Number: None
> Family: Virtual Machine
>
> Handle 0x0010, DMI type 32, 11 bytes
> System Boot Information
> Status: No errors detected
>
> gnos@vEdge:~$
>
> Thanks a lot for the quick analysis!
>
> Peter.
Hi Peter,
Thanks for confirming. I am discussing this with Long Li, to hear his
thoughts on this, and have kept the patch ready.
Porting the same on 6.6 and older kernels would be a little different
since we don't have commit 547fa4ffd799 ("uio_hv_generic: Enable
interrupt for low speed VMBus devices") on these kernels and this would
lead to merge conflicts, which needs to be handled separately.
Meanwhile, if I should be including any tags in the fix patch for debian
bug, please let me know.
Regards,
Naman
[toc] | [prev] | [next] | [standalone]
| From | Salvatore Bonaccorso <carnil@debian.org> |
|---|---|
| Date | 2025-11-14 22:50 +0100 |
| Subject | Bug#1120602: [REGRESSION 6.12.y] hyper-v: BUG: kernel NULL pointer dereference, address: 00000000000000a0: RIP: 0010:hv_uio_channel_cb+0xd/0x20 [uio_hv_generic] |
| Message-ID | <LR9Q5-d046-1@gated-at.bofh.it> |
| In reply to | #90028 |
Hi,
On Fri, Nov 14, 2025 at 08:05:55PM +0530, Naman Jain wrote:
>
>
> On 11/14/2025 5:19 PM, Peter Morrow wrote:
> > Hi Naman,
> >
> > On Fri, 14 Nov 2025 at 06:03, Naman Jain <namjain@linux.microsoft.com> wrote:
> > >
> > >
> > >
> > > On 11/13/2025 11:59 PM, Salvatore Bonaccorso wrote:
> > > > Peter Morrow reported in Debian a regression, reported in
> > > > https://bugs.debian.org/1120602 . The regression was seen after
> > > > updating, to 6.12.57-1 in Debian, but details on the offending commit
> > > > follows.
> > > >
> > > > His report was as follows:
> > > >
> > > > > Dear Maintainer,
> > > > >
> > > > > I'm seeing a kernel crash quite soon after boot on a debian trixie based
> > > > > system running 6.12.57+deb13-amd64, unfortunately the kernel panics before
> > > > > I can access the system to gather more information. Thus I'll provide details
> > > > > of the system using a previously known good version. The panic is happening
> > > > > 100% of the time unfortunately. I have access to the serial console however
> > > > > so can enable any required verbose logging during boot if necessary.
> > > > >
> > > > > Crucially the crash is not seen with kernel version 6.12.41+deb13-amd64 with the
> > > > > same userspace. We had pinned to that version until very recently to in order
> > > > > to work around https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1109676
> > > > >
> > > > > I'm running a dpdk application here (VPP) on Azure, VM form factor is a
> > > > > "Standard DS3 v2 (4 vcpus, 14 GiB memory)".
> > > > >
> > > > > The only relevant upstream commit in this area (as far as I can see) is:
> > > > >
> > > > > https://lore.kernel.org/linux-hyperv/1bb599ee-fe28-409d-b430-2fc086268936@linux.microsoft.com/
> > > > >
> > > > > The comment regarding avoiding races at start adds a bit more weight behind this
> > > > > hunch, though it's only a hunch as I am most definitely nowhere near an expert
> > > > > in this area.
> > > > >
> > > > > -- Package-specific info:
> > > > >
> > > > > [ 19.625535] BUG: kernel NULL pointer dereference, address: 00000000000000a0
> > > > > [ 19.628874] #PF: supervisor read access in kernel mode
> > > > > [ 19.630841] #PF: error_code(0x0000) - not-present page
> > > > > [ 19.632788] PGD 0 P4D 0
> > > > > [ 19.633905] Oops: Oops: 0000 [#1] PREEMPT SMP PTI
> > > > > [ 19.635586] CPU: 3 UID: 0 PID: 0 Comm: swapper/3 Not tainted 6.12.57+deb13-amd64 #1 Debian 6.12.57-1
> > > > > [ 19.640216] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.1 09/28/2024
> > > > > [ 19.644514] RIP: 0010:hv_uio_channel_cb+0xd/0x20 [uio_hv_generic]
> > > > > [ 19.646994] Code: 02 00 00 5b 5d e9 53 98 69 e9 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 48 8b 47 10 <48> 8b b8 a0 00 00 00 f0 83 44 24 fc 00 e9 51 6f fa ff 90 90 90 90
> > > > > [ 19.654377] RSP: 0018:ffffb15ac01a4fa8 EFLAGS: 00010046
> > > > > [ 19.656385] RAX: 0000000000000000 RBX: 0000000000000015 RCX: 0000000000000015
> > > > > [ 19.659240] RDX: 0000000000000001 RSI: ffffffffffffffff RDI: ffff8ff69c759400
> > > > > [ 19.662168] RBP: ffff8ff548790200 R08: ffff8ff548790200 R09: 00fca75150b080e9
> > > > > [ 19.665239] R10: 0000000000000000 R11: ffffb15ac01a4ff8 R12: ffff8ff871dc1480
> > > > > [ 19.668193] R13: ffff8ff69c759400 R14: ffff8ff69c7596a0 R15: ffffffffc106e160
> > > > > [ 19.671106] FS: 0000000000000000(0000) GS:ffff8ff871d80000(0000) knlGS:0000000000000000
> > > > > [ 19.674281] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > > > [ 19.676533] CR2: 00000000000000a0 CR3: 0000000100ba6003 CR4: 00000000003706f0
> > > > > [ 19.679385] Call Trace:
> > > > > [ 19.680361] <IRQ>
> > > > > [ 19.681181] vmbus_isr+0x1a5/0x210 [hv_vmbus]
> > > > > [ 19.682916] __sysvec_hyperv_callback+0x32/0x60
> > > > > [ 19.684991] sysvec_hyperv_callback+0x6c/0x90
> > > > > [ 19.686665] </IRQ>
> > > > > [ 19.687509] <TASK>
> > > > > [ 19.688366] asm_sysvec_hyperv_callback+0x1a/0x20
> > > > > [ 19.690262] RIP: 0010:pv_native_safe_halt+0xf/0x20
> > > > > [ 19.692067] Code: 09 e9 c5 08 01 00 0f 1f 44 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 66 90 0f 00 2d e5 3b 31 00 fb f4 <c3> cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90
> > > > > [ 19.699119] RSP: 0018:ffffb15ac0103ed8 EFLAGS: 00000246
> > > > > [ 19.701412] RAX: 0000000000000003 RBX: ffff8ff5403b1fc0 RCX: ffff8ff54c64ce30
> > > > > [ 19.704328] RDX: 0000000000000000 RSI: 0000000000000003 RDI: 000000000001f894
> > > > > [ 19.706910] RBP: 0000000000000003 R08: 000000000bb760d9 R09: 00fca75150b080e9
> > > > > [ 19.709762] R10: 0000000000000003 R11: 0000000000000001 R12: 0000000000000000
> > > > > [ 19.712510] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
> > > > > [ 19.715173] default_idle+0x9/0x20
> > > > > [ 19.716846] default_idle_call+0x29/0x100
> > > > > [ 19.718623] do_idle+0x1fe/0x240
> > > > > [ 19.720045] cpu_startup_entry+0x29/0x30
> > > > > [ 19.721595] start_secondary+0x11e/0x140
> > > > > [ 19.723080] common_startup_64+0x13e/0x141
> > > > > [ 19.725222] </TASK>
> > > > > [ 19.726387] Modules linked in: isofs cdrom uio_hv_generic uio binfmt_misc intel_rapl_msr intel_rapl_common intel_uncore_frequency_common isst_if_mbox_msr isst_if_common rpcrdma skx_edac_common nfit sunrpc libnvdimm crct10dif_pclmul ghash_clmulni_intel sha512_ssse3 sha256_ssse3 rdma_ucm ib_iser sha1_ssse3 rdma_cm aesni_intel iw_cm gf128mul crypto_simd libiscsi cryptd ib_umad ib_ipoib scsi_transport_iscsi ib_cm rapl sg hv_utils hv_balloon evdev pcspkr joydev mpls_router ip_tunnel ramoops configfs pstore_blk efi_pstore pstore_zone nfnetlink vsock_loopback vmw_vsock_virtio_transport_common hv_sock vmw_vsock_vmci_transport vsock vmw_vmci efivarfs ip_tables x_tables autofs4 overlay squashfs dm_verity dm_bufio reed_solomon dm_mod loop ext4 crc16 mbcache jbd2 crc32c_generic mlx5_ib ib_uverbs ib_core mlx5_core mlxfw pci_hyperv pci_hyperv_intf hyperv_drm drm_shmem_helper sd_mod drm_kms_helper hv_storvsc scsi_transport_fc drm scsi_mod hid_generic hid_hyperv hid serio_raw hv_netvsc hyperv_keyboard scsi_common hv_vmbus
> > > > > [ 19.726466] crc32_pclmul crc32c_intel
> > > > > [ 19.765771] CR2: 00000000000000a0
> > > > > [ 19.767524] ---[ end trace 0000000000000000 ]---
> > > > > [ 19.800433] RIP: 0010:hv_uio_channel_cb+0xd/0x20 [uio_hv_generic]
> > > > > [ 19.803170] Code: 02 00 00 5b 5d e9 53 98 69 e9 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 48 8b 47 10 <48> 8b b8 a0 00 00 00 f0 83 44 24 fc 00 e9 51 6f fa ff 90 90 90 90
> > > > > [ 19.811041] RSP: 0018:ffffb15ac01a4fa8 EFLAGS: 00010046
> > > > > [ 19.813466] RAX: 0000000000000000 RBX: 0000000000000015 RCX: 0000000000000015
> > > > > [ 19.816504] RDX: 0000000000000001 RSI: ffffffffffffffff RDI: ffff8ff69c759400
> > > > > [ 19.819484] RBP: ffff8ff548790200 R08: ffff8ff548790200 R09: 00fca75150b080e9
> > > > > [ 19.822625] R10: 0000000000000000 R11: ffffb15ac01a4ff8 R12: ffff8ff871dc1480
> > > > > [ 19.825569] R13: ffff8ff69c759400 R14: ffff8ff69c7596a0 R15: ffffffffc106e160
> > > > > [ 19.828804] FS: 0000000000000000(0000) GS:ffff8ff871d80000(0000) knlGS:0000000000000000
> > > > > [ 19.832214] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > > > [ 19.834709] CR2: 00000000000000a0 CR3: 0000000100ba6003 CR4: 00000000003706f0
> > > > > [ 19.837976] Kernel panic - not syncing: Fatal exception in interrupt
> > > > > [ 19.841825] Kernel Offset: 0x28a00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
> > > > > [ 19.896620] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---
> > > > >
> > >
> > > <snip>
> > >
> > > > The offending commit appers to be the backport of b15b7d2a1b09
> > > > ("uio_hv_generic: Let userspace take care of interrupt mask") for
> > > > 6.12.y.
> > > >
> > > > Peter confirmed that reverting this commit on top of 6.12.57-1 as
> > > > packaged in Debian resolves indeed the issue. Interestingly the issue
> > > > is *not* seen with 6.17.7 based kernel in Debian.
> > > >
> > > > #regzbot introduced: 37bd91f22794dc05436130d6983302cb90ecfe7e
> > > > #regzbot monitor: https://bugs.debian.org/1120602
> > > >
> > > > Thank you already!
> > > >
> > > > Regards,
> > > > Salvatore
> > >
> > > Hi Peter, Salvatore,
> > > Thanks for reporting this crash, and sorry for the trouble. Here is my
> > > analysis.
> > >
> > > On 6.17.7, where commit d062463edf17 ("uio_hv_generic: Set event for all
> > > channels on the device") is present, hv_uio_irqcontrol() supports
> > > setting of interrupt mask from userspace for sub-channels as well.
> > >
> > > This aligns with commit e29587c07537 ("uio_hv_generic: Let userspace
> > > take care of interrupt mask") which relies on userspace to manage
> > > interrupt mask, so it safely removes the interrupt mask management logic
> > > in the driver.
> > >
> > > However, in 6.12.57, the first commit is not present, but the second one
> > > is, so there is no way to disable interrupt mask for sub-channels and
> > > interrupt_mask stays 0, which means interrupts are not masked. So we may
> > > be having an interrupt callback being handled for a sub-channel, where
> > > we do not expect it to come. This may be causing this issue.
> > >
> > > This would have led to a crash in hv_uio_channel_cb() for sub-channels:
> > > struct hv_device *hv_dev = chan->device_obj;
> > >
> > >
> > > I have ported commit d062463edf17 ("uio_hv_generic: Set event for all
> > > channels on the device") on 6.12.57, and resolved some merge conflicts.
> > > Could you please help with testing this, if it works for you.
> >
> > Applying the patch against the debian 6.12.57 kernel worked, I am no
> > longer seeing that panic on boot:
> >
> > gnos@vEdge:~$ uname -a
> > Linux vEdge 6.12+unreleased-amd64 #1 SMP PREEMPT_DYNAMIC Debian
> > 6.12.57-1a~test (2025-11-14) x86_64 GNU/Linux
> > gnos@vEdge:~$ uptime
> > 11:46:33 up 4 min, 1 user, load average: 3.31, 2.07, 0.89
> > gnos@vEdge:~$ sudo dmidecode -t system
> > # dmidecode 3.6
> > Getting SMBIOS data from sysfs.
> > SMBIOS 3.1.0 present.
> >
> > Handle 0x0001, DMI type 1, 27 bytes
> > System Information
> > Manufacturer: Microsoft Corporation
> > Product Name: Virtual Machine
> > Version: Hyper-V UEFI Release v4.1
> > Serial Number: 0000-0002-8036-1108-7588-3134-50
> > UUID: 26e86d6e-140c-496a-862c-a3b3bbcd16ad
> > Wake-up Type: Power Switch
> > SKU Number: None
> > Family: Virtual Machine
> >
> > Handle 0x0010, DMI type 32, 11 bytes
> > System Boot Information
> > Status: No errors detected
> >
> > gnos@vEdge:~$
> >
> > Thanks a lot for the quick analysis!
> >
> > Peter.
>
> Hi Peter,
>
> Thanks for confirming. I am discussing this with Long Li, to hear his
> thoughts on this, and have kept the patch ready.
> Porting the same on 6.6 and older kernels would be a little different since
> we don't have commit 547fa4ffd799 ("uio_hv_generic: Enable interrupt for low
> speed VMBus devices") on these kernels and this would lead to merge
> conflicts, which needs to be handled separately.
>
> Meanwhile, if I should be including any tags in the fix patch for debian
> bug, please let me know.
Thank you very much for the quick analysis and fix.
If you can add a Closes: https://bugs.debian.org/1120602 that would
make our tracking for the fixes easier. But not sure if this is
allowed for proposing the backport for a stable series, as it did not
affect the upper releases.
In any case your work is much appreciated!
Regards,
Salvatore
[toc] | [prev] | [next] | [standalone]
| From | Naman Jain <namjain@linux.microsoft.com> |
|---|---|
| Date | 2025-11-15 10:10 +0100 |
| Subject | Bug#1120602: [REGRESSION 6.12.y] hyper-v: BUG: kernel NULL pointer dereference, address: 00000000000000a0: RIP: 0010:hv_uio_channel_cb+0xd/0x20 [uio_hv_generic] |
| Message-ID | <LRks9-d7yY-3@gated-at.bofh.it> |
| In reply to | #90041 |
On 11/15/2025 3:14 AM, Salvatore Bonaccorso wrote:
> Hi,
>
> On Fri, Nov 14, 2025 at 08:05:55PM +0530, Naman Jain wrote:
>>
>>
>> On 11/14/2025 5:19 PM, Peter Morrow wrote:
>>> Hi Naman,
>>>
>>> On Fri, 14 Nov 2025 at 06:03, Naman Jain <namjain@linux.microsoft.com> wrote:
>>>>
>>>>
>>>>
>>>> On 11/13/2025 11:59 PM, Salvatore Bonaccorso wrote:
>>>>> Peter Morrow reported in Debian a regression, reported in
>>>>> https://bugs.debian.org/1120602 . The regression was seen after
>>>>> updating, to 6.12.57-1 in Debian, but details on the offending commit
>>>>> follows.
>>>>>
>>>>> His report was as follows:
>>>>>
>>>>>> Dear Maintainer,
>>>>>>
>>>>>> I'm seeing a kernel crash quite soon after boot on a debian trixie based
>>>>>> system running 6.12.57+deb13-amd64, unfortunately the kernel panics before
>>>>>> I can access the system to gather more information. Thus I'll provide details
>>>>>> of the system using a previously known good version. The panic is happening
>>>>>> 100% of the time unfortunately. I have access to the serial console however
>>>>>> so can enable any required verbose logging during boot if necessary.
>>>>>>
>>>>>> Crucially the crash is not seen with kernel version 6.12.41+deb13-amd64 with the
>>>>>> same userspace. We had pinned to that version until very recently to in order
>>>>>> to work around https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1109676
>>>>>>
>>>>>> I'm running a dpdk application here (VPP) on Azure, VM form factor is a
>>>>>> "Standard DS3 v2 (4 vcpus, 14 GiB memory)".
>>>>>>
>>>>>> The only relevant upstream commit in this area (as far as I can see) is:
>>>>>>
>>>>>> https://lore.kernel.org/linux-hyperv/1bb599ee-fe28-409d-b430-2fc086268936@linux.microsoft.com/
>>>>>>
>>>>>> The comment regarding avoiding races at start adds a bit more weight behind this
>>>>>> hunch, though it's only a hunch as I am most definitely nowhere near an expert
>>>>>> in this area.
>>>>>>
>>>>>> -- Package-specific info:
>>>>>>
>>>>>> [ 19.625535] BUG: kernel NULL pointer dereference, address: 00000000000000a0
>>>>>> [ 19.628874] #PF: supervisor read access in kernel mode
>>>>>> [ 19.630841] #PF: error_code(0x0000) - not-present page
>>>>>> [ 19.632788] PGD 0 P4D 0
>>>>>> [ 19.633905] Oops: Oops: 0000 [#1] PREEMPT SMP PTI
>>>>>> [ 19.635586] CPU: 3 UID: 0 PID: 0 Comm: swapper/3 Not tainted 6.12.57+deb13-amd64 #1 Debian 6.12.57-1
>>>>>> [ 19.640216] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.1 09/28/2024
>>>>>> [ 19.644514] RIP: 0010:hv_uio_channel_cb+0xd/0x20 [uio_hv_generic]
>>>>>> [ 19.646994] Code: 02 00 00 5b 5d e9 53 98 69 e9 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 48 8b 47 10 <48> 8b b8 a0 00 00 00 f0 83 44 24 fc 00 e9 51 6f fa ff 90 90 90 90
>>>>>> [ 19.654377] RSP: 0018:ffffb15ac01a4fa8 EFLAGS: 00010046
>>>>>> [ 19.656385] RAX: 0000000000000000 RBX: 0000000000000015 RCX: 0000000000000015
>>>>>> [ 19.659240] RDX: 0000000000000001 RSI: ffffffffffffffff RDI: ffff8ff69c759400
>>>>>> [ 19.662168] RBP: ffff8ff548790200 R08: ffff8ff548790200 R09: 00fca75150b080e9
>>>>>> [ 19.665239] R10: 0000000000000000 R11: ffffb15ac01a4ff8 R12: ffff8ff871dc1480
>>>>>> [ 19.668193] R13: ffff8ff69c759400 R14: ffff8ff69c7596a0 R15: ffffffffc106e160
>>>>>> [ 19.671106] FS: 0000000000000000(0000) GS:ffff8ff871d80000(0000) knlGS:0000000000000000
>>>>>> [ 19.674281] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>>>> [ 19.676533] CR2: 00000000000000a0 CR3: 0000000100ba6003 CR4: 00000000003706f0
>>>>>> [ 19.679385] Call Trace:
>>>>>> [ 19.680361] <IRQ>
>>>>>> [ 19.681181] vmbus_isr+0x1a5/0x210 [hv_vmbus]
>>>>>> [ 19.682916] __sysvec_hyperv_callback+0x32/0x60
>>>>>> [ 19.684991] sysvec_hyperv_callback+0x6c/0x90
>>>>>> [ 19.686665] </IRQ>
>>>>>> [ 19.687509] <TASK>
>>>>>> [ 19.688366] asm_sysvec_hyperv_callback+0x1a/0x20
>>>>>> [ 19.690262] RIP: 0010:pv_native_safe_halt+0xf/0x20
>>>>>> [ 19.692067] Code: 09 e9 c5 08 01 00 0f 1f 44 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 66 90 0f 00 2d e5 3b 31 00 fb f4 <c3> cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90
>>>>>> [ 19.699119] RSP: 0018:ffffb15ac0103ed8 EFLAGS: 00000246
>>>>>> [ 19.701412] RAX: 0000000000000003 RBX: ffff8ff5403b1fc0 RCX: ffff8ff54c64ce30
>>>>>> [ 19.704328] RDX: 0000000000000000 RSI: 0000000000000003 RDI: 000000000001f894
>>>>>> [ 19.706910] RBP: 0000000000000003 R08: 000000000bb760d9 R09: 00fca75150b080e9
>>>>>> [ 19.709762] R10: 0000000000000003 R11: 0000000000000001 R12: 0000000000000000
>>>>>> [ 19.712510] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
>>>>>> [ 19.715173] default_idle+0x9/0x20
>>>>>> [ 19.716846] default_idle_call+0x29/0x100
>>>>>> [ 19.718623] do_idle+0x1fe/0x240
>>>>>> [ 19.720045] cpu_startup_entry+0x29/0x30
>>>>>> [ 19.721595] start_secondary+0x11e/0x140
>>>>>> [ 19.723080] common_startup_64+0x13e/0x141
>>>>>> [ 19.725222] </TASK>
>>>>>> [ 19.726387] Modules linked in: isofs cdrom uio_hv_generic uio binfmt_misc intel_rapl_msr intel_rapl_common intel_uncore_frequency_common isst_if_mbox_msr isst_if_common rpcrdma skx_edac_common nfit sunrpc libnvdimm crct10dif_pclmul ghash_clmulni_intel sha512_ssse3 sha256_ssse3 rdma_ucm ib_iser sha1_ssse3 rdma_cm aesni_intel iw_cm gf128mul crypto_simd libiscsi cryptd ib_umad ib_ipoib scsi_transport_iscsi ib_cm rapl sg hv_utils hv_balloon evdev pcspkr joydev mpls_router ip_tunnel ramoops configfs pstore_blk efi_pstore pstore_zone nfnetlink vsock_loopback vmw_vsock_virtio_transport_common hv_sock vmw_vsock_vmci_transport vsock vmw_vmci efivarfs ip_tables x_tables autofs4 overlay squashfs dm_verity dm_bufio reed_solomon dm_mod loop ext4 crc16 mbcache jbd2 crc32c_generic mlx5_ib ib_uverbs ib_core mlx5_core mlxfw pci_hyperv pci_hyperv_intf hyperv_drm drm_shmem_helper sd_mod drm_kms_helper hv_storvsc scsi_transport_fc drm scsi_mod hid_generic hid_hyperv hid serio_raw hv_netvsc hyperv_keyboard scsi_common hv_vmbus
>>>>>> [ 19.726466] crc32_pclmul crc32c_intel
>>>>>> [ 19.765771] CR2: 00000000000000a0
>>>>>> [ 19.767524] ---[ end trace 0000000000000000 ]---
>>>>>> [ 19.800433] RIP: 0010:hv_uio_channel_cb+0xd/0x20 [uio_hv_generic]
>>>>>> [ 19.803170] Code: 02 00 00 5b 5d e9 53 98 69 e9 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 48 8b 47 10 <48> 8b b8 a0 00 00 00 f0 83 44 24 fc 00 e9 51 6f fa ff 90 90 90 90
>>>>>> [ 19.811041] RSP: 0018:ffffb15ac01a4fa8 EFLAGS: 00010046
>>>>>> [ 19.813466] RAX: 0000000000000000 RBX: 0000000000000015 RCX: 0000000000000015
>>>>>> [ 19.816504] RDX: 0000000000000001 RSI: ffffffffffffffff RDI: ffff8ff69c759400
>>>>>> [ 19.819484] RBP: ffff8ff548790200 R08: ffff8ff548790200 R09: 00fca75150b080e9
>>>>>> [ 19.822625] R10: 0000000000000000 R11: ffffb15ac01a4ff8 R12: ffff8ff871dc1480
>>>>>> [ 19.825569] R13: ffff8ff69c759400 R14: ffff8ff69c7596a0 R15: ffffffffc106e160
>>>>>> [ 19.828804] FS: 0000000000000000(0000) GS:ffff8ff871d80000(0000) knlGS:0000000000000000
>>>>>> [ 19.832214] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>>>> [ 19.834709] CR2: 00000000000000a0 CR3: 0000000100ba6003 CR4: 00000000003706f0
>>>>>> [ 19.837976] Kernel panic - not syncing: Fatal exception in interrupt
>>>>>> [ 19.841825] Kernel Offset: 0x28a00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
>>>>>> [ 19.896620] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---
>>>>>>
>>>>
>>>> <snip>
>>>>
>>>>> The offending commit appers to be the backport of b15b7d2a1b09
>>>>> ("uio_hv_generic: Let userspace take care of interrupt mask") for
>>>>> 6.12.y.
>>>>>
>>>>> Peter confirmed that reverting this commit on top of 6.12.57-1 as
>>>>> packaged in Debian resolves indeed the issue. Interestingly the issue
>>>>> is *not* seen with 6.17.7 based kernel in Debian.
>>>>>
>>>>> #regzbot introduced: 37bd91f22794dc05436130d6983302cb90ecfe7e
>>>>> #regzbot monitor: https://bugs.debian.org/1120602
>>>>>
>>>>> Thank you already!
>>>>>
>>>>> Regards,
>>>>> Salvatore
>>>>
>>>> Hi Peter, Salvatore,
>>>> Thanks for reporting this crash, and sorry for the trouble. Here is my
>>>> analysis.
>>>>
>>>> On 6.17.7, where commit d062463edf17 ("uio_hv_generic: Set event for all
>>>> channels on the device") is present, hv_uio_irqcontrol() supports
>>>> setting of interrupt mask from userspace for sub-channels as well.
>>>>
>>>> This aligns with commit e29587c07537 ("uio_hv_generic: Let userspace
>>>> take care of interrupt mask") which relies on userspace to manage
>>>> interrupt mask, so it safely removes the interrupt mask management logic
>>>> in the driver.
>>>>
>>>> However, in 6.12.57, the first commit is not present, but the second one
>>>> is, so there is no way to disable interrupt mask for sub-channels and
>>>> interrupt_mask stays 0, which means interrupts are not masked. So we may
>>>> be having an interrupt callback being handled for a sub-channel, where
>>>> we do not expect it to come. This may be causing this issue.
>>>>
>>>> This would have led to a crash in hv_uio_channel_cb() for sub-channels:
>>>> struct hv_device *hv_dev = chan->device_obj;
>>>>
>>>>
>>>> I have ported commit d062463edf17 ("uio_hv_generic: Set event for all
>>>> channels on the device") on 6.12.57, and resolved some merge conflicts.
>>>> Could you please help with testing this, if it works for you.
>>>
>>> Applying the patch against the debian 6.12.57 kernel worked, I am no
>>> longer seeing that panic on boot:
>>>
>>> gnos@vEdge:~$ uname -a
>>> Linux vEdge 6.12+unreleased-amd64 #1 SMP PREEMPT_DYNAMIC Debian
>>> 6.12.57-1a~test (2025-11-14) x86_64 GNU/Linux
>>> gnos@vEdge:~$ uptime
>>> 11:46:33 up 4 min, 1 user, load average: 3.31, 2.07, 0.89
>>> gnos@vEdge:~$ sudo dmidecode -t system
>>> # dmidecode 3.6
>>> Getting SMBIOS data from sysfs.
>>> SMBIOS 3.1.0 present.
>>>
>>> Handle 0x0001, DMI type 1, 27 bytes
>>> System Information
>>> Manufacturer: Microsoft Corporation
>>> Product Name: Virtual Machine
>>> Version: Hyper-V UEFI Release v4.1
>>> Serial Number: 0000-0002-8036-1108-7588-3134-50
>>> UUID: 26e86d6e-140c-496a-862c-a3b3bbcd16ad
>>> Wake-up Type: Power Switch
>>> SKU Number: None
>>> Family: Virtual Machine
>>>
>>> Handle 0x0010, DMI type 32, 11 bytes
>>> System Boot Information
>>> Status: No errors detected
>>>
>>> gnos@vEdge:~$
>>>
>>> Thanks a lot for the quick analysis!
>>>
>>> Peter.
>>
>> Hi Peter,
>>
>> Thanks for confirming. I am discussing this with Long Li, to hear his
>> thoughts on this, and have kept the patch ready.
>> Porting the same on 6.6 and older kernels would be a little different since
>> we don't have commit 547fa4ffd799 ("uio_hv_generic: Enable interrupt for low
>> speed VMBus devices") on these kernels and this would lead to merge
>> conflicts, which needs to be handled separately.
>>
>> Meanwhile, if I should be including any tags in the fix patch for debian
>> bug, please let me know.
>
> Thank you very much for the quick analysis and fix.
>
> If you can add a Closes: https://bugs.debian.org/1120602 that would
> make our tracking for the fixes easier. But not sure if this is
> allowed for proposing the backport for a stable series, as it did not
> affect the upper releases.
>
> In any case your work is much appreciated!
>
> Regards,
> Salvatore
Hi,
I have sent the patches now to the list. Please consider adding your
tested-by if you find it alright.
Thanks.
Regards,
Naman
[toc] | [prev] | [next] | [standalone]
| From | Naman Jain <namjain@linux.microsoft.com> |
|---|---|
| Date | 2025-11-26 07:20 +0100 |
| Subject | Bug#1120602: [REGRESSION 6.12.y] hyper-v: BUG: kernel NULL pointer dereference, address: 00000000000000a0: RIP: 0010:hv_uio_channel_cb+0xd/0x20 [uio_hv_generic] |
| Message-ID | <LVh2F-fOPj-1@gated-at.bofh.it> |
| In reply to | #90005 |
On 11/21/2025 3:34 PM, Peter Morrow wrote: > Hi Naman/Salvatore, > > Is it possible to get this fixed in the 6.1 LTS series too? I just ran > into this crash when moving from bookworm based Debian kernel > 6.1.153-1 to 6.1.158-1. I saw that "uio_hv_generic: Let userspace take > care of interrupt mask" appeared in 6.1.156. > > Thanks, > Peter. > Hi Peter, Yes, I have sent a patch for older kernel versions as well. I am working to fix the review comments and send new revisions. Here is the link: https://lore.kernel.org/all/20251115085937.2237-1-namjain@linux.microsoft.com/ Regards, Naman
[toc] | [prev] | [standalone]
Back to top | Article view | linux.debian.kernel
csiph-web