Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #71220 > unrolled thread

Bug#988044: linux-image-5.10.0-0.bpo.4-amd64: Kernel panic due nfsd error: refcount_t: addition on 0; use-after-free

Started bySalvatore Bonaccorso <carnil@debian.org>
First post2021-05-04 21:00 +0200
Last post2021-12-14 10:00 +0100
Articles 4 — 2 participants

Back to article view | Back to linux.debian.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Bug#988044: linux-image-5.10.0-0.bpo.4-amd64: Kernel panic due nfsd error: refcount_t: addition on 0; use-after-free Salvatore Bonaccorso <carnil@debian.org> - 2021-05-04 21:00 +0200
    Bug#988044: linux-image-5.10.0-0.bpo.4-amd64: Kernel panic due nfsd error: refcount_t: addition on 0; use-after-free Olivier Monaco <olivier@bm-services.com> - 2021-11-22 10:40 +0100
      Bug#988044: linux-image-5.10.0-0.bpo.4-amd64: Kernel panic due nfsd error: refcount_t: addition on 0; use-after-free Salvatore Bonaccorso <carnil@debian.org> - 2021-12-13 21:40 +0100
        Bug#988044: linux-image-5.10.0-0.bpo.4-amd64: Kernel panic due nfsd error: refcount_t: addition on 0; use-after-free Olivier Monaco <olivier@bm-services.com> - 2021-12-14 10:00 +0100

#71220 — Bug#988044: linux-image-5.10.0-0.bpo.4-amd64: Kernel panic due nfsd error: refcount_t: addition on 0; use-after-free

FromSalvatore Bonaccorso <carnil@debian.org>
Date2021-05-04 21:00 +0200
SubjectBug#988044: linux-image-5.10.0-0.bpo.4-amd64: Kernel panic due nfsd error: refcount_t: addition on 0; use-after-free
Message-ID<Cb8qZ-5NL-1@gated-at.bofh.it>
Control: tags -1 + moreinfo

Hi Olivier,

On Tue, May 04, 2021 at 10:01:17AM +0200, Olivier Monaco wrote:
> Package: src:linux
> Version: 5.10.19-1~bpo10+1
> Severity: important
> 
> On a virtual machine running a NFS server the following kernel panic occurs:
> 2021-05-04T02:28:21.051193+02:00 storage-t20 kernel: [1736623.921391] ------------[ cut here ]------------
> 2021-05-04T02:28:21.051214+02:00 storage-t20 kernel: [1736623.921406] refcount_t: addition on 0; use-after-free.
> 2021-05-04T02:28:21.051215+02:00 storage-t20 kernel: [1736623.921416] WARNING: CPU: 0 PID: 675 at lib/refcount.c:25 refcount_warn_saturate+0x6d/0xf0
> 2021-05-04T02:28:21.051216+02:00 storage-t20 kernel: [1736623.921417] Modules linked in: binfmt_misc vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock intel_rapl_msr intel_rapl_common nfit libnvdimm crc32_pclmul ghash_clmulni_intel aesni_intel libaes crypto_simd cryptd glue_helper rapl vm
> w_balloon vmwgfx joydev evdev serio_raw pcspkr ttm sg drm_kms_helper vmw_vmci cec ac button nfsd auth_rpcgss nfs_acl lockd grace drm sunrpc fuse configfs ip_tables x_tables autofs4 btrfs blake2b_generic xor raid6_pq libcrc32c crc32c_generic dm_mod sd_mod t10_pi crc_t10dif crct10dif_generic ata_generic crct10dif_pclmul
>  crct10dif_common crc32c_intel psmouse vmxnet3 ata_piix libata vmw_pvscsi scsi_mod i2c_piix4
> 2021-05-04T02:28:21.051217+02:00 storage-t20 kernel: [1736623.921488] CPU: 0 PID: 675 Comm: nfsd Not tainted 5.10.0-0.bpo.4-amd64 #1 Debian 5.10.19-1~bpo10+1
> 2021-05-04T02:28:21.051218+02:00 storage-t20 kernel: [1736623.921488] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
> 2021-05-04T02:28:21.051219+02:00 storage-t20 kernel: [1736623.921491] RIP: 0010:refcount_warn_saturate+0x6d/0xf0
> 2021-05-04T02:28:21.051219+02:00 storage-t20 kernel: [1736623.921492] Code: 05 d8 be 3f 01 01 e8 c3 0a 40 00 0f 0b c3 80 3d c8 be 3f 01 00 75 ce 48 c7 c7 30 6c 92 86 c6 05 b8 be 3f 01 01 e8 a4 0a 40 00 <0f> 0b c3 80 3d ab be 3f 01 00 75 af 48 c7 c7 08 6c 92 86 c6 05 9b
> 2021-05-04T02:28:21.051220+02:00 storage-t20 kernel: [1736623.921493] RSP: 0018:ffffb93f412b3c28 EFLAGS: 00010282
> 2021-05-04T02:28:21.051234+02:00 storage-t20 kernel: [1736623.921494] RAX: 0000000000000000 RBX: ffff9c2c913a0f80 RCX: 0000000000000027
> 2021-05-04T02:28:21.051236+02:00 storage-t20 kernel: [1736623.921495] RDX: 0000000000000027 RSI: ffff9c2d39e18a00 RDI: ffff9c2d39e18a08
> 2021-05-04T02:28:21.051237+02:00 storage-t20 kernel: [1736623.921495] RBP: ffff9c2c96e4f2a4 R08: 0000000000000000 R09: c0000000ffff7fff
> 2021-05-04T02:28:21.051238+02:00 storage-t20 kernel: [1736623.921496] R10: 0000000000000001 R11: ffffb93f412b3a30 R12: ffff9c2c96e4f2a0
> 2021-05-04T02:28:21.051238+02:00 storage-t20 kernel: [1736623.921496] R13: ffff9c2c375f5450 R14: ffff9c2cb4f9fde8 R15: ffffffff86f75300
> 2021-05-04T02:28:21.051244+02:00 storage-t20 kernel: [1736623.921497] FS:  0000000000000000(0000) GS:ffff9c2d39e00000(0000) knlGS:0000000000000000
> 2021-05-04T02:28:21.051245+02:00 storage-t20 kernel: [1736623.921498] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> 2021-05-04T02:28:21.051245+02:00 storage-t20 kernel: [1736623.921499] CR2: 00007f424807d5b9 CR3: 0000000103084001 CR4: 00000000007706f0
> 2021-05-04T02:28:21.051246+02:00 storage-t20 kernel: [1736623.921522] PKRU: 55555554
> 2021-05-04T02:28:21.051246+02:00 storage-t20 kernel: [1736623.921523] Call Trace:
> 2021-05-04T02:28:21.051246+02:00 storage-t20 kernel: [1736623.921546]  nfsd_break_deleg_cb+0xb5/0xc0 [nfsd]
> 2021-05-04T02:28:21.051247+02:00 storage-t20 kernel: [1736623.921553]  __break_lease+0x148/0x500
> 2021-05-04T02:28:21.051249+02:00 storage-t20 kernel: [1736623.921564]  ? fill_pre_wcc+0x8f/0x180 [nfsd]
> 2021-05-04T02:28:21.051250+02:00 storage-t20 kernel: [1736623.921566]  notify_change+0x196/0x4c0
> 2021-05-04T02:28:21.051250+02:00 storage-t20 kernel: [1736623.921575]  ? nfsd_setattr+0x2e6/0x470 [nfsd]
> 2021-05-04T02:28:21.051250+02:00 storage-t20 kernel: [1736623.921586]  nfsd_setattr+0x2e6/0x470 [nfsd]
> 2021-05-04T02:28:21.051251+02:00 storage-t20 kernel: [1736623.921597]  nfsd4_setattr+0x7b/0x140 [nfsd]
> 2021-05-04T02:28:21.051251+02:00 storage-t20 kernel: [1736623.921611]  nfsd4_proc_compound+0x355/0x680 [nfsd]
> 2021-05-04T02:28:21.051251+02:00 storage-t20 kernel: [1736623.921623]  nfsd_dispatch+0xd4/0x180 [nfsd]
> 2021-05-04T02:28:21.051253+02:00 storage-t20 kernel: [1736623.921661]  svc_process_common+0x390/0x6c0 [sunrpc]
> 2021-05-04T02:28:21.051253+02:00 storage-t20 kernel: [1736623.921680]  ? svc_recv+0x3c4/0x8a0 [sunrpc]
> 2021-05-04T02:28:21.051254+02:00 storage-t20 kernel: [1736623.921688]  ? nfsd_svc+0x300/0x300 [nfsd]
> 2021-05-04T02:28:21.051254+02:00 storage-t20 kernel: [1736623.921695]  ? nfsd_destroy+0x60/0x60 [nfsd]
> 2021-05-04T02:28:21.051255+02:00 storage-t20 kernel: [1736623.921710]  svc_process+0xb7/0xf0 [sunrpc]
> 2021-05-04T02:28:21.051255+02:00 storage-t20 kernel: [1736623.921734]  nfsd+0xe8/0x140 [nfsd]
> 2021-05-04T02:28:21.051257+02:00 storage-t20 kernel: [1736623.921737]  kthread+0x116/0x130
> 2021-05-04T02:28:21.051258+02:00 storage-t20 kernel: [1736623.921738]  ? kthread_park+0x80/0x80
> 2021-05-04T02:28:21.051258+02:00 storage-t20 kernel: [1736623.921741]  ret_from_fork+0x1f/0x30
> 2021-05-04T02:28:21.051259+02:00 storage-t20 kernel: [1736623.921743] ---[ end trace f6e153631af275dc ]---
> 
> It is followed by:
> 2021-05-04T02:28:21.101162+02:00 storage-t20 kernel: [1736623.971161] list_add corruption. prev->next should be next (ffff9c2d0875ecb8), but was ffff9c2c913a0fe8. (prev=ffff9c2c913a0fe8).
> 2021-05-04T02:28:21.101176+02:00 storage-t20 kernel: [1736623.971315] ------------[ cut here ]------------
> 2021-05-04T02:28:21.101177+02:00 storage-t20 kernel: [1736623.971317] kernel BUG at lib/list_debug.c:28!
> 2021-05-04T02:28:21.101178+02:00 storage-t20 kernel: [1736623.971362] invalid opcode: 0000 [#1] SMP NOPTI
> 2021-05-04T02:28:21.101178+02:00 storage-t20 kernel: [1736623.971402] CPU: 1 PID: 2435711 Comm: kworker/u256:5 Tainted: G        W         5.10.0-0.bpo.4-amd64 #1 Debian 5.10.19-1~bpo10+1
> 2021-05-04T02:28:21.101179+02:00 storage-t20 kernel: [1736623.971456] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
> 2021-05-04T02:28:21.101180+02:00 storage-t20 kernel: [1736623.971499] Workqueue: nfsd4_callbacks nfsd4_run_cb_work [nfsd]
> 2021-05-04T02:28:21.101180+02:00 storage-t20 kernel: [1736623.971515] RIP: 0010:__list_add_valid.cold.0+0x26/0x28
> 2021-05-04T02:28:21.101181+02:00 storage-t20 kernel: [1736623.971527] Code: 7b 1c bf ff 48 89 d1 48 c7 c7 18 71 92 86 48 89 c2 e8 02 2a ff ff 0f 0b 48 89 c1 4c 89 c6 48 c7 c7 70 71 92 86 e8 ee 29 ff ff <0f> 0b 48 89 fe 48 89 c2 48 c7 c7 00 72 92 86 e8 da 29 ff ff 0f 0b
> 2021-05-04T02:28:21.101181+02:00 storage-t20 kernel: [1736623.971564] RSP: 0018:ffffb93f4075fe48 EFLAGS: 00010246
> 2021-05-04T02:28:21.101182+02:00 storage-t20 kernel: [1736623.971579] RAX: 0000000000000075 RBX: ffff9c2c913a0fe8 RCX: 0000000000000000
> 2021-05-04T02:28:21.101182+02:00 storage-t20 kernel: [1736623.971594] RDX: 0000000000000000 RSI: ffff9c2d39e58a00 RDI: ffff9c2d39e58a00
> 2021-05-04T02:28:21.101183+02:00 storage-t20 kernel: [1736623.971608] RBP: ffff9c2c913a1018 R08: 0000000000000000 R09: c0000000ffff7fff
> 2021-05-04T02:28:21.101183+02:00 storage-t20 kernel: [1736623.971623] R10: 0000000000000001 R11: ffffb93f4075fc58 R12: ffff9c2d0875ec00
> 2021-05-04T02:28:21.101183+02:00 storage-t20 kernel: [1736623.971637] R13: ffff9c2c913a0fe8 R14: ffff9c2d0875ecb8 R15: ffff9c2c913a1050
> 2021-05-04T02:28:21.101184+02:00 storage-t20 kernel: [1736623.971653] FS:  0000000000000000(0000) GS:ffff9c2d39e40000(0000) knlGS:0000000000000000
> 2021-05-04T02:28:21.101184+02:00 storage-t20 kernel: [1736623.971684] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> 2021-05-04T02:28:21.101184+02:00 storage-t20 kernel: [1736623.971735] CR2: 00007f8d98002698 CR3: 0000000103904005 CR4: 00000000007706e0
> 2021-05-04T02:28:21.101185+02:00 storage-t20 kernel: [1736623.971774] PKRU: 55555554
> 2021-05-04T02:28:21.101185+02:00 storage-t20 kernel: [1736623.971781] Call Trace:
> 2021-05-04T02:28:21.101186+02:00 storage-t20 kernel: [1736623.971805]  nfsd4_cb_recall_prepare+0x2aa/0x2f0 [nfsd]
> 2021-05-04T02:28:21.101186+02:00 storage-t20 kernel: [1736623.971829]  nfsd4_run_cb_work+0xe9/0x150 [nfsd]
> 2021-05-04T02:28:21.101186+02:00 storage-t20 kernel: [1736623.971843]  process_one_work+0x1aa/0x340
> 2021-05-04T02:28:21.101187+02:00 storage-t20 kernel: [1736623.971855]  ? create_worker+0x1a0/0x1a0
> 2021-05-04T02:28:21.101187+02:00 storage-t20 kernel: [1736623.971865]  worker_thread+0x30/0x390
> 2021-05-04T02:28:21.101188+02:00 storage-t20 kernel: [1736623.971875]  ? create_worker+0x1a0/0x1a0
> 2021-05-04T02:28:21.101188+02:00 storage-t20 kernel: [1736623.972279]  kthread+0x116/0x130
> 2021-05-04T02:28:21.101188+02:00 storage-t20 kernel: [1736623.972663]  ? kthread_park+0x80/0x80
> 2021-05-04T02:28:21.101189+02:00 storage-t20 kernel: [1736623.973043]  ret_from_fork+0x1f/0x30
> 2021-05-04T02:28:21.101189+02:00 storage-t20 kernel: [1736623.973411] Modules linked in: binfmt_misc vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock intel_rapl_msr intel_rapl_common nfit libnvdimm crc32_pclmul ghash_clmulni_intel aesni_intel libaes crypto_simd cryptd glue_helper rapl vmw_balloon vmwgfx joydev evdev serio_raw pcspkr ttm sg drm_kms_helper vmw_vmci cec ac button nfsd auth_rpcgss nfs_acl lockd grace drm sunrpc fuse configfs ip_tables x_tables autofs4 btrfs blake2b_generic xor raid6_pq libcrc32c crc32c_generic dm_mod sd_mod t10_pi crc_t10dif crct10dif_generic ata_generic crct10dif_pclmul crct10dif_common crc32c_intel psmouse vmxnet3 ata_piix libata vmw_pvscsi scsi_mod i2c_piix4
> 2021-05-04T02:28:21.101190+02:00 storage-t20 kernel: [1736623.976175] ---[ end trace f6e153631af275dd ]---
> 
> We are running a VMware vSphere platform running 9 groups of virtual machines. Each group include a VM with NFS for file sharing and 3 VM with NFS clients, so we are running 9 independent file servers. This issue occured on 2 different file servers with the same kernel version and the same error. There is no direct link between the two servers except the fact they are running the same software, on the same hadware for the same pupose.
> 
> It also occured earlier 4 times on 3 different servers which was running kernel 5.10.13-1~bpo10+1 (package linux-image-5.10.0-0.bpo.3-amd64).

From the above report I suspect you have not a easy way to trigger the
issue right? Did you see the issue as well with the most current
version in buster-backports, 5.10.24-1~bpo10+1.

Ideally I think this issue should be just be forwarded upstream, but
keep us in the loop accordingly, could you do that?

I did not not found immediately something similarly on
https://lore.kernel.org/linux-nfs/ (a recent one about doing
inter-server copy, but that is/looks different here).

So this would be, mailing

"J. Bruce Fields" <bfields@fieldses.org> (supporter:KERNEL NFSD, SUNRPC, AND LOCKD SERVERS)
Chuck Lever <chuck.lever@oracle.com> (supporter:KERNEL NFSD, SUNRPC, AND LOCKD SERVERS)
linux-nfs@vger.kernel.org (open list:KERNEL NFSD, SUNRPC, AND LOCKD SERVERS)
linux-kernel@vger.kernel.org (open list)

Regards,
Salvatore

[toc] | [next] | [standalone]

#73758

FromOlivier Monaco <olivier@bm-services.com>
Date2021-11-22 10:40 +0100
Message-ID<DmdXQ-5UA-15@gated-at.bofh.it>
In reply to#71220
Hi Salvatore,

Sorry for my very very late answer.

I tried many actions to reproduce and/or fix this problem. We upgraded our kernel to the lastest backport version for Buster but the issue was still there. We upgraded to Bullseye at the end of October but today this issue occurred again. 

I think it related to https://lore.kernel.org/linux-nfs/YV3vDQOPVgxc%2FJ99@eldamar.lan/

I just send an email to this thread. 

Something that may be important: our groups of servers are used to host websites, some with only one website, some with many. For now, this issue only occurred on NFS servers part of a group with many websites. But one of the most loaded NFS server is also used only for one website and has never had this issue. May be this issue occurs only when the NFS server is serving many different files.

Regards,

Olivier

-----Message d'origine-----
De : Salvatore Bonaccorso <salvatore.bonaccorso@gmail.com> De la part de Salvatore Bonaccorso
Envoyé : mardi 4 mai 2021 20:47
À : Olivier Monaco <olivier@bm-services.com>; 988044@bugs.debian.org
Objet : Re: Bug#988044: linux-image-5.10.0-0.bpo.4-amd64: Kernel panic due nfsd error: refcount_t: addition on 0; use-after-free

Control: tags -1 + moreinfo

Hi Olivier,

On Tue, May 04, 2021 at 10:01:17AM +0200, Olivier Monaco wrote:
> Package: src:linux
> Version: 5.10.19-1~bpo10+1
> Severity: important
> 
> On a virtual machine running a NFS server the following kernel panic occurs:
> 2021-05-04T02:28:21.051193+02:00 storage-t20 kernel: [1736623.921391] 
> ------------[ cut here ]------------
> 2021-05-04T02:28:21.051214+02:00 storage-t20 kernel: [1736623.921406] refcount_t: addition on 0; use-after-free.
> 2021-05-04T02:28:21.051215+02:00 storage-t20 kernel: [1736623.921416] 
> WARNING: CPU: 0 PID: 675 at lib/refcount.c:25 
> refcount_warn_saturate+0x6d/0xf0
> 2021-05-04T02:28:21.051216+02:00 storage-t20 kernel: [1736623.921417] 
> Modules linked in: binfmt_misc vsock_loopback 
> vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock 
> intel_rapl_msr intel_rapl_common nfit libnvdimm crc32_pclmul 
> ghash_clmulni_intel aesni_intel libaes crypto_simd cryptd glue_helper 
> rapl vm w_balloon vmwgfx joydev evdev serio_raw pcspkr ttm sg 
> drm_kms_helper vmw_vmci cec ac button nfsd auth_rpcgss nfs_acl lockd 
> grace drm sunrpc fuse configfs ip_tables x_tables autofs4 btrfs 
> blake2b_generic xor raid6_pq libcrc32c crc32c_generic dm_mod sd_mod 
> t10_pi crc_t10dif crct10dif_generic ata_generic crct10dif_pclmul  
> crct10dif_common crc32c_intel psmouse vmxnet3 ata_piix libata 
> vmw_pvscsi scsi_mod i2c_piix4
> 2021-05-04T02:28:21.051217+02:00 storage-t20 kernel: [1736623.921488] 
> CPU: 0 PID: 675 Comm: nfsd Not tainted 5.10.0-0.bpo.4-amd64 #1 Debian 
> 5.10.19-1~bpo10+1
> 2021-05-04T02:28:21.051218+02:00 storage-t20 kernel: [1736623.921488] 
> Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop 
> Reference Platform, BIOS 6.00 12/12/2018
> 2021-05-04T02:28:21.051219+02:00 storage-t20 kernel: [1736623.921491] 
> RIP: 0010:refcount_warn_saturate+0x6d/0xf0
> 2021-05-04T02:28:21.051219+02:00 storage-t20 kernel: [1736623.921492] 
> Code: 05 d8 be 3f 01 01 e8 c3 0a 40 00 0f 0b c3 80 3d c8 be 3f 01 00 
> 75 ce 48 c7 c7 30 6c 92 86 c6 05 b8 be 3f 01 01 e8 a4 0a 40 00 <0f> 0b 
> c3 80 3d ab be 3f 01 00 75 af 48 c7 c7 08 6c 92 86 c6 05 9b
> 2021-05-04T02:28:21.051220+02:00 storage-t20 kernel: [1736623.921493] 
> RSP: 0018:ffffb93f412b3c28 EFLAGS: 00010282
> 2021-05-04T02:28:21.051234+02:00 storage-t20 kernel: [1736623.921494] 
> RAX: 0000000000000000 RBX: ffff9c2c913a0f80 RCX: 0000000000000027
> 2021-05-04T02:28:21.051236+02:00 storage-t20 kernel: [1736623.921495] 
> RDX: 0000000000000027 RSI: ffff9c2d39e18a00 RDI: ffff9c2d39e18a08
> 2021-05-04T02:28:21.051237+02:00 storage-t20 kernel: [1736623.921495] 
> RBP: ffff9c2c96e4f2a4 R08: 0000000000000000 R09: c0000000ffff7fff
> 2021-05-04T02:28:21.051238+02:00 storage-t20 kernel: [1736623.921496] 
> R10: 0000000000000001 R11: ffffb93f412b3a30 R12: ffff9c2c96e4f2a0
> 2021-05-04T02:28:21.051238+02:00 storage-t20 kernel: [1736623.921496] 
> R13: ffff9c2c375f5450 R14: ffff9c2cb4f9fde8 R15: ffffffff86f75300
> 2021-05-04T02:28:21.051244+02:00 storage-t20 kernel: [1736623.921497] 
> FS:  0000000000000000(0000) GS:ffff9c2d39e00000(0000) 
> knlGS:0000000000000000
> 2021-05-04T02:28:21.051245+02:00 storage-t20 kernel: [1736623.921498] 
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> 2021-05-04T02:28:21.051245+02:00 storage-t20 kernel: [1736623.921499] 
> CR2: 00007f424807d5b9 CR3: 0000000103084001 CR4: 00000000007706f0
> 2021-05-04T02:28:21.051246+02:00 storage-t20 kernel: [1736623.921522] 
> PKRU: 55555554
> 2021-05-04T02:28:21.051246+02:00 storage-t20 kernel: [1736623.921523] Call Trace:
> 2021-05-04T02:28:21.051246+02:00 storage-t20 kernel: [1736623.921546]  
> nfsd_break_deleg_cb+0xb5/0xc0 [nfsd]
> 2021-05-04T02:28:21.051247+02:00 storage-t20 kernel: [1736623.921553]  
> __break_lease+0x148/0x500
> 2021-05-04T02:28:21.051249+02:00 storage-t20 kernel: [1736623.921564]  
> ? fill_pre_wcc+0x8f/0x180 [nfsd]
> 2021-05-04T02:28:21.051250+02:00 storage-t20 kernel: [1736623.921566]  
> notify_change+0x196/0x4c0
> 2021-05-04T02:28:21.051250+02:00 storage-t20 kernel: [1736623.921575]  
> ? nfsd_setattr+0x2e6/0x470 [nfsd]
> 2021-05-04T02:28:21.051250+02:00 storage-t20 kernel: [1736623.921586]  
> nfsd_setattr+0x2e6/0x470 [nfsd]
> 2021-05-04T02:28:21.051251+02:00 storage-t20 kernel: [1736623.921597]  
> nfsd4_setattr+0x7b/0x140 [nfsd]
> 2021-05-04T02:28:21.051251+02:00 storage-t20 kernel: [1736623.921611]  
> nfsd4_proc_compound+0x355/0x680 [nfsd]
> 2021-05-04T02:28:21.051251+02:00 storage-t20 kernel: [1736623.921623]  
> nfsd_dispatch+0xd4/0x180 [nfsd]
> 2021-05-04T02:28:21.051253+02:00 storage-t20 kernel: [1736623.921661]  
> svc_process_common+0x390/0x6c0 [sunrpc]
> 2021-05-04T02:28:21.051253+02:00 storage-t20 kernel: [1736623.921680]  
> ? svc_recv+0x3c4/0x8a0 [sunrpc]
> 2021-05-04T02:28:21.051254+02:00 storage-t20 kernel: [1736623.921688]  
> ? nfsd_svc+0x300/0x300 [nfsd]
> 2021-05-04T02:28:21.051254+02:00 storage-t20 kernel: [1736623.921695]  
> ? nfsd_destroy+0x60/0x60 [nfsd]
> 2021-05-04T02:28:21.051255+02:00 storage-t20 kernel: [1736623.921710]  
> svc_process+0xb7/0xf0 [sunrpc]
> 2021-05-04T02:28:21.051255+02:00 storage-t20 kernel: [1736623.921734]  
> nfsd+0xe8/0x140 [nfsd]
> 2021-05-04T02:28:21.051257+02:00 storage-t20 kernel: [1736623.921737]  
> kthread+0x116/0x130
> 2021-05-04T02:28:21.051258+02:00 storage-t20 kernel: [1736623.921738]  
> ? kthread_park+0x80/0x80
> 2021-05-04T02:28:21.051258+02:00 storage-t20 kernel: [1736623.921741]  
> ret_from_fork+0x1f/0x30
> 2021-05-04T02:28:21.051259+02:00 storage-t20 kernel: [1736623.921743] 
> ---[ end trace f6e153631af275dc ]---
> 
> It is followed by:
> 2021-05-04T02:28:21.101162+02:00 storage-t20 kernel: [1736623.971161] list_add corruption. prev->next should be next (ffff9c2d0875ecb8), but was ffff9c2c913a0fe8. (prev=ffff9c2c913a0fe8).
> 2021-05-04T02:28:21.101176+02:00 storage-t20 kernel: [1736623.971315] 
> ------------[ cut here ]------------
> 2021-05-04T02:28:21.101177+02:00 storage-t20 kernel: [1736623.971317] kernel BUG at lib/list_debug.c:28!
> 2021-05-04T02:28:21.101178+02:00 storage-t20 kernel: [1736623.971362] invalid opcode: 0000 [#1] SMP NOPTI
> 2021-05-04T02:28:21.101178+02:00 storage-t20 kernel: [1736623.971402] CPU: 1 PID: 2435711 Comm: kworker/u256:5 Tainted: G        W         5.10.0-0.bpo.4-amd64 #1 Debian 5.10.19-1~bpo10+1
> 2021-05-04T02:28:21.101179+02:00 storage-t20 kernel: [1736623.971456] 
> Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop 
> Reference Platform, BIOS 6.00 12/12/2018
> 2021-05-04T02:28:21.101180+02:00 storage-t20 kernel: [1736623.971499] 
> Workqueue: nfsd4_callbacks nfsd4_run_cb_work [nfsd]
> 2021-05-04T02:28:21.101180+02:00 storage-t20 kernel: [1736623.971515] 
> RIP: 0010:__list_add_valid.cold.0+0x26/0x28
> 2021-05-04T02:28:21.101181+02:00 storage-t20 kernel: [1736623.971527] 
> Code: 7b 1c bf ff 48 89 d1 48 c7 c7 18 71 92 86 48 89 c2 e8 02 2a ff 
> ff 0f 0b 48 89 c1 4c 89 c6 48 c7 c7 70 71 92 86 e8 ee 29 ff ff <0f> 0b 
> 48 89 fe 48 89 c2 48 c7 c7 00 72 92 86 e8 da 29 ff ff 0f 0b
> 2021-05-04T02:28:21.101181+02:00 storage-t20 kernel: [1736623.971564] 
> RSP: 0018:ffffb93f4075fe48 EFLAGS: 00010246
> 2021-05-04T02:28:21.101182+02:00 storage-t20 kernel: [1736623.971579] 
> RAX: 0000000000000075 RBX: ffff9c2c913a0fe8 RCX: 0000000000000000
> 2021-05-04T02:28:21.101182+02:00 storage-t20 kernel: [1736623.971594] 
> RDX: 0000000000000000 RSI: ffff9c2d39e58a00 RDI: ffff9c2d39e58a00
> 2021-05-04T02:28:21.101183+02:00 storage-t20 kernel: [1736623.971608] 
> RBP: ffff9c2c913a1018 R08: 0000000000000000 R09: c0000000ffff7fff
> 2021-05-04T02:28:21.101183+02:00 storage-t20 kernel: [1736623.971623] 
> R10: 0000000000000001 R11: ffffb93f4075fc58 R12: ffff9c2d0875ec00
> 2021-05-04T02:28:21.101183+02:00 storage-t20 kernel: [1736623.971637] 
> R13: ffff9c2c913a0fe8 R14: ffff9c2d0875ecb8 R15: ffff9c2c913a1050
> 2021-05-04T02:28:21.101184+02:00 storage-t20 kernel: [1736623.971653] 
> FS:  0000000000000000(0000) GS:ffff9c2d39e40000(0000) 
> knlGS:0000000000000000
> 2021-05-04T02:28:21.101184+02:00 storage-t20 kernel: [1736623.971684] 
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> 2021-05-04T02:28:21.101184+02:00 storage-t20 kernel: [1736623.971735] 
> CR2: 00007f8d98002698 CR3: 0000000103904005 CR4: 00000000007706e0
> 2021-05-04T02:28:21.101185+02:00 storage-t20 kernel: [1736623.971774] 
> PKRU: 55555554
> 2021-05-04T02:28:21.101185+02:00 storage-t20 kernel: [1736623.971781] Call Trace:
> 2021-05-04T02:28:21.101186+02:00 storage-t20 kernel: [1736623.971805]  
> nfsd4_cb_recall_prepare+0x2aa/0x2f0 [nfsd]
> 2021-05-04T02:28:21.101186+02:00 storage-t20 kernel: [1736623.971829]  
> nfsd4_run_cb_work+0xe9/0x150 [nfsd]
> 2021-05-04T02:28:21.101186+02:00 storage-t20 kernel: [1736623.971843]  
> process_one_work+0x1aa/0x340
> 2021-05-04T02:28:21.101187+02:00 storage-t20 kernel: [1736623.971855]  
> ? create_worker+0x1a0/0x1a0
> 2021-05-04T02:28:21.101187+02:00 storage-t20 kernel: [1736623.971865]  
> worker_thread+0x30/0x390
> 2021-05-04T02:28:21.101188+02:00 storage-t20 kernel: [1736623.971875]  
> ? create_worker+0x1a0/0x1a0
> 2021-05-04T02:28:21.101188+02:00 storage-t20 kernel: [1736623.972279]  
> kthread+0x116/0x130
> 2021-05-04T02:28:21.101188+02:00 storage-t20 kernel: [1736623.972663]  
> ? kthread_park+0x80/0x80
> 2021-05-04T02:28:21.101189+02:00 storage-t20 kernel: [1736623.973043]  
> ret_from_fork+0x1f/0x30
> 2021-05-04T02:28:21.101189+02:00 storage-t20 kernel: [1736623.973411] 
> Modules linked in: binfmt_misc vsock_loopback 
> vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock 
> intel_rapl_msr intel_rapl_common nfit libnvdimm crc32_pclmul 
> ghash_clmulni_intel aesni_intel libaes crypto_simd cryptd glue_helper 
> rapl vmw_balloon vmwgfx joydev evdev serio_raw pcspkr ttm sg 
> drm_kms_helper vmw_vmci cec ac button nfsd auth_rpcgss nfs_acl lockd 
> grace drm sunrpc fuse configfs ip_tables x_tables autofs4 btrfs 
> blake2b_generic xor raid6_pq libcrc32c crc32c_generic dm_mod sd_mod 
> t10_pi crc_t10dif crct10dif_generic ata_generic crct10dif_pclmul 
> crct10dif_common crc32c_intel psmouse vmxnet3 ata_piix libata 
> vmw_pvscsi scsi_mod i2c_piix4
> 2021-05-04T02:28:21.101190+02:00 storage-t20 kernel: [1736623.976175] 
> ---[ end trace f6e153631af275dd ]---
> 
> We are running a VMware vSphere platform running 9 groups of virtual machines. Each group include a VM with NFS for file sharing and 3 VM with NFS clients, so we are running 9 independent file servers. This issue occured on 2 different file servers with the same kernel version and the same error. There is no direct link between the two servers except the fact they are running the same software, on the same hadware for the same pupose.
> 
> It also occured earlier 4 times on 3 different servers which was running kernel 5.10.13-1~bpo10+1 (package linux-image-5.10.0-0.bpo.3-amd64).

From the above report I suspect you have not a easy way to trigger the issue right? Did you see the issue as well with the most current version in buster-backports, 5.10.24-1~bpo10+1.

Ideally I think this issue should be just be forwarded upstream, but keep us in the loop accordingly, could you do that?

I did not not found immediately something similarly on https://lore.kernel.org/linux-nfs/ (a recent one about doing inter-server copy, but that is/looks different here).

So this would be, mailing

"J. Bruce Fields" <bfields@fieldses.org> (supporter:KERNEL NFSD, SUNRPC, AND LOCKD SERVERS) Chuck Lever <chuck.lever@oracle.com> (supporter:KERNEL NFSD, SUNRPC, AND LOCKD SERVERS) linux-nfs@vger.kernel.org (open list:KERNEL NFSD, SUNRPC, AND LOCKD SERVERS) linux-kernel@vger.kernel.org (open list)

Regards,
Salvatore

[toc] | [prev] | [next] | [standalone]

#73938

FromSalvatore Bonaccorso <carnil@debian.org>
Date2021-12-13 21:40 +0100
Message-ID<Du0h3-1sv-1@gated-at.bofh.it>
In reply to#73758
Hi Olivier,

On Mon, Nov 22, 2021 at 09:27:26AM +0000, Olivier Monaco wrote:
> Hi Salvatore,
> 
> Sorry for my very very late answer.
> 
> I tried many actions to reproduce and/or fix this problem. We
> upgraded our kernel to the lastest backport version for Buster but
> the issue was still there. We upgraded to Bullseye at the end of
> October but today this issue occurred again. 
> 
> I think it related to https://lore.kernel.org/linux-nfs/YV3vDQOPVgxc%2FJ99@eldamar.lan/
> 
> I just send an email to this thread. 
> 
> Something that may be important: our groups of servers are used to
> host websites, some with only one website, some with many. For now,
> this issue only occurred on NFS servers part of a group with many
> websites. But one of the most loaded NFS server is also used only
> for one website and has never had this issue. May be this issue
> occurs only when the NFS server is serving many different files.

I was never able to reliably reproduce the issue myself, but upstream
now thinks to have isolated the issue and should be fixed with:

https://git.kernel.org/linus/548ec0805c399c65ed66c6641be467f717833ab5

Fixes are as well queued to stable series.

Regards,
Salvatore

[toc] | [prev] | [next] | [standalone]

#73941

FromOlivier Monaco <olivier@bm-services.com>
Date2021-12-14 10:00 +0100
Message-ID<DubPc-8oI-5@gated-at.bofh.it>
In reply to#73938
Hi Salvatore,

I try also to reproduce the issue using multiple ways without success. I hope the modification will fix this issue.

Thank you,

Regards,
Olivier

-----Message d'origine-----
De : Salvatore Bonaccorso <salvatore.bonaccorso@gmail.com> De la part de Salvatore Bonaccorso
Envoyé : lundi 13 décembre 2021 21:34
À : Olivier Monaco <olivier@bm-services.com>; 988044@bugs.debian.org
Objet : Re: Bug#988044: linux-image-5.10.0-0.bpo.4-amd64: Kernel panic due nfsd error: refcount_t: addition on 0; use-after-free

Hi Olivier,

On Mon, Nov 22, 2021 at 09:27:26AM +0000, Olivier Monaco wrote:
> Hi Salvatore,
> 
> Sorry for my very very late answer.
> 
> I tried many actions to reproduce and/or fix this problem. We upgraded 
> our kernel to the lastest backport version for Buster but the issue 
> was still there. We upgraded to Bullseye at the end of October but 
> today this issue occurred again.
> 
> I think it related to 
> https://lore.kernel.org/linux-nfs/YV3vDQOPVgxc%2FJ99@eldamar.lan/
> 
> I just send an email to this thread. 
> 
> Something that may be important: our groups of servers are used to 
> host websites, some with only one website, some with many. For now, 
> this issue only occurred on NFS servers part of a group with many 
> websites. But one of the most loaded NFS server is also used only for 
> one website and has never had this issue. May be this issue occurs 
> only when the NFS server is serving many different files.

I was never able to reliably reproduce the issue myself, but upstream now thinks to have isolated the issue and should be fixed with:

https://git.kernel.org/linus/548ec0805c399c65ed66c6641be467f717833ab5

Fixes are as well queued to stable series.

Regards,
Salvatore

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.kernel

csiph-web