Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #67673 > unrolled thread

Bug#966846: Kernel panic (4.19.0-10): RIP __cgroup_bpf_run_filter_skb

Back to article view | Back to linux.debian.kernel

Contents

  Bug#966846: Kernel panic (4.19.0-10): RIP __cgroup_bpf_run_filter_skb Cédric Dufour <cedric.dufour@ced-network.net> - 2020-08-03 10:10 +0200
    Bug#966846: Kernel panic (4.19.0-10): RIP __cgroup_bpf_run_filter_skb Shane Frasier <jeremy.frasier@trio.dhs.gov> - 2020-08-10 22:00 +0200
    Bug#966846: Kernel panic (4.19.0-10): RIP __cgroup_bpf_run_filter_skb Kevin Tran <ktomega@gmail.com> - 2020-08-20 00:40 +0200
    Bug#966846: Kernel panic (4.19.0-10): RIP __cgroup_bpf_run_filter_skb Salvatore Bonaccorso <carnil@debian.org> - 2020-08-30 17:20 +0200
      Bug#966846: Kernel panic (4.19.0-10): RIP __cgroup_bpf_run_filter_skb Sébastien NOBILI <s-liste-debian-bugs@pipoprods.org> - 2020-09-03 14:10 +0200
        Bug#966846: Kernel panic (4.19.0-10): RIP __cgroup_bpf_run_filter_skb Sébastien NOBILI <s-liste-debian-bugs@pipoprods.org> - 2020-09-05 19:20 +0200
          Bug#966846: Kernel panic (4.19.0-10): RIP __cgroup_bpf_run_filter_skb Sébastien NOBILI <s-liste-debian-bugs@pipoprods.org> - 2020-09-11 11:50 +0200
            Bug#966846: Kernel panic (4.19.0-10): RIP __cgroup_bpf_run_filter_skb Salvatore Bonaccorso <carnil@debian.org> - 2020-09-11 14:00 +0200
    Bug#966846: Kernel panic (4.19.0-10): RIP __cgroup_bpf_run_filter_skb Michel Le Bihan <michel@lebihan.pl> - 2020-09-23 00:10 +0200
      Bug#966846: Kernel panic (4.19.0-10): RIP __cgroup_bpf_run_filter_skb Ben Hutchings <ben@decadent.org.uk> - 2020-09-23 18:20 +0200

#67673 — Bug#966846: Kernel panic (4.19.0-10): RIP __cgroup_bpf_run_filter_skb

Package: linux-source-4.19
Version: 4.19.132-1
Severity: important

Hello,

Since linux-image-4.19.0-10-amd64, I'm facing regular Kernel panics - "RIP: 0010:__cgroup_bpf_run_filter_skb+0x26d/0x3d0" - resulting in full (file) *server freeze*.

The issue is pretty well described and summarized in https://forum.proxmox.com/threads/kernel-5-4-44-causes-system-freeze-on-hp-microserver-gen8.72050/page-2#post-323498

The "culprit" commit - "netprio_cgroup: Fix unlimited memory leak of v2 cgroups" - is indeed included in Debian kernel (4.19) since changelog entry 4.19.131-1

It *seems* there is already a patch proposed upstream (although here for kernel 4.9): https://lkml.org/lkml/2020/7/20/883

Best regards,

Cédric

-- 
Cédric Dufour

[toc] | [next] | [standalone]

#67741

[Multipart message — attachments visible in raw view] — view raw

On Mon, 3 Aug 2020 09:55:28 +0200 =?UTF-8?Q?C=c3=a9dric_Dufour?= <
cedric.dufour@ced-network.net> wrote:
> Package: linux-source-4.19
> Version: 4.19.132-1
> Severity: important
>
> Hello,
>
> Since linux-image-4.19.0-10-amd64, I'm facing regular Kernel panics -
"RIP: 0010:__cgroup_bpf_run_filter_skb+0x26d/0x3d0" - resulting in full
(file) *server freeze*.
>
> The issue is pretty well described and summarized in
https://forum.proxmox.com/threads/kernel-5-4-44-causes-system-freeze-on-hp-microserver-gen8.72050/page-2#post-323498
>
> The "culprit" commit - "netprio_cgroup: Fix unlimited memory leak of v2
cgroups" - is indeed included in Debian kernel (4.19) since changelog entry
4.19.131-1
>
> It *seems* there is already a patch proposed upstream (although here for
kernel 4.9): https://lkml.org/lkml/2020/7/20/883
>
> Best regards,
>
> Cédric
>
> --
> Cédric Dufour
>
>

FWIW, I am seeing a very similar issue.  Some Debian 10 AWS instances used
to run Guacamole via Docker recently started randomly freezing up on me.  I
enabled kernel dumps and finally caught one of the machines misbehaving.
Looking at the kdump I see this:
      KERNEL: /usr/lib/debug/vmlinux-4.19.0-10-cloud-amd64
    DUMPFILE: dump.202008101612  [PARTIAL DUMP]
        CPUS: 2
        DATE: Mon Aug 10 16:11:47 2020
      UPTIME: 00:05:44
LOAD AVERAGE: 0.21, 0.11, 0.04
       TASKS: 261
    NODENAME: guac.env0.staging.cool.cyber.dhs.gov
     RELEASE: 4.19.0-10-cloud-amd64
     VERSION: #1 SMP Debian 4.19.132-1 (2020-07-24)
     MACHINE: x86_64  (2499 Mhz)
      MEMORY: 4 GB
       PANIC: "BUG: unable to handle kernel NULL pointer dereference at
0000000000000010"
         PID: 1453
     COMMAND: "sshd"
        TASK: ffff8a3f695115c0  [THREAD_INFO: ffff8a3f695115c0]
         CPU: 0
       STATE: TASK_RUNNING (PANIC)

crash> bt
PID: 1453   TASK: ffff8a3f695115c0  CPU: 0   COMMAND: "sshd"
 #0 [ffffb37740c77800] machine_kexec at ffffffff97a4b297
 #1 [ffffb37740c77858] __crash_kexec at ffffffff97b0e7dd
 #2 [ffffb37740c77920] crash_kexec at ffffffff97b0f62d
 #3 [ffffb37740c77938] oops_end at ffffffff97a2907d
 #4 [ffffb37740c77958] no_context at ffffffff97a5858e
 #5 [ffffb37740c779b0] __do_page_fault at ffffffff97a58c42
 #6 [ffffb37740c77a20] async_page_fault at ffffffff982010be
    [exception RIP: __cgroup_bpf_run_filter_skb+189]
    RIP: ffffffff97b94ffd  RSP: ffffb37740c77ad0  RFLAGS: 00010286
    RAX: 0000000000000000  RBX: ffff8a3ff55e5ee8  RCX: 0000000000000000
    RDX: 0000000000000001  RSI: ffff8a3ff3d49800  RDI: ffff8a3ff52fd500
    RBP: ffff8a3ff52fd500   R8: ffff8a3ff55e5ee8   R9: 0000000000010000
    R10: 0000000000000001  R11: ffff8a3ef6dd7500  R12: 0000000000000000
    R13: 0000000000000000  R14: ffff8a3ff52fd840  R15: ffff8a3ff55e5ee8
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #7 [ffffb37740c77b30] ip_finish_output at ffffffff97f65988
 #8 [ffffb37740c77b68] ip_output at ffffffff97f6640c
 #9 [ffffb37740c77bc0] __ip_queue_xmit at ffffffff97f65e6d
#10 [ffffb37740c77c18] __tcp_transmit_skb at ffffffff97f80557
#11 [ffffb37740c77c88] tcp_write_xmit at ffffffff97f81e34
#12 [ffffb37740c77cf0] __tcp_push_pending_frames at ffffffff97f82ae1
#13 [ffffb37740c77d00] tcp_sendmsg_locked at ffffffff97f733ac
#14 [ffffb37740c77da8] tcp_sendmsg at ffffffff97f73507
#15 [ffffb37740c77dc8] sock_sendmsg at ffffffff97ee8aa6
#16 [ffffb37740c77de0] sock_write_iter at ffffffff97ee8b47
#17 [ffffb37740c77e50] new_sync_write at ffffffff97c49bfb
#18 [ffffb37740c77ed0] vfs_write at ffffffff97c4c7d5
#19 [ffffb37740c77f00] ksys_write at ffffffff97c4ca77
#20 [ffffb37740c77f38] do_syscall_64 at ffffffff97a04140
#21 [ffffb37740c77f50] entry_SYSCALL_64_after_hwframe at ffffffff98200088
    RIP: 00007fd74beba504  RSP: 00007ffc1d456638  RFLAGS: 00000246
    RAX: ffffffffffffffda  RBX: 0000000000000084  RCX: 00007fd74beba504
    RDX: 0000000000000084  RSI: 000055785f33bb90  RDI: 0000000000000003
    RBP: 000055785f31d630   R8: 0000000000000000   R9: 0000000000001000
    R10: 0000000000000008  R11: 0000000000000246  R12: 00000000000001dd
    R13: 000055785ddc9b00  R14: 0000000000000003  R15: 00007ffc1d4566e0
    ORIG_RAX: 0000000000000001  CS: 0033  SS: 002b

crash> sym ffffffff97b94ffd
ffffffff97b94ffd (T) __cgroup_bpf_run_filter_skb+189
./debian/build/build_amd64_none_cloud-amd64/./kernel/bpf/cgroup.c: 539

crash> log
[    0.000000] Linux version 4.19.0-10-cloud-amd64 (
debian-kernel@lists.debian.org) (gcc version 8.3.0 (Debian 8.3.0-6)) #1 SMP
Debian 4.19.132-1 (2020-07-24)
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.19.0-10-cloud-amd64
root=UUID=9ac8f5bd-5b64-48cd-9efd-2b2d35a30500 ro console=tty0
console=ttyS0,115200 earlyprintk=ttyS0,115200 nmi_watchdog=1 elevator=noop
scsi_mod.use_blk_mq=Y crashkernel=384M-:128M
<SNIP>
[  478.686368] BUG: unable to handle kernel NULL pointer dereference at
0000000000000010
[  478.693551] PGD 0 P4D 0
[  478.696291] Oops: 0000 [#1] SMP PTI
[  478.699431] CPU: 0 PID: 1453 Comm: sshd Kdump: loaded Not tainted
4.19.0-10-cloud-amd64 #1 Debian 4.19.132-1
[  478.706782] Hardware name: Amazon EC2 t3.medium/, BIOS 1.0 10/16/2017
[  478.711129] RIP: 0010:__cgroup_bpf_run_filter_skb+0xbd/0x1e0
[  478.715172] Code: 00 00 00 49 89 7f 18 48 89 0c 24 44 89 e1 48 29 c8 48
89 4c 24 08 49 89 87 d8 00 00 00 89 d2 48 8d 84 d6 b0 03 00 00 48 8b 00
<48> 8b 58 10 4c 8d 70 10 48 85 db 0f 84 01 01 00 00 4d 8d 6f 30 bd
[  478.727711] RSP: 0018:ffffb37740c77ad0 EFLAGS: 00010286
[  478.731595] RAX: 0000000000000000 RBX: ffff8a3ff55e5ee8 RCX:
0000000000000000
[  478.736351] RDX: 0000000000000001 RSI: ffff8a3ff3d49800 RDI:
ffff8a3ff52fd500
[  478.741042] RBP: ffff8a3ff52fd500 R08: ffff8a3ff55e5ee8 R09:
0000000000010000
[  478.745697] R10: 0000000000000001 R11: ffff8a3ef6dd7500 R12:
0000000000000000
[  478.750446] R13: 0000000000000000 R14: ffff8a3ff52fd840 R15:
ffff8a3ff55e5ee8
[  478.755161] FS:  00007fd74bb17e40(0000) GS:ffff8a3ff7e00000(0000)
knlGS:0000000000000000
[  478.761724] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  478.765853] CR2: 0000000000000010 CR3: 00000000a94e6005 CR4:
00000000007606b0
[  478.770524] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[  478.775273] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[  478.779984] PKRU: 55555554
[  478.782901] Call Trace:
[  478.785756]  ip_finish_output+0x228/0x270
[  478.789204]  ? nf_hook_slow+0x44/0xc0
[  478.792490]  ip_output+0x6c/0xe0
[  478.795685]  ? ip_append_data.part.49+0xd0/0xd0
[  478.799403]  __ip_queue_xmit+0x15d/0x410
[  478.802945]  ? set_fd_set.part.7+0x40/0x40
[  478.806411]  __tcp_transmit_skb+0x527/0xb10
[  478.810032]  tcp_write_xmit+0x384/0x1000
[  478.813636]  ? _copy_from_iter_full+0x94/0x240
[  478.817438]  __tcp_push_pending_frames+0x31/0xd0
[  478.821170]  tcp_sendmsg_locked+0xc1c/0xd50
[  478.824714]  tcp_sendmsg+0x27/0x40
[  478.827921]  sock_sendmsg+0x36/0x40
[  478.831280]  sock_write_iter+0x97/0x100
[  478.834714]  new_sync_write+0xfb/0x160
[  478.838010]  vfs_write+0xa5/0x1a0
[  478.841129]  ksys_write+0x57/0xd0
[  478.844250]  do_syscall_64+0x50/0xf0
[  478.847526]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[  478.851385] RIP: 0033:0x7fd74beba504
[  478.854598] Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80
00 00 00 00 48 8d 05 f9 61 0d 00 8b 00 85 c0 75 13 b8 01 00 00 00 0f 05
<48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 41 54 49 89 d4 55 48 89 f5 53
[  478.867315] RSP: 002b:00007ffc1d456638 EFLAGS: 00000246 ORIG_RAX:
0000000000000001
[  478.873758] RAX: ffffffffffffffda RBX: 0000000000000084 RCX:
00007fd74beba504
[  478.878456] RDX: 0000000000000084 RSI: 000055785f33bb90 RDI:
0000000000000003
[  478.883176] RBP: 000055785f31d630 R08: 0000000000000000 R09:
0000000000001000
[  478.887885] R10: 0000000000000008 R11: 0000000000000246 R12:
00000000000001dd
[  478.892646] R13: 000055785ddc9b00 R14: 0000000000000003 R15:
00007ffc1d4566e0
[  478.897480] Modules linked in: xt_nat xt_tcpudp veth xt_conntrack
ipt_MASQUERADE nf_conntrack_netlink xfrm_user xfrm_algo nft_counter
xt_addrtype nft_compat nft_chain_nat_ipv4 nf_nat_ipv4 nf_nat nf_conntrack
nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c nf_tables nfnetlink br_netfilter
bridge stp llc binfmt_misc overlay crct10dif_pclmul crc32_pclmul
ghash_clmulni_intel nls_ascii nls_cp437 vfat fat intel_rapl_perf evdev
serio_raw button ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2
crc32c_generic fscrypto ecb crc32c_intel aesni_intel nvme aes_x86_64
crypto_simd ena nvme_core cryptd glue_helper
[  478.931979] CR2: 0000000000000010

Let me know if I can provide any other information that may be of use.

Shane Frasier

[toc] | [prev] | [next] | [standalone]

#67802

[Multipart message — attachments visible in raw view] — view raw

Also seeing this issue. Long kernel panic backtrace at the end of the
message. The kernel seems to fully crash only when I have Docker running in
a volume with an NFS export, and then I mount the NFS volume. The
underlying volume is ZFS, but I'm not entirely sure if that has much to do
with it. I can consistently reproduce this by spamming umounts and mounts:

for i in $(seq 1 100); do echo $i: umount; sudo umount -l -f /data; echo
$i: mount; mount /data; echo; done;

The spam doesn't crash the system when Docker is stopped - only when Docker
is actively running.

I saw this related bug in the kernel:
https://bugzilla.kernel.org/show_bug.cgi?id=208003
And this fix in mainline:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v5.9-rc1&id=ad0f75e5f57ccbceec13274e1e242f2b5a6397ed

Looks like the fix is in kernel 4.19.134:
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/log/?h=v4.19.134
("cgroup: fix cgroup_sk_alloc() for sk_clone_lock()")

Since the latest Debian buster kernel is based on 4.19.132, I suppose the
best thing to do is to wait for a new kernel version to be minted that
(hopefully) contains the fix?

Aug 19 14:32:11 cascadia kernel: general protection fault: 0000 [#1] SMP PTI
Aug 19 14:32:11 cascadia kernel: CPU: 9 PID: 4424 Comm: nfsd Tainted: P
      OE     4.19.0-10-amd64 #1 Debian 4.19.132-1
Aug 19 14:32:11 cascadia kernel: Hardware name: Supermicro Super
Server/X10SDV-TLN4F, BIOS 1.3 02/13/2018
Aug 19 14:32:11 cascadia kernel: RIP:
0010:__cgroup_bpf_run_filter_skb+0xbd/0x1e0
Aug 19 14:32:11 cascadia kernel: Code: 00 00 00 49 89 7f 18 48 89 0c 24 44
89 e1 48 29 c8 48 89 4c 24 08 49 89 87 d8 00 00 00 89 d2 48 8d 84 d6 b0 03
00 00 48 8b 00 <48> 8b 58 10 4c 8d 70 10 48 85 db 0f 84 01 01 00 00 4d 8d
6f 30 bd
Aug 19 14:32:11 cascadia kernel: RSP: 0000:ffffb5290ff9fa08 EFLAGS: 00010286
Aug 19 14:32:11 cascadia kernel: RAX: f70b7912beaa0ef3 RBX:
ffff9bb45783eee8 RCX: 0000000000000000
Aug 19 14:32:11 cascadia kernel: RDX: 0000000000000001 RSI:
ffff9bb45b922800 RDI: ffff9bb3f78eee80
Aug 19 14:32:11 cascadia kernel: RBP: ffff9bb3f78eee80 R08:
ffff9bb45783eee8 R09: 0000000000000c32
Aug 19 14:32:11 cascadia kernel: R10: 0000000000000001 R11:
ffff9bb373296780 R12: 0000000000000000
Aug 19 14:32:11 cascadia kernel: R13: 0000000000000000 R14:
ffff9bb3f78ef1c0 R15: ffff9bb45783eee8
Aug 19 14:32:11 cascadia kernel: FS:  0000000000000000(0000)
GS:ffff9bb45fa40000(0000) knlGS:0000000000000000
Aug 19 14:32:11 cascadia kernel: CS:  0010 DS: 0000 ES: 0000 CR0:
0000000080050033
Aug 19 14:32:11 cascadia kernel: CR2: 000055f6f1a7b010 CR3:
00000007ccc0a004 CR4: 00000000003606e0
Aug 19 14:32:11 cascadia kernel: DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
Aug 19 14:32:11 cascadia kernel: DR3: 0000000000000000 DR6:
00000000fffe0ff0 DR7: 0000000000000400
Aug 19 14:32:11 cascadia kernel: Call Trace:
Aug 19 14:32:11 cascadia kernel:  ip_finish_output+0x228/0x270
Aug 19 14:32:11 cascadia kernel:  ? nf_hook_slow+0x44/0xc0
Aug 19 14:32:11 cascadia kernel:  ip_output+0x6c/0xe0
Aug 19 14:32:11 cascadia kernel:  ? ip_append_data.part.49+0xd0/0xd0
Aug 19 14:32:11 cascadia kernel:  __ip_queue_xmit+0x15d/0x410
Aug 19 14:32:11 cascadia kernel:  __tcp_transmit_skb+0x527/0xb10
Aug 19 14:32:11 cascadia kernel:  tcp_write_xmit+0x384/0x1000
Aug 19 14:32:11 cascadia kernel:  do_tcp_sendpages+0x2d4/0x600
Aug 19 14:32:11 cascadia kernel:  tcp_sendpage_locked+0x44/0x60
Aug 19 14:32:11 cascadia kernel:  tcp_sendpage+0x37/0x50
Aug 19 14:32:11 cascadia kernel:  inet_sendpage+0x3f/0xf0
Aug 19 14:32:11 cascadia kernel:  kernel_sendpage+0x1a/0x30
Aug 19 14:32:11 cascadia kernel:  svc_send_common+0xf5/0x150 [sunrpc]
Aug 19 14:32:11 cascadia kernel:  ? nfsd_destroy+0x50/0x50 [nfsd]
Aug 19 14:32:11 cascadia kernel:  svc_sendto+0x117/0x220 [sunrpc]
Aug 19 14:32:11 cascadia kernel:  ? dput.part.33+0x32/0x140
Aug 19 14:32:11 cascadia kernel:  ? fh_put+0x93/0xf0 [nfsd]
Aug 19 14:32:11 cascadia kernel:  ? copy_cred+0x3f/0xb0 [nfsd]
Aug 19 14:32:11 cascadia kernel:  ? put_client_renew_locked+0xcd/0xe0 [nfsd]
Aug 19 14:32:11 cascadia kernel:  ? refcount_dec_and_lock+0xd/0x40
Aug 19 14:32:11 cascadia kernel:  ? auth_domain_put+ 0x15/0x70 [sunrpc]
Aug 19 14:32:11 cascadia kernel:  ? svc_authorise+0x34/0x40 [sunrpc]
Aug 19 14:32:11 cascadia kernel:  ? svc_process_common+0x11e/0x740 [sunrpc]
Aug 19 14:32:11 cascadia kernel:  ? svc_tcp_sendto+0x36/0x50 [sunrpc]
Aug 19 14:32:11 cascadia kernel:  svc_tcp_sendto+0x36/0x50 [sunrpc]
Aug 19 14:32:11 cascadia kernel:  svc_send+0x73/0x160 [sunrpc]
Aug 19 14:32:11 cascadia kernel:  nfsd+0xe3/0x140 [nfsd]
Aug 19 14:32:11 cascadia kernel:  kthread+0x112/0x130
Aug 19 14:32:11 cascadia kernel:  ? kthread_bind+0x30/0x30
Aug 19 14:32:11 cascadia kernel:  ret_from_fork+0x35/0x40
Aug 19 14:32:11 cascadia kernel: Modules linked in: veth xt_nat xt_tcpudp
xt_conntrack ipt_MASQUERADE nf_conntrack_netlink xfrm_user xfrm_algo
nft_counter xt_addrtype nft_compat nft_chain_nat_ipv4 nf_nat_ipv4 nf_nat
nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 br_netfilter bridge stp llc
rpcsec_gss_krb5 devlink nf_tables nfnetlink aufs(OE) wireguard(E)
ip6_udp_tunnel udp_tunnel overlay intel_rapl sb_edac x86_pkg_temp_thermal
intel_powerclamp coretemp kvm_intel pcc_cpufreq zfs(POE) kvm irqbypass
zunicode(POE) intel_cstate zlua(POE) zavl(POE) intel_uncore icp(POE)
ipmi_ssif mxm_wmi ast ttm drm_kms_helper intel_rapl_perf pcspkr drm
intel_pch_thermal iTCO_wdt iTCO_vendor_support joydev cdc_acm evdev sg
mei_me mei ioatdma ipmi_si wmi acpi_pad button nfsd auth_rpcgss nfs_acl
lockd grace zcommon(POE) sunrpc znvpair(POE) spl(OE)
Aug 19 14:32:11 cascadia kernel:  ipmi_devintf ipmi_msghandler ip_tables
x_tables autofs4 ext4 crc16 mbcache jbd2 fscrypto ecb btrfs xor
zstd_decompress zstd_compress xxhash raid6_pq libcrc32c crc32c_generic
algif_skcipher af_alg dm_crypt dm_mod hid_generic usbhid hid sd_mod
crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc ahci
xhci_pci libahci xhci_hcd mpt3sas ehci_pci libata ehci_hcd usbcore
raid_class aesni_intel ixgbe scsi_transport_sas igb aes_x86_64 scsi_mod
nvme crypto_simd cryptd glue_helper lpc_ich nvme_core i2c_i801 mfd_core
i2c_algo_bit dca usb_common mdio
Aug 19 14:32:11 cascadia kernel: ---[ end trace acf9c2d544d7fd2c ]---
Aug 19 14:32:11 cascadia kernel: RIP:
0010:__cgroup_bpf_run_filter_skb+0xbd/0x1e0
Aug 19 14:32:11 cascadia kernel: Code: 00 00 00 49 89 7f 18 48 89 0c 24 44
89 e1 48 29 c8 48 89 4c 24 08 49 89 87 d8 00 00 00 89 d2 48 8d 84 d6 b0 03
00 00 48 8b 00 <48> 8b 58 10 4c 8d 70 10 48 85 db 0f 84 01 01 00 00 4d 8d
6f 30 bd
Aug 19 14:32:11 cascadia kernel: RSP: 0000:ffffb5290ff9fa08 EFLAGS: 00010286
Aug 19 14:32:11 cascadia kernel: RAX: f70b7912beaa0ef3 RBX:
ffff9bb45783eee8 RCX: 0000000000000000
Aug 19 14:32:11 cascadia kernel: RDX: 0000000000000001 RSI:
ffff9bb45b922800 RDI: ffff9bb3f78eee80
Aug 19 14:32:11 cascadia kernel: RBP: ffff9bb3f78eee80 R08:
ffff9bb45783eee8 R09: 0000000000000c32
Aug 19 14:32:11 cascadia kernel: R10: 0000000000000001 R11:
ffff9bb373296780 R12: 0000000000000000
Aug 19 14:32:11 cascadia kernel: R13: 0000000000000000 R14:
ffff9bb3f78ef1c0 R15: ffff9bb45783eee8
Aug 19 14:32:11 cascadia kernel: FS:  0000000000000000(0000)
GS:ffff9bb45fa40000(0000) knlGS:0000000000000000
Aug 19 14:32:11 cascadia kernel: CS:  0010 DS: 0000 ES: 0000 CR0:
0000000080050033
Aug 19 14:32:11 cascadia kernel: CR2: 000055f6f1a7b010 CR3:
00000007ccc0a004 CR4: 00000000003606e0
Aug 19 14:32:11 cascadia kernel: DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
Aug 19 14:32:11 cascadia kernel: DR3: 0000000000000000 DR6:
00000000fffe0ff0 DR7: 0000000000000400
Aug 19 14:32:11 cascadia kernel: general protection fault: 0000 [#2] SMP PTI
Aug 19 14:32:11 cascadia kernel: CPU: 2 PID: 13041 Comm: C2 CompilerThre
Tainted: P      D    OE     4.19.0-10-amd64 #1 Debian 4.19.132-1
Aug 19 14:32:11 cascadia kernel: Hardware name: Supermicro Super
Server/X10SDV-TLN4F, BIOS 1.3 02/13/2018
Aug 19 14:32:11 cascadia kernel: RIP:
0010:__cgroup_bpf_run_filter_skb+0xbd/0x1e0
Aug 19 14:32:11 cascadia kernel: Code: 00 00 00 49 89 7f 18 48 89 0c 24 44
89 e1 48 29 c8 48 89 4c 24 08 49 89 87 d8 00 00 00 89 d2 48 8d 84 d6 b0 03
00 00 48 8b 00 <48> 8b 58 10 4c 8d 70 10 48 85 db 0f 84 01 01 00 00 4d 8d
6f 30 bd
Aug 19 14:32:11 cascadia kernel: RSP: 0000:ffff9bb45f883c00 EFLAGS: 00010296
Aug 19 14:32:11 cascadia kernel: RAX: f3f4ae25351c25db RBX:
ffff9bb453a7c300 RCX: 0000000000000014
Aug 19 14:32:11 cascadia kernel: RDX: 0000000000000000 RSI:
ffff9bb45b922800 RDI: ffff9bb3f78eee80
Aug 19 14:32:11 cascadia kernel: RBP: ffff9bb3f78eee80 R08:
0000000000000001 R09: 0000000000000801
Aug 19 14:32:11 cascadia kernel: R10: 0000000000000001 R11:
0000000000000000 R12: 0000000000000014
Aug 19 14:32:11 cascadia kernel: R13: 0000000000000020 R14:
ffff9bb43d882862 R15: ffff9bb453a7c300
Aug 19 14:32:11 cascadia kernel: FS:  00007f28a1043b10(0000)
GS:ffff9bb45f880000(0000) knlGS:0000000000000000
Aug 19 14:32:11 cascadia kernel: CS:  0010 DS: 0000 ES: 0000 CR0:
0000000080050033
Aug 19 14:32:11 cascadia kernel: CR2: 000055b71d84fb10 CR3:
0000000749b5a004 CR4: 00000000003606e0
Aug 19 14:32:11 cascadia kernel: DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
Aug 19 14:32:11 cascadia kernel: DR3: 0000000000000000 DR6:
00000000fffe0ff0 DR7: 0000000000000400
Aug 19 14:32:11 cascadia kernel: Call Trace:
Aug 19 14:32:11 cascadia kernel:  <IRQ>
Aug 19 14:32:11 cascadia kernel:  sk_filter_trim_cap+0xcd/0x1b0
Aug 19 14:32:11 cascadia kernel:  tcp_v4_rcv+0xa41/0xc20
Aug 19 14:32:11 cascadia kernel:  ip_local_deliver_finish+0x63/0x1e0
Aug 19 14:32:11 cascadia kernel:  ip_local_deliver+0xe0/0xf0
Aug 19 14:32:11 cascadia kernel:  ? ip_sublist_rcv_finish+0x80/0x80
Aug 19 14:32:11 cascadia kernel:  ip_rcv+0xbc/0xd0
Aug 19 14:32:11 cascadia kernel:  ? ip_rcv_finish_core.isra.18+0x360/0x360
Aug 19 14:32:11 cascadia kernel:  __netif_receive_skb_one_core+0x5a/0x80
Aug 19 14:32:11 cascadia kernel:  netif_receive_skb_internal+0x2f/0xa0
Aug 19 14:32:11 cascadia kernel:  napi_gro_receive+0xba/0xe0
Aug 19 14:32:11 cascadia kernel:  igb_poll+0x481/0xeb0 [igb]
Aug 19 14:32:11 cascadia kernel:  net_rx_action+0x149/0x3b0
Aug 19 14:32:11 cascadia kernel:  __do_softirq+0xde/0x2d8
Aug 19 14:32:11 cascadia kernel:  irq_exit+0xba/0xc0
Aug 19 14:32:11 cascadia kernel:  do_IRQ+0x7f/0xe0
Aug 19 14:32:11 cascadia kernel:  common_interrupt+0xf/0xf
Aug 19 14:32:11 cascadia kernel:  </IRQ>
Aug 19 14:32:11 cascadia kernel: RIP: 0033:0x7f28b5f3147a
Aug 19 14:32:11 cascadia kernel: Code: 89 c2 48 c1 e0 02 48 8b 4d 18 39 d3
7c 0f c7 04 01 00 00 00 00 ff c2 48 83 c0 04 eb e9 8d 43 01 89 45 00 48 8b
45 18 8b 04 98 <5a> 5b 5d c3 31 c0 48 39 f1 0f 84 d6 00 00 00 66 44 8b 41
2c 41 80
Aug 19 14:32:11 cascadia kernel: RSP: 002b:00007f28a1040560 EFLAGS:
00000297 ORIG_RAX: ffffffffffffffdb
Aug 19 14:32:11 cascadia kernel: RAX: 0000000000000000 RBX:
0000000000003a31 RCX: 000055b71dfb23a8
Aug 19 14:32:11 cascadia kernel: RDX: 000055b71dfb23a8 RSI:
0000000000003a31 RDI: 000055b7202930b0
Aug 19 14:32:11 cascadia kernel: RBP: 000055b7202930b0 R08:
000055b71e475e50 R09: 000055b71dfb2010
Aug 19 14:32:11 cascadia kernel: R10: 0000000000000018 R11:
000055b71df0b480 R12: 000055b71dfb23a8
Aug 19 14:32:11 cascadia kernel: R13: 000055b7212a1f10 R14:
0000000000000000 R15: 00007f28a1040790
Aug 19 14:32:11 cascadia kernel: Modules linked in: veth xt_nat xt_tcpudp
xt_conntrack ipt_MASQUERADE nf_conntrack_netlink xfrm_user xfrm_algo
nft_counter xt_addrtype nft_compat nft_chain_nat_ipv4 nf_nat_ipv4 nf_nat
nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 br_netfilter bridge stp llc
rpcsec_gss_krb5 devlink nf_tables nfnetlink aufs(OE) wireguard(E)
ip6_udp_tunnel udp_tunnel overlay intel_rapl sb_edac x86_pkg_temp_thermal
intel_powerclamp coretemp kvm_intel pcc_cpufreq zfs(POE) kvm irqbypass
zunicode(POE) intel_cstate zlua(POE) zavl(POE) intel_uncore icp(POE)
ipmi_ssif mxm_wmi ast ttm drm_kms_helper intel_rapl_perf pcspkr drm
intel_pch_thermal iTCO_wdt iTCO_vendor_support joydev cdc_acm evdev sg
mei_me mei ioatdma ipmi_si wmi acpi_pad button nfsd auth_rpcgss nfs_acl
lockd grace zcommon(POE) sunrpc znvpair(POE) spl(OE)
Aug 19 14:32:11 cascadia kernel:  ipmi_devintf ipmi_msghandler ip_tables
x_tables autofs4 ext4 crc16 mbcache jbd2 fscrypto ecb btrfs xor
zstd_decompress zstd_compress xxhash raid6_pq libcrc32c crc32c_generic
algif_skcipher af_alg dm_crypt dm_mod hid_generic usbhid hid sd_mod
crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc ahci
xhci_pci libahci xhci_hcd mpt3sas ehci_pci libata ehci_hcd usbcore
raid_class aesni_intel ixgbe scsi_transport_sas igb aes_x86_64 scsi_mod
nvme crypto_simd cryptd glue_helper lpc_ich nvme_core i2c_i801 mfd_core
i2c_algo_bit dca usb_common mdio
Aug 19 14:32:11 cascadia kernel: ---[ end trace acf9c2d544d7fd2d ]---

Kevin Tran

[toc] | [prev] | [next] | [standalone]

#67919

Hi Immanuel,

On Sun, Aug 30, 2020 at 04:49:51PM +0200, ischbug@der-ball-ist-rund.net wrote:
> I can confirm this bug. For me the panic is not exclusively related
> with docker.  I stopped my docker daemon - and I'm still suffering
> from random freezes of the kernel 4.19.0-10. Kernel is unusable.
> 
> But I'm still running two lxc-containers and two virtual machines kvm/qemu.
> 
> enclosed you'll find the "screenshot"

Yes, the issue would not be exclusively with docker.

The issue should be pending fixed, but if you can I would appreciate
if you can test (warning: temporary and unofficial build!) packages
rebased to 4.19.142 upstream:

https://people.debian.org/~carnil/tmp/linux/4.19.142-1/

Regards,
Salvatore

[toc] | [prev] | [next] | [standalone]

#67979

Hi Salvatore,

Le 2020-08-30 17:08, Salvatore Bonaccorso a écrit :
> On Sun, Aug 30, 2020 at 04:49:51PM +0200, ischbug@der-ball-ist-rund.net 
> wrote:
>> But I'm still running two lxc-containers and two virtual machines 
>> kvm/qemu.
> 
> Yes, the issue would not be exclusively with docker.

I'm facing this bug as well on a server with LXC containers (no 
Docker/KVM/Qemu
at all).

> The issue should be pending fixed, but if you can I would appreciate
> if you can test (warning: temporary and unofficial build!) packages
> rebased to 4.19.142 upstream:
> 
> https://people.debian.org/~carnil/tmp/linux/4.19.142-1/

I've installed it (amd64 version) and will let you know how things are 
going on.

Sébastien

[toc] | [prev] | [next] | [standalone]

#67992

Hi Salvatore,

No crash for two days with this build. I'll send an update in a few 
days.

Sébastien

[toc] | [prev] | [next] | [standalone]

#68089

Hi,

More than a week after, no problem with this build, working fine 24/7.

Sébastien

[toc] | [prev] | [next] | [standalone]

#68091

Hi Sébastien,

On Fri, Sep 11, 2020 at 11:41:16AM +0200, Sébastien NOBILI wrote:
> Hi,
> 
> More than a week after, no problem with this build, working fine
> 24/7.

Thanks for confirming that. We are planning to rebase the version for
the next point release and so will contain the fix.

Regards,
Salvatore

[toc] | [prev] | [next] | [standalone]

#68237

Hello,

I'm a bit late but I also have this issue and it occurs every several
hours on my server. Here is the trace if anybody is still interested: 
https://lebihan.pl/files/trace.txt

When can I expect the new package to be uploaded into stable?

Michel Le Bihan

[toc] | [prev] | [next] | [standalone]

#68254

[Multipart message — attachments visible in raw view] — view raw

On Tue, 2020-09-22 at 23:52 +0200, Michel Le Bihan wrote:
> Hello,
> 
> I'm a bit late but I also have this issue and it occurs every several
> hours on my server. Here is the trace if anybody is still interested: 
> https://lebihan.pl/files/trace.txt
> 
> When can I expect the new package to be uploaded into stable?

In the point release, at the weekend.

Ben.

-- 
Ben Hutchings
The generation of random numbers is too important to be left to chance.
                                                       - Robert Coveyou

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.kernel

csiph-web