Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.debian.bugs.rc > #304401
| From | Paul Gevers <elbrus@debian.org> |
|---|---|
| Newsgroups | linux.debian.bugs.rc |
| Subject | Bug#993978: linux-image-5.10.0-8-arm64: host hangs after some time of use |
| Date | 2021-09-09 09:40 +0200 |
| Message-ID | <CVmP8-4GP-5@gated-at.bofh.it> (permalink) |
| Organization | linux.* mail to news gateway |
[Multipart message — attachments visible in raw view] - view raw
Package: src:linux
Version: 5.10.46-4
X-Debbugs-CC: debian-ci@lists.debian.org, aron@debian.org
Severity: serious
Justification: data loss
Hi,
As discussed over IRC, here is the bug report for one of the hanging
arm64 hosts we have for ci.debian.net.
Since the upgrade of our hosts to bullseye (days before the bullseye
release) we have been experiencing random loss of access to our hosts.
For the hosts that have some form of out-of-bound access, I tried to use
that to see what's going on, but at AWS our account doesn't have the
right permissions to use the serial port out-of-bound access and all
other forms that I tried on all hosts that I have access to some for of
out-of-bound access that didn't work.
Since the bullseye release I've rebooted (externally triggered) already
dozens of times and for those host that don't allow rebooting (AWS
again) I had to reprovision the hosts.
All the architectures (amd64, arm64, ppc64el and s390x) that we have
experience these hangs. I'm absolutely not claiming that the root cause
is the same, but on buster we didn't experience this (our s390x host
never workerd on buster so I don't claim regression there), so there is
a pattern. However, the symptoms don't look completely the same everywhere.
On one of our arm64 hosts (we call ci-worker-armel-01) I found the
attached logging as the final logs in the journal.
Paul
-- Package-specific info:
** Version:
Linux version 5.10.0-8-arm64 (debian-kernel@lists.debian.org) (gcc-10
(Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian)
2.35.2) #1 SMP Debian 5.10.46-4 (2021-08-03)
** Command line:
BOOT_IMAGE=/boot/vmlinuz-5.10.0-8-arm64
root=UUID=05ed5059-7820-4ab3-a3b7-37cc7dede2cf ro console=tty0
console=ttyS1,115200n8 quiet
** Tainted: W (512)
* kernel issued warning
** Kernel log:
Unable to read kernel log; any relevant messages should be attached
** Model information
** Loaded modules:
cfg80211
rdma_ucm
streebog_generic
sha3_generic
rmd320
rmd256
rmd160
rmd128
poly1305_generic
poly1305_neon
nhpoly1305_neon
nhpoly1305
libpoly1305
michael_mic
crc32_generic
blake2b_generic
twofish_generic
twofish_common
serpent_generic
fcrypt
cast6_generic
cast5_generic
cast_common
camellia_generic
blowfish_generic
blowfish_common
aes_arm64
ghash_generic
gcm
aegis128
aes_generic
nf_log_ipv6
nf_conntrack_ftp
nf_log_ipv4
nf_log_common
nft_log
nft_synproxy
nf_synproxy_core
nft_limit
nft_objref
nft_osf
nfnetlink_osf
nft_numgen
nft_nat
nf_conntrack_sip
nft_quota
nft_meta_bridge
nf_flow_table_ipv4
nft_flow_offload
nf_flow_table_inet
nf_flow_table
aes_ce_ccm
cbc
des_generic
libdes
ecb
sha512_generic
sha512_arm64
md4
af_packet_diag
netlink_diag
nft_masq
nft_fib_inet
nft_fib_ipv4
nft_fib_ipv6
nft_fib
nft_reject_inet
nf_reject_ipv6
nft_reject
nft_ct
sctp_diag
udp_diag
raw_diag
unix_diag
iptable_mangle
iptable_filter
tcp_diag
inet_diag
8021q
garp
mrp
nfsd
auth_rpcgss
nfs_acl
lockd
grace
sunrpc
nf_conntrack_netlink
xfrm_user
xfrm_algo
overlay
algif_rng
algif_skcipher
algif_aead
can_bcm
sctp
bluetooth
jitterentropy_rng
drbg
ansi_cprng
ecdh_generic
rfkill
ecc
qrtr
ns
can_j1939
can_isotp
can_raw
can
algif_hash
af_alg
xt_nat
xt_addrtype
iptable_nat
dummy
xt_multiport
vhost_net
vhost
vhost_iotlb
tap
tun
veth
xt_conntrack
ipt_REJECT
nf_reject_ipv4
xt_CHECKSUM
nft_chain_nat
xt_MASQUERADE
nf_nat
nf_conntrack
nf_defrag_ipv6
nf_defrag_ipv4
nft_counter
xt_tcpudp
nft_compat
bridge
stp
llc
nf_tables
nfnetlink
binfmt_misc
nls_ascii
nls_cp437
vfat
fat
dm_multipath
scsi_dh_rdac
scsi_dh_emc
scsi_dh_alua
aes_ce_blk
crypto_simd
cryptd
aes_ce_cipher
ghash_ce
efi_pstore
gf128mul
libaes
sha2_ce
sha256_arm64
sha1_ce
sbsa_gwdt
acpi_ipmi
ipmi_ssif
dm_mod
evdev
ipmi_devintf
ipmi_msghandler
arm_cmn
xgene_hwmon
cppc_cpufreq
ib_iser
rdma_cm
iw_cm
ib_cm
iscsi_tcp
libiscsi_tcp
libiscsi
scsi_transport_iscsi
scsi_mod
bonding
drm
fuse
configfs
efivarfs
ip_tables
x_tables
autofs4
ext4
crc16
mbcache
jbd2
raid10
raid456
async_raid6_recov
async_memcpy
async_pq
async_xor
async_tx
xor
xor_neon
raid6_pq
crc32c_generic
libcrc32c
raid1
raid0
multipath
linear
md_mod
mlx5_ib
ib_uverbs
ib_core
mlx5_core
nvme
nvme_core
igb
t10_pi
crc_t10dif
mlxfw
crct10dif_generic
i2c_algo_bit
ptp
crct10dif_ce
pps_core
crct10dif_common
i2c_designware_platform
i2c_designware_core
** Network interface configuration:
*** /etc/network/interfaces:
auto lo
iface lo inet loopback
auto bond0
iface bond0 inet static
address 139.178.85.85
netmask 255.255.255.254
gateway 139.178.85.84
bond-downdelay 200
bond-miimon 100
bond-mode 4
bond-updelay 200
bond-xmit_hash_policy layer3+4
bond-slaves enP7p1s0f0 enP7p1s0f1
dns-nameservers 147.75.207.207 147.75.207.208
iface bond0 inet6 static
address 2604:1380:4111:f200::1
netmask 127
gateway 2604:1380:4111:f200::
auto bond0:0
iface bond0:0 inet static
address 10.32.131.1
netmask 255.255.255.254
post-up route add -net 10.0.0.0/8 gw 10.32.131.0
post-down route del -net 10.0.0.0/8 gw 10.32.131.0
** Network status:
*** IP interfaces and addresses:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enP7p2s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group
default qlen 1000
link/ether 00:1b:21:e4:b6:5e brd ff:ff:ff:ff:ff:ff
3: enP7p1s0f0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq
master bond0 state UP group default qlen 1000
link/ether 76:9b:20:ae:91:80 brd ff:ff:ff:ff:ff:ff permaddr
0c:42:a1:49:d5:f8
altname enP7p1s0f0np0
4: enP7p1s0f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq
master bond0 state UP group default qlen 1000
link/ether 76:9b:20:ae:91:80 brd ff:ff:ff:ff:ff:ff permaddr
0c:42:a1:49:d5:f9
altname enP7p1s0f1np1
5: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc
noqueue state UP group default qlen 1000
link/ether 76:9b:20:ae:91:80 brd ff:ff:ff:ff:ff:ff
inet 139.178.85.85/31 brd 255.255.255.255 scope global bond0
valid_lft forever preferred_lft forever
inet 10.32.131.1/31 brd 255.255.255.255 scope global bond0:0
valid_lft forever preferred_lft forever
inet6 2604:1380:4111:f200::1/127 scope global
valid_lft forever preferred_lft forever
inet6 fe80::749b:20ff:feae:9180/64 scope link
valid_lft forever preferred_lft forever
6: lxcbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
state DOWN group default qlen 1000
link/ether 00:16:3e:00:00:00 brd ff:ff:ff:ff:ff:ff
inet 10.0.3.1/24 brd 10.0.3.255 scope global lxcbr0
valid_lft forever preferred_lft forever
7: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
state UP group default qlen 1000
link/ether 52:54:00:b2:8a:76 brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
88: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
master virbr0 state UNKNOWN group default qlen 1000
link/ether fe:54:00:02:ea:09 brd ff:ff:ff:ff:ff:ff
inet6 fe80::fc54:ff:fe02:ea09/64 scope link
valid_lft forever preferred_lft forever
9314: vethwTl3LI@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
noqueue master virbr0 state UP group default qlen 1000
link/ether fe:8e:b7:a4:0f:8a brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::fc8e:b7ff:fea4:f8a/64 scope link
valid_lft forever preferred_lft forever
9358: veth1Ii4zY@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
noqueue master virbr0 state UP group default qlen 1000
link/ether fe:57:eb:89:3e:73 brd ff:ff:ff:ff:ff:ff link-netnsid 9
inet6 fe80::fc57:ebff:fe89:3e73/64 scope link
valid_lft forever preferred_lft forever
9395: vethtTId3j@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
noqueue master virbr0 state UP group default qlen 1000
link/ether fe:7d:c2:58:f2:d4 brd ff:ff:ff:ff:ff:ff link-netnsid 4
inet6 fe80::fc7d:c2ff:fe58:f2d4/64 scope link
valid_lft forever preferred_lft forever
9396: vethJPSJdQ@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
noqueue master virbr0 state UP group default qlen 1000
link/ether fe:33:77:82:52:4c brd ff:ff:ff:ff:ff:ff link-netnsid 5
inet6 fe80::fc33:77ff:fe82:524c/64 scope link
valid_lft forever preferred_lft forever
9406: vethMGmicQ@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
noqueue master virbr0 state UP group default qlen 1000
link/ether fe:ed:1e:e1:4c:56 brd ff:ff:ff:ff:ff:ff link-netnsid 1
inet6 fe80::fced:1eff:fee1:4c56/64 scope link
valid_lft forever preferred_lft forever
9417: vethzNIjxf@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
noqueue master virbr0 state UP group default qlen 1000
link/ether fe:63:65:c0:6b:f9 brd ff:ff:ff:ff:ff:ff link-netnsid 8
inet6 fe80::fc63:65ff:fec0:6bf9/64 scope link
valid_lft forever preferred_lft forever
*** Device statistics:
Inter-| Receive | Transmit
face |bytes packets errs drop fifo frame compressed multicast|bytes
packets errs drop fifo colls carrier compressed
lo: 9756936 43339 0 0 0 0 0 0
9756936 43339 0 0 0 0 0 0
enP7p2s0: 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
enP7p1s0f0: 459927613599 306790587 0 1802 0 0 0
5724 1059056101 11880163 0 0 0 0 0 0
enP7p1s0f1: 479413404341 319814129 0 2954 0 0 0
5578 1070041065 11815207 0 0 0 0 0 0
bond0: 939341017940 626604716 0 4756 0 0 0 11302
2129097166 23695370 0 0 0 0 0 0
lxcbr0: 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
virbr0: 1412550333 23306162 0 0 0 0 0 38556
899850355543 38198226 0 0 0 0 0 0
vnet0: 636832 6977 0 0 0 0 0 0
80183868 189767 0 0 0 0 0 0
vethwTl3LI: 159192 2048 0 0 0 0 0 0
86926884 9251 0 0 0 0 0 0
veth1Ii4zY: 383259 5312 0 0 0 0 0 0
153748418 14784 0 0 0 0 0 0
vethtTId3j: 297532 3671 0 0 0 0 0 0
211773613 8664 0 0 0 0 0 0
vethJPSJdQ: 346012 4405 0 0 0 0 0 0
211833586 9595 0 0 0 0 0 0
vethMGmicQ: 301910 3765 0 0 0 0 0 0
209812125 8565 0 0 0 0 0 0
vethzNIjxf: 173512 2313 0 0 0 0 0 0
93115403 3828 0 0 0 0 0 0
*** Protocol statistics:
Ip:
Forwarding: 1
61304574 total packets received
60248201 forwarded
3 with unknown protocol
0 incoming packets discarded
1049474 incoming packets delivered
61315940 requests sent out
Icmp:
5910 ICMP messages received
22 input ICMP message failed
ICMP input histogram:
destination unreachable: 4228
timeout in transit: 3
redirects: 2
echo requests: 1676
echo replies: 1
20285 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
destination unreachable: 9649
echo requests: 8960
echo replies: 1676
IcmpMsg:
InType0: 1
InType3: 4228
InType5: 2
InType8: 1676
InType11: 3
OutType0: 1676
OutType3: 9649
OutType8: 8960
Tcp:
7898 active connection openings
3342 passive connection openings
242 failed connection attempts
27 connection resets received
13 connections established
569052 segments received
705696 segments sent out
1660 segments retransmitted
287 bad segments received
88097 resets sent
InCsumErrors: 287
Udp:
560374 packets received
867 packets to unknown port received
0 packet receive errors
550902 packets sent
0 receive buffer errors
0 send buffer errors
IgnoredMulti: 1073
UdpLite:
TcpExt:
216 resets received for embryonic SYN_RECV sockets
25 ICMP packets dropped because they were out-of-window
1276 TCP sockets finished time wait in fast timer
53 packetes rejected in established connections because of timestamp
5119 delayed acks sent
20 delayed acks further delayed because of locked socket
Quick ack mode was activated 106 times
201668 packet headers predicted
67204 acknowledgments not containing data payload received
186917 predicted acknowledgments
TCPSackRecovery: 24
TCPDSACKUndo: 4
8 congestion windows recovered without slow start after partial ack
TCPLostRetransmit: 599
222 fast retransmits
TCPTimeouts: 1203
TCPLossProbes: 296
TCPLossProbeRecovery: 60
TCPSackRecoveryFail: 6
TCPBacklogCoalesce: 851
TCPDSACKOldSent: 101
TCPDSACKRecv: 76
6682 connections reset due to unexpected data
4 connections reset due to early user close
87 connections aborted due to timeout
TCPDSACKIgnoredNoUndo: 38
TCPSackShifted: 308
TCPSackMerged: 1259
TCPSackShiftFallback: 184
TCPRcvCoalesce: 26158
TCPOFOQueue: 224
TCPAutoCorking: 28571
TCPFromZeroWindowAdv: 5
TCPToZeroWindowAdv: 5
TCPWantZeroWindowAdv: 33
TCPSynRetrans: 446
TCPOrigDataSent: 447757
TCPHystartDelayDetect: 3
TCPHystartDelayCwnd: 760
TCPACKSkippedPAWS: 34
TCPACKSkippedSeq: 64
TCPKeepAlive: 111
TCPDelivered: 455398
TcpTimeoutRehash: 741
TcpDuplicateDataRehash: 6
TCPDSACKRecvSegs: 76
IpExt:
OutMcastPkts: 4
InBcastPkts: 21625
OutBcastPkts: 385
InOctets: 900725598967
OutOctets: 1801685659170
OutMcastOctets: 160
InBcastOctets: 7060774
OutBcastOctets: 126289
InNoECTPkts: 649366854
InECT0Pkts: 12
Sctp:
0 Current Associations
0 Active Associations
0 Passive Associations
0 Number of Aborteds
0 Number of Graceful Terminations
0 Number of Out of Blue packets
0 Number of Packets with invalid Checksum
0 Number of control chunks sent
0 Number of ordered chunks sent
0 Number of Unordered chunks sent
0 Number of control chunks received
0 Number of ordered chunks received
0 Number of Unordered chunks received
0 Number of messages fragmented
0 Number of messages reassembled
0 Number of SCTP packets sent
0 Number of SCTP packets received
** PCI devices:
not available
** USB devices:
not available
-- System Information:
Debian Release: 11.0
APT prefers stable-security
APT policy: (990, 'stable-security'), (990, 'stable'), (500,
'unstable'), (1, 'experimental')
Architecture: arm64 (aarch64)
Kernel: Linux 5.10.0-8-arm64 (SMP w/160 CPU threads)
Kernel taint flags: TAINT_WARN
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE
not set
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Versions of packages linux-image-5.10.0-8-arm64 depends on:
ii initramfs-tools [linux-initramfs-tool] 0.140
ii kmod 28-1
ii linux-base 4.6
Versions of packages linux-image-5.10.0-8-arm64 recommends:
ii apparmor 2.13.6-10
ii firmware-linux-free 20200122-1
Versions of packages linux-image-5.10.0-8-arm64 suggests:
pn debian-kernel-handbook <none>
pn linux-doc-5.10 <none>
Versions of packages linux-image-5.10.0-8-arm64 is related to:
pn firmware-amd-graphics <none>
pn firmware-atheros <none>
pn firmware-bnx2 <none>
pn firmware-bnx2x <none>
pn firmware-brcm80211 <none>
pn firmware-cavium <none>
pn firmware-intel-sound <none>
pn firmware-intelwimax <none>
pn firmware-ipw2x00 <none>
pn firmware-ivtv <none>
pn firmware-iwlwifi <none>
pn firmware-libertas <none>
pn firmware-linux-nonfree <none>
pn firmware-misc-nonfree <none>
pn firmware-myricom <none>
pn firmware-netxen <none>
pn firmware-qlogic <none>
pn firmware-realtek <none>
pn firmware-samsung <none>
pn firmware-siano <none>
pn firmware-ti-connectivity <none>
pn xen-hypervisor <none>
-- no debconf information
Back to linux.debian.bugs.rc | Previous | Next — Next in thread | Find similar | Unroll thread
Bug#993978: linux-image-5.10.0-8-arm64: host hangs after some time of use Paul Gevers <elbrus@debian.org> - 2021-09-09 09:40 +0200
Bug#993978: linux-image-5.10.0-8-arm64: host hangs after some time of use Salvatore Bonaccorso <carnil@debian.org> - 2021-09-09 11:20 +0200
Bug#993978: linux-image-5.10.0-8-arm64: host hangs after some time of use Salvatore Bonaccorso <carnil@debian.org> - 2021-09-09 17:20 +0200
Bug#993978: linux-image-5.10.0-8-arm64: host hangs after some time of use Salvatore Bonaccorso <carnil@debian.org> - 2021-09-09 19:40 +0200
Bug#993978: linux-image-5.10.0-8-arm64: host hangs after some time of use Paul Gevers <elbrus@debian.org> - 2021-09-09 12:30 +0200
Bug#993978: closing 993978 Salvatore Bonaccorso <carnil@debian.org> - 2021-09-09 18:00 +0200
csiph-web