Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.bugs.dist > #1074910

Bug#990642: linux-image-4.19.0-17-amd64: kernel panic on xen dom0 with Broadcom Limited NetXtreme II BCM5709

Cross-posted to 2 groups.

Show all headers | View raw

Am 30.09.21 um 22:03 schrieb Hans van Kranenburg:
> Hi spi, Salvatore,
>
> On 8/5/21 1:58 PM, spi@gmxpro.de wrote:
>>
>> In preparation for the bug report for upstream I did some more
>> investigation.
>>
>> The kernel panic also occurs without bonding interfaces but needs much
>> more time to happen. With a bonding interface it happens within some
>> seconds. Without bonding interfaces it needs like a minute with the
>> network discovery being re-launched for 2 or 3 times. The kernel panic
>> is still the same about the bnx2 driver.
>>
>> In the constellation without a bonding interface the kernel panic only
>> occurs if
>> - opnsense as a domU is running (this domU bounds all bridged interfaces
>> as default gateway for all networks)
>
> Just FWIW, I'm seeing this bug-mail-thread now, and it rings a bell.
>
> I spent some time in the past to debug crashing BCM5719 (4x1G) nics in
> HP DL360 G8/9 series servers. In this case, the firmware inside the nic
> crashed, so the symptoms were different. This happened only when having
> a Xen domU active as router, which was routing incoming traffic packets
> (from outside the box) back to the outside again.
>
> 02:00.0 Ethernet controller: Broadcom Limited NetXtreme BCM5719 Gigabit
> Ethernet PCIe (rev 01)
> 02:00.1 Ethernet controller: Broadcom Limited NetXtreme BCM5719 Gigabit
> Ethernet PCIe (rev 01)
> 02:00.2 Ethernet controller: Broadcom Limited NetXtreme BCM5719 Gigabit
> Ethernet PCIe (rev 01)
> 02:00.3 Ethernet controller: Broadcom Limited NetXtreme BCM5719 Gigabit
> Ethernet PCIe (rev 01)
>
> Also, 2x 1G were bonded, I use openvswitch with LACP for that.
>
> The symptoms are obviously different, mine looked like this:
>
> tg3 0000:02:00.2 eth1: transmit timed out, resetting
> tg3 0000:02:00.2 eth1: 0x00000000: 0x165714e4, 0x00100546, 0x02000001,
> 0x00800010
> tg3 0000:02:00.2 eth1: 0x00000010: 0x92b3000c, 0x00000000, 0x92b4000c,
> 0x00000000
> tg3 0000:02:00.2 eth1: 0x00000020: 0x92b5000c, 0x00000000, 0x00000000,
> 0x22be103c
>
> tg3 0000:02:00.2 eth1: 0x00007000: 0x08000008, 0x00000000, 0x00000000,
> 0x00004cd8
> tg3 0000:02:00.2 eth1: 0x00007010: 0xdbbd2b97, 0x010080f3, 0x00d70081,
> 0x03008200
> tg3 0000:02:00.2 eth1: 0x00007020: 0x00000000, 0x00000000, 0x00000406,
> 0x10004000
> tg3 0000:02:00.2 eth1: 0x00007030: 0x00020000, 0x00004cdc, 0x001f0000,
> 0x00000000
> tg3 0000:02:00.2 eth1: 0: Host status block
> [00000001:00000070:(0000:0563:0000):(0000:0094)]
> tg3 0000:02:00.2 eth1: 0: NAPI info
> [00000070:00000070:(016a:0094:01ff):0000:(068c:0000:0000:0000)]
> tg3 0000:02:00.2 eth1: 1: Host status block
> [00000001:00000083:(0000:0000:0000):(015b:0000)]
> tg3 0000:02:00.2 eth1: 1: NAPI info
> [00000051:00000051:(0000:0000:01ff):0124:(0124:0124:0000:0000)]
> tg3 0000:02:00.2 eth1: 2: Host status block
> [00000001:000000d8:(0e96:0000:0000):(0000:0000)]
> tg3 0000:02:00.2 eth1: 2: NAPI info
> [000000a4:000000a4:(0000:0000:01ff):0e5b:(065b:065b:0000:0000)]
> tg3 0000:02:00.2 eth1: 3: Host status block
> [00000001:00000013:(0000:0000:0000):(0000:0000)]
> tg3 0000:02:00.2 eth1: 3: NAPI info
> [000000f8:000000f8:(0000:0000:01ff):072f:(072f:072f:0000:0000)]
> tg3 0000:02:00.2 eth1: 4: Host status block
> [00000001:0000009c:(0000:0000:0736):(0000:0000)]
> tg3 0000:02:00.2 eth1: 4: NAPI info
> [0000007c:0000007c:(0000:0000:01ff):0716:(0716:0716:0000:0000)]
> tg3 0000:02:00.2: tg3_stop_block timed out, ofs=1400 enable_bit=2
> tg3 0000:02:00.2: tg3_stop_block timed out, ofs=c00 enable_bit=2
> tg3 0000:02:00.2 eth1: Link is down
> tg3 0000:02:00.2 eth1: Link is up at 1000 Mbps, full duplex
> tg3 0000:02:00.2 eth1: Flow control is off for TX and off for RX
> tg3 0000:02:00.2 eth1: EEE is disabled
>
>> - sysctl parameter net.bridge.bridge-nf-call-ip6tables is set to 0.
>>
>> If both conditions are not met no kernel panic oaccurs.
>
> What I found out in the end is that using `ethtool -K $iface tso off` is
> a workaround to not make it trigger some obscure bug inside the nic that
> makes it crash.
>
> So, I think my actual suggestion would be, even while it does not look
> like the same thing, but it's still Broadcom stuff which can have
> *cough* weird issues... if you can reliably reproduce the problem, then
> can you try setting tso off on the physical interfaces in dom0 and try
> again? In Dutch we say "nooit geschoten altijd mis".
>

The next time I do maintenance I'll rerun the tests and set "ethtool -K
$iface tso off". "ethtool -K ${int} tx off" is alreday configured on the
server and I also tried tso off and others as well but didn't pay
attention to this bug at that time - as said the kernel panic doesn't
occur immediately but only after some minutes.

In German it is "wer nicht wagt, der nicht gewinnt" ;-)

>> Other IPv6 related sysctl parameters are set on dom0 like
>> net.ipv6.conf.all.disable_ipv6 = 1
>> net.ipv6.conf.default.disable_ipv6 = 1
>> net.ipv6.conf.lo.disable_ipv6 = 1
>>
>>
>> The layer2-iptables settings are
>> net.bridge.bridge-nf-call-ip6tables = 0 ***
>>

Do you recall what your setting for net.bridge.bridge-nf-call-ip6tables was?

>>
>> net.bridge.bridge-nf-call-iptables = 1
>>
>>
>> net.bridge.bridge-nf-call-arptables = 0
>>
>>
>>
>>
>> As said, if I don't set the one marked with *** to 0 there is no kernel
>> panic.
>>
>> I wonder if this still is a kernel issue but still wouldn't expect a
>> kernel panic to happen.
>>
>> Cheers,
>> spi
>>
>
> Have fun,
> Hans
>
Cheers,
Sebastian

Back to linux.debian.bugs.dist | Previous | Next — Previous in thread | Find similar | Unroll thread

Thread

Bug#990642: linux-image-4.19.0-17-amd64: kernel panic on xen dom0 with Broadcom Limited NetXtreme II BCM5709 Hans van Kranenburg <hans@knorrie.org> - 2021-09-30 22:20 +0200
  Bug#990642: linux-image-4.19.0-17-amd64: kernel panic on xen dom0 with Broadcom Limited NetXtreme II BCM5709 spi@gmxpro.de - 2021-10-09 13:40 +0200

csiph-web