Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.bugs.dist > #814017 > unrolled thread

Bug#858125: e1000: ethernet interface hangs occasionally, kernel reports hang

Back to article view | Back to linux.debian.bugs.dist

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Bug#858125: e1000: ethernet interface hangs occasionally, kernel reports hang Ben Hutchings <ben@decadent.org.uk> - 2017-03-18 22:10 +0100
    Bug#858125: e1000: ethernet interface hangs occasionally, kernel reports hang "Bruce Momjian,,," <bruce@momjian.us> - 2017-03-18 22:20 +0100
      Bug#858125: e1000: ethernet interface hangs occasionally, kernel reports hang Ben Hutchings <ben@decadent.org.uk> - 2017-03-18 23:10 +0100
        Bug#858125: e1000: ethernet interface hangs occasionally, kernel reports hang "Bruce Momjian,,," <bruce@momjian.us> - 2017-03-18 23:20 +0100
          Bug#858125: e1000: ethernet interface hangs occasionally, kernel reports hang "Bruce Momjian,,," <bruce@momjian.us> - 2017-03-18 23:40 +0100
            Bug#858125: e1000: ethernet interface hangs occasionally, kernel reports hang "Bruce Momjian,,," <bruce@momjian.us> - 2017-03-21 21:10 +0100
              Bug#858125: e1000: ethernet interface hangs occasionally, kernel reports hang "Bruce Momjian,,," <bruce@momjian.us> - 2017-03-21 23:40 +0100
                Bug#858125: e1000: ethernet interface hangs occasionally, kernel reports hang Ben Hutchings <ben@decadent.org.uk> - 2017-03-22 03:50 +0100
                  Bug#858125: e1000: ethernet interface hangs occasionally, kernel reports hang "Bruce Momjian,,," <bruce@momjian.us> - 2017-03-22 04:20 +0100
                    Bug#858125: e1000: ethernet interface hangs occasionally, kernel reports hang "Bruce Momjian,,," <bruce@momjian.us> - 2017-03-23 04:50 +0100
                      Bug#858125: e1000: ethernet interface hangs occasionally, kernel reports hang "Bruce Momjian,,," <bruce@momjian.us> - 2017-03-23 20:30 +0100
                        Bug#858125: e1000: ethernet interface hangs occasionally, kernel reports hang "Bruce Momjian,,," <bruce@momjian.us> - 2017-04-11 00:10 +0200
    Bug#858125: e1000: ethernet interface hangs occasionally, kernel reports hang "Bruce Momjian,,," <bruce@momjian.us> - 2017-03-18 23:00 +0100

#814017 — Bug#858125: e1000: ethernet interface hangs occasionally, kernel reports hang

[Multipart message — attachments visible in raw view] — view raw

Control: retitle -1 TX watchdog fires on e1000e interface 
Control: tag -1 moreinfo

Please don't confuse e1000 and e1000e; they are quite different drivers
for different chips.

Does it help if you disable flow control by running:

    ethtool -A eth0 rx off tx off

(Check the current settings first by running 'ethtool -a eth0'.)

Have you tried a newer kernel version?  Linux 4.9 is available in
jessie-backports.

Ben.

-- 
Ben Hutchings
Editing code like this is akin to sticking plasters on the bleeding
stump
of a severed limb. - me, 29 June 1999

[toc] | [next] | [standalone]

#814022

On Sat, Mar 18, 2017 at 09:02:10PM +0000, Ben Hutchings wrote:
> Control: retitle -1 TX watchdog fires on e1000e interface 
> Control: tag -1 moreinfo
> 
> Please don't confuse e1000 and e1000e; they are quite different drivers
> for different chips.

Oh, thanks for the fix.  I think I used e1000 so it would match a kernel
file when I entered the bug report.  You are right that e1000e is what
the kernel reports on the error lines:

	[ 1351.850362] NETDEV WATCHDOG: eth0 (e1000e): transmit queue 0	timed out
	                                      ^^^^^^

> Does it help if you disable flow control by running:
> 
>     ethtool -A eth0 rx off tx off
> 
> (Check the current settings first by running 'ethtool -a eth0'.)

Current settings:

	$ ethtool -a eth0
	Pause parameters for eth0:
	Autonegotiate:  on
	RX:             on
	TX:             on

I then ran:

	$ ethtool -A eth0 rx off tx off

	$ ethtool -a eth0
	Pause parameters for eth0:
	Autonegotiate:  on
	RX:             on
	TX:             on

It seems nothing changed.

> Have you tried a newer kernel version?  Linux 4.9 is available in
> jessie-backports.

Wow, that seems more risky than just buying a dual-port PCI-E ethernet
card and using that.  FYI, this server is from 2012 so I am worried such
an upgrade would break things more than fix them.

If there is no known fix I think that will be my next try, in a long
list of many attempts to fix this.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

[toc] | [prev] | [next] | [standalone]

#814031

[Multipart message — attachments visible in raw view] — view raw

On Sat, Mar 18, 2017 at 05:10:50PM -0400, Bruce Momjian,,, wrote:
[...]
> I then ran:
> 
> 	$ ethtool -A eth0 rx off tx off
> 
> 	$ ethtool -a eth0
> 	Pause parameters for eth0:
> 	Autonegotiate:  on
> 	RX:             on
> 	TX:             on
> 
> It seems nothing changed.

Try:
    ethtool -A eth0 rx off tx off autoneg off
instead.

> > Have you tried a newer kernel version?  Linux 4.9 is available in
> > jessie-backports.
> 
> Wow, that seems more risky than just buying a dual-port PCI-E ethernet
> card and using that.  FYI, this server is from 2012 so I am worried such
> an upgrade would break things more than fix them.
[...]

I wasn't suggesting it as a long-term fix but just to check whether a
fix has been implemented.  (And I think it is unlikely to break
anything.)

Ben.

-- 
Ben Hutchings
Life is like a sewer:
what you get out of it depends on what you put into it.

[toc] | [prev] | [next] | [standalone]

#814033

On Sat, Mar 18, 2017 at 10:06:53PM +0000, Ben Hutchings wrote:
> On Sat, Mar 18, 2017 at 05:10:50PM -0400, Bruce Momjian,,, wrote:
> [...]
> > I then ran:
> > 
> > 	$ ethtool -A eth0 rx off tx off
> > 
> > 	$ ethtool -a eth0
> > 	Pause parameters for eth0:
> > 	Autonegotiate:  on
> > 	RX:             on
> > 	TX:             on
> > 
> > It seems nothing changed.
> 
> Try:
>     ethtool -A eth0 rx off tx off autoneg off
> instead.

OK, that worked:

	$ ethtool -A eth0 rx off tx off autoneg off
	$ ethtool -a eth0
	Pause parameters for eth0:
	Autonegotiate:  off
	RX:             off
	TX:             off

I will report back as soon as another failure happens, or doesn't
happen.  :-)  Thanks.

> > > Have you tried a newer kernel version?  Linux 4.9 is available in
> > > jessie-backports.
> > 
> > Wow, that seems more risky than just buying a dual-port PCI-E ethernet
> > card and using that.  FYI, this server is from 2012 so I am worried such
> > an upgrade would break things more than fix them.
> [...]
> 
> I wasn't suggesting it as a long-term fix but just to check whether a
> fix has been implemented.  (And I think it is unlikely to break
> anything.)

Oh, I guess I am not sure how I could test that easily.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

[toc] | [prev] | [next] | [standalone]

#814035

On Sat, Mar 18, 2017 at 06:09:33PM -0400, Bruce Momjian,,, wrote:
> On Sat, Mar 18, 2017 at 10:06:53PM +0000, Ben Hutchings wrote:
> > > It seems nothing changed.
> > 
> > Try:
> >     ethtool -A eth0 rx off tx off autoneg off
> > instead.
> 
> OK, that worked:
> 
> 	$ ethtool -A eth0 rx off tx off autoneg off
> 	$ ethtool -a eth0
> 	Pause parameters for eth0:
> 	Autonegotiate:  off
> 	RX:             off
> 	TX:             off
> 
> I will report back as soon as another failure happens, or doesn't
> happen.  :-)  Thanks.

I turned off flow control on the Debian server _and_ the switch port,
because I am told they should match.  I will report back.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

[toc] | [prev] | [next] | [standalone]

#814624

On Sat, Mar 18, 2017 at 06:33:30PM -0400, Bruce Momjian,,, wrote:
> On Sat, Mar 18, 2017 at 06:09:33PM -0400, Bruce Momjian,,, wrote:
> > On Sat, Mar 18, 2017 at 10:06:53PM +0000, Ben Hutchings wrote:
> > > > It seems nothing changed.
> > > 
> > > Try:
> > >     ethtool -A eth0 rx off tx off autoneg off
> > > instead.
> > 
> > OK, that worked:
> > 
> > 	$ ethtool -A eth0 rx off tx off autoneg off
> > 	$ ethtool -a eth0
> > 	Pause parameters for eth0:
> > 	Autonegotiate:  off
> > 	RX:             off
> > 	TX:             off
> > 
> > I will report back as soon as another failure happens, or doesn't
> > happen.  :-)  Thanks.
> 
> I turned off flow control on the Debian server _and_ the switch port,
> because I am told they should match.  I will report back.

OK, I turned off flow control, rebooted, and ran for 48 hours with no
problems.  I then turned on flow control (the default), rebooted, and
after 21 hours saw a failure, then two minutes later, another one.

I think this proves my problems are related to flow control.  How would
you like to proceed?  Is there a patch or change you would like me to
test?  Just close the ticket?

I have a fix, but it is likely others would not know they had this
problem unless they were monitoring their kernel logs or their network
traffic for lag.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

[toc] | [prev] | [next] | [standalone]

#814647

On Tue, Mar 21, 2017 at 04:04:11PM -0400, Bruce Momjian,,, wrote:
> I think this proves my problems are related to flow control.  How would
> you like to proceed?  Is there a patch or change you would like me to
> test?  Just close the ticket?
> 
> I have a fix, but it is likely others would not know they had this
> problem unless they were monitoring their kernel logs or their network
> traffic for lag.

Oh, I should also mention the port that is having problems is connected
to a NetGear GS108Ev3 switch, with current firmware, version 2.00.09. 
The port connected to my Actiontec FIOS router is not having problems.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

[toc] | [prev] | [next] | [standalone]

#814666

[Multipart message — attachments visible in raw view] — view raw

Control: retitle -1 TX watchdog fires on e1000e interface with flow control enabled

On Tue, 2017-03-21 at 18:36 -0400, Bruce Momjian,,, wrote:
> On Tue, Mar 21, 2017 at 04:04:11PM -0400, Bruce Momjian,,, wrote:
> > I think this proves my problems are related to flow control.  How would
> > you like to proceed?  Is there a patch or change you would like me to
> > test?  Just close the ticket?
> > 
> > I have a fix, but it is likely others would not know they had this
> > problem unless they were monitoring their kernel logs or their network
> > traffic for lag.
> 
> Oh, I should also mention the port that is having problems is connected
> to a NetGear GS108Ev3 switch, with current firmware, version 2.00.09. 
> The port connected to my Actiontec FIOS router is not having problems.

I don't know about any specific bug, but if the switch sends flow
control XOFF frames continually for long enough (usually 5 seconds)
this will trigger the TX watchdog.

It sounds like your switch implements flow control properly (some
broken switches auto-negotiate it but actually flood flow control
frames).  However, if a device on some other port (that also has flow
control enabled) sends XOFF frames continually *and* your server sends
frames that should go to that other port, the switch will do the same
to the server once the switch's internal queue has filled up.

If the switch has port statistics including numbers of pause frames
then you can see where they are coming from, but I think it doesn't.
Without that information it's going to be hard to tell exactly where
the fault lies.

The e1000e driver *does* have statistics for pause frames transmitted
and received (run: "ethtool -S eth0| grep flow_control").  If you log
these every second then it should be possible to see what happens
around the time the TX watchdog fires.  That could provide some clues
as to whether the NIC is behaving correctly.

Ben.

-- 
Ben Hutchings
Power corrupts.  Absolute power is kind of neat.
                           - John Lehman, Secretary of the US Navy
1981-1987

[toc] | [prev] | [next] | [standalone]

#814671

On Wed, Mar 22, 2017 at 02:42:30AM +0000, Ben Hutchings wrote:
> Control: retitle -1 TX watchdog fires on e1000e interface with flow control enabled
> 
> On Tue, 2017-03-21 at 18:36 -0400, Bruce Momjian,,, wrote:
> > On Tue, Mar 21, 2017 at 04:04:11PM -0400, Bruce Momjian,,, wrote:
> > > I think this proves my problems are related to flow control.  How would
> > > you like to proceed?  Is there a patch or change you would like me to
> > > test?  Just close the ticket?
> > > 
> > > I have a fix, but it is likely others would not know they had this
> > > problem unless they were monitoring their kernel logs or their network
> > > traffic for lag.
> > 
> > Oh, I should also mention the port that is having problems is connected
> > to a NetGear GS108Ev3 switch, with current firmware, version 2.00.09. 
> > The port connected to my Actiontec FIOS router is not having problems.
> 
> I don't know about any specific bug, but if the switch sends flow
> control XOFF frames continually for long enough (usually 5 seconds)
> this will trigger the TX watchdog.

Makes sense.

> It sounds like your switch implements flow control properly (some
> broken switches auto-negotiate it but actually flood flow control
> frames).  However, if a device on some other port (that also has flow

If I turn off flow control on the switch port, and leave the Debian
server at defaults, the Debian port automatically turns off flow
control, which must be what 'autoneg' is meant to do.

> control enabled) sends XOFF frames continually *and* your server sends
> frames that should go to that other port, the switch will do the same
> to the server once the switch's internal queue has filled up.

What I could do it to turn off flow control on all switch ports _except_
the Debian server.  The switch has per-port flow control management
control.

> If the switch has port statistics including numbers of pause frames
> then you can see where they are coming from, but I think it doesn't.
> Without that information it's going to be hard to tell exactly where
> the fault lies.

Yeah, I don't see flow control stats on the switch, just CRC error
reporting.

> The e1000e driver *does* have statistics for pause frames transmitted
> and received (run: "ethtool -S eth0| grep flow_control").  If you log
> these every second then it should be possible to see what happens
> around the time the TX watchdog fires.  That could provide some clues
> as to whether the NIC is behaving correctly.

OK, I am running this after setting flow control on/default on the
switch and Debian, and rebooting:

	daemon -- sh -c "while :; do date;ethtool -S eth0| grep flow_control;
	sleep 1;done > /root/ethtool"

I will report back with the relevant logging lines once it hangs again. 
Thanks.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

[toc] | [prev] | [next] | [standalone]

#814873

On Tue, Mar 21, 2017 at 11:08:34PM -0400, Bruce Momjian,,, wrote:
> > The e1000e driver *does* have statistics for pause frames transmitted
> > and received (run: "ethtool -S eth0| grep flow_control").  If you log
> > these every second then it should be possible to see what happens
> > around the time the TX watchdog fires.  That could provide some clues
> > as to whether the NIC is behaving correctly.
> 
> OK, I am running this after setting flow control on/default on the
> switch and Debian, and rebooting:
> 
> 	daemon -- sh -c "while :; do date;ethtool -S eth0| grep flow_control;
> 	sleep 1;done > /root/ethtool"
> 
> I will report back with the relevant logging lines once it hangs again. 

OK, I have results of a hang after 24 hours of uptime.  The hangs are
listed here via dmesg -T:

	http://momjian.us/expire/eth0/dmesg.txt

showing the watchdog warning/hang/reset at 23:01 and port hang/reset at
23:10.

I have also produced the ethtool -S output every second for the entire
24-hour period, gziped, at:

	http://momjian.us/expire/eth0/ethtool.gz

You will see reception of a large number of rx_flow_control_xoff
messages about 50 minutes before the hangs, and just before the hangs.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

[toc] | [prev] | [next] | [standalone]

#815006

On Wed, Mar 22, 2017 at 11:41:57PM -0400, Bruce Momjian,,, wrote:
> > OK, I am running this after setting flow control on/default on the
> > switch and Debian, and rebooting:
> > 
> > 	daemon -- sh -c "while :; do date;ethtool -S eth0| grep flow_control;
> > 	sleep 1;done > /root/ethtool"
> > 
> > I will report back with the relevant logging lines once it hangs again. 
> 
> OK, I have results of a hang after 24 hours of uptime.  The hangs are
> listed here via dmesg -T:
> 
> 	http://momjian.us/expire/eth0/dmesg.txt
> 
> showing the watchdog warning/hang/reset at 23:01 and port hang/reset at
> 23:10.
> 
> I have also produced the ethtool -S output every second for the entire
> 24-hour period, gziped, at:
> 
> 	http://momjian.us/expire/eth0/ethtool.gz
> 
> You will see reception of a large number of rx_flow_control_xoff
> messages about 50 minutes before the hangs, and just before the hangs.

I had four more 14 hours later so I created new files that also include
the earlier ones:

	http://momjian.us/expire/eth0/dmesg2.txt
	http://momjian.us/expire/eth0/ethtool2.gz

The last two dmesg lines at 13:29  are me turning of flow control on the
switch so they are not problems.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

[toc] | [prev] | [next] | [standalone]

#818189

On Thu, Mar 23, 2017 at 03:25:15PM -0400, Bruce Momjian,,, wrote:
> I had four more 14 hours later so I created new files that also include
> the earlier ones:
> 
> 	http://momjian.us/expire/eth0/dmesg2.txt
> 	http://momjian.us/expire/eth0/ethtool2.gz
> 
> The last two dmesg lines at 13:29  are me turning of flow control on the
> switch so they are not problems.

My system is working fine with flow control turned off.  What are my
next steps?

* Additional debugging
* Patched or updated Ethernet driver
* Try a new Ethernet card
* Nothing?

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

[toc] | [prev] | [next] | [standalone]

#814027

Here is the dmesg output from just this afternoon:

	[Mar18 12:40] e1000e 0000:03:00.0 eth0: Reset adapter unexpectedly
	[  +3.891401] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
	[Mar18 13:40] e1000e 0000:03:00.0 eth0: Reset adapter unexpectedly
	[  +3.827380] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
	[Mar18 14:43] e1000e 0000:03:00.0 eth0: Reset adapter unexpectedly
	[  +3.871385] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
	[Mar18 14:47] e1000e 0000:03:00.0 eth0: Reset adapter unexpectedly
	[  +3.431026] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
	[Mar18 16:13] e1000e 0000:03:00.0 eth0: Reset adapter unexpectedly
	[  +3.799385] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
	[Mar18 17:04] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
	[  +9.870235] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
	[Mar18 17:05] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
	[Mar18 17:09] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx

which is pretty extreme.  Eight resets in five hours!  This is a home
server (momjian.us) so it isn't like it is pushing huge amounts of data
or overloaded.

I did so much research on this before reporting it because this is
common hardware (SuperMicro) and Intel PRO/1000 ethernet adaptors, which
are also popular.   There must be something odd about my server, but I
have no idea what it is.  Also, if I buy a dual PCI-E adaptor, it will
also be an Intel PRO/1000.  Will I get the same errors on that?

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.bugs.dist

csiph-web