Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1531293 > unrolled thread

Re: stmmac ethernet in kernel 4.4: coalescing related pauses?

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: stmmac ethernet in kernel 4.4: coalescing related pauses? Lino Sanfilippo <lsanfil@marvell.com> - 2016-11-28 14:10 +0100
    Re: stmmac ethernet in kernel 4.4: coalescing related pauses? David Miller <davem@davemloft.net> - 2016-11-28 16:00 +0100
      Re: stmmac ethernet in kernel 4.4: coalescing related pauses? Lino Sanfilippo <lsanfil@marvell.com> - 2016-11-28 16:40 +0100
      Re: stmmac ethernet in kernel 4.4: coalescing related pauses? Eric Dumazet <eric.dumazet@gmail.com> - 2016-11-28 16:40 +0100
        Re: stmmac ethernet in kernel 4.4: coalescing related pauses? Lino Sanfilippo <lsanfil@marvell.com> - 2016-11-28 17:00 +0100
          Re: stmmac ethernet in kernel 4.4: coalescing related pauses? David Miller <davem@davemloft.net> - 2016-11-28 17:40 +0100
            Re: stmmac ethernet in kernel 4.4: coalescing related pauses? Lino Sanfilippo <lsanfil@marvell.com> - 2016-11-28 18:10 +0100
        Re: stmmac ethernet in kernel 4.4: coalescing related pauses? Pavel Machek <pavel@ucw.cz> - 2016-11-30 11:30 +0100

#1531293 — Re: stmmac ethernet in kernel 4.4: coalescing related pauses?

Hi Pavel,

On 23.11.2016 11:51, Pavel Machek wrote:

> I'm debugging strange delays during transmit in stmmac driver. They
> seem to be present in 4.4 kernel (and older kernels, too). Workload is
> burst of udp packets being sent, pause, burst of udp packets, ...
>
> Test code is attached, I use these parameters for testing:
>
> ./udp-test raw 10.0.0.6 1234 1000 100 30
>
> The delays seem to be related to coalescing:
>
> drivers/net/ethernet/stmicro/stmmac/common.h
> #define STMMAC_COAL_TX_TIMER    40000
> #define STMMAC_MAX_COAL_TX_TICK 100000
> #define STMMAC_TX_MAX_FRAMES    256
>
> If I lower the parameters, delays are gone, but I get netdev watchdog
> backtrace followed by broken driver.
>
> Any ideas what is going on there?
>
> [I'm currently trying to get newer kernels working on affected
> hardware.]
>
> Best regards,
>
> 									Pavel

I once encountered a similar behaviour with a driver. The reason was that the socket
queue limit was temporarily exhausted because the irq handler did not free the tx skbs
fast enough (that driver also used irq coalescing).
Calling skb_orphan() in the xmit handler made this issue disappear.

Regards,
Lino  

[toc] | [next] | [standalone]

#1531380

From: Lino Sanfilippo <lsanfil@marvell.com>
Date: Mon, 28 Nov 2016 14:07:51 +0100

> Calling skb_orphan() in the xmit handler made this issue disappear.

This is not the way to handle this problem.

The solution is to free the SKBs in a timely manner after the
chip has transmitted the frame.

[toc] | [prev] | [next] | [standalone]

#1531395

Hi,

On 28.11.2016 15:54, David Miller wrote:
> From: Lino Sanfilippo <lsanfil@marvell.com>
> Date: Mon, 28 Nov 2016 14:07:51 +0100
>
>> Calling skb_orphan() in the xmit handler made this issue disappear.
>
> This is not the way to handle this problem.
>

I agree, its not a clean solution. But if the use of skb_orphan makes the delays
disappear, it can at least be a hint that socket queue limits are involved.

Regards,
Lino

[toc] | [prev] | [next] | [standalone]

#1531399

On Mon, 2016-11-28 at 09:54 -0500, David Miller wrote:
> From: Lino Sanfilippo <lsanfil@marvell.com>
> Date: Mon, 28 Nov 2016 14:07:51 +0100
> 
> > Calling skb_orphan() in the xmit handler made this issue disappear.
> 
> This is not the way to handle this problem.
> 
> The solution is to free the SKBs in a timely manner after the
> chip has transmitted the frame.

Note that the 'pauses' described by Pavel are also caused by a too small
SO_SNDBUF value on the UDP socket.

An immediate fix, with no kernel change is to increase it.

echo 1000000 >/proc/sys/net/core/wmem_default

or

val = 1000000;
setsockopt(fd, SOL_SOCKET, SO_SNDBUF, &val, sizeof(val));

[toc] | [prev] | [next] | [standalone]

#1531408

On 28.11.2016 16:31, Eric Dumazet wrote:
> On Mon, 2016-11-28 at 09:54 -0500, David Miller wrote:
>> From: Lino Sanfilippo <lsanfil@marvell.com>
>> Date: Mon, 28 Nov 2016 14:07:51 +0100
>>
>>> Calling skb_orphan() in the xmit handler made this issue disappear.
>>
>> This is not the way to handle this problem.
>>
>> The solution is to free the SKBs in a timely manner after the
>> chip has transmitted the frame.
>
> Note that the 'pauses' described by Pavel are also caused by a too small
> SO_SNDBUF value on the UDP socket.
>
> An immediate fix, with no kernel change is to increase it.
>
> echo 1000000 >/proc/sys/net/core/wmem_default
>
> or
>
> val = 1000000;
> setsockopt(fd, SOL_SOCKET, SO_SNDBUF, &val, sizeof(val));
>

I wonder if the best fix would be indeed to deactivate irq coalescing completely.
Does it make any sense at all to use it if a driver uses NAPI already?

Regards,
Lino

[toc] | [prev] | [next] | [standalone]

#1531431

From: Lino Sanfilippo <lsanfil@marvell.com>
Date: Mon, 28 Nov 2016 16:57:35 +0100

> I wonder if the best fix would be indeed to deactivate irq coalescing
> completely.
> Does it make any sense at all to use it if a driver uses NAPI already?

It absolutely does make sense, when it is implemented and functions
properly.

[toc] | [prev] | [next] | [standalone]

#1531449

On 28.11.2016 17:30, David Miller wrote:
> From: Lino Sanfilippo <lsanfil@marvell.com>
> Date: Mon, 28 Nov 2016 16:57:35 +0100
>
>> I wonder if the best fix would be indeed to deactivate irq coalescing
>> completely.
>> Does it make any sense at all to use it if a driver uses NAPI already?
>
> It absolutely does make sense, when it is implemented and functions
> properly.
>

Interesting. I always thought both (NAPI and irq coalescing) are essentially doing the same thing only
one time in software and one time with hw support. Did I misunderstand NAPI?

Regards,
Lino

[toc] | [prev] | [next] | [standalone]

#1533198

[Multipart message — attachments visible in raw view] — view raw

On Mon 2016-11-28 07:31:43, Eric Dumazet wrote:
> On Mon, 2016-11-28 at 09:54 -0500, David Miller wrote:
> > From: Lino Sanfilippo <lsanfil@marvell.com>
> > Date: Mon, 28 Nov 2016 14:07:51 +0100
> > 
> > > Calling skb_orphan() in the xmit handler made this issue disappear.
> > 
> > This is not the way to handle this problem.
> > 
> > The solution is to free the SKBs in a timely manner after the
> > chip has transmitted the frame.
> 
> Note that the 'pauses' described by Pavel are also caused by a too small
> SO_SNDBUF value on the UDP socket.
> 
> An immediate fix, with no kernel change is to increase it.
> 
> echo 1000000 >/proc/sys/net/core/wmem_default

Thanks a lot. For the record, that works around the problem, too. (Or
at least helps a lot; it may be possible that problem still remains if
continuous stream of packets is going to trigger this, if I read the
sources correctly.)

Best regards,
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web