Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1575053 > unrolled thread

Re: Regression: Failed boots bisected to 4cd13c21b207 "softirq: Let ksoftirqd do its job"

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Regression: Failed boots bisected to 4cd13c21b207 "softirq: Let  ksoftirqd do its job" Will Deacon <will.deacon@arm.com> - 2017-02-06 19:50 +0100
    Re: Regression: Failed boots bisected to 4cd13c21b207 "softirq: Let  ksoftirqd do its job" Russell King - ARM Linux <linux@armlinux.org.uk> - 2017-02-06 20:00 +0100
      Re: Regression: Failed boots bisected to 4cd13c21b207 "softirq: Let  ksoftirqd do its job" Will Deacon <will.deacon@arm.com> - 2017-02-08 10:50 +0100

#1575053 — Re: Regression: Failed boots bisected to 4cd13c21b207 "softirq: Let ksoftirqd do its job"

Hi all,

I've also stumbled over this issue with the ARM fastmodel and, somewhat
embarrassingly, blamed the model developers for the regression. I'm using
NFS and copying ~14MB file from NFS to a virtio-blk device which takes
over 20 minutes with 4cd13c21b207, but <1 min with it reverted.

I also think I've figured out what's going on. See below.

On Fri, Nov 25, 2016 at 01:14:03PM +0000, Brian Starkey wrote:
> On Wed, Nov 23, 2016 at 12:03:28PM -0800, Eric Dumazet wrote:
> >On Wed, Nov 23, 2016 at 10:21 AM, Brian Starkey <brian.starkey@arm.com> wrote:
> >
> >>This patch didn't help.
> >>
> >>I did get some new traces though - I've attached the diff for the
> >>trace_printks I added.
> >>
> >>Before 4cd13c21b207:
> >>https://drive.google.com/open?id=0B8siaK6ZjvEwcEtOeFQzTmY0Nnc
> >>After 4cd13c21b207:
> >>https://drive.google.com/open?id=0B8siaK6ZjvEwZnQ4MVg1d3d1Tm8
> >>
> >>It looks like the difference is that after 4cd13c21b207 the RX softirq
> >>isn't running, and RX interrupts don't call softirq_raise anymore -
> >>presumably because there's one pending, but I didn't have time to
> >>track that down to a code-path.
> >>
> >>Cheers,
> >>-Brian
> >>
> >
> >Hi Brian
> >
> >Looks like netif_rx() drops the incoming packets then ?
> >
> >Maybe netif_running() is not happy :(
> >
> >Could you trace netif_rx() return value (NET_RX_SUCCESS or NET_RX_DROP)
> 
> Some packets are dropped, but not very many:
> 
>   $ grep NET_RX_SUCCESS trace_netif_rx.txt | wc -l
>   14399
>   $ grep NET_RX_DROP trace_netif_rx.txt | wc -l
>   22
> 
> Without the ksoftirqd change there were zero NET_RX_DROPs.

The SMC91x has an on-chip 8KB FIFO (i.e. there's no DMA going on here).
When the FIFO is full (every 4 TCP packets in my case), we get an
interrupt and run down the smc_rcv path. There, we allocate an skb for
the data (netdev_alloc_skb) and copy the data out of the FIFO
(SMC_PULL_DATA) into the buffer, which we hand over the network core via
netif_rx.

The problem is that netif_rx defers to ksoftirqd to process the packet
and more crucially *free* the skb after it's been consumed. Since the
thing was allocated in IRQ context, we end up exhausting our GFP_ATOMIC
memory because ksoftirqd gets interrupted so frequently due to the tiny
FIFO depth that buffers are allocated at a much higher frequency than
they are freed. This may be exagerated by the relative speed of the model
emulated CPU with respect to the network interface, but I'd expect this
to be reproducible on real hardware too (rmk, cc'd, was going to give that
a go).

Prior to 4cd13c21b207, we'd always run softirqs synchronously on the
hardirq exit path and therefore have a chance to free some skbs before
actually EOI'ing the hardirq and allowing the FIFO-full interrupt to
interrupt us again.

Converting the smc91x driver over to NAPI would probably solve this problem,
but given the "vintage" of this code, I'd be more tempted by a simpler
point fix if only I could think of one.

Any ideas?

Will

[toc] | [next] | [standalone]

#1575063

On Mon, Feb 06, 2017 at 06:46:19PM +0000, Will Deacon wrote:
> Converting the smc91x driver over to NAPI would probably solve this problem,
> but given the "vintage" of this code, I'd be more tempted by a simpler
> point fix if only I could think of one.

I'm not sure if converting it to NAPI would solve it, or just move
the problem elsewhere - IOW, move it from "we need to drop the packet
because we couldn't allocate a skb" to "the hardware dropped the packed
because the FIFO was full."

Yes, I'm intending giving it a go, once I've a spare moment to build
a kernel for the platform etc.  It runs root NFS, so should be a good
test for it.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

[toc] | [prev] | [next] | [standalone]

#1576393

On Mon, Feb 06, 2017 at 06:49:42PM +0000, Russell King - ARM Linux wrote:
> On Mon, Feb 06, 2017 at 06:46:19PM +0000, Will Deacon wrote:
> > Converting the smc91x driver over to NAPI would probably solve this problem,
> > but given the "vintage" of this code, I'd be more tempted by a simpler
> > point fix if only I could think of one.
> 
> I'm not sure if converting it to NAPI would solve it, or just move
> the problem elsewhere - IOW, move it from "we need to drop the packet
> because we couldn't allocate a skb" to "the hardware dropped the packed
> because the FIFO was full."

That's quite possible. I did a quick hack using a threaded irq handler,
with the thread basically running a modified version of smc_rcv using
GFP_KERNEL allocations. Whilst this improves things significantly, I do
still see rx drops, probably for the reason you mention above.

Still, NAPI should be better than what mainline is currently doing because
it won't continuously interrupt ksoftirqd when in polling mode. It's all
a rather delicate balancing act and getting back to the old behaviour might
not be possible after 4cd13c21b207.

> Yes, I'm intending giving it a go, once I've a spare moment to build
> a kernel for the platform etc.  It runs root NFS, so should be a good
> test for it.

Thanks, that would be interesting. We resurrected one of our realview-eb
machines with this NIC, but I think it's all on an FPGA so the relative
speed of the NIC vs the CPU isn't different enough that we see the problem.

Will

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web