Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1345055 > unrolled thread

Softirq priority inversion from "softirq: reduce latencies"

Back to article view | Back to linux.kernel

Contents

  Softirq priority inversion from "softirq: reduce latencies" Peter Hurley <peter@hurleysoftware.com> - 2016-02-27 19:20 +0100
    Re: Softirq priority inversion from "softirq: reduce latencies" Eric Dumazet <eric.dumazet@gmail.com> - 2016-02-27 21:20 +0100
      Re: Softirq priority inversion from "softirq: reduce latencies" Peter Hurley <peter@hurleysoftware.com> - 2016-02-27 21:40 +0100
        Re: Softirq priority inversion from "softirq: reduce latencies" David Miller <davem@davemloft.net> - 2016-02-28 00:10 +0100
          Re: Softirq priority inversion from "softirq: reduce latencies" Peter Hurley <peter@hurleysoftware.com> - 2016-02-28 00:40 +0100
            Re: Softirq priority inversion from "softirq: reduce latencies" Eric Dumazet <eric.dumazet@gmail.com> - 2016-02-28 03:10 +0100
              Re: Softirq priority inversion from "softirq: reduce latencies" Peter Hurley <peter@hurleysoftware.com> - 2016-02-28 03:20 +0100
                Re: Softirq priority inversion from "softirq: reduce latencies" Eric Dumazet <eric.dumazet@gmail.com> - 2016-02-28 03:20 +0100
                Re: Softirq priority inversion from "softirq: reduce latencies" David Miller <davem@davemloft.net> - 2016-02-28 05:50 +0100
    Re: Softirq priority inversion from "softirq: reduce latencies" Mike Galbraith <umgwanakikbuti@gmail.com> - 2016-02-28 07:00 +0100
      Re: Softirq priority inversion from "softirq: reduce latencies" Francois Romieu <romieu@fr.zoreil.com> - 2016-02-28 18:10 +0100
        Re: Softirq priority inversion from "softirq: reduce latencies" Mike Galbraith <umgwanakikbuti@gmail.com> - 2016-02-29 06:00 +0100
          Re: Softirq priority inversion from "softirq: reduce latencies" Peter Hurley <peter@hurleysoftware.com> - 2016-02-29 16:10 +0100
            Re: Softirq priority inversion from "softirq: reduce latencies" Eric Dumazet <eric.dumazet@gmail.com> - 2016-02-29 16:30 +0100
              Re: Softirq priority inversion from "softirq: reduce latencies" Peter Hurley <peter@hurleysoftware.com> - 2016-02-29 17:00 +0100
                Re: Softirq priority inversion from "softirq: reduce latencies" Eric Dumazet <eric.dumazet@gmail.com> - 2016-02-29 17:30 +0100
                  Re: Softirq priority inversion from "softirq: reduce latencies" Peter Hurley <peter@hurleysoftware.com> - 2016-02-29 19:10 +0100
                    Re: Softirq priority inversion from "softirq: reduce latencies" Eric Dumazet <eric.dumazet@gmail.com> - 2016-02-29 19:30 +0100
                      Re: Softirq priority inversion from "softirq: reduce latencies" Peter Hurley <peter@hurleysoftware.com> - 2016-02-29 20:00 +0100
                        Re: Softirq priority inversion from "softirq: reduce latencies" Thomas Gleixner <tglx@linutronix.de> - 2016-02-29 20:20 +0100
                          Re: Softirq priority inversion from "softirq: reduce latencies" David Miller <davem@davemloft.net> - 2016-02-29 21:30 +0100
                          Re: Softirq priority inversion from "softirq: reduce latencies" Peter Hurley <peter@hurleysoftware.com> - 2016-03-01 00:10 +0100
            Re: Softirq priority inversion from "softirq: reduce latencies" Eric Dumazet <eric.dumazet@gmail.com> - 2016-02-29 16:30 +0100
              Re: Softirq priority inversion from "softirq: reduce latencies" Peter Hurley <peter@hurleysoftware.com> - 2016-02-29 20:20 +0100
                Re: Softirq priority inversion from "softirq: reduce latencies" Eric Dumazet <eric.dumazet@gmail.com> - 2016-02-29 20:50 +0100
            Re: Softirq priority inversion from "softirq: reduce latencies" Mike Galbraith <umgwanakikbuti@gmail.com> - 2016-02-29 16:50 +0100
              Re: Softirq priority inversion from "softirq: reduce latencies" Peter Hurley <peter@hurleysoftware.com> - 2016-02-29 17:00 +0100
                Re: Softirq priority inversion from "softirq: reduce latencies" Eric Dumazet <eric.dumazet@gmail.com> - 2016-02-29 17:30 +0100
            Re: Softirq priority inversion from "softirq: reduce latencies" David Miller <davem@davemloft.net> - 2016-02-29 18:20 +0100

Page 2 of 2 — ← Prev page 1 [2]

#1346140

From: Thomas Gleixner <tglx@linutronix.de>
Date: Mon, 29 Feb 2016 20:14:36 +0100 (CET)

> On Mon, 29 Feb 2016, Peter Hurley wrote:
>> Or flipping your argument on its head, why not just _always_ execute
>> softirq in ksoftirqd?
> 
> Which is what that change effectivley does. And that makes a lot of sense,
> because you get the softirq load under scheduler control and do not let the
> softirq run as a context stealing entity which is completely uncontrollable by
> the scheduler.

+1

[toc] | [prev] | [next] | [standalone]

#1346234

On 02/29/2016 11:14 AM, Thomas Gleixner wrote:
> On Mon, 29 Feb 2016, Peter Hurley wrote:
>> On 02/29/2016 10:24 AM, Eric Dumazet wrote:
>>>> Just to be clear
>>>>
>>>> 		if (time_before(jiffies, end) && !need_resched() &&
>>>> 		    --max_restart)
>>>> 			goto restart;
>>>>
>>>> aborts softirq *even if 0ns have elapsed*, if NET_RX has woken a process.
>>>
>>> Sure, now remove the 1st and 2nd condition.
>>
>> Well just removing the 2nd condition has everything working fine,
>> because that fixes the priority inversion.
> 
> No. It does not fix anything. It hides the shortcomings of the driver.
>  
>> However, when system resources are _not_ contended, it makes no
>> sense to be forced to revert to ksoftirqd resolution, which is strictly
>> intended as fallback.
> 
> No. You claim it is simply because your driver does not handle that situation
> properly.
>  
>> Or flipping your argument on its head, why not just _always_ execute
>> softirq in ksoftirqd?
> 
> Which is what that change effectivley does. And that makes a lot of sense,
> because you get the softirq load under scheduler control and do not let the
> softirq run as a context stealing entity which is completely uncontrollable by
> the scheduler.

Ok, fair enough.

However, charging [in the scheduler sense] very lightweight DMA completion for
one subsystem collectively with very heavyweight NET_RX (doing garbage collection
in softirq!) is hardly ideal.

The alternative being threaded interrupt handlers (which are essentially treated
as 0.000000 scheduler cost).

I just want to make sure that's the conscious choice being made, when the
patches for converting from tasklet to threaded irq start hitting subsystem
maintainers.

Regards,
Peter Hurley

[toc] | [prev] | [next] | [standalone]

#1345907

On lun., 2016-02-29 at 07:03 -0800, Peter Hurley wrote:

> The reason why Eric's change is so effective for Eric's workload is
> that it fixes the problem where NET_RX keeps getting new network packets
> so it keeps looping, servicing more NET_RX softirq.

You have very little idea of what is happening in networking land.

Once hard irq for RX has triggered, we arm a NAPI (NET_RX softirq), and
no more irq will come unless the napi handler ran. Then when NAPI is
complete, we re-allow interrupt to be delivered when a new packet is
coming.

Yes, ksoftirqd runs under load, and this is _wanted_.

Sure, it might add a latency if some high prio task is wanting the same
cpu, but this is exactly the purpose of having multi tasking.

[toc] | [prev] | [next] | [standalone]

#1346103

On 02/29/2016 07:27 AM, Eric Dumazet wrote:
> On lun., 2016-02-29 at 07:03 -0800, Peter Hurley wrote:
> 
>> The reason why Eric's change is so effective for Eric's workload is
>> that it fixes the problem where NET_RX keeps getting new network packets
>> so it keeps looping, servicing more NET_RX softirq.
> 
> You have very little idea of what is happening in networking land.

While that is true, I can read a trace:

  ** already in NET_RX softirq **

  <idle>-0       0..s2   15us : kmem_cache_alloc: call_site=c08378e4 ptr=de55d7c0 bytes_req=192 bytes_alloc=192 gfp_flags=GFP_ATOMIC
  <idle>-0       0..s2   23us : netif_receive_skb_entry: dev=eth0 napi_id=0x0 queue_mapping=0 skbaddr=dca04400 vlan_tagged=0 vlan_proto=0x0000 vlan_tci=0x000
0 protocol=0x0800 ip_summed=0 hash=0x00000000 l4_hash=0 len=88 data_len=0 truesize=1984 mac_header_valid=1 mac_header=-14 nr_frags=0 gso_size=0 gso_type=0x0
  <idle>-0       0..s2   30us+: netif_receive_skb: dev=eth0 skbaddr=dca04400 len=88
  <idle>-0       0d.s5   98us : sched_waking: comm=sshd pid=750 prio=120 target_cpu=000
  <idle>-0       0d.s6  105us : sched_stat_sleep: comm=sshd pid=750 delay=3125230447 [ns]
  <idle>-0       0dns6  110us+: sched_wakeup: comm=sshd pid=750 prio=120 target_cpu=000
  <idle>-0       0dns4  123us+: timer_start: timer=dc940e9c function=tcp_delack_timer expires=9746 [timeout=10] flags=0x00000000
  <idle>-0       0dnH3  150us : irq_handler_entry: irq=176 name=4a100000.ethernet
  <idle>-0       0dnH3  153us : softirq_raise: vec=3 [action=NET_RX]
  <idle>-0       0dnH3  155us : irq_handler_exit: irq=176 ret=handled
  <idle>-0       0dnH3  160us : irq_handler_entry: irq=20 name=49000000.edma_ccint
  <idle>-0       0dnH3  163us : irq_handler_exit: irq=20 ret=handled
  <idle>-0       0.ns2  169us : napi_poll: napi poll on napi struct de465c30 for device eth0
  <idle>-0       0.ns2  171us : softirq_exit: vec=3 [action=NET_RX]


As you can see, NET_RX softirq is re-raised while in NET_RX softirq,
as a result of receiving new packets. So NET_RX will keep looping,
which is what I wrote.


> Once hard irq for RX has triggered, we arm a NAPI (NET_RX softirq), and
> no more irq will come unless the napi handler ran. Then when NAPI is
> complete, we re-allow interrupt to be delivered when a new packet is
> coming.
> 
> Yes, ksoftirqd runs under load, and this is _wanted_.
> 
> Sure, it might add a latency if some high prio task is wanting the same
> cpu, but this is exactly the purpose of having multi tasking.
> 
> 

[toc] | [prev] | [next] | [standalone]

#1346127

On lun., 2016-02-29 at 11:13 -0800, Peter Hurley wrote:
> On 02/29/2016 07:27 AM, Eric Dumazet wrote:
> > On lun., 2016-02-29 at 07:03 -0800, Peter Hurley wrote:
> > 
> >> The reason why Eric's change is so effective for Eric's workload is
> >> that it fixes the problem where NET_RX keeps getting new network packets
> >> so it keeps looping, servicing more NET_RX softirq.
> > 
> > You have very little idea of what is happening in networking land.
> 
> While that is true, I can read a trace:
> 
>   ** already in NET_RX softirq **
> 
>   <idle>-0       0..s2   15us : kmem_cache_alloc: call_site=c08378e4 ptr=de55d7c0 bytes_req=192 bytes_alloc=192 gfp_flags=GFP_ATOMIC
>   <idle>-0       0..s2   23us : netif_receive_skb_entry: dev=eth0 napi_id=0x0 queue_mapping=0 skbaddr=dca04400 vlan_tagged=0 vlan_proto=0x0000 vlan_tci=0x000
> 0 protocol=0x0800 ip_summed=0 hash=0x00000000 l4_hash=0 len=88 data_len=0 truesize=1984 mac_header_valid=1 mac_header=-14 nr_frags=0 gso_size=0 gso_type=0x0
>   <idle>-0       0..s2   30us+: netif_receive_skb: dev=eth0 skbaddr=dca04400 len=88
>   <idle>-0       0d.s5   98us : sched_waking: comm=sshd pid=750 prio=120 target_cpu=000
>   <idle>-0       0d.s6  105us : sched_stat_sleep: comm=sshd pid=750 delay=3125230447 [ns]
>   <idle>-0       0dns6  110us+: sched_wakeup: comm=sshd pid=750 prio=120 target_cpu=000
>   <idle>-0       0dns4  123us+: timer_start: timer=dc940e9c function=tcp_delack_timer expires=9746 [timeout=10] flags=0x00000000
>   <idle>-0       0dnH3  150us : irq_handler_entry: irq=176 name=4a100000.ethernet
>   <idle>-0       0dnH3  153us : softirq_raise: vec=3 [action=NET_RX]
>   <idle>-0       0dnH3  155us : irq_handler_exit: irq=176 ret=handled
>   <idle>-0       0dnH3  160us : irq_handler_entry: irq=20 name=49000000.edma_ccint
>   <idle>-0       0dnH3  163us : irq_handler_exit: irq=20 ret=handled
>   <idle>-0       0.ns2  169us : napi_poll: napi poll on napi struct de465c30 for device eth0
>   <idle>-0       0.ns2  171us : softirq_exit: vec=3 [action=NET_RX]
> 
> 
> As you can see, NET_RX softirq is re-raised while in NET_RX softirq,
> as a result of receiving new packets. So NET_RX will keep looping,
> which is what I wrote.

Well, NET_RX can not be re-raised, it is a single bit flip.

It is 'raised' on this trace because the driver already rearmed the IRQ
so that hard irq handler could fire.

Anyway, it seems you know much better than me, so I will stop answering
your mails on this topic.

[toc] | [prev] | [next] | [standalone]

#1345923

On Mon, 2016-02-29 at 07:03 -0800, Peter Hurley wrote:

> > If I'm listening properly, the root cause is that there is a timing
> > constraint involved, which is being exposed because one softirq raises
> > another (ew).
> 
> Not the case. The softirq is raised from interrupt.

Yeah, saw that on re-read.

> Before Eric's change, when an interrupt raises a new softirq
> while processing another softirq, the new softirq is immediately
> processed *after the existing softirq completes*.

Not necessarily, Eric only changed it from an arbitrary count to an
arbitrary time, so your irq could just as well land when there's no
count left and be up the same creek.

I was more infatuated by the constraint that's left dangling in the
breeze any time processing is deferred to ksoftirqd.

	-Mike

[toc] | [prev] | [next] | [standalone]

#1345937

On 02/29/2016 07:40 AM, Mike Galbraith wrote:
> On Mon, 2016-02-29 at 07:03 -0800, Peter Hurley wrote:
> 
>>> If I'm listening properly, the root cause is that there is a timing
>>> constraint involved, which is being exposed because one softirq raises
>>> another (ew).
>>
>> Not the case. The softirq is raised from interrupt.
> 
> Yeah, saw that on re-read.
> 
>> Before Eric's change, when an interrupt raises a new softirq
>> while processing another softirq, the new softirq is immediately
>> processed *after the existing softirq completes*.
> 
> Not necessarily, Eric only changed it from an arbitrary count to an
> arbitrary time, so your irq could just as well land when there's no
> count left and be up the same creek.

Your misreading the softirq abort logic:
neither 2ms nor a fixed number of loops has elapsed.

All that's happened is the first loop of NET_RX softirq has woken a
process; that is sufficient to abort softirq and defer it for ksoftirqd.

That's why I'm saying this is a priority inversion, and one that
will happen a lot.


> I was more infatuated by the constraint that's left dangling in the
> breeze any time processing is deferred to ksoftirqd.
> 
> 	-Mike
> 

[toc] | [prev] | [next] | [standalone]

#1345959

On lun., 2016-02-29 at 07:58 -0800, Peter Hurley wrote:

> All that's happened is the first loop of NET_RX softirq has woken a
> process; that is sufficient to abort softirq and defer it for ksoftirqd.
> 
> That's why I'm saying this is a priority inversion, and one that
> will happen a lot.

Sure. This will happen every time ksoftirqd is launched.

Get rid of ksoftirqd or renice it so that you can easily be killed by
softirq storm.

[toc] | [prev] | [next] | [standalone]

#1346010

From: Peter Hurley <peter@hurleysoftware.com>
Date: Mon, 29 Feb 2016 07:03:11 -0800

> However, I'm pointing out that Eric's sledgehammer approach to fixing
> the NET_RX softirq bug is having significant side-effects in other
> subsystems.

Either your hardware can handle arbitrary latencies and thus can use
softirqs for event completion successfully, or it can't.

You, my friend, are the one using the sledgehammer.

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | linux.kernel

csiph-web