Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1473748 > unrolled thread

[PATCH] softirq: let ksoftirqd do its job

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  [PATCH] softirq: let ksoftirqd do its job Eric Dumazet <eric.dumazet@gmail.com> - 2016-08-31 19:50 +0200
    Re: [PATCH] softirq: let ksoftirqd do its job Jesper Dangaard Brouer <jbrouer@redhat.com> - 2016-08-31 21:50 +0200
      Re: [PATCH] softirq: let ksoftirqd do its job Eric Dumazet <eric.dumazet@gmail.com> - 2016-08-31 22:50 +0200
        Re: [PATCH] softirq: let ksoftirqd do its job Jesper Dangaard Brouer <jbrouer@redhat.com> - 2016-09-01 00:00 +0200
          Re: [PATCH] softirq: let ksoftirqd do its job Eric Dumazet <eric.dumazet@gmail.com> - 2016-09-01 00:30 +0200
            Re: [PATCH] softirq: let ksoftirqd do its job Rick Jones <rick.jones2@hpe.com> - 2016-09-01 00:50 +0200
              Re: [PATCH] softirq: let ksoftirqd do its job Eric Dumazet <eric.dumazet@gmail.com> - 2016-09-01 01:20 +0200
                Re: [PATCH] softirq: let ksoftirqd do its job Rick Jones <rick.jones2@hpe.com> - 2016-09-01 01:40 +0200
                  Re: [PATCH] softirq: let ksoftirqd do its job Jesper Dangaard Brouer <brouer@redhat.com> - 2016-09-01 12:40 +0200
                    Re: [PATCH] softirq: let ksoftirqd do its job Eric Dumazet <eric.dumazet@gmail.com> - 2016-09-01 15:10 +0200
          Re: [PATCH] softirq: let ksoftirqd do its job Jesper Dangaard Brouer <brouer@redhat.com> - 2016-09-01 13:10 +0200
            Re: [PATCH] softirq: let ksoftirqd do its job Hannes Frederic Sowa <hannes@redhat.com> - 2016-09-01 13:20 +0200
            Re: [PATCH] softirq: let ksoftirqd do its job Peter Zijlstra <peterz@infradead.org> - 2016-09-01 14:00 +0200
              Re: [PATCH] softirq: let ksoftirqd do its job Jesper Dangaard Brouer <brouer@redhat.com> - 2016-09-01 14:40 +0200
                Re: [PATCH] softirq: let ksoftirqd do its job Jesper Dangaard Brouer <brouer@redhat.com> - 2016-09-01 14:40 +0200
                  Re: [PATCH] softirq: let ksoftirqd do its job Peter Zijlstra <peterz@infradead.org> - 2016-09-01 14:50 +0200
                    Re: [PATCH] softirq: let ksoftirqd do its job Jesper Dangaard Brouer <brouer@redhat.com> - 2016-09-01 15:40 +0200
                      Re: [PATCH] softirq: let ksoftirqd do its job Peter Zijlstra <peterz@infradead.org> - 2016-09-01 17:30 +0200
                        Re: [PATCH] softirq: let ksoftirqd do its job Jesper Dangaard Brouer <brouer@redhat.com> - 2016-09-02 10:40 +0200
                  Re: [PATCH] softirq: let ksoftirqd do its job Eric Dumazet <eric.dumazet@gmail.com> - 2016-09-01 15:10 +0200
                    Re: [PATCH] softirq: let ksoftirqd do its job Hannes Frederic Sowa <hannes@redhat.com> - 2016-09-01 15:10 +0200
                      Re: [PATCH] softirq: let ksoftirqd do its job Eric Dumazet <eric.dumazet@gmail.com> - 2016-09-01 15:30 +0200
        Re: [PATCH] softirq: let ksoftirqd do its job Hannes Frederic Sowa <hannes@redhat.com> - 2016-09-01 14:10 +0200
          Re: [PATCH] softirq: let ksoftirqd do its job Eric Dumazet <eric.dumazet@gmail.com> - 2016-09-01 15:00 +0200
    Re: [PATCH] softirq: let ksoftirqd do its job Hannes Frederic Sowa <hannes@redhat.com> - 2016-09-01 14:10 +0200
    Re: [PATCH] softirq: let ksoftirqd do its job David Miller <davem@davemloft.net> - 2016-09-02 08:40 +0200

Page 2 of 2 — ← Prev page 1 [2]

#1474397

On 01.09.2016 14:57, Eric Dumazet wrote:
> On Thu, 2016-09-01 at 14:38 +0200, Jesper Dangaard Brouer wrote:
> 
>> Correction, on the server-under-test, I'm actually running RHEL7.2
>>
>>
>>> How do I verify/check if I have enabled a cpu-cgroup?
>>
>> Hannes says I can look in "/proc/self/cgroup"
>>
>>  $ cat /proc/self/cgroup
>>  7:net_cls:/
>>  6:blkio:/
>>  5:devices:/
>>  4:perf_event:/
>>  3:cpu,cpuacct:/
>>  2:cpuset:/
>>  1:name=systemd:/user.slice/user-1000.slice/session-c1.scope
>>  
>> And that "/" indicate I've not enabled cgroups, right?
>>
> 
> In my experience, I found that times displayed by top are often off for
> softirq processing.
> 
> Before applying my patch, top shows very small amount of cpu time for
> udp_rcv and ksoftirqd/0 , while obviously cpu 0 is completely busy.
> 
> Make sure to try latest Linus tree, as I did yesterday, because
> apparently things are better than a few weeks back.
> 
> BTW, even 'perf top' has sometimes problems showing me cycles spent in
> softirq. I need to make sure the cpu processing NIC interrupts also
> spend cycles in some user space program to get meaningful results.

I think that ksoftirqd time is actually accounted to system:

excerpt from irqtime_account_process_tick in kernel/sched/cputime.c

	if (this_cpu_ksoftirqd() == p) {
		/*
		 * ksoftirqd time do not get accounted in cpu_softirq_time.
		 * So, we have to handle it separately here.
		 * Also, p->stime needs to be updated for ksoftirqd.
		 */
		__account_system_time(p, cputime, scaled, CPUTIME_SOFTIRQ);
	} else if (user_tick) {

[toc] | [prev] | [next] | [standalone]

#1474402

On Thu, 2016-09-01 at 15:00 +0200, Hannes Frederic Sowa wrote:
> On 01.09.2016 14:57, Eric Dumazet wrote:
> > On Thu, 2016-09-01 at 14:38 +0200, Jesper Dangaard Brouer wrote:
> > 
> >> Correction, on the server-under-test, I'm actually running RHEL7.2
> >>
> >>
> >>> How do I verify/check if I have enabled a cpu-cgroup?
> >>
> >> Hannes says I can look in "/proc/self/cgroup"
> >>
> >>  $ cat /proc/self/cgroup
> >>  7:net_cls:/
> >>  6:blkio:/
> >>  5:devices:/
> >>  4:perf_event:/
> >>  3:cpu,cpuacct:/
> >>  2:cpuset:/
> >>  1:name=systemd:/user.slice/user-1000.slice/session-c1.scope
> >>  
> >> And that "/" indicate I've not enabled cgroups, right?
> >>
> > 
> > In my experience, I found that times displayed by top are often off for
> > softirq processing.
> > 
> > Before applying my patch, top shows very small amount of cpu time for
> > udp_rcv and ksoftirqd/0 , while obviously cpu 0 is completely busy.
> > 
> > Make sure to try latest Linus tree, as I did yesterday, because
> > apparently things are better than a few weeks back.
> > 
> > BTW, even 'perf top' has sometimes problems showing me cycles spent in
> > softirq. I need to make sure the cpu processing NIC interrupts also
> > spend cycles in some user space program to get meaningful results.
> 
> I think that ksoftirqd time is actually accounted to system:
> 
> excerpt from irqtime_account_process_tick in kernel/sched/cputime.c
> 
> 	if (this_cpu_ksoftirqd() == p) {
> 		/*
> 		 * ksoftirqd time do not get accounted in cpu_softirq_time.
> 		 * So, we have to handle it separately here.
> 		 * Also, p->stime needs to be updated for ksoftirqd.
> 		 */
> 		__account_system_time(p, cputime, scaled, CPUTIME_SOFTIRQ);
> 	} else if (user_tick) {
> 

Tell me more about kernel/sched/cputime.c stability over recent linux
versions ;)

git log --oneline v4.2.. kernel/sched/cputime.c
03cbc732639ddcad15218c4b2046d255851ff1e3 sched/cputime: Resync steal time when guest & host lose sync
173be9a14f7b2e901cf77c18b1aafd4d672e9d9e sched/cputime: Fix NO_HZ_FULL getrusage() monotonicity regression
26f2c75cd2cf10a6120ef02ca9a94db77cc9c8e0 sched/cputime: Fix omitted ticks passed in parameter
f9bcf1e0e0145323ba2cf72ecad5264ff3883eb1 sched/cputime: Fix steal time accounting
08fd8c17686c6b09fa410a26d516548dd80ff147 Merge tag 'for-linus-4.8-rc0-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip
553bf6bbfd8a540c70aee28eb50e24caff456a03 sched/cputime: Drop local_irq_save/restore from irqtime_account_irq()
0cfdf9a198b0d4f5ad6c87d894db7830b796b2cc sched/cputime: Clean up the old vtime gen irqtime accounting completely
b58c35840521bb02b150e1d0d34ca9197f8b7145 sched/cputime: Replace VTIME_GEN irq time code with IRQ_TIME_ACCOUNTING code
57430218317e5b280a80582a139b26029c25de6c sched/cputime: Count actually elapsed irq & softirq time
ecb23dc6f2eff0ce64dd60351a81f376f13b12cc xen: add steal_clock support on x86
807e5b80687c06715d62df51a5473b231e3e8b15 sched/cputime: Add steal time support to full dynticks CPU time accounting
f9c904b7613b8b4c85b10cd6b33ad41b2843fa9d sched/cputime: Fix steal_account_process_tick() to always return jiffies
ff9a9b4c4334b53b52ee9279f30bd5dd92ea9bdd sched, time: Switch VIRT_CPU_ACCOUNTING_GEN to jiffy granularity
c9bed1cf51011c815d88288b774865d013ca78a8 Merge tag 'for-linus-4.5-rc0-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip
1fe7c4ef88bd32e039f5f4126537c3f20c340414 missing include asm/paravirt.h in cputime.c
b7ce2277f087fd052e7e1bbf432f7fecbee82bb6 sched/cputime: Convert vtime_seqlock to seqcount
e592539466380279a9e6e6fdfe4545aa54f22593 sched/cputime: Introduce vtime accounting check for readers
55dbdcfa05533f44c9416070b8a9f6432b22314a sched/cputime: Rename vtime_accounting_enabled() to vtime_accounting_cpu_enabled()
cab245d68c38afff1a4c4d018ab7e1d316982f5d sched/cputime: Correctly handle task guest time on housekeepers
7098c1eac75dc03fdbb7249171a6e68ce6044a5a sched/cputime: Clarify vtime symbols and document them
7877a0ba5ec63c7b0111b06c773f1696fa17b35a sched/cputime: Remove extra cost in task_cputime()
2541117b0cf79977fa11a0d6e17d61010677bd7b sched/cputime: Fix invalid gtime in proc
9eec50b8bbe1535c440a1ee88c1958f78fc55957 kvm/x86: Hyper-V HV_X64_MSR_VP_RUNTIME support
9d7fb04276481c59610983362d8e023d262b58ca sched/cputime: Guarantee stime + utime == rtime

[toc] | [prev] | [next] | [standalone]

#1474361

On 31.08.2016 22:42, Eric Dumazet wrote:
> On Wed, 2016-08-31 at 21:40 +0200, Jesper Dangaard Brouer wrote:
> 
>> I can confirm the improvement of approx 900Kpps (no wonder people have
>> been complaining about DoS against UDP/DNS servers).
>>
>> BUT during my extensive testing, of this patch, I also think that we
>> have not gotten to the bottom of this.  I was expecting to see a higher
>> (collective) PPS number as I add more UDP servers, but I don't.
>>
>> Running many UDP netperf's with command:
>>  super_netperf 4 -H 198.18.50.3 -l 120 -t UDP_STREAM -T 0,0 -- -m 1472 -n -N
> 
> Are you sure sender can send fast enough ?
> 
>>
>> With 'top' I can see ksoftirq are still getting a higher %CPU time:
>>
>>     PID   %CPU     TIME+  COMMAND
>>      3   36.5   2:28.98  ksoftirqd/0
>>  10724    9.6   0:01.05  netserver
>>  10722    9.3   0:01.05  netserver
>>  10723    9.3   0:01.05  netserver
>>  10725    9.3   0:01.05  netserver
> 
> Looks much better on my machine, with "udprcv -n 4" (using 4 threads,
> and 4 sockets using SO_REUSEPORT)

Would it make sense to include used socket backlog in udp socket lookup
compute_score calculation? Just want to throw out the idea, I actually
could imagine to also cause bad side effects.

[toc] | [prev] | [next] | [standalone]

#1474388

On Thu, 2016-09-01 at 14:05 +0200, Hannes Frederic Sowa wrote:

> Would it make sense to include used socket backlog in udp socket lookup
> compute_score calculation? Just want to throw out the idea, I actually
> could imagine to also cause bad side effects.

Hopefully we can get rid of the backlog for UDP, by no longer having to
lock the socket in RX path, and perform memory charging in a better way.

The backlog for TCP is problematic for high speed flows, and for UDP it
is problematic in flood situations as a single recvmsg() might have to
process thousands of skbs before returning to user space.

What you suggest is going to be difficult :

1) Packets of a 5-tuple (eg QUIC flow) wont all land to the same silo,
and will cause reorders or application issues.

2) SO_ATTACH_REUSEPORT_CBPF wont have access to the socket(s) backlog to
perform the choice.

Thanks.

[toc] | [prev] | [next] | [standalone]

#1474362

On 31.08.2016 19:42, Eric Dumazet wrote:
> From: Eric Dumazet <edumazet@google.com>
> 
> A while back, Paolo and Hannes sent an RFC patch adding threaded-able
> napi poll loop support : (https://patchwork.ozlabs.org/patch/620657/) 
> 
> The problem seems to be that softirqs are very aggressive and are often
> handled by the current process, even if we are under stress and that
> ksoftirqd was scheduled, so that innocent threads would have more chance
> to make progress.
> 
> This patch makes sure that if ksoftirq is running, we let it
> perform the softirq work.
> 
> Jonathan Corbet summarized the issue in https://lwn.net/Articles/687617/
> 
> Tested:
> 
>  - NIC receiving traffic handled by CPU 0
>  - UDP receiver running on CPU 0, using a single UDP socket.
>  - Incoming flood of UDP packets targeting the UDP socket.
> 
> Before the patch, the UDP receiver could almost never get cpu cycles and
> could only receive ~2,000 packets per second.
> 
> After the patch, cpu cycles are split 50/50 between user application and
> ksoftirqd/0, and we can effectively read ~900,000 packets per second,
> a huge improvement in DOS situation. (Note that more packets are now
> dropped by the NIC itself, since the BH handlers get less cpu cycles to
> drain RX ring buffer)
> 
> Since the load runs in well identified threads context, an admin can
> more easily tune process scheduling parameters if needed.
> 
> Reported-by: Paolo Abeni <pabeni@redhat.com>
> Reported-by: Hannes Frederic Sowa <hannes@stressinduktion.org>
> Signed-off-by: Eric Dumazet <edumazet@google.com>
> Cc: David Miller <davem@davemloft.net
> Cc: Jesper Dangaard Brouer <jbrouer@redhat.com>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Rik van Riel <riel@redhat.com>

Acked-by: Hannes Frederic Sowa <hannes@stressinduktion.org>

Thanks,
Hannes

[toc] | [prev] | [next] | [standalone]

#1474911

From: Eric Dumazet <eric.dumazet@gmail.com>
Date: Wed, 31 Aug 2016 10:42:29 -0700

> From: Eric Dumazet <edumazet@google.com>
> 
> A while back, Paolo and Hannes sent an RFC patch adding threaded-able
> napi poll loop support : (https://patchwork.ozlabs.org/patch/620657/) 
> 
> The problem seems to be that softirqs are very aggressive and are often
> handled by the current process, even if we are under stress and that
> ksoftirqd was scheduled, so that innocent threads would have more chance
> to make progress.
> 
> This patch makes sure that if ksoftirq is running, we let it
> perform the softirq work.
> 
> Jonathan Corbet summarized the issue in https://lwn.net/Articles/687617/
> 
> Tested:
> 
>  - NIC receiving traffic handled by CPU 0
>  - UDP receiver running on CPU 0, using a single UDP socket.
>  - Incoming flood of UDP packets targeting the UDP socket.
> 
> Before the patch, the UDP receiver could almost never get cpu cycles and
> could only receive ~2,000 packets per second.
> 
> After the patch, cpu cycles are split 50/50 between user application and
> ksoftirqd/0, and we can effectively read ~900,000 packets per second,
> a huge improvement in DOS situation. (Note that more packets are now
> dropped by the NIC itself, since the BH handlers get less cpu cycles to
> drain RX ring buffer)
> 
> Since the load runs in well identified threads context, an admin can
> more easily tune process scheduling parameters if needed.
> 
> Reported-by: Paolo Abeni <pabeni@redhat.com>
> Reported-by: Hannes Frederic Sowa <hannes@stressinduktion.org>
> Signed-off-by: Eric Dumazet <edumazet@google.com>

I'm just kind of assuming this won't go through my tree, but I can take
it if that's what everyone agrees to.

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | linux.kernel

csiph-web