Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1371865 > unrolled thread

[PATCH RFC] select_idle_sibling experiments

Back to article view | Back to linux.kernel

Contents

  [PATCH RFC] select_idle_sibling experiments Chris Mason <clm@fb.com> - 2016-04-05 20:20 +0200
    Re: [PATCH RFC] select_idle_sibling experiments Bastien Bastien Philbert <bastienphilbert@gmail.com> - 2016-04-05 20:50 +0200
      Re: [PATCH RFC] select_idle_sibling experiments Chris Mason <clm@fb.com> - 2016-04-05 21:30 +0200
    Re: [PATCH RFC] select_idle_sibling experiments Matt Fleming <matt@codeblueprint.co.uk> - 2016-04-05 22:10 +0200
      Re: [PATCH RFC] select_idle_sibling experiments Bastien Philbert <bastienphilbert@gmail.com> - 2016-04-05 23:10 +0200
      Re: [PATCH RFC] select_idle_sibling experiments Chris Mason <clm@fb.com> - 2016-04-06 02:50 +0200
    Re: [PATCH RFC] select_idle_sibling experiments Mike Galbraith <mgalbraith@suse.de> - 2016-04-06 09:30 +0200
      Re: [PATCH RFC] select_idle_sibling experiments Chris Mason <clm@fb.com> - 2016-04-06 15:40 +0200
      Re: [PATCH RFC] select_idle_sibling experiments Chris Mason <clm@fb.com> - 2016-04-09 19:40 +0200
        Re: [PATCH RFC] select_idle_sibling experiments Matt Fleming <matt@codeblueprint.co.uk> - 2016-04-12 23:50 +0200
          Re: [PATCH RFC] select_idle_sibling experiments Mike Galbraith <mgalbraith@suse.de> - 2016-04-13 05:50 +0200
            Re: [PATCH RFC] select_idle_sibling experiments Chris Mason <clm@fb.com> - 2016-04-13 18:00 +0200
    Re: [PATCH RFC] select_idle_sibling experiments Chris Mason <clm@fb.com> - 2016-04-07 17:20 +0200
    sched: tweak select_idle_sibling to look for idle threads Chris Mason <clm@fb.com> - 2016-04-09 21:10 +0200
      Re: sched: tweak select_idle_sibling to look for idle threads Mike Galbraith <mgalbraith@suse.de> - 2016-04-10 12:10 +0200
        Re: sched: tweak select_idle_sibling to look for idle threads Chris Mason <clm@fb.com> - 2016-04-10 14:40 +0200
          Re: sched: tweak select_idle_sibling to look for idle threads Mike Galbraith <mgalbraith@suse.de> - 2016-04-10 14:50 +0200
        Re: sched: tweak select_idle_sibling to look for idle threads Chris Mason <clm@fb.com> - 2016-04-10 22:00 +0200
          Re: sched: tweak select_idle_sibling to look for idle threads Mike Galbraith <mgalbraith@suse.de> - 2016-04-11 07:00 +0200
            Re: sched: tweak select_idle_sibling to look for idle threads Chris Mason <clm@fb.com> - 2016-04-12 02:40 +0200
              Re: sched: tweak select_idle_sibling to look for idle threads Mike Galbraith <mgalbraith@suse.de> - 2016-04-12 06:50 +0200
                Re: sched: tweak select_idle_sibling to look for idle threads Chris Mason <clm@fb.com> - 2016-04-12 15:30 +0200
                  Re: sched: tweak select_idle_sibling to look for idle threads Mike Galbraith <mgalbraith@suse.de> - 2016-04-12 20:20 +0200
                    Re: sched: tweak select_idle_sibling to look for idle threads Chris Mason <clm@fb.com> - 2016-04-12 22:10 +0200
                      Re: sched: tweak select_idle_sibling to look for idle threads Mike Galbraith <mgalbraith@suse.de> - 2016-04-13 05:20 +0200
                        Re: sched: tweak select_idle_sibling to look for idle threads Chris Mason <clm@fb.com> - 2016-04-13 15:50 +0200
                          Re: sched: tweak select_idle_sibling to look for idle threads Mike Galbraith <mgalbraith@suse.de> - 2016-04-13 16:30 +0200
                            Re: sched: tweak select_idle_sibling to look for idle threads Chris Mason <clm@fb.com> - 2016-04-13 16:40 +0200
                              Re: sched: tweak select_idle_sibling to look for idle threads Mike Galbraith <mgalbraith@suse.de> - 2016-04-13 17:10 +0200
                                Re: sched: tweak select_idle_sibling to look for idle threads Mike Galbraith <mgalbraith@suse.de> - 2016-04-13 17:40 +0200

Page 2 of 2 — ← Prev page 1 [2]

#1376446 — Re: sched: tweak select_idle_sibling to look for idle threads

On Mon, 2016-04-11 at 20:30 -0400, Chris Mason wrote:
> On Mon, Apr 11, 2016 at 06:54:21AM +0200, Mike Galbraith wrote:

> > > Ok, I was able to reproduce this by stuffing tbench_srv and tbench onto
> > > just socket 0.  Version 2 below fixes things for me, but I'm hoping
> > > someone can suggest a way to get task_hot() buddy checks without the rq
> > > lock.
> > > 
> > > I haven't run this on production loads yet, but our 4.0 patch for this
> > > uses task_hot(), so I'd expect it to be on par.  If this doesn't fix it
> > > for you, I'll dig up a similar machine on Monday.
> > 
> > My box stopped caring.  I personally would be reluctant to apply it
> > without a "you asked for it" button or a large pile of benchmark
> > results.  Lock banging or not, full scan existing makes me nervous.
> 
> 
> We can use a bitmap at the socket level to keep track of which cpus are
> idle.  I'm sure there are better places for the array and better ways to
> allocate, this is just a rough cut to make sure the idle tracking works.

See e0a79f529d5b:

      pre   15.22 MB/sec 1 procs
      post 252.01 MB/sec 1 procs

You can make traverse cycles go away, but those cycles, while precious,
are not the most costly cycles.  The above was 1 tbench pair in an
otherwise idle box.. ie it wasn't traverse cycles that demolished it.

	-Mike

(p.s. SCHED_IDLE is dinky bandwidth fair class)

[toc] | [prev] | [next] | [standalone]

#1376795 — Re: sched: tweak select_idle_sibling to look for idle threads

On Tue, Apr 12, 2016 at 06:44:08AM +0200, Mike Galbraith wrote:
> On Mon, 2016-04-11 at 20:30 -0400, Chris Mason wrote:
> > On Mon, Apr 11, 2016 at 06:54:21AM +0200, Mike Galbraith wrote:
> 
> > > > Ok, I was able to reproduce this by stuffing tbench_srv and tbench onto
> > > > just socket 0.  Version 2 below fixes things for me, but I'm hoping
> > > > someone can suggest a way to get task_hot() buddy checks without the rq
> > > > lock.
> > > > 
> > > > I haven't run this on production loads yet, but our 4.0 patch for this
> > > > uses task_hot(), so I'd expect it to be on par.  If this doesn't fix it
> > > > for you, I'll dig up a similar machine on Monday.
> > > 
> > > My box stopped caring.  I personally would be reluctant to apply it
> > > without a "you asked for it" button or a large pile of benchmark
> > > results.  Lock banging or not, full scan existing makes me nervous.
> > 
> > 
> > We can use a bitmap at the socket level to keep track of which cpus are
> > idle.  I'm sure there are better places for the array and better ways to
> > allocate, this is just a rough cut to make sure the idle tracking works.
> 
> See e0a79f529d5b:
> 
>       pre   15.22 MB/sec 1 procs
>       post 252.01 MB/sec 1 procs
> 
> You can make traverse cycles go away, but those cycles, while precious,
> are not the most costly cycles.  The above was 1 tbench pair in an
> otherwise idle box.. ie it wasn't traverse cycles that demolished it.

Agreed, this is why the decision not to scan is so important.  But while
I've been describing this patch in terms of latency, latency is really
the symptom instead of the goal.  Without these patches, workloads that
do want to fully utilize the hardware are basically getting one fewer
core of utilization.  It's true that we define 'fully utilize' with an
upper bound on application response time, but we're not talking high
frequency trading here.

It clearly shows up in our graphs.  CPU idle is higher (the lost core),
CPU user time is lower, average system load is higher (procs waiting on
a fewer number of core).

We measure this internally with scheduling latency because that's the
easiest way to talk about it across a wide variety of hardware.

> 
> 	-Mike
> 
> (p.s. SCHED_IDLE is dinky bandwidth fair class)

Ugh, not my best quick patch, but you get the idea I was going for.  I
can always add the tunable to flip things on/off but I'd prefer that we
find a good set of defaults, mostly so the FB production runtime is the
common config instead of the special snowflake.

-chris

[toc] | [prev] | [next] | [standalone]

#1377084 — Re: sched: tweak select_idle_sibling to look for idle threads

On Tue, 2016-04-12 at 09:27 -0400, Chris Mason wrote:
> I
> can always add the tunable to flip things on/off but I'd prefer that we
> find a good set of defaults, mostly so the FB production runtime is the
> common config instead of the special snowflake.

Yeah, generic has a much better chance to actually get merged, just
need a very solid chain on the lurking beast from hell.  Hm...

The last time we went through this, the problem child was the waker of
many in your load.  With tiny twiddle to wake_wide(), all was allegedly
well, or at least that's the impression I was left with.  That leads me
to a pseudo-random thought: iff that waker of many is still at the
root, you could try using wake_wide() as the full search trigger, which
should shrink the attack surface available to the horror-from-hell
quite a lot.  Just a thought.

	-Mike

[toc] | [prev] | [next] | [standalone]

#1377139 — Re: sched: tweak select_idle_sibling to look for idle threads

On Tue, Apr 12, 2016 at 08:16:17PM +0200, Mike Galbraith wrote:
> On Tue, 2016-04-12 at 09:27 -0400, Chris Mason wrote:
> > I
> > can always add the tunable to flip things on/off but I'd prefer that we
> > find a good set of defaults, mostly so the FB production runtime is the
> > common config instead of the special snowflake.
> 
> Yeah, generic has a much better chance to actually get merged, just
> need a very solid chain on the lurking beast from hell.  Hm...
> 
> The last time we went through this, the problem child was the waker of
> many in your load.  With tiny twiddle to wake_wide(), all was allegedly
> well, or at least that's the impression I was left with.  That leads me
> to a pseudo-random thought: iff that waker of many is still at the
> root, you could try using wake_wide() as the full search trigger, which
> should shrink the attack surface available to the horror-from-hell
> quite a lot.  Just a thought.

I think that if we're worried about the cost of the idle scan for this
workload, find_idlest_group() is either going to hurt much more, or not
search enough CPUs to find the idle one.

But I'm happy to try patches or other ideas, I have a fixed version of
the bitmap one going through production benchmarks now.

-chris

[toc] | [prev] | [next] | [standalone]

#1377485 — Re: sched: tweak select_idle_sibling to look for idle threads

On Tue, 2016-04-12 at 16:07 -0400, Chris Mason wrote:

> I think that if we're worried about the cost of the idle scan for this
> workload, find_idlest_group() is either going to hurt much more, or not
> search enough CPUs to find the idle one.

find_idlest_group()?  No no no, that's not what I mean at all.

wake_wide() identifies loads that really want to spread out, thus turns
off affine wakeups.  We still call select_idle_sibling(), only
difference being that target is the original cpu, not the waking cpu. 
 Given making that wide connection bidirectional helped FB's load, it
seems reasonable that passing wide information to select_idle_sibling()
would have a good chance of hitting the candidate that stands to gain
from a full socket scan, while also keeping that cache scrambling scan
far away from the rest. 

> But I'm happy to try patches or other ideas, I have a fixed version of
> the bitmap one going through production benchmarks now.

Making that wide/full search cheap is still good, because wake_wide()
also identifies interrupt sources that are waking many, so cheap wide
search should increase utilization there as well.  The thought was to
just make the wide thing have a tad wider effect on what it already
does affect.. and hope that doesn't demolish anything.

	-Mike

[toc] | [prev] | [next] | [standalone]

#1377956 — Re: sched: tweak select_idle_sibling to look for idle threads

On Wed, Apr 13, 2016 at 05:18:51AM +0200, Mike Galbraith wrote:
> On Tue, 2016-04-12 at 16:07 -0400, Chris Mason wrote:
> 
> > I think that if we're worried about the cost of the idle scan for this
> > workload, find_idlest_group() is either going to hurt much more, or not
> > search enough CPUs to find the idle one.
> 
> find_idlest_group()?  No no no, that's not what I mean at all.
> 
> wake_wide() identifies loads that really want to spread out, thus turns
> off affine wakeups.  We still call select_idle_sibling(), only
> difference being that target is the original cpu, not the waking cpu. 

Ah ok, I see what you mean now.

>  Given making that wide connection bidirectional helped FB's load, it
> seems reasonable that passing wide information to select_idle_sibling()
> would have a good chance of hitting the candidate that stands to gain
> from a full socket scan, while also keeping that cache scrambling scan
> far away from the rest. 
> 
> > But I'm happy to try patches or other ideas, I have a fixed version of
> > the bitmap one going through production benchmarks now.

[ benchmarks say it needs more fixing, ick ]

> 
> Making that wide/full search cheap is still good, because wake_wide()
> also identifies interrupt sources that are waking many, so cheap wide
> search should increase utilization there as well.  The thought was to
> just make the wide thing have a tad wider effect on what it already
> does affect.. and hope that doesn't demolish anything.

So you're interested in numbers where we pass the wake_wide decision
into select_idle_sibling(), and then use that instead of (or in addition
to?) my should_scan_idle() function?

I agree we may need to tweak wake_wide, since most of our wakeups now
are failed affine wakeups.  But, the differences are in p99, so I'll
probably need to get some better metrics.

-chris

[toc] | [prev] | [next] | [standalone]

#1377988 — Re: sched: tweak select_idle_sibling to look for idle threads

On Wed, 2016-04-13 at 09:44 -0400, Chris Mason wrote:

> So you're interested in numbers where we pass the wake_wide decision
> into select_idle_sibling(), and then use that instead of (or in addition
> to?) my should_scan_idle() function?

Yeah, I was thinking instead of, and hoping that would be enough.

> I agree we may need to tweak wake_wide, since most of our wakeups now
> are failed affine wakeups.

What exactly do you mean by failed affine wakeups?  Failed because
wake_wide() said we don't want one, or because wake_affine() said we
can't have one?  If the later, my thought bubble may have just burst,
but it still "feels" right.

	-Mike

[toc] | [prev] | [next] | [standalone]

#1377993 — Re: sched: tweak select_idle_sibling to look for idle threads

On Wed, Apr 13, 2016 at 04:22:58PM +0200, Mike Galbraith wrote:
> On Wed, 2016-04-13 at 09:44 -0400, Chris Mason wrote:
> 
> > So you're interested in numbers where we pass the wake_wide decision
> > into select_idle_sibling(), and then use that instead of (or in addition
> > to?) my should_scan_idle() function?
> 
> Yeah, I was thinking instead of, and hoping that would be enough.

I'm definitely up for experimenting with different tests to decide when
to scan idle.  I'll have to wait until after lsf/vault, but I can layout
a bunch of tests.

> 
> > I agree we may need to tweak wake_wide, since most of our wakeups now
> > are failed affine wakeups.
> 
> What exactly do you mean by failed affine wakeups?  Failed because
> wake_wide() said we don't want one, or because wake_affine() said we
> can't have one?  If the later, my thought bubble may have just burst,
> but it still "feels" right.

I mean this number:

schedstat_inc(p, se.statistics.nr_wakeups_affine_attempts);

Is much much much higher than this number:

schedstat_inc(p, se.statistics.nr_wakeups_affine);

So, wake_affine said we can't have one.  I made a script to sum it up
across all the threads of the webserver workload.

-chris

[toc] | [prev] | [next] | [standalone]

#1378029 — Re: sched: tweak select_idle_sibling to look for idle threads

On Wed, 2016-04-13 at 10:36 -0400, Chris Mason wrote:
> On Wed, Apr 13, 2016 at 04:22:58PM +0200, Mike Galbraith wrote:

> > What exactly do you mean by failed affine wakeups?  Failed because
> > wake_wide() said we don't want one, or because wake_affine() said we
> > can't have one?  If the later, my thought bubble may have just burst,
> > but it still "feels" right.
> 
> I mean this number:
> 
> schedstat_inc(p, se.statistics.nr_wakeups_affine_attempts);
> 
> Is much much much higher than this number:
> 
> schedstat_inc(p, se.statistics.nr_wakeups_affine);
> 
> So, wake_affine said we can't have one.  I made a script to sum it up
> across all the threads of the webserver workload.

Hm, ok, that doesn't really tell us more than there's more to the load
than the 1:N bits that wake_wide() apparently did identify fairly well
last go, so targeting them still might help.

	-Mike

[toc] | [prev] | [next] | [standalone]

#1378055 — Re: sched: tweak select_idle_sibling to look for idle threads

Another thing you could try is looking at your avg_idle, and twiddling
sched_migration_cost_ns to crank up idle balancing a bit.

	-Mike

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | linux.kernel

csiph-web