Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1400430

Re: [RFC][PATCH 4/7] sched: Replace sd_busy/nr_busy_cpus with sched_domain_shared

Show all headers | View raw

On Thu, 2016-05-12 at 13:33 +0200, Peter Zijlstra wrote:
> On Thu, May 12, 2016 at 09:07:52PM +1000, Michael Neuling wrote:
> > 
> > On Thu, 2016-05-12 at 07:07 +0200, Peter Zijlstra wrote:
> > 
> > > 
> > > But as per the above, Power7 and Power8 have explicit logic to share
> > > the
> > > per-core L3 with the other cores.
> > > 
> > > How effective is that? From some of the slides/documents i've looked
> > > at
> > > the L3s are connected with a high-speed fabric. Suggesting that the
> > > cross-core sharing should be fairly efficient.
> > I'm not sure.  I thought it was mostly private but if another core was
> > sleeping or not experiencing much cache pressure, another core could
> > use it
> > for some things. But I'm fuzzy on the the exact properties, sorry.
> Right; I'm going by bits and pieces found on the tubes, so I'm just
> guessing ;-)
> 
> But it sounds like these L3s are nowhere close to what Intel does with
> their L3, where each core has an L3 slice, and slices are connected on a
> ring to form a unified/shared cache across all cores.
> 
> http://www.realworldtech.com/sandy-bridge/8/

The POWER8 user manual is what you want to look at:

https://www.setphaserstostun.org/power8/POWER8_UM_v1.3_16MAR2016_pub.pdf

There is a section 10. "L3 Cache Overview" starting on page 128.  In there
it talks about L3.0 which is using the local cores L3.  L3.1 which is using
some other cores L3.

Once the L3.0 is full, we can cast out to an L3.1 (ie. the cache on another
core).  L3.1 can also provide data for reads.

ECO mode (section 10.4) is what I was talking about for sleeping/unused
cores.  That's more of a boot time (firmware option) than something we can
dynamically play with at runtime (I believe), so it's not something I think
is relevant here.

> > 
> > > 
> > > In which case it would make sense to treat/model the combined L3 as a
> > > single large LLC covering all cores.
> > Are you thinking it would be much cheaper to migrate a task to another
> > core
> > inside this chip, than to off chip?
> Basically; and if so, if its cheap enough to shoot a task to an idle
> core to avoid queueing. Assuming there still is some cache residency on
> the old core, the inter-core fill should be much cheaper than fetching
> it off package (either remote cache or dram).

So I think that will apply on POWER8.

In 10.4.2 it says "The L3.1 ECO Caches will be snooped and provide
intervention data similar to the L2 and L3.0 caches on the
chip"  That should be much faster than going to another chip or DIMM.

So migrating to another core on the same chip should be faster than off
chip.

Mikey




> Or at least; so goes my reasoning based on my google results.
> 

Back to linux.kernel | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

[RFC][PATCH 0/7] sched: select_idle_siblings rewrite Peter Zijlstra <peterz@infradead.org> - 2016-05-09 13:00 +0200
  [RFC][PATCH 4/7] sched: Replace sd_busy/nr_busy_cpus with sched_domain_shared Peter Zijlstra <peterz@infradead.org> - 2016-05-09 13:00 +0200
    Re: [RFC][PATCH 4/7] sched: Replace sd_busy/nr_busy_cpus with  sched_domain_shared Matt Fleming <matt@codeblueprint.co.uk> - 2016-05-11 14:00 +0200
      Re: [RFC][PATCH 4/7] sched: Replace sd_busy/nr_busy_cpus with  sched_domain_shared Peter Zijlstra <peterz@infradead.org> - 2016-05-11 14:40 +0200
        Re: [RFC][PATCH 4/7] sched: Replace sd_busy/nr_busy_cpus with  sched_domain_shared Peter Zijlstra <peterz@infradead.org> - 2016-05-11 20:20 +0200
        Re: [RFC][PATCH 4/7] sched: Replace sd_busy/nr_busy_cpus with  sched_domain_shared Peter Zijlstra <peterz@infradead.org> - 2016-05-11 20:30 +0200
          Re: [RFC][PATCH 4/7] sched: Replace sd_busy/nr_busy_cpus with  sched_domain_shared Michael Neuling <mikey@neuling.org> - 2016-05-12 04:10 +0200
            Re: [RFC][PATCH 4/7] sched: Replace sd_busy/nr_busy_cpus with  sched_domain_shared Peter Zijlstra <peterz@infradead.org> - 2016-05-12 07:10 +0200
              Re: [RFC][PATCH 4/7] sched: Replace sd_busy/nr_busy_cpus with  sched_domain_shared Michael Neuling <mikey@neuling.org> - 2016-05-12 13:10 +0200
                Re: [RFC][PATCH 4/7] sched: Replace sd_busy/nr_busy_cpus with  sched_domain_shared Peter Zijlstra <peterz@infradead.org> - 2016-05-12 13:40 +0200
                Re: [RFC][PATCH 4/7] sched: Replace sd_busy/nr_busy_cpus with  sched_domain_shared Michael Neuling <mikey@neuling.org> - 2016-05-13 02:20 +0200
                Re: [RFC][PATCH 4/7] sched: Replace sd_busy/nr_busy_cpus with  sched_domain_shared Peter Zijlstra <peterz@infradead.org> - 2016-05-16 16:10 +0200
                Re: [RFC][PATCH 4/7] sched: Replace sd_busy/nr_busy_cpus with  sched_domain_shared Peter Zijlstra <peterz@infradead.org> - 2016-05-17 12:30 +0200
                Re: [RFC][PATCH 4/7] sched: Replace sd_busy/nr_busy_cpus with  sched_domain_shared Srikar Dronamraju <srikar@linux.vnet.ibm.com> - 2016-05-17 13:00 +0200
                Re: [RFC][PATCH 4/7] sched: Replace sd_busy/nr_busy_cpus with  sched_domain_shared Peter Zijlstra <peterz@infradead.org> - 2016-05-17 13:20 +0200
      Re: [RFC][PATCH 4/7] sched: Replace sd_busy/nr_busy_cpus with  sched_domain_shared Peter Zijlstra <peterz@infradead.org> - 2016-05-11 19:40 +0200
        Re: [RFC][PATCH 4/7] sched: Replace sd_busy/nr_busy_cpus with  sched_domain_shared Matt Fleming <matt@codeblueprint.co.uk> - 2016-05-11 20:10 +0200
    Re: [RFC][PATCH 4/7] sched: Replace sd_busy/nr_busy_cpus with  sched_domain_shared Dietmar Eggemann <dietmar.eggemann@arm.com> - 2016-05-16 17:40 +0200
      Re: [RFC][PATCH 4/7] sched: Replace sd_busy/nr_busy_cpus with  sched_domain_shared Peter Zijlstra <peterz@infradead.org> - 2016-05-16 19:10 +0200
        Re: [RFC][PATCH 4/7] sched: Replace sd_busy/nr_busy_cpus with  sched_domain_shared Dietmar Eggemann <dietmar.eggemann@arm.com> - 2016-05-16 19:30 +0200
  [RFC][PATCH 1/7] sched: Remove unused @cpu argument from destroy_sched_domain*() Peter Zijlstra <peterz@infradead.org> - 2016-05-09 13:00 +0200
  [RFC][PATCH 3/7] sched: Introduce struct sched_domain_shared Peter Zijlstra <peterz@infradead.org> - 2016-05-09 13:00 +0200
  [RFC][PATCH 6/7] sched: Optimize SCHED_SMT Peter Zijlstra <peterz@infradead.org> - 2016-05-09 13:00 +0200
  Re: [RFC][PATCH 0/7] sched: select_idle_siblings rewrite Chris Mason <clm@fb.com> - 2016-05-10 03:00 +0200
  Re: [RFC][PATCH 0/7] sched: select_idle_siblings rewrite Chris Mason <clm@fb.com> - 2016-05-11 16:30 +0200
  [RFC][PATCH 8/7] sched/fair: Use utilization distance to filter  affine sync wakeups Mike Galbraith <mgalbraith@suse.de> - 2016-05-18 08:00 +0200
    Re: [RFC][PATCH 8/7] sched/fair: Use utilization distance to filter  affine sync wakeups Rik van Riel <riel@redhat.com> - 2016-05-19 23:50 +0200
      Re: [RFC][PATCH 8/7] sched/fair: Use utilization distance to filter  affine sync wakeups Mike Galbraith <mgalbraith@suse.de> - 2016-05-20 05:00 +0200
  Re: [RFC][PATCH 0/7] sched: select_idle_siblings rewrite Chris Mason <clm@fb.com> - 2016-05-25 17:00 +0200
    Re: [RFC][PATCH 0/7] sched: select_idle_siblings rewrite Peter Zijlstra <peterz@infradead.org> - 2016-05-25 18:30 +0200

csiph-web