Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1196401 > unrolled thread

Re: [PATCH 5/9] x86/intel_rdt: Add new cgroup and Class of service management

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH 5/9] x86/intel_rdt: Add new cgroup and Class of service  management Tejun Heo <tj@kernel.org> - 2015-07-30 21:50 +0200
    Re: [PATCH 5/9] x86/intel_rdt: Add new cgroup and Class of service  management Marcelo Tosatti <mtosatti@redhat.com> - 2015-07-31 17:20 +0200
      Re: [PATCH 5/9] x86/intel_rdt: Add new cgroup and Class of service  management Tejun Heo <tj@kernel.org> - 2015-08-02 18:30 +0200
        Re: [PATCH 5/9] x86/intel_rdt: Add new cgroup and Class of service  management Marcelo Tosatti <mtosatti@redhat.com> - 2015-08-04 00:50 +0200
          Re: [PATCH 5/9] x86/intel_rdt: Add new cgroup and Class of service  management Marcelo Tosatti <mtosatti@redhat.com> - 2015-08-04 15:00 +0200
    Re: [PATCH 5/9] x86/intel_rdt: Add new cgroup and Class of service  management Vikas Shivappa <vikas.shivappa@intel.com> - 2015-07-31 18:30 +0200
      Re: [PATCH 5/9] x86/intel_rdt: Add new cgroup and Class of service  management Tejun Heo <tj@kernel.org> - 2015-08-02 18:40 +0200

#1196401 — Re: [PATCH 5/9] x86/intel_rdt: Add new cgroup and Class of service management

Hello, Vikas.

On Wed, Jul 01, 2015 at 03:21:06PM -0700, Vikas Shivappa wrote:
> This patch adds a cgroup subsystem for Intel Resource Director
> Technology(RDT) feature and Class of service(CLOSid) management which is
> part of common RDT framework.  This cgroup would eventually be used by
> all sub-features of RDT and hence be associated with the common RDT
> framework as well as sub-feature specific framework.  However current
> patch series only adds cache allocation sub-feature specific code.
> 
> When a cgroup directory is created it has a CLOSid associated with it
> which is inherited from its parent.  The Closid is mapped to a
> cache_mask which represents the L3 cache allocation to the cgroup.
> Tasks belonging to the cgroup get to fill the cache represented by the
> cache_mask.

First of all, I apologize for being so late.  I've been thinking about
it but the thoughts didn't quite crystalize (which isn't to say that
it's very crystal now) until recently.  If I understand correctly,
there are a couple suggested use cases for explicitly managing cache
usage.

1. Pinning known hot areas of memory in cache.

2. Explicitly regulating cache usage so that cacheline allocation can
   be better than CPU itself doing it.

#1 isn't part of this patchset, right?  Is there any plan for working
towards this too?

For #2, it is likely that the targeted use cases would involve threads
of a process or at least cooperating processes and having a simple API
which just goes "this (or the current) thread is only gonna use this
part of cache" would be a lot easier to use and actually beneficial.

I don't really think it makes sense to implement a fully hierarchical
cgroup solution when there isn't the basic affinity-adjusting
interface and it isn't clear whether fully hierarchical resource
distribution would be necessary especially given that the granularity
of the target resource is very coarse.

I can see that how cpuset would seem to invite this sort of usage but
cpuset itself is more of an arbitrary outgrowth (regardless of
history) in terms of resource control and most things controlled by
cpuset already have countepart interface which is readily accessible
to the normal applications.

Given that what the feature allows is restricting usage rather than
granting anything exclusively, a programmable interface wouldn't need
to worry about complications around priviledges while being able to
reap most of the benefits in an a lot easier way.  Am I missing
something?

Thanks.

-- 
tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1197092

On Thu, Jul 30, 2015 at 03:44:58PM -0400, Tejun Heo wrote:
> Hello, Vikas.
> 
> On Wed, Jul 01, 2015 at 03:21:06PM -0700, Vikas Shivappa wrote:
> > This patch adds a cgroup subsystem for Intel Resource Director
> > Technology(RDT) feature and Class of service(CLOSid) management which is
> > part of common RDT framework.  This cgroup would eventually be used by
> > all sub-features of RDT and hence be associated with the common RDT
> > framework as well as sub-feature specific framework.  However current
> > patch series only adds cache allocation sub-feature specific code.
> > 
> > When a cgroup directory is created it has a CLOSid associated with it
> > which is inherited from its parent.  The Closid is mapped to a
> > cache_mask which represents the L3 cache allocation to the cgroup.
> > Tasks belonging to the cgroup get to fill the cache represented by the
> > cache_mask.
> 
> First of all, I apologize for being so late.  I've been thinking about
> it but the thoughts didn't quite crystalize (which isn't to say that
> it's very crystal now) until recently.  If I understand correctly,
> there are a couple suggested use cases for explicitly managing cache
> usage.
> 
> 1. Pinning known hot areas of memory in cache.
> 
> 2. Explicitly regulating cache usage so that cacheline allocation can
>    be better than CPU itself doing it.
> 
> #1 isn't part of this patchset, right?  Is there any plan for working
> towards this too?
> 
> For #2, it is likely that the targeted use cases would involve threads
> of a process or at least cooperating processes and having a simple API
> which just goes "this (or the current) thread is only gonna use this
> part of cache" would be a lot easier to use and actually beneficial.
> 
> I don't really think it makes sense to implement a fully hierarchical
> cgroup solution when there isn't the basic affinity-adjusting
> interface 

What is an "affinity adjusting interface" ? Can you give an example
please?

> and it isn't clear whether fully hierarchical resource
> distribution would be necessary especially given that the granularity
> of the target resource is very coarse.

As i see it, the benefit of the hierarchical structure to the CAT
configuration is simply to organize sharing of cache ways in subtrees
- two cgroups can share a given cache way only if they have a common
parent. 

That is the only benefit. Vikas, please correct me if i'm wrong.

> I can see that how cpuset would seem to invite this sort of usage but
> cpuset itself is more of an arbitrary outgrowth (regardless of
> history) in terms of resource control and most things controlled by
> cpuset already have countepart interface which is readily accessible
> to the normal applications.

I can't parse that phrase (due to ignorance). Please educate.

> Given that what the feature allows is restricting usage rather than
> granting anything exclusively, a programmable interface wouldn't need
> to worry about complications around priviledges

What complications about priviledges you refer to?

> while being able to reap most of the benefits in an a lot easier way.
> Am I missing something?

The interface does allow for exclusive cache usage by an application.
Please read the Intel manual, section 17, it is very instructive.

The use cases we have now are the following:

Scenario 1: Consider a system with 4 high performance applications
running, one of which is a streaming application that manages a very
large address space from which it reads and writes as it does its processing.
As such the application will use all the cache it can get but does
not need much if any cache. So, it spoils the cache for everyone for no
gain on its own. In this case we'd like to constrain it to the
smallest possible amount of cache while at the same time constraining
the other 3 applications to stay out of this thrashed area of the
cache.

Scenario 2: We have a numeric application that has been highly optimized
to fit in the L2 cache (2M for example). We want to ensure that its
cached data does not get flushed from the cache hierarchy while it is
scheduled out. In this case we exclusively allocate enough L3 cache to
hold all of the L2 cache.

Scenario 3: Latency sensitive application executing in a shared
environment, where memory to handle an event must be in L3 cache
for latency requirements to be met.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1198340

Hello,

On Fri, Jul 31, 2015 at 12:12:18PM -0300, Marcelo Tosatti wrote:
> > I don't really think it makes sense to implement a fully hierarchical
> > cgroup solution when there isn't the basic affinity-adjusting
> > interface 
> 
> What is an "affinity adjusting interface" ? Can you give an example
> please?

Something similar to sched_setaffinity().  Just a syscall / prctl or
whatever programmable interface which sets per-task attribute.

> > and it isn't clear whether fully hierarchical resource
> > distribution would be necessary especially given that the granularity
> > of the target resource is very coarse.
> 
> As i see it, the benefit of the hierarchical structure to the CAT
> configuration is simply to organize sharing of cache ways in subtrees
> - two cgroups can share a given cache way only if they have a common
> parent. 
> 
> That is the only benefit. Vikas, please correct me if i'm wrong.

cgroups is not a superset of a programmable interface.  It has
distinctive disadvantages and not a substitute with hirearchy support
for regular systemcall-like interface.  I don't think it makes sense
to go full-on hierarchical cgroups when we don't have basic interface
which is likely to cover many use cases better.  A syscall-like
interface combined with a tool similar to taskset would cover a lot in
a more accessible way.

> > I can see that how cpuset would seem to invite this sort of usage but
> > cpuset itself is more of an arbitrary outgrowth (regardless of
> > history) in terms of resource control and most things controlled by
> > cpuset already have countepart interface which is readily accessible
> > to the normal applications.
> 
> I can't parse that phrase (due to ignorance). Please educate.

Hmmm... consider CPU affinity.  cpuset definitely is useful for some
use cases as a management tool especially if the workloads are not
cooperative or delegated; however, it's no substitute for a proper
syscall interface and it'd be silly to try to replace that with
cpuset.

> > Given that what the feature allows is restricting usage rather than
> > granting anything exclusively, a programmable interface wouldn't need
> > to worry about complications around priviledges
> 
> What complications about priviledges you refer to?

It's not granting exclusive access, so individual user applications
can be allowed to do whatever it wanna do as long as the issuer has
enough priv over the target task.

> > while being able to reap most of the benefits in an a lot easier way.
> > Am I missing something?
> 
> The interface does allow for exclusive cache usage by an application.
> Please read the Intel manual, section 17, it is very instructive.

For that, it'd have to require some CAP but I think just having
restrictive interface in the style of CPU or NUMA affinity would go a
long way.

> The use cases we have now are the following:
> 
> Scenario 1: Consider a system with 4 high performance applications
> running, one of which is a streaming application that manages a very
> large address space from which it reads and writes as it does its processing.
> As such the application will use all the cache it can get but does
> not need much if any cache. So, it spoils the cache for everyone for no
> gain on its own. In this case we'd like to constrain it to the
> smallest possible amount of cache while at the same time constraining
> the other 3 applications to stay out of this thrashed area of the
> cache.

A tool in the style of taskset should be enough for the above
scenario.

> Scenario 2: We have a numeric application that has been highly optimized
> to fit in the L2 cache (2M for example). We want to ensure that its
> cached data does not get flushed from the cache hierarchy while it is
> scheduled out. In this case we exclusively allocate enough L3 cache to
> hold all of the L2 cache.
>
> Scenario 3: Latency sensitive application executing in a shared
> environment, where memory to handle an event must be in L3 cache
> for latency requirements to be met.

Either isolate CPUs or run other stuff with affinity restricted.

cpuset-style allocation can be easier for things like this but that
should be an addition on top not the one and only interface.  How is
it gonna handle if multiple threads of a process want to restrict
cache usages to avoid stepping on each other's toes?  Delegate the
subdirectory and let the process itself open it and write to files to
configure when there isn't even a way to atomically access the
process's own directory or a way to synchronize against migration?
cgroups may be an okay management interface but a horrible
programmable interface.

Sure, if this turns out to be as important as cpu or numa affinity and
gets widely used creating management burden in many use cases, we sure
can add cgroups controller for it but that's a remote possibility at
this point and the current attempt is over-engineering solution for
problems which haven't been shown to exist.  Let's please first
implement something simple and easy to use.

Thanks.

-- 
tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1199308

On Sun, Aug 02, 2015 at 12:23:25PM -0400, Tejun Heo wrote:
> Hello,
> 
> On Fri, Jul 31, 2015 at 12:12:18PM -0300, Marcelo Tosatti wrote:
> > > I don't really think it makes sense to implement a fully hierarchical
> > > cgroup solution when there isn't the basic affinity-adjusting
> > > interface 
> > 
> > What is an "affinity adjusting interface" ? Can you give an example
> > please?
> 
> Something similar to sched_setaffinity().  Just a syscall / prctl or
> whatever programmable interface which sets per-task attribute.

You really want to specify the cache configuration "at once": 
having process-A exclusive access to 2MB of cache at all times,
and process-B 4MB exclusive, means you can't have process-C use 4MB of 
cache exclusively (consider 8MB cache machine).

But the syscall allows processes to set and retrieve

> > > and it isn't clear whether fully hierarchical resource
> > > distribution would be necessary especially given that the granularity
> > > of the target resource is very coarse.
> > 
> > As i see it, the benefit of the hierarchical structure to the CAT
> > configuration is simply to organize sharing of cache ways in subtrees
> > - two cgroups can share a given cache way only if they have a common
> > parent. 
> > 
> > That is the only benefit. Vikas, please correct me if i'm wrong.
> 
> cgroups is not a superset of a programmable interface.  It has
> distinctive disadvantages and not a substitute with hirearchy support
> for regular systemcall-like interface.  I don't think it makes sense
> to go full-on hierarchical cgroups when we don't have basic interface
> which is likely to cover many use cases better.  A syscall-like
> interface combined with a tool similar to taskset would cover a lot in
> a more accessible way.

How are you going to specify sharing of portions of cache by two sets
of tasks with a syscall interface?

> > > I can see that how cpuset would seem to invite this sort of usage but
> > > cpuset itself is more of an arbitrary outgrowth (regardless of
> > > history) in terms of resource control and most things controlled by
> > > cpuset already have countepart interface which is readily accessible
> > > to the normal applications.
> > 
> > I can't parse that phrase (due to ignorance). Please educate.
> 
> Hmmm... consider CPU affinity.  cpuset definitely is useful for some
> use cases as a management tool especially if the workloads are not
> cooperative or delegated; however, it's no substitute for a proper
> syscall interface and it'd be silly to try to replace that with
> cpuset.
> 
> > > Given that what the feature allows is restricting usage rather than
> > > granting anything exclusively, a programmable interface wouldn't need
> > > to worry about complications around priviledges
> > 
> > What complications about priviledges you refer to?
> 
> It's not granting exclusive access, so individual user applications
> can be allowed to do whatever it wanna do as long as the issuer has
> enough priv over the target task.

Priviledge management with cgroup system: to change cache allocation
requires priviledge over cgroups.

Priviledge management with system call interface: applications 
could be allowed to reserve up to a certain percentage of the cache.

> > > while being able to reap most of the benefits in an a lot easier way.
> > > Am I missing something?
> > 
> > The interface does allow for exclusive cache usage by an application.
> > Please read the Intel manual, section 17, it is very instructive.
> 
> For that, it'd have to require some CAP but I think just having
> restrictive interface in the style of CPU or NUMA affinity would go a
> long way.
> 
> > The use cases we have now are the following:
> > 
> > Scenario 1: Consider a system with 4 high performance applications
> > running, one of which is a streaming application that manages a very
> > large address space from which it reads and writes as it does its processing.
> > As such the application will use all the cache it can get but does
> > not need much if any cache. So, it spoils the cache for everyone for no
> > gain on its own. In this case we'd like to constrain it to the
> > smallest possible amount of cache while at the same time constraining
> > the other 3 applications to stay out of this thrashed area of the
> > cache.
> 
> A tool in the style of taskset should be enough for the above
> scenario.
> 
> > Scenario 2: We have a numeric application that has been highly optimized
> > to fit in the L2 cache (2M for example). We want to ensure that its
> > cached data does not get flushed from the cache hierarchy while it is
> > scheduled out. In this case we exclusively allocate enough L3 cache to
> > hold all of the L2 cache.
> >
> > Scenario 3: Latency sensitive application executing in a shared
> > environment, where memory to handle an event must be in L3 cache
> > for latency requirements to be met.
> 
> Either isolate CPUs or run other stuff with affinity restricted.
> 
> cpuset-style allocation can be easier for things like this but that
> should be an addition on top not the one and only interface.  How is
> it gonna handle if multiple threads of a process want to restrict
> cache usages to avoid stepping on each other's toes?  Delegate the
> subdirectory and let the process itself open it and write to files to
> configure when there isn't even a way to atomically access the
> process's own directory or a way to synchronize against migration?

One would preconfigure that in advance - but you are right, a 
syscall interface is more flexible in that respect.

> cgroups may be an okay management interface but a horrible
> programmable interface.
> 
> Sure, if this turns out to be as important as cpu or numa affinity and
> gets widely used creating management burden in many use cases, we sure
> can add cgroups controller for it but that's a remote possibility at
> this point and the current attempt is over-engineering solution for
> problems which haven't been shown to exist.  Let's please first
> implement something simple and easy to use.
> 
> Thanks.
> 
> -- 
> tejun


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1199791

On Mon, Aug 03, 2015 at 05:32:50PM -0300, Marcelo Tosatti wrote:
> On Sun, Aug 02, 2015 at 12:23:25PM -0400, Tejun Heo wrote:
> > Hello,
> > 
> > On Fri, Jul 31, 2015 at 12:12:18PM -0300, Marcelo Tosatti wrote:
> > > > I don't really think it makes sense to implement a fully hierarchical
> > > > cgroup solution when there isn't the basic affinity-adjusting
> > > > interface 
> > > 
> > > What is an "affinity adjusting interface" ? Can you give an example
> > > please?
> > 
> > Something similar to sched_setaffinity().  Just a syscall / prctl or
> > whatever programmable interface which sets per-task attribute.
> 
> You really want to specify the cache configuration "at once": 
> having process-A exclusive access to 2MB of cache at all times,
> and process-B 4MB exclusive, means you can't have process-C use 4MB of 
> cache exclusively (consider 8MB cache machine).

Thats not true. Its fine to setup the 

	task set <--> cache portion

mapping in pieces.

In fact, its more natural because you don't necessarily know in advance
the entire cache allocation (think of "cp largefile /destination" with
sequential use-once behavior).

However, there is a use-case for sharing: in scenario 1 it might be
possible (and desired) to share code between applications.

> > > > and it isn't clear whether fully hierarchical resource
> > > > distribution would be necessary especially given that the granularity
> > > > of the target resource is very coarse.
> > > 
> > > As i see it, the benefit of the hierarchical structure to the CAT
> > > configuration is simply to organize sharing of cache ways in subtrees
> > > - two cgroups can share a given cache way only if they have a common
> > > parent. 
> > > 
> > > That is the only benefit. Vikas, please correct me if i'm wrong.
> > 
> > cgroups is not a superset of a programmable interface.  It has
> > distinctive disadvantages and not a substitute with hirearchy support
> > for regular systemcall-like interface.  I don't think it makes sense
> > to go full-on hierarchical cgroups when we don't have basic interface
> > which is likely to cover many use cases better.  A syscall-like
> > interface combined with a tool similar to taskset would cover a lot in
> > a more accessible way.
> 
> How are you going to specify sharing of portions of cache by two sets
> of tasks with a syscall interface?
> 
> > > > I can see that how cpuset would seem to invite this sort of usage but
> > > > cpuset itself is more of an arbitrary outgrowth (regardless of
> > > > history) in terms of resource control and most things controlled by
> > > > cpuset already have countepart interface which is readily accessible
> > > > to the normal applications.
> > > 
> > > I can't parse that phrase (due to ignorance). Please educate.
> > 
> > Hmmm... consider CPU affinity.  cpuset definitely is useful for some
> > use cases as a management tool especially if the workloads are not
> > cooperative or delegated; however, it's no substitute for a proper
> > syscall interface and it'd be silly to try to replace that with
> > cpuset.
> > 
> > > > Given that what the feature allows is restricting usage rather than
> > > > granting anything exclusively, a programmable interface wouldn't need
> > > > to worry about complications around priviledges
> > > 
> > > What complications about priviledges you refer to?
> > 
> > It's not granting exclusive access, so individual user applications
> > can be allowed to do whatever it wanna do as long as the issuer has
> > enough priv over the target task.
> 
> Priviledge management with cgroup system: to change cache allocation
> requires priviledge over cgroups.
> 
> Priviledge management with system call interface: applications 
> could be allowed to reserve up to a certain percentage of the cache.
> 
> > > > while being able to reap most of the benefits in an a lot easier way.
> > > > Am I missing something?
> > > 
> > > The interface does allow for exclusive cache usage by an application.
> > > Please read the Intel manual, section 17, it is very instructive.
> > 
> > For that, it'd have to require some CAP but I think just having
> > restrictive interface in the style of CPU or NUMA affinity would go a
> > long way.
> > 
> > > The use cases we have now are the following:
> > > 
> > > Scenario 1: Consider a system with 4 high performance applications
> > > running, one of which is a streaming application that manages a very
> > > large address space from which it reads and writes as it does its processing.
> > > As such the application will use all the cache it can get but does
> > > not need much if any cache. So, it spoils the cache for everyone for no
> > > gain on its own. In this case we'd like to constrain it to the
> > > smallest possible amount of cache while at the same time constraining
> > > the other 3 applications to stay out of this thrashed area of the
> > > cache.
> > 
> > A tool in the style of taskset should be enough for the above
> > scenario.
> > 
> > > Scenario 2: We have a numeric application that has been highly optimized
> > > to fit in the L2 cache (2M for example). We want to ensure that its
> > > cached data does not get flushed from the cache hierarchy while it is
> > > scheduled out. In this case we exclusively allocate enough L3 cache to
> > > hold all of the L2 cache.
> > >
> > > Scenario 3: Latency sensitive application executing in a shared
> > > environment, where memory to handle an event must be in L3 cache
> > > for latency requirements to be met.
> > 
> > Either isolate CPUs or run other stuff with affinity restricted.
> > 
> > cpuset-style allocation can be easier for things like this but that
> > should be an addition on top not the one and only interface.  How is
> > it gonna handle if multiple threads of a process want to restrict
> > cache usages to avoid stepping on each other's toes?  Delegate the
> > subdirectory and let the process itself open it and write to files to
> > configure when there isn't even a way to atomically access the
> > process's own directory or a way to synchronize against migration?
> 
> One would preconfigure that in advance - but you are right, a 
> syscall interface is more flexible in that respect.

So, systemd is responsible for locking.

> > cgroups may be an okay management interface but a horrible
> > programmable interface.
> > 
> > Sure, if this turns out to be as important as cpu or numa affinity and
> > gets widely used creating management burden in many use cases, we sure
> > can add cgroups controller for it but that's a remote possibility at
> > this point and the current attempt is over-engineering solution for
> > problems which haven't been shown to exist.  Let's please first
> > implement something simple and easy to use.
> > 
> > Thanks.
> > 
> > -- 
> > tejun

Don't see an easy way to fix the sharing use-case (it would require
exposing the "intersection" between two task sets).

Can't "cacheset" helper (similar to taskset) talk to systemd
to achieve the flexibility you point ?

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1197166

On Thu, 30 Jul 2015, Tejun Heo wrote:

> Hello, Vikas.
>
> On Wed, Jul 01, 2015 at 03:21:06PM -0700, Vikas Shivappa wrote:
>> This patch adds a cgroup subsystem for Intel Resource Director
>> Technology(RDT) feature and Class of service(CLOSid) management which is
>> part of common RDT framework.  This cgroup would eventually be used by
>> all sub-features of RDT and hence be associated with the common RDT
>> framework as well as sub-feature specific framework.  However current
>> patch series only adds cache allocation sub-feature specific code.
>>
>> When a cgroup directory is created it has a CLOSid associated with it
>> which is inherited from its parent.  The Closid is mapped to a
>> cache_mask which represents the L3 cache allocation to the cgroup.
>> Tasks belonging to the cgroup get to fill the cache represented by the
>> cache_mask.
>
> First of all, I apologize for being so late.  I've been thinking about
> it but the thoughts didn't quite crystalize (which isn't to say that
> it's very crystal now) until recently.  If I understand correctly,
> there are a couple suggested use cases for explicitly managing cache
> usage.
>
> 1. Pinning known hot areas of memory in cache.

No , the cache allocation doesnt do this. (or it isn't expected to do)

>
> 2. Explicitly regulating cache usage so that cacheline allocation can
>   be better than CPU itself doing it.

yes , this is what we want to do using cache alloc.

>
> #1 isn't part of this patchset, right?  Is there any plan for working
> towards this too?

cache allocation is not intended to do #1 , so we dont have to support this.

>
> For #2, it is likely that the targeted use cases would involve threads
> of a process or at least cooperating processes and having a simple API
> which just goes "this (or the current) thread is only gonna use this
> part of cache" would be a lot easier to use and actually beneficial.
>
> I don't really think it makes sense to implement a fully hierarchical
> cgroup solution when there isn't the basic affinity-adjusting
> interface and it isn't clear whether fully hierarchical resource
> distribution would be necessary especially given that the granularity
> of the target resource is very coarse.
>
> I can see that how cpuset would seem to invite this sort of usage but
> cpuset itself is more of an arbitrary outgrowth (regardless of
> history) in terms of resource control and most things controlled by
> cpuset already have countepart interface which is readily accessible
> to the normal applications.

Yes today we dont have an alternative interface - but we can always build one. 
We simply dont have it because till now Linux kernel just tolerated the 
degradation that could have occured by cache contention and this is the first 
interface we are building.

>
> Given that what the feature allows is restricting usage rather than
> granting anything exclusively, a programmable interface wouldn't need
> to worry about complications around priviledges while being able to
> reap most of the benefits in an a lot easier way.  Am I missing
> something?
>

For #2 , from the intel_rdt cgroup we develop a framework where the user can 
regulate the cache allocation. A user space app could also eventually use this 
as underlying support and then do things on top of it depending on the 
enterprise or other requirements.

A typical use case would be that an application which 
is say continuously polluting the cache(low priority app from cache usage 
perspective) by bringing in data from the network (copying/streaming app) and 
and not letting an app to use the cache which has legitimate requirement of 
cache usage(high priority app).

We need to map the group of tasks to a particular class of service and way for 
the user to specify the cache capacity for that class of service . Also a 
default cgroup which could have all the tasks and use all the cache.
The hierarchical interface can be used by the user as required and does not 
really interfere with allocating exclusive blocks of cache - all the user needs 
to do is make sure the masks dont overlap.
The user can configure the masks to be exclusive from others.
But note that overlapping mask provides a very easy way to share the cache usage 
which is what you may want to do sometimes. The current implementation can be 
easily extended to *enforce* exclusive capacity masks between child nodes if 
required. But since its expected for the super user to be using this , the usage 
may be limited as well or the user can still care of it like i said above. Some 
of the emails may have been confusing that we cannot do exclusive allocations - 
but thats not true all together : we can do canfigure the masks to have 
exclusive cache blocks for different cgroups but its just left to the user...


We did have a lot of discussions during the design and V3 if you remember and 
were closed on using a seperate controller ... Below is one such thread where 
we discussed the same . Dont want to loop throug again with this already full 
marathon patch :)

https://lkml.org/lkml/2015/1/27/846

quick copy from V3 thread  -
"

> proposal but was removed as we did not get agreement on lkml.
>
> the original lkml thread is here from 10/2014 for your reference -
> https://lkml.org/lkml/2014/10/16/568

Yeap, I followed that thread and this being a separate controller 
definitely makes a lot more sense.

"



Thanks,
Vikas

> Thanks.
>
> -- 
> tejun
>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1198404

Hello, Vikas.

On Fri, Jul 31, 2015 at 09:24:58AM -0700, Vikas Shivappa wrote:
> Yes today we dont have an alternative interface - but we can always build
> one. We simply dont have it because till now Linux kernel just tolerated the
> degradation that could have occured by cache contention and this is the
> first interface we are building.

But we're doing it the wrong way around.  You can do most of what
cgroup interface can do with systemcall-like interface with some
inconvenience.  The other way doesn't really work.  As I wrote in the
other reply, cgroups is a horrible programmable interface and we don't
want individual applications to interact with it directly and CAT's
use cases most definitely include each application programming its own
cache mask.  Let's build something which is simple and can be used
easily first.  If this turns out to be widely useful and an overall
management capability over it is wanted, we can consider cgroups then.

Thanks.

-- 
tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web