Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1272457 > unrolled thread

[RFD] CAT user space interface revisited

Back to article view | Back to linux.kernel

Contents

  [RFD] CAT user space interface revisited Thomas Gleixner <tglx@linutronix.de> - 2015-11-18 19:30 +0100
    Re: [RFD] CAT user space interface revisited Luiz Capitulino <lcapitulino@redhat.com> - 2015-11-18 20:40 +0100
      RE: [RFD] CAT user space interface revisited "Auld, Will" <will.auld@intel.com> - 2015-11-18 21:00 +0100
    Re: [RFD] CAT user space interface revisited Marcelo Tosatti <mtosatti@redhat.com> - 2015-11-18 23:40 +0100
      Re: [RFD] CAT user space interface revisited Marcelo Tosatti <mtosatti@redhat.com> - 2015-11-19 01:40 +0100
        Re: [RFD] CAT user space interface revisited Thomas Gleixner <tglx@linutronix.de> - 2015-11-19 09:40 +0100
          Re: [RFD] CAT user space interface revisited Luiz Capitulino <lcapitulino@redhat.com> - 2015-11-19 14:50 +0100
          Re: [RFD] CAT user space interface revisited Marcelo Tosatti <mtosatti@redhat.com> - 2015-11-20 17:40 +0100
      Re: [RFD] CAT user space interface revisited Thomas Gleixner <tglx@linutronix.de> - 2015-11-19 09:20 +0100
    Re: [RFD] CAT user space interface revisited Marcelo Tosatti <mtosatti@redhat.com> - 2015-11-19 01:20 +0100
      Re: [RFD] CAT user space interface revisited Marcelo Tosatti <mtosatti@redhat.com> - 2015-11-19 02:10 +0100
        Re: [RFD] CAT user space interface revisited Thomas Gleixner <tglx@linutronix.de> - 2015-11-19 10:10 +0100
          Re: [RFD] CAT user space interface revisited Marcelo Tosatti <mtosatti@redhat.com> - 2015-11-19 23:30 +0100
            Re: [RFD] CAT user space interface revisited Thomas Gleixner <tglx@linutronix.de> - 2015-11-20 09:00 +0100
              Re: [RFD] CAT user space interface revisited Marcelo Tosatti <mtosatti@redhat.com> - 2015-11-20 20:30 +0100
        Re: [RFD] CAT user space interface revisited Marcelo Tosatti <mtosatti@redhat.com> - 2015-11-19 21:40 +0100
      Re: [RFD] CAT user space interface revisited Thomas Gleixner <tglx@linutronix.de> - 2015-11-19 10:10 +0100
      Re: [RFD] CAT user space interface revisited Chao Peng <chao.p.peng@linux.intel.com> - 2015-11-24 09:40 +0100
        Re: [RFD] CAT user space interface revisited Marcelo Tosatti <mtosatti@redhat.com> - 2015-11-25 23:10 +0100
    Re: [RFD] CAT user space interface revisited Chao Peng <chao.p.peng@linux.intel.com> - 2015-11-24 08:40 +0100
      Re: [RFD] CAT user space interface revisited Marcelo Tosatti <mtosatti@redhat.com> - 2015-11-25 23:10 +0100

Page 1 of 2  [1] 2  Next page →

#1272457 — [RFD] CAT user space interface revisited

Folks!

After rereading the mail flood on CAT and staring into the SDM for a
while, I think we all should sit back and look at it from scratch
again w/o our preconceptions - I certainly had to put my own away.

Let's look at the properties of CAT again:

   - It's a per socket facility

   - CAT slots can be associated to external hardware. This
     association is per socket as well, so different sockets can have
     different behaviour. I missed that detail when staring the first
     time, thanks for the pointer!

   - The association ifself is per cpu. The COS selection happens on a
     CPU while the set of masks which are selected via COS are shared
     by all CPUs on a socket.

There are restrictions which CAT imposes in terms of configurability:

   - The bits which select a cache partition need to be consecutive

   - The number of possible cache association masks is limited

Let's look at the configurations (CDP omitted and size restricted)

Default:   1 1 1 1 1 1 1 1
	   1 1 1 1 1 1 1 1
	   1 1 1 1 1 1 1 1
	   1 1 1 1 1 1 1 1

Shared:	   1 1 1 1 1 1 1 1
	   0 0 1 1 1 1 1 1
	   0 0 0 0 1 1 1 1
	   0 0 0 0 0 0 1 1

Isolated:  1 1 1 1 0 0 0 0
	   0 0 0 0 1 1 0 0
	   0 0 0 0 0 0 1 0
	   0 0 0 0 0 0 0 1

Or any combination thereof. Surely some combinations will not make any
sense, but we really should not make any restrictions on the stupidity
of a sysadmin. The worst outcome might be L3 disabled for everything,
so what?

Now that gets even more convoluted if CDP comes into play and we
really need to look at CDP right now. We might end up with something
which looks like this:

   	   1 1 1 1 0 0 0 0	Code
	   1 1 1 1 0 0 0 0	Data
	   0 0 0 0 0 0 1 0	Code
	   0 0 0 0 1 1 0 0	Data
	   0 0 0 0 0 0 0 1	Code
	   0 0 0 0 1 1 0 0	Data
or 
	   0 0 0 0 0 0 0 1	Code
	   0 0 0 0 1 1 0 0	Data
	   0 0 0 0 0 0 0 1	Code
	   0 0 0 0 0 1 1 0	Data

Let's look at partitioning itself. We have two options:

   1) Per task partitioning

   2) Per CPU partitioning

So far we only talked about #1, but I think that #2 has a value as
well. Let me give you a simple example.

Assume that you have isolated a CPU and run your important task on
it. You give that task a slice of cache. Now that task needs kernel
services which run in kernel threads on that CPU. We really don't want
to (and cannot) hunt down random kernel threads (think cpu bound
worker threads, softirq threads ....) and give them another slice of
cache. What we really want is:

    	 1 1 1 1 0 0 0 0    <- Default cache
	 0 0 0 0 1 1 1 0    <- Cache for important task
	 0 0 0 0 0 0 0 1    <- Cache for CPU of important task

It would even be sufficient for particular use cases to just associate
a piece of cache to a given CPU and do not bother with tasks at all.

We really need to make this as configurable as possible from userspace
without imposing random restrictions to it. I played around with it on
my new intel toy and the restriction to 16 COS ids (that's 8 with CDP
enabled) makes it really useless if we force the ids to have the same
meaning on all sockets and restrict it to per task partitioning.

Even if next generation systems will have more COS ids available,
there are not going to be enough to have a system wide consistent
view unless we have COS ids > nr_cpus.

Aside of that I don't think that a system wide consistent view is
useful at all.

 - If a task migrates between sockets, it's going to suffer anyway.
   Real sensitive applications will simply pin tasks on a socket to
   avoid that in the first place. If we make the whole thing
   configurable enough then the sysadmin can set it up to support
   even the nonsensical case of identical cache partitions on all
   sockets and let tasks use the corresponding partitions when
   migrating.

 - The number of cache slices is going to be limited no matter what,
   so one still has to come up with a sensible partitioning scheme.

 - Even if we have enough cos ids the system wide view will not make
   the configuration problem any simpler as it remains per socket.

It's hard. Policies are hard by definition, but this one is harder
than most other policies due to the inherent limitations.

So now to the interface part. Unfortunately we need to expose this
very close to the hardware implementation as there are really no
abstractions which allow us to express the various bitmap
combinations. Any abstraction I tried to come up with renders that
thing completely useless.

I was not able to identify any existing infrastructure where this
really fits in. I chose a directory/file based representation. We
certainly could do the same with a syscall, but that's just an
implementation detail.

At top level:

   xxxxxxx/cat/max_cosids		<- Assume that all CPUs are the same
   xxxxxxx/cat/max_maskbits		<- Assume that all CPUs are the same
   xxxxxxx/cat/cdp_enable		<- Depends on CDP availability

Per socket data:

   xxxxxxx/cat/socket-0/
   ...
   xxxxxxx/cat/socket-N/l3_size
   xxxxxxx/cat/socket-N/hwsharedbits

Per socket mask data:

   xxxxxxx/cat/socket-N/cos-id-0/
   ...
   xxxxxxx/cat/socket-N/cos-id-N/inuse
				/cat_mask	
				/cdp_mask	<- Data mask if CDP enabled

Per cpu default cos id for the cpus on that socket:

   xxxxxxx/cat/socket-N/cpu-x/default_cosid
   ...
   xxxxxxx/cat/socket-N/cpu-N/default_cosid

The above allows a simple cpu based partitioning. All tasks which do
not have a cache partition assigned on a particular socket use the
default one of the cpu they are running on.

Now for the task(s) partitioning:

   xxxxxxx/cat/partitions/

Under that directory one can create partitions

   xxxxxxx/cat/partitions/p1/tasks
			    /socket-0/cosid
			    ...
			    /socket-n/cosid

   The default value for the per socket cosid is COSID_DEFAULT, which
   causes the task(s) to use the per cpu default id.

Thoughts?

Thanks,

	tglx
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1272503

On Wed, 18 Nov 2015 19:25:03 +0100 (CET)
Thomas Gleixner <tglx@linutronix.de> wrote:

> We really need to make this as configurable as possible from userspace
> without imposing random restrictions to it. I played around with it on
> my new intel toy and the restriction to 16 COS ids (that's 8 with CDP
> enabled) makes it really useless if we force the ids to have the same
> meaning on all sockets and restrict it to per task partitioning.
> 
> Even if next generation systems will have more COS ids available,
> there are not going to be enough to have a system wide consistent
> view unless we have COS ids > nr_cpus.
> 
> Aside of that I don't think that a system wide consistent view is
> useful at all.

This is a great writeup! I agree with everything you said.

> So now to the interface part. Unfortunately we need to expose this
> very close to the hardware implementation as there are really no
> abstractions which allow us to express the various bitmap
> combinations. Any abstraction I tried to come up with renders that
> thing completely useless.
> 
> I was not able to identify any existing infrastructure where this
> really fits in. I chose a directory/file based representation. We
> certainly could do the same with a syscall, but that's just an
> implementation detail.
> 
> At top level:
> 
>    xxxxxxx/cat/max_cosids		<- Assume that all CPUs are the same
>    xxxxxxx/cat/max_maskbits		<- Assume that all CPUs are the same
>    xxxxxxx/cat/cdp_enable		<- Depends on CDP availability
> 
> Per socket data:
> 
>    xxxxxxx/cat/socket-0/
>    ...
>    xxxxxxx/cat/socket-N/l3_size
>    xxxxxxx/cat/socket-N/hwsharedbits
> 
> Per socket mask data:
> 
>    xxxxxxx/cat/socket-N/cos-id-0/
>    ...
>    xxxxxxx/cat/socket-N/cos-id-N/inuse
> 				/cat_mask	
> 				/cdp_mask	<- Data mask if CDP enabled
> 
> Per cpu default cos id for the cpus on that socket:
> 
>    xxxxxxx/cat/socket-N/cpu-x/default_cosid
>    ...
>    xxxxxxx/cat/socket-N/cpu-N/default_cosid
> 
> The above allows a simple cpu based partitioning. All tasks which do
> not have a cache partition assigned on a particular socket use the
> default one of the cpu they are running on.
> 
> Now for the task(s) partitioning:
> 
>    xxxxxxx/cat/partitions/
> 
> Under that directory one can create partitions
> 
>    xxxxxxx/cat/partitions/p1/tasks
> 			    /socket-0/cosid
> 			    ...
> 			    /socket-n/cosid
> 
>    The default value for the per socket cosid is COSID_DEFAULT, which
>    causes the task(s) to use the per cpu default id.

I hope I've got all the details right, but this proposal looks awesome.
There's more people who seem to agree with something like this.

Btw, I think it should be possible to implement this with cgroups. But
I too don't care that much on cgroups vs. syscalls.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1272517

+Tony

> -----Original Message-----
> From: Luiz Capitulino [mailto:lcapitulino@redhat.com]
> Sent: Wednesday, November 18, 2015 11:38 AM
> To: Thomas Gleixner
> Cc: LKML; Peter Zijlstra; x86@kernel.org; Marcelo Tosatti; Shivappa, Vikas; Tejun
> Heo; Yu, Fenghua; Auld, Will; Dugger, Donald D; riel@redhat.com
> Subject: Re: [RFD] CAT user space interface revisited
> 
> On Wed, 18 Nov 2015 19:25:03 +0100 (CET) Thomas Gleixner
> <tglx@linutronix.de> wrote:
> 
> > We really need to make this as configurable as possible from userspace
> > without imposing random restrictions to it. I played around with it on
> > my new intel toy and the restriction to 16 COS ids (that's 8 with CDP
> > enabled) makes it really useless if we force the ids to have the same
> > meaning on all sockets and restrict it to per task partitioning.
> >
> > Even if next generation systems will have more COS ids available,
> > there are not going to be enough to have a system wide consistent view
> > unless we have COS ids > nr_cpus.
> >
> > Aside of that I don't think that a system wide consistent view is
> > useful at all.
> 
> This is a great writeup! I agree with everything you said.
> 
> > So now to the interface part. Unfortunately we need to expose this
> > very close to the hardware implementation as there are really no
> > abstractions which allow us to express the various bitmap
> > combinations. Any abstraction I tried to come up with renders that
> > thing completely useless.
> >
> > I was not able to identify any existing infrastructure where this
> > really fits in. I chose a directory/file based representation. We
> > certainly could do the same with a syscall, but that's just an
> > implementation detail.
> >
> > At top level:
> >
> >    xxxxxxx/cat/max_cosids		<- Assume that all CPUs are the same
> >    xxxxxxx/cat/max_maskbits		<- Assume that all CPUs are the same
> >    xxxxxxx/cat/cdp_enable		<- Depends on CDP availability
> >
> > Per socket data:
> >
> >    xxxxxxx/cat/socket-0/
> >    ...
> >    xxxxxxx/cat/socket-N/l3_size
> >    xxxxxxx/cat/socket-N/hwsharedbits
> >
> > Per socket mask data:
> >
> >    xxxxxxx/cat/socket-N/cos-id-0/
> >    ...
> >    xxxxxxx/cat/socket-N/cos-id-N/inuse
> > 				/cat_mask
> > 				/cdp_mask	<- Data mask if CDP enabled
> >
> > Per cpu default cos id for the cpus on that socket:
> >
> >    xxxxxxx/cat/socket-N/cpu-x/default_cosid
> >    ...
> >    xxxxxxx/cat/socket-N/cpu-N/default_cosid
> >
> > The above allows a simple cpu based partitioning. All tasks which do
> > not have a cache partition assigned on a particular socket use the
> > default one of the cpu they are running on.
> >
> > Now for the task(s) partitioning:
> >
> >    xxxxxxx/cat/partitions/
> >
> > Under that directory one can create partitions
> >
> >    xxxxxxx/cat/partitions/p1/tasks
> > 			    /socket-0/cosid
> > 			    ...
> > 			    /socket-n/cosid
> >
> >    The default value for the per socket cosid is COSID_DEFAULT, which
> >    causes the task(s) to use the per cpu default id.
> 
> I hope I've got all the details right, but this proposal looks awesome.
> There's more people who seem to agree with something like this.
> 
> Btw, I think it should be possible to implement this with cgroups. But I too don't
> care that much on cgroups vs. syscalls.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1272636

On Wed, Nov 18, 2015 at 07:25:03PM +0100, Thomas Gleixner wrote:
> Folks!
> 
> After rereading the mail flood on CAT and staring into the SDM for a
> while, I think we all should sit back and look at it from scratch
> again w/o our preconceptions - I certainly had to put my own away.
> 
> Let's look at the properties of CAT again:
> 
>    - It's a per socket facility
> 
>    - CAT slots can be associated to external hardware. This
>      association is per socket as well, so different sockets can have
>      different behaviour. I missed that detail when staring the first
>      time, thanks for the pointer!
> 
>    - The association ifself is per cpu. The COS selection happens on a
>      CPU while the set of masks which are selected via COS are shared
>      by all CPUs on a socket.
> 
> There are restrictions which CAT imposes in terms of configurability:
> 
>    - The bits which select a cache partition need to be consecutive
> 
>    - The number of possible cache association masks is limited
> 
> Let's look at the configurations (CDP omitted and size restricted)
> 
> Default:   1 1 1 1 1 1 1 1
> 	   1 1 1 1 1 1 1 1
> 	   1 1 1 1 1 1 1 1
> 	   1 1 1 1 1 1 1 1
> 
> Shared:	   1 1 1 1 1 1 1 1
> 	   0 0 1 1 1 1 1 1
> 	   0 0 0 0 1 1 1 1
> 	   0 0 0 0 0 0 1 1
> 
> Isolated:  1 1 1 1 0 0 0 0
> 	   0 0 0 0 1 1 0 0
> 	   0 0 0 0 0 0 1 0
> 	   0 0 0 0 0 0 0 1
> 
> Or any combination thereof. Surely some combinations will not make any
> sense, but we really should not make any restrictions on the stupidity
> of a sysadmin. The worst outcome might be L3 disabled for everything,
> so what?
> 
> Now that gets even more convoluted if CDP comes into play and we
> really need to look at CDP right now. We might end up with something
> which looks like this:
> 
>    	   1 1 1 1 0 0 0 0	Code
> 	   1 1 1 1 0 0 0 0	Data
> 	   0 0 0 0 0 0 1 0	Code
> 	   0 0 0 0 1 1 0 0	Data
> 	   0 0 0 0 0 0 0 1	Code
> 	   0 0 0 0 1 1 0 0	Data
> or 
> 	   0 0 0 0 0 0 0 1	Code
> 	   0 0 0 0 1 1 0 0	Data
> 	   0 0 0 0 0 0 0 1	Code
> 	   0 0 0 0 0 1 1 0	Data
> 
> Let's look at partitioning itself. We have two options:
> 
>    1) Per task partitioning
> 
>    2) Per CPU partitioning
> 
> So far we only talked about #1, but I think that #2 has a value as
> well. Let me give you a simple example.
> 
> Assume that you have isolated a CPU and run your important task on
> it. You give that task a slice of cache. Now that task needs kernel
> services which run in kernel threads on that CPU. We really don't want
> to (and cannot) hunt down random kernel threads (think cpu bound
> worker threads, softirq threads ....) and give them another slice of
> cache. What we really want is:
> 
>     	 1 1 1 1 0 0 0 0    <- Default cache
> 	 0 0 0 0 1 1 1 0    <- Cache for important task
> 	 0 0 0 0 0 0 0 1    <- Cache for CPU of important task
> 
> It would even be sufficient for particular use cases to just associate
> a piece of cache to a given CPU and do not bother with tasks at all.
> 
> We really need to make this as configurable as possible from userspace
> without imposing random restrictions to it. I played around with it on
> my new intel toy and the restriction to 16 COS ids (that's 8 with CDP
> enabled) makes it really useless if we force the ids to have the same
> meaning on all sockets and restrict it to per task partitioning.
> 
> Even if next generation systems will have more COS ids available,
> there are not going to be enough to have a system wide consistent
> view unless we have COS ids > nr_cpus.
> 
> Aside of that I don't think that a system wide consistent view is
> useful at all.
> 
>  - If a task migrates between sockets, it's going to suffer anyway.
>    Real sensitive applications will simply pin tasks on a socket to
>    avoid that in the first place. If we make the whole thing
>    configurable enough then the sysadmin can set it up to support
>    even the nonsensical case of identical cache partitions on all
>    sockets and let tasks use the corresponding partitions when
>    migrating.
> 
>  - The number of cache slices is going to be limited no matter what,
>    so one still has to come up with a sensible partitioning scheme.
> 
>  - Even if we have enough cos ids the system wide view will not make
>    the configuration problem any simpler as it remains per socket.
> 
> It's hard. Policies are hard by definition, but this one is harder
> than most other policies due to the inherent limitations.
> 
> So now to the interface part. Unfortunately we need to expose this
> very close to the hardware implementation as there are really no
> abstractions which allow us to express the various bitmap
> combinations. Any abstraction I tried to come up with renders that
> thing completely useless.
> 
> I was not able to identify any existing infrastructure where this
> really fits in. I chose a directory/file based representation. We
> certainly could do the same with a syscall, but that's just an
> implementation detail.
> 
> At top level:
> 
>    xxxxxxx/cat/max_cosids		<- Assume that all CPUs are the same
>    xxxxxxx/cat/max_maskbits		<- Assume that all CPUs are the same
>    xxxxxxx/cat/cdp_enable		<- Depends on CDP availability
> 
> Per socket data:
> 
>    xxxxxxx/cat/socket-0/
>    ...
>    xxxxxxx/cat/socket-N/l3_size
>    xxxxxxx/cat/socket-N/hwsharedbits
> 
> Per socket mask data:
> 
>    xxxxxxx/cat/socket-N/cos-id-0/
>    ...
>    xxxxxxx/cat/socket-N/cos-id-N/inuse
> 				/cat_mask	
> 				/cdp_mask	<- Data mask if CDP enabled
> 
> Per cpu default cos id for the cpus on that socket:
> 
>    xxxxxxx/cat/socket-N/cpu-x/default_cosid
>    ...
>    xxxxxxx/cat/socket-N/cpu-N/default_cosid
> 
> The above allows a simple cpu based partitioning. All tasks which do
> not have a cache partition assigned on a particular socket use the
> default one of the cpu they are running on.
> 
> Now for the task(s) partitioning:
> 
>    xxxxxxx/cat/partitions/
> 
> Under that directory one can create partitions
> 
>    xxxxxxx/cat/partitions/p1/tasks
> 			    /socket-0/cosid
> 			    ...
> 			    /socket-n/cosid
> 
>    The default value for the per socket cosid is COSID_DEFAULT, which
>    causes the task(s) to use the per cpu default id.
> 
> Thoughts?
> 
> Thanks,
> 
> 	tglx

The cgroups interface works, but moves the problem of contiguous
allocation to userspace, and is incompatible with cache allocations
on demand.

Have to solve the kernel threads VS cgroups issue...

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1272717

On Wed, Nov 18, 2015 at 08:34:07PM -0200, Marcelo Tosatti wrote:
> On Wed, Nov 18, 2015 at 07:25:03PM +0100, Thomas Gleixner wrote:
> > Folks!
> > 
> > After rereading the mail flood on CAT and staring into the SDM for a
> > while, I think we all should sit back and look at it from scratch
> > again w/o our preconceptions - I certainly had to put my own away.
> > 
> > Let's look at the properties of CAT again:
> > 
> >    - It's a per socket facility
> > 
> >    - CAT slots can be associated to external hardware. This
> >      association is per socket as well, so different sockets can have
> >      different behaviour. I missed that detail when staring the first
> >      time, thanks for the pointer!
> > 
> >    - The association ifself is per cpu. The COS selection happens on a
> >      CPU while the set of masks which are selected via COS are shared
> >      by all CPUs on a socket.
> > 
> > There are restrictions which CAT imposes in terms of configurability:
> > 
> >    - The bits which select a cache partition need to be consecutive
> > 
> >    - The number of possible cache association masks is limited
> > 
> > Let's look at the configurations (CDP omitted and size restricted)
> > 
> > Default:   1 1 1 1 1 1 1 1
> > 	   1 1 1 1 1 1 1 1
> > 	   1 1 1 1 1 1 1 1
> > 	   1 1 1 1 1 1 1 1
> > 
> > Shared:	   1 1 1 1 1 1 1 1
> > 	   0 0 1 1 1 1 1 1
> > 	   0 0 0 0 1 1 1 1
> > 	   0 0 0 0 0 0 1 1
> > 
> > Isolated:  1 1 1 1 0 0 0 0
> > 	   0 0 0 0 1 1 0 0
> > 	   0 0 0 0 0 0 1 0
> > 	   0 0 0 0 0 0 0 1
> > 
> > Or any combination thereof. Surely some combinations will not make any
> > sense, but we really should not make any restrictions on the stupidity
> > of a sysadmin. The worst outcome might be L3 disabled for everything,
> > so what?
> > 
> > Now that gets even more convoluted if CDP comes into play and we
> > really need to look at CDP right now. We might end up with something
> > which looks like this:
> > 
> >    	   1 1 1 1 0 0 0 0	Code
> > 	   1 1 1 1 0 0 0 0	Data
> > 	   0 0 0 0 0 0 1 0	Code
> > 	   0 0 0 0 1 1 0 0	Data
> > 	   0 0 0 0 0 0 0 1	Code
> > 	   0 0 0 0 1 1 0 0	Data
> > or 
> > 	   0 0 0 0 0 0 0 1	Code
> > 	   0 0 0 0 1 1 0 0	Data
> > 	   0 0 0 0 0 0 0 1	Code
> > 	   0 0 0 0 0 1 1 0	Data
> > 
> > Let's look at partitioning itself. We have two options:
> > 
> >    1) Per task partitioning
> > 
> >    2) Per CPU partitioning
> > 
> > So far we only talked about #1, but I think that #2 has a value as
> > well. Let me give you a simple example.
> > 
> > Assume that you have isolated a CPU and run your important task on
> > it. You give that task a slice of cache. Now that task needs kernel
> > services which run in kernel threads on that CPU. We really don't want
> > to (and cannot) hunt down random kernel threads (think cpu bound
> > worker threads, softirq threads ....) and give them another slice of
> > cache. What we really want is:
> > 
> >     	 1 1 1 1 0 0 0 0    <- Default cache
> > 	 0 0 0 0 1 1 1 0    <- Cache for important task
> > 	 0 0 0 0 0 0 0 1    <- Cache for CPU of important task
> > 
> > It would even be sufficient for particular use cases to just associate
> > a piece of cache to a given CPU and do not bother with tasks at all.

Well any work on behalf of the important task, should have its cache
protected as well (example irq handling threads). 

But for certain kernel tasks for which L3 cache is not beneficial
(eg: kernel samepage merging), it might useful to exclude such tasks
from the "important, do not flush" L3 cache portion.

> > We really need to make this as configurable as possible from userspace
> > without imposing random restrictions to it. I played around with it on
> > my new intel toy and the restriction to 16 COS ids (that's 8 with CDP
> > enabled) makes it really useless if we force the ids to have the same
> > meaning on all sockets and restrict it to per task partitioning.
> > 
> > Even if next generation systems will have more COS ids available,
> > there are not going to be enough to have a system wide consistent
> > view unless we have COS ids > nr_cpus.
> > 
> > Aside of that I don't think that a system wide consistent view is
> > useful at all.
> > 
> >  - If a task migrates between sockets, it's going to suffer anyway.
> >    Real sensitive applications will simply pin tasks on a socket to
> >    avoid that in the first place. If we make the whole thing
> >    configurable enough then the sysadmin can set it up to support
> >    even the nonsensical case of identical cache partitions on all
> >    sockets and let tasks use the corresponding partitions when
> >    migrating.
> > 
> >  - The number of cache slices is going to be limited no matter what,
> >    so one still has to come up with a sensible partitioning scheme.
> > 
> >  - Even if we have enough cos ids the system wide view will not make
> >    the configuration problem any simpler as it remains per socket.
> > 
> > It's hard. Policies are hard by definition, but this one is harder
> > than most other policies due to the inherent limitations.

That is exactly why it should be allowed for software to automatically 
configure the policies.

> > So now to the interface part. Unfortunately we need to expose this
> > very close to the hardware implementation as there are really no
> > abstractions which allow us to express the various bitmap
> > combinations. Any abstraction I tried to come up with renders that
> > thing completely useless.
> > 
> > I was not able to identify any existing infrastructure where this
> > really fits in. I chose a directory/file based representation. We
> > certainly could do the same with a syscall, but that's just an
> > implementation detail.
> > 
> > At top level:
> > 
> >    xxxxxxx/cat/max_cosids		<- Assume that all CPUs are the same
> >    xxxxxxx/cat/max_maskbits		<- Assume that all CPUs are the same
> >    xxxxxxx/cat/cdp_enable		<- Depends on CDP availability
> > 
> > Per socket data:
> > 
> >    xxxxxxx/cat/socket-0/
> >    ...
> >    xxxxxxx/cat/socket-N/l3_size
> >    xxxxxxx/cat/socket-N/hwsharedbits
> > 
> > Per socket mask data:
> > 
> >    xxxxxxx/cat/socket-N/cos-id-0/
> >    ...
> >    xxxxxxx/cat/socket-N/cos-id-N/inuse
> > 				/cat_mask	
> > 				/cdp_mask	<- Data mask if CDP enabled
> > 
> > Per cpu default cos id for the cpus on that socket:
> > 
> >    xxxxxxx/cat/socket-N/cpu-x/default_cosid
> >    ...
> >    xxxxxxx/cat/socket-N/cpu-N/default_cosid
> > 
> > The above allows a simple cpu based partitioning. All tasks which do
> > not have a cache partition assigned on a particular socket use the
> > default one of the cpu they are running on.
> > 
> > Now for the task(s) partitioning:
> > 
> >    xxxxxxx/cat/partitions/
> > 
> > Under that directory one can create partitions
> > 
> >    xxxxxxx/cat/partitions/p1/tasks
> > 			    /socket-0/cosid
> > 			    ...
> > 			    /socket-n/cosid
> > 
> >    The default value for the per socket cosid is COSID_DEFAULT, which
> >    causes the task(s) to use the per cpu default id.
> > 
> > Thoughts?
> > 
> > Thanks,
> > 
> > 	tglx
> 
> The cgroups interface works, but moves the problem of contiguous
> allocation to userspace, and is incompatible with cache allocations
> on demand.
> 
> Have to solve the kernel threads VS cgroups issue...
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1272929

On Wed, 18 Nov 2015, Marcelo Tosatti wrote:
> On Wed, Nov 18, 2015 at 08:34:07PM -0200, Marcelo Tosatti wrote:
> > On Wed, Nov 18, 2015 at 07:25:03PM +0100, Thomas Gleixner wrote:
> > > Assume that you have isolated a CPU and run your important task on
> > > it. You give that task a slice of cache. Now that task needs kernel
> > > services which run in kernel threads on that CPU. We really don't want
> > > to (and cannot) hunt down random kernel threads (think cpu bound
> > > worker threads, softirq threads ....) and give them another slice of
> > > cache. What we really want is:
> > > 
> > >     	 1 1 1 1 0 0 0 0    <- Default cache
> > > 	 0 0 0 0 1 1 1 0    <- Cache for important task
> > > 	 0 0 0 0 0 0 0 1    <- Cache for CPU of important task
> > > 
> > > It would even be sufficient for particular use cases to just associate
> > > a piece of cache to a given CPU and do not bother with tasks at all.
> 
> Well any work on behalf of the important task, should have its cache
> protected as well (example irq handling threads). 

Right, but that's nothing you can do automatically and certainly not
from a random application.

> But for certain kernel tasks for which L3 cache is not beneficial
> (eg: kernel samepage merging), it might useful to exclude such tasks
> from the "important, do not flush" L3 cache portion.

Sure it might be useful, but this needs to be done on a case by case
basis and there is no way to do this in any automated way.
 
> > > It's hard. Policies are hard by definition, but this one is harder
> > > than most other policies due to the inherent limitations.
> 
> That is exactly why it should be allowed for software to automatically 
> configure the policies.

There is nothing you can do automatically. If you want to allow
applications to set the policies themself, then you need to assign a
portion of the bitmask space and a portion of the cos id space to that
application and then let it do with that space what it wants.

That's where cgroups come into play. But that does not solve the other
issues of "global" configuration, i.e. CPU defaults etc.

Thanks,

	tglx
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1273138

On Thu, 19 Nov 2015 09:35:34 +0100 (CET)
Thomas Gleixner <tglx@linutronix.de> wrote:

> > Well any work on behalf of the important task, should have its cache
> > protected as well (example irq handling threads). 
> 
> Right, but that's nothing you can do automatically and certainly not
> from a random application.

Right and that's not a problem. For the use-cases CAT is intended to,
manual and per-workload system setup is very common. Things like
thread pinning, hugepages reservation, CPU isolation, nohz_full, etc
require manual setup too.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1274232

On Thu, Nov 19, 2015 at 09:35:34AM +0100, Thomas Gleixner wrote:
> On Wed, 18 Nov 2015, Marcelo Tosatti wrote:
> > On Wed, Nov 18, 2015 at 08:34:07PM -0200, Marcelo Tosatti wrote:
> > > On Wed, Nov 18, 2015 at 07:25:03PM +0100, Thomas Gleixner wrote:
> > > > Assume that you have isolated a CPU and run your important task on
> > > > it. You give that task a slice of cache. Now that task needs kernel
> > > > services which run in kernel threads on that CPU. We really don't want
> > > > to (and cannot) hunt down random kernel threads (think cpu bound
> > > > worker threads, softirq threads ....) and give them another slice of
> > > > cache. What we really want is:
> > > > 
> > > >     	 1 1 1 1 0 0 0 0    <- Default cache
> > > > 	 0 0 0 0 1 1 1 0    <- Cache for important task
> > > > 	 0 0 0 0 0 0 0 1    <- Cache for CPU of important task
> > > > 
> > > > It would even be sufficient for particular use cases to just associate
> > > > a piece of cache to a given CPU and do not bother with tasks at all.
> > 
> > Well any work on behalf of the important task, should have its cache
> > protected as well (example irq handling threads). 
> 
> Right, but that's nothing you can do automatically and certainly not
> from a random application.
> 
> > But for certain kernel tasks for which L3 cache is not beneficial
> > (eg: kernel samepage merging), it might useful to exclude such tasks
> > from the "important, do not flush" L3 cache portion.
> 
> Sure it might be useful, but this needs to be done on a case by case
> basis and there is no way to do this in any automated way.
>  
> > > > It's hard. Policies are hard by definition, but this one is harder
> > > > than most other policies due to the inherent limitations.
> > 
> > That is exactly why it should be allowed for software to automatically 
> > configure the policies.
> 
> There is nothing you can do automatically. 

Every cacheline brought in the L3 has a reaccess time (the time when it
was first brought in to the time it was reaccessed).

Assume you have a single threaded app, a sequence of cacheline
accesses.

Now if there are groups of accesses which have long reaccess times
(meaning that keeping them in L3 is not beneficial), that are large
enough to justify the OS notification, the application can notify the OS
to switch to a constrained COSid (so that L3 misses reclaim from that
small portion of the L3 cache).

> If you want to allow
> applications to set the policies themself, then you need to assign a
> portion of the bitmask space and a portion of the cos id space to that
> application and then let it do with that space what it wants.

Thats why you should specify the requirements independently of each
other (the requirement in this case the size of the reservation and
type, which is tied to the application), and let something else figure
out how they all fit together.

> That's where cgroups come into play. But that does not solve the other
> issues of "global" configuration, i.e. CPU defaults etc.

I don't understand what you mean issues of global configuration.

CPU defaults: A task is associated with a COSid. A COSid points to 
a set of CBMs (one CBM per socket). What defaults are you talking about?

But the interfaces do not exclude each other (the ioctl or syscall
interfaces and the manual direct MSR interface can coexist). There is
time pressure to integrate something workable for the present use cases
(none are in the class "applications set reservation themselves").

Peter has some objection against ioctls. So for something workable,
well have to handle the numbered issues pointed in the other e-mail
(2,3,4), in userspace.


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1272920

Marcelo,

On Wed, 18 Nov 2015, Marcelo Tosatti wrote:

Can you please trim your replies? It's really annoying having to
search for a single line of reply.

> The cgroups interface works, but moves the problem of contiguous
> allocation to userspace, and is incompatible with cache allocations
> on demand.
>
> Have to solve the kernel threads VS cgroups issue...

Sorry, I have no idea what you want to tell me.

Thanks,

	tglx
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1272704

On Wed, Nov 18, 2015 at 07:25:03PM +0100, Thomas Gleixner wrote:
> Folks!
> 
> After rereading the mail flood on CAT and staring into the SDM for a
> while, I think we all should sit back and look at it from scratch
> again w/o our preconceptions - I certainly had to put my own away.
> 
> Let's look at the properties of CAT again:
> 
>    - It's a per socket facility
> 
>    - CAT slots can be associated to external hardware. This
>      association is per socket as well, so different sockets can have
>      different behaviour. I missed that detail when staring the first
>      time, thanks for the pointer!
> 
>    - The association ifself is per cpu. The COS selection happens on a
>      CPU while the set of masks which are selected via COS are shared
>      by all CPUs on a socket.
> 
> There are restrictions which CAT imposes in terms of configurability:
> 
>    - The bits which select a cache partition need to be consecutive
> 
>    - The number of possible cache association masks is limited
> 
> Let's look at the configurations (CDP omitted and size restricted)
> 
> Default:   1 1 1 1 1 1 1 1
> 	   1 1 1 1 1 1 1 1
> 	   1 1 1 1 1 1 1 1
> 	   1 1 1 1 1 1 1 1
> 
> Shared:	   1 1 1 1 1 1 1 1
> 	   0 0 1 1 1 1 1 1
> 	   0 0 0 0 1 1 1 1
> 	   0 0 0 0 0 0 1 1
> 
> Isolated:  1 1 1 1 0 0 0 0
> 	   0 0 0 0 1 1 0 0
> 	   0 0 0 0 0 0 1 0
> 	   0 0 0 0 0 0 0 1
> 
> Or any combination thereof. Surely some combinations will not make any
> sense, but we really should not make any restrictions on the stupidity
> of a sysadmin. The worst outcome might be L3 disabled for everything,
> so what?
> 
> Now that gets even more convoluted if CDP comes into play and we
> really need to look at CDP right now. We might end up with something
> which looks like this:
> 
>    	   1 1 1 1 0 0 0 0	Code
> 	   1 1 1 1 0 0 0 0	Data
> 	   0 0 0 0 0 0 1 0	Code
> 	   0 0 0 0 1 1 0 0	Data
> 	   0 0 0 0 0 0 0 1	Code
> 	   0 0 0 0 1 1 0 0	Data
> or 
> 	   0 0 0 0 0 0 0 1	Code
> 	   0 0 0 0 1 1 0 0	Data
> 	   0 0 0 0 0 0 0 1	Code
> 	   0 0 0 0 0 1 1 0	Data
> 
> Let's look at partitioning itself. We have two options:
> 
>    1) Per task partitioning
> 
>    2) Per CPU partitioning
> 
> So far we only talked about #1, but I think that #2 has a value as
> well. Let me give you a simple example.
> 
> Assume that you have isolated a CPU and run your important task on
> it. You give that task a slice of cache. Now that task needs kernel
> services which run in kernel threads on that CPU. We really don't want
> to (and cannot) hunt down random kernel threads (think cpu bound
> worker threads, softirq threads ....) and give them another slice of
> cache. What we really want is:
> 
>     	 1 1 1 1 0 0 0 0    <- Default cache
> 	 0 0 0 0 1 1 1 0    <- Cache for important task
> 	 0 0 0 0 0 0 0 1    <- Cache for CPU of important task
> 
> It would even be sufficient for particular use cases to just associate
> a piece of cache to a given CPU and do not bother with tasks at all.
> 
> We really need to make this as configurable as possible from userspace
> without imposing random restrictions to it. I played around with it on
> my new intel toy and the restriction to 16 COS ids (that's 8 with CDP
> enabled) makes it really useless if we force the ids to have the same
> meaning on all sockets and restrict it to per task partitioning.
> 
> Even if next generation systems will have more COS ids available,
> there are not going to be enough to have a system wide consistent
> view unless we have COS ids > nr_cpus.
> 
> Aside of that I don't think that a system wide consistent view is
> useful at all.
> 
>  - If a task migrates between sockets, it's going to suffer anyway.
>    Real sensitive applications will simply pin tasks on a socket to
>    avoid that in the first place. If we make the whole thing
>    configurable enough then the sysadmin can set it up to support
>    even the nonsensical case of identical cache partitions on all
>    sockets and let tasks use the corresponding partitions when
>    migrating.
> 
>  - The number of cache slices is going to be limited no matter what,
>    so one still has to come up with a sensible partitioning scheme.
> 
>  - Even if we have enough cos ids the system wide view will not make
>    the configuration problem any simpler as it remains per socket.
> 
> It's hard. Policies are hard by definition, but this one is harder
> than most other policies due to the inherent limitations.
> 
> So now to the interface part. Unfortunately we need to expose this
> very close to the hardware implementation as there are really no
> abstractions which allow us to express the various bitmap
> combinations. Any abstraction I tried to come up with renders that
> thing completely useless.

No you don't.

> I was not able to identify any existing infrastructure where this
> really fits in. I chose a directory/file based representation. We
> certainly could do the same with a syscall, but that's just an
> implementation detail.
> 
> At top level:
> 
>    xxxxxxx/cat/max_cosids		<- Assume that all CPUs are the same
>    xxxxxxx/cat/max_maskbits		<- Assume that all CPUs are the same
>    xxxxxxx/cat/cdp_enable		<- Depends on CDP availability
> 
> Per socket data:
> 
>    xxxxxxx/cat/socket-0/
>    ...
>    xxxxxxx/cat/socket-N/l3_size
>    xxxxxxx/cat/socket-N/hwsharedbits
> 
> Per socket mask data:
> 
>    xxxxxxx/cat/socket-N/cos-id-0/
>    ...
>    xxxxxxx/cat/socket-N/cos-id-N/inuse
> 				/cat_mask	
> 				/cdp_mask	<- Data mask if CDP enabled

There is no need to expose all this to userspace, but for some unknown 
reason people seem to be fond of that, so lets pretend its necessary.

> Per cpu default cos id for the cpus on that socket:
> 
>    xxxxxxx/cat/socket-N/cpu-x/default_cosid
>    ...
>    xxxxxxx/cat/socket-N/cpu-N/default_cosid
> 
> The above allows a simple cpu based partitioning. All tasks which do
> not have a cache partition assigned on a particular socket use the
> default one of the cpu they are running on.

A tasks which does not have a partition assigned to it 
has to use the "other tasks" group (COSid0), so that it does 
not interfere with the cache reservations of other tasks.

All is necessary are reservations {size,type}, and lists of reservations
per tasks. This is the right level to expose this to userspace without
userspace having to care about unnecessary HW details.

> Now for the task(s) partitioning:
> 
>    xxxxxxx/cat/partitions/
> 
> Under that directory one can create partitions
> 
>    xxxxxxx/cat/partitions/p1/tasks
> 			    /socket-0/cosid
> 			    ...
> 			    /socket-n/cosid
> 
>    The default value for the per socket cosid is COSID_DEFAULT, which
>    causes the task(s) to use the per cpu default id.
> 
> Thoughts?
> 
> Thanks,
> 
> 	tglx

Again: you don't need to look into the MSR table and relate it 
to tasks if you store the data as:

	task group 1 = {
			reservation-1 = {size = 80Kb, type = data, socketmask = 0xffff},
			reservation-2 = {size = 100Kb, type = code, socketmask = 0xffff}
	}
	
	task group 2 = {
			reservation-1 = {size = 80Kb, type = data, socketmask = 0xffff},
			reservation-3 = {size = 200Kb, type = code, socketmask = 0xffff}
	}

Task group 1 and task group 2 share reservation-1.

This is what userspace is going to expose to users, of course.

If you expose the MSRs to userspace, you force userspace to convert
from this format to the MSRs (minding whether there
are contiguous regions available, and the region shared with HW).

    - The bits which select a cache partition need to be consecutive

BUT, for our usecase the cgroups interface works as well, so lets
go with that (Tejun apparently had a usecase where tasks were allowed to 
set reservations themselves, on response to external events).


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1272728

On Wed, Nov 18, 2015 at 10:01:53PM -0200, Marcelo Tosatti wrote:
> On Wed, Nov 18, 2015 at 07:25:03PM +0100, Thomas Gleixner wrote:
> > Folks!
> > 
> > After rereading the mail flood on CAT and staring into the SDM for a
> > while, I think we all should sit back and look at it from scratch
> > again w/o our preconceptions - I certainly had to put my own away.
> > 
> > Let's look at the properties of CAT again:
> > 
> >    - It's a per socket facility
> > 
> >    - CAT slots can be associated to external hardware. This
> >      association is per socket as well, so different sockets can have
> >      different behaviour. I missed that detail when staring the first
> >      time, thanks for the pointer!
> > 
> >    - The association ifself is per cpu. The COS selection happens on a
> >      CPU while the set of masks which are selected via COS are shared
> >      by all CPUs on a socket.
> > 
> > There are restrictions which CAT imposes in terms of configurability:
> > 
> >    - The bits which select a cache partition need to be consecutive
> > 
> >    - The number of possible cache association masks is limited
> > 
> > Let's look at the configurations (CDP omitted and size restricted)
> > 
> > Default:   1 1 1 1 1 1 1 1
> > 	   1 1 1 1 1 1 1 1
> > 	   1 1 1 1 1 1 1 1
> > 	   1 1 1 1 1 1 1 1
> > 
> > Shared:	   1 1 1 1 1 1 1 1
> > 	   0 0 1 1 1 1 1 1
> > 	   0 0 0 0 1 1 1 1
> > 	   0 0 0 0 0 0 1 1
> > 
> > Isolated:  1 1 1 1 0 0 0 0
> > 	   0 0 0 0 1 1 0 0
> > 	   0 0 0 0 0 0 1 0
> > 	   0 0 0 0 0 0 0 1
> > 
> > Or any combination thereof. Surely some combinations will not make any
> > sense, but we really should not make any restrictions on the stupidity
> > of a sysadmin. The worst outcome might be L3 disabled for everything,
> > so what?
> > 
> > Now that gets even more convoluted if CDP comes into play and we
> > really need to look at CDP right now. We might end up with something
> > which looks like this:
> > 
> >    	   1 1 1 1 0 0 0 0	Code
> > 	   1 1 1 1 0 0 0 0	Data
> > 	   0 0 0 0 0 0 1 0	Code
> > 	   0 0 0 0 1 1 0 0	Data
> > 	   0 0 0 0 0 0 0 1	Code
> > 	   0 0 0 0 1 1 0 0	Data
> > or 
> > 	   0 0 0 0 0 0 0 1	Code
> > 	   0 0 0 0 1 1 0 0	Data
> > 	   0 0 0 0 0 0 0 1	Code
> > 	   0 0 0 0 0 1 1 0	Data
> > 
> > Let's look at partitioning itself. We have two options:
> > 
> >    1) Per task partitioning
> > 
> >    2) Per CPU partitioning
> > 
> > So far we only talked about #1, but I think that #2 has a value as
> > well. Let me give you a simple example.
> > 
> > Assume that you have isolated a CPU and run your important task on
> > it. You give that task a slice of cache. Now that task needs kernel
> > services which run in kernel threads on that CPU. We really don't want
> > to (and cannot) hunt down random kernel threads (think cpu bound
> > worker threads, softirq threads ....) and give them another slice of
> > cache. What we really want is:
> > 
> >     	 1 1 1 1 0 0 0 0    <- Default cache
> > 	 0 0 0 0 1 1 1 0    <- Cache for important task
> > 	 0 0 0 0 0 0 0 1    <- Cache for CPU of important task
> > 
> > It would even be sufficient for particular use cases to just associate
> > a piece of cache to a given CPU and do not bother with tasks at all.
> > 
> > We really need to make this as configurable as possible from userspace
> > without imposing random restrictions to it. I played around with it on
> > my new intel toy and the restriction to 16 COS ids (that's 8 with CDP
> > enabled) makes it really useless if we force the ids to have the same
> > meaning on all sockets and restrict it to per task partitioning.
> > 
> > Even if next generation systems will have more COS ids available,
> > there are not going to be enough to have a system wide consistent
> > view unless we have COS ids > nr_cpus.
> > 
> > Aside of that I don't think that a system wide consistent view is
> > useful at all.
> > 
> >  - If a task migrates between sockets, it's going to suffer anyway.
> >    Real sensitive applications will simply pin tasks on a socket to
> >    avoid that in the first place. If we make the whole thing
> >    configurable enough then the sysadmin can set it up to support
> >    even the nonsensical case of identical cache partitions on all
> >    sockets and let tasks use the corresponding partitions when
> >    migrating.
> > 
> >  - The number of cache slices is going to be limited no matter what,
> >    so one still has to come up with a sensible partitioning scheme.
> > 
> >  - Even if we have enough cos ids the system wide view will not make
> >    the configuration problem any simpler as it remains per socket.
> > 
> > It's hard. Policies are hard by definition, but this one is harder
> > than most other policies due to the inherent limitations.
> > 
> > So now to the interface part. Unfortunately we need to expose this
> > very close to the hardware implementation as there are really no
> > abstractions which allow us to express the various bitmap
> > combinations. Any abstraction I tried to come up with renders that
> > thing completely useless.
> 
> No you don't.

Actually, there is a point that is useful: you might want the important
application to share the L3 portion with HW (that HW DMAs into), and
have only the application and the HW use that region.

So its a good point that controlling the exact position of the reservation 
is important.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1272950

On Wed, 18 Nov 2015, Marcelo Tosatti wrote
> Actually, there is a point that is useful: you might want the important
> application to share the L3 portion with HW (that HW DMAs into), and
> have only the application and the HW use that region.
> 
> So its a good point that controlling the exact position of the reservation 
> is important.

I'm glad you figured that out yourself. :)

Thanks,

	tglx
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1273555

On Thu, Nov 19, 2015 at 10:09:03AM +0100, Thomas Gleixner wrote:
> On Wed, 18 Nov 2015, Marcelo Tosatti wrote
> > Actually, there is a point that is useful: you might want the important
> > application to share the L3 portion with HW (that HW DMAs into), and
> > have only the application and the HW use that region.
> > 
> > So its a good point that controlling the exact position of the reservation 
> > is important.
> 
> I'm glad you figured that out yourself. :)
> 
> Thanks,
> 
> 	tglx

The HW is a reclaimer of the L3 region shared with HW.

You might want to remove any threads from reclaiming from 
that region.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1273771

On Thu, 19 Nov 2015, Marcelo Tosatti wrote:
> On Thu, Nov 19, 2015 at 10:09:03AM +0100, Thomas Gleixner wrote:
> > On Wed, 18 Nov 2015, Marcelo Tosatti wrote
> > > Actually, there is a point that is useful: you might want the important
> > > application to share the L3 portion with HW (that HW DMAs into), and
> > > have only the application and the HW use that region.
> > > 
> > > So its a good point that controlling the exact position of the reservation 
> > > is important.
> > 
> > I'm glad you figured that out yourself. :)
> > 
> > Thanks,
> > 
> > 	tglx
> 
> The HW is a reclaimer of the L3 region shared with HW.
> 
> You might want to remove any threads from reclaiming from 
> that region.

I might for some threads, but certainly not for those which need to
access DMA buffers. Throwing away 10% of L3 just because you don't
want to deal with it at the interface level is hillarious.

Thanks,

	tglx
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1274367

On Fri, Nov 20, 2015 at 08:53:34AM +0100, Thomas Gleixner wrote:
> On Thu, 19 Nov 2015, Marcelo Tosatti wrote:
> > On Thu, Nov 19, 2015 at 10:09:03AM +0100, Thomas Gleixner wrote:
> > > On Wed, 18 Nov 2015, Marcelo Tosatti wrote
> > > > Actually, there is a point that is useful: you might want the important
> > > > application to share the L3 portion with HW (that HW DMAs into), and
> > > > have only the application and the HW use that region.
> > > > 
> > > > So its a good point that controlling the exact position of the reservation 
> > > > is important.
> > > 
> > > I'm glad you figured that out yourself. :)
> > > 
> > > Thanks,
> > > 
> > > 	tglx
> > 
> > The HW is a reclaimer of the L3 region shared with HW.
> > 
> > You might want to remove any threads from reclaiming from 
> > that region.
> 
> I might for some threads, but certainly not for those which need to
> access DMA buffers.

Yes, when i wrote "its a good point that controlling the exact position
of the reservation is important" i had that in mind as well.

But its wrong: not having a bit set in the CBM for the portion of L3
cache which is shared with HW only means "for cacheline misses of the
application, evict cachelines from this portion".

So yes, you might want to exclude the application which accesses DMA
buffers from reclaiming cachelines in the portion shared with HW,
to keep those cachelines longer in L3.

> Throwing away 10% of L3 just because you don't
> want to deal with it at the interface level is hillarious.

If there is interest on per-application configuration then it can 
be integrated as well.

Thanks for your time.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1273480

On Wed, Nov 18, 2015 at 11:05:35PM -0200, Marcelo Tosatti wrote:
> On Wed, Nov 18, 2015 at 10:01:53PM -0200, Marcelo Tosatti wrote:
> > On Wed, Nov 18, 2015 at 07:25:03PM +0100, Thomas Gleixner wrote:
> > > Folks!
> > > 
> > > After rereading the mail flood on CAT and staring into the SDM for a
> > > while, I think we all should sit back and look at it from scratch
> > > again w/o our preconceptions - I certainly had to put my own away.
> > > 
> > > Let's look at the properties of CAT again:
> > > 
> > >    - It's a per socket facility
> > > 
> > >    - CAT slots can be associated to external hardware. This
> > >      association is per socket as well, so different sockets can have
> > >      different behaviour. I missed that detail when staring the first
> > >      time, thanks for the pointer!
> > > 
> > >    - The association ifself is per cpu. The COS selection happens on a
> > >      CPU while the set of masks which are selected via COS are shared
> > >      by all CPUs on a socket.
> > > 
> > > There are restrictions which CAT imposes in terms of configurability:
> > > 
> > >    - The bits which select a cache partition need to be consecutive
> > > 
> > >    - The number of possible cache association masks is limited
> > > 
> > > Let's look at the configurations (CDP omitted and size restricted)
> > > 
> > > Default:   1 1 1 1 1 1 1 1
> > > 	   1 1 1 1 1 1 1 1
> > > 	   1 1 1 1 1 1 1 1
> > > 	   1 1 1 1 1 1 1 1
> > > 
> > > Shared:	   1 1 1 1 1 1 1 1
> > > 	   0 0 1 1 1 1 1 1
> > > 	   0 0 0 0 1 1 1 1
> > > 	   0 0 0 0 0 0 1 1
> > > 
> > > Isolated:  1 1 1 1 0 0 0 0
> > > 	   0 0 0 0 1 1 0 0
> > > 	   0 0 0 0 0 0 1 0
> > > 	   0 0 0 0 0 0 0 1
> > > 
> > > Or any combination thereof. Surely some combinations will not make any
> > > sense, but we really should not make any restrictions on the stupidity
> > > of a sysadmin. The worst outcome might be L3 disabled for everything,
> > > so what?
> > > 
> > > Now that gets even more convoluted if CDP comes into play and we
> > > really need to look at CDP right now. We might end up with something
> > > which looks like this:
> > > 
> > >    	   1 1 1 1 0 0 0 0	Code
> > > 	   1 1 1 1 0 0 0 0	Data
> > > 	   0 0 0 0 0 0 1 0	Code
> > > 	   0 0 0 0 1 1 0 0	Data
> > > 	   0 0 0 0 0 0 0 1	Code
> > > 	   0 0 0 0 1 1 0 0	Data
> > > or 
> > > 	   0 0 0 0 0 0 0 1	Code
> > > 	   0 0 0 0 1 1 0 0	Data
> > > 	   0 0 0 0 0 0 0 1	Code
> > > 	   0 0 0 0 0 1 1 0	Data
> > > 
> > > Let's look at partitioning itself. We have two options:
> > > 
> > >    1) Per task partitioning
> > > 
> > >    2) Per CPU partitioning
> > > 
> > > So far we only talked about #1, but I think that #2 has a value as
> > > well. Let me give you a simple example.
> > > 
> > > Assume that you have isolated a CPU and run your important task on
> > > it. You give that task a slice of cache. Now that task needs kernel
> > > services which run in kernel threads on that CPU. We really don't want
> > > to (and cannot) hunt down random kernel threads (think cpu bound
> > > worker threads, softirq threads ....) and give them another slice of
> > > cache. What we really want is:
> > > 
> > >     	 1 1 1 1 0 0 0 0    <- Default cache
> > > 	 0 0 0 0 1 1 1 0    <- Cache for important task
> > > 	 0 0 0 0 0 0 0 1    <- Cache for CPU of important task
> > > 
> > > It would even be sufficient for particular use cases to just associate
> > > a piece of cache to a given CPU and do not bother with tasks at all.
> > > 
> > > We really need to make this as configurable as possible from userspace
> > > without imposing random restrictions to it. I played around with it on
> > > my new intel toy and the restriction to 16 COS ids (that's 8 with CDP
> > > enabled) makes it really useless if we force the ids to have the same
> > > meaning on all sockets and restrict it to per task partitioning.
> > > 
> > > Even if next generation systems will have more COS ids available,
> > > there are not going to be enough to have a system wide consistent
> > > view unless we have COS ids > nr_cpus.
> > > 
> > > Aside of that I don't think that a system wide consistent view is
> > > useful at all.
> > > 
> > >  - If a task migrates between sockets, it's going to suffer anyway.
> > >    Real sensitive applications will simply pin tasks on a socket to
> > >    avoid that in the first place. If we make the whole thing
> > >    configurable enough then the sysadmin can set it up to support
> > >    even the nonsensical case of identical cache partitions on all
> > >    sockets and let tasks use the corresponding partitions when
> > >    migrating.
> > > 
> > >  - The number of cache slices is going to be limited no matter what,
> > >    so one still has to come up with a sensible partitioning scheme.
> > > 
> > >  - Even if we have enough cos ids the system wide view will not make
> > >    the configuration problem any simpler as it remains per socket.
> > > 
> > > It's hard. Policies are hard by definition, but this one is harder
> > > than most other policies due to the inherent limitations.
> > > 
> > > So now to the interface part. Unfortunately we need to expose this
> > > very close to the hardware implementation as there are really no
> > > abstractions which allow us to express the various bitmap
> > > combinations. Any abstraction I tried to come up with renders that
> > > thing completely useless.
> > 
> > No you don't.
> 
> Actually, there is a point that is useful: you might want the important
> application to share the L3 portion with HW (that HW DMAs into), and
> have only the application and the HW use that region.

Actually, don't see why that makes sense.

So "share the L3 portion" means being allowed to reclaim data from that
portion of L3 cache.

Why would you want to allow application and HW to reclaim from the same 
region? I don't know.

But exposing the HW interface allows you to do that, if some reason 
for doing so exists.

Exposing the HW interface:
--------------------------

Pros: *1) Can do whatever combination necessary.
Cons: *2) Userspace has to deal with the contiguity issue
      (example: upon allocation request, "compacting" the cbm bits
       can allow for allocation request to be succesful (that is
       enough contiguous bits available), but "compacting" means 
       moving CBM bits around which means applications will lose 
       their reservation at the time the CBM bit positions are moved, 
       so it can affect ongoing code).
      *3) Userspace has to deal with convertion from kbytes to cache ways.
      *4) Userspace has to deal with locking access to the interface.
      * Userspace has no access to the timing of schedin/schedouts, 
        so it cannot perform optimizations based on that information.

Not exposing the HW interface:
------------------------------

Pros: *10) Can use whatever combination necessary, provided that you 
extend the interface.
*11) Allows the kernel to optimize usage of the reservations, because only 
the kernel knows the times of scheduling.
*12) Allows the kernel to handle 2,3,4, rather than having userspace
handle it.
*13) Allows applications to set cache reservations themselves, directly 
via an ioctl or system call.

Cons:
* There are users of the cgroups interface today, they will have 
to change.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1272948

On Wed, 18 Nov 2015, Marcelo Tosatti wrote:
> On Wed, Nov 18, 2015 at 07:25:03PM +0100, Thomas Gleixner wrote:
> > So now to the interface part. Unfortunately we need to expose this
> > very close to the hardware implementation as there are really no
> > abstractions which allow us to express the various bitmap
> > combinations. Any abstraction I tried to come up with renders that
> > thing completely useless.
> 
> No you don't.

Because you have a use case which allows you to write some policy
translator? I seriously doubt that it is general enough.
 
> Again: you don't need to look into the MSR table and relate it 
> to tasks if you store the data as:
> 
> 	task group 1 = {
> 			reservation-1 = {size = 80Kb, type = data, socketmask = 0xffff},
> 			reservation-2 = {size = 100Kb, type = code, socketmask = 0xffff}
> 	}
> 	
> 	task group 2 = {
> 			reservation-1 = {size = 80Kb, type = data, socketmask = 0xffff},
> 			reservation-3 = {size = 200Kb, type = code, socketmask = 0xffff}
> 	}
> 
> Task group 1 and task group 2 share reservation-1.
> 
> This is what userspace is going to expose to users, of course.


 
> If you expose the MSRs to userspace, you force userspace to convert
> from this format to the MSRs (minding whether there
> are contiguous regions available, and the region shared with HW).

Fair enough. I'm not too fond about the exposure of the MSRs, but I
chose this just to explain the full problem space and the various
requirements we might have accross the full application space.

If we can come up with an abstract way which does not impose
restrictions on the overall configuration abilities, I'm all for it.

>     - The bits which select a cache partition need to be consecutive
> 
> BUT, for our usecase the cgroups interface works as well, so lets
> go with that (Tejun apparently had a usecase where tasks were allowed to 
> set reservations themselves, on response to external events).

Can you please set aside your narrow use case view for a moment and
just think about the full application space? We are not designing such
an interface for a single use case.

Thanks,

	tglx
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1276160

On Wed, Nov 18, 2015 at 10:01:54PM -0200, Marcelo Tosatti wrote:
> > 	tglx
> 
> Again: you don't need to look into the MSR table and relate it 
> to tasks if you store the data as:
> 
> 	task group 1 = {
> 			reservation-1 = {size = 80Kb, type = data, socketmask = 0xffff},
> 			reservation-2 = {size = 100Kb, type = code, socketmask = 0xffff}
> 	}
> 	
> 	task group 2 = {
> 			reservation-1 = {size = 80Kb, type = data, socketmask = 0xffff},
> 			reservation-3 = {size = 200Kb, type = code, socketmask = 0xffff}
> 	}
> 
> Task group 1 and task group 2 share reservation-1.

Because there is only size but not CBM position info, I guess for
different reservations they will not overlap each other, right?

Personally I like this way of exposing minimal information to userspace.
I can think it working well except for one concern of losing flexibility:

For instance, there is a box for which the full CBM is 0xfffff. After
cache reservation creating/freeing for a while we then have reservations:

reservation1: 0xf0000
reservation2: 0x00ff0

Now people want to request a reservation which size is 0xff, so how
will kernel do at this time? It could return just error or do some
moving/merging (e.g. for reservation2: 0x00ff0 => 0x0ff00) and then
satisfy the request. But I don't know if the moving/merging will cause
delay for tasks that is using it.

Thanks,
Chao
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1277832

On Tue, Nov 24, 2015 at 07:25:43PM -0200, Marcelo Tosatti wrote:
> On Tue, Nov 24, 2015 at 04:27:54PM +0800, Chao Peng wrote:
> > On Wed, Nov 18, 2015 at 10:01:54PM -0200, Marcelo Tosatti wrote:
> > > > 	tglx
> > > 
> > > Again: you don't need to look into the MSR table and relate it 
> > > to tasks if you store the data as:
> > > 
> > > 	task group 1 = {
> > > 			reservation-1 = {size = 80Kb, type = data, socketmask = 0xffff},
> > > 			reservation-2 = {size = 100Kb, type = code, socketmask = 0xffff}
> > > 	}
> > > 	
> > > 	task group 2 = {
> > > 			reservation-1 = {size = 80Kb, type = data, socketmask = 0xffff},
> > > 			reservation-3 = {size = 200Kb, type = code, socketmask = 0xffff}
> > > 	}
> > > 
> > > Task group 1 and task group 2 share reservation-1.
> > 
> > Because there is only size but not CBM position info, I guess for
> > different reservations they will not overlap each other, right?
> 
> Reservation 1 is shared between task group 1 and task group 2 
> so the CBMs overlap (by 80Kb, rounded).
> 
> > Personally I like this way of exposing minimal information to userspace.
> > I can think it working well except for one concern of losing flexibility:
> > 
> > For instance, there is a box for which the full CBM is 0xfffff. After
> > cache reservation creating/freeing for a while we then have reservations:
> > 
> > reservation1: 0xf0000
> > reservation2: 0x00ff0
> > 
> > Now people want to request a reservation which size is 0xff, so how
> > will kernel do at this time? It could return just error or do some
> > moving/merging (e.g. for reservation2: 0x00ff0 => 0x0ff00) and then
> > satisfy the request. But I don't know if the moving/merging will cause
> > delay for tasks that is using it.
> 
> Right, i was thinking of adding a "force" parameter. 
> 
> So, default behaviour of attach: do not merge.
> "force" behaviour of attach: move reservations around and merge if
> necessary.

To make the decision userspace would need the know that a merge can
be performed if particular reservations can be moved (that is, the
moveable property is per-reservation, depending on whether its ok 
for the given app to cacheline fault or not).
Anyway, thats for later.






--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1276106

On Wed, Nov 18, 2015 at 07:25:03PM +0100, Thomas Gleixner wrote:
> 
> Let's look at partitioning itself. We have two options:
> 
>    1) Per task partitioning
> 
>    2) Per CPU partitioning
> 
> So far we only talked about #1, but I think that #2 has a value as
> well. Let me give you a simple example.

I would second this. In practice per CPU partitioning is useful for
realtime as well. And I can see three possible solutions:

     1) What you suggested below, to address both problems in one
        framework. But I wonder if it would end with too complex.

     2) Achieve per CPU partitioning with per task partitioning. For
        example, if current CAT patch can solve the kernel threads
	problem, together with CPU pinning, we then can set a same CBM
	for all the tasks/kernel threads run on an isolated CPU. 

     3) I wonder if it feasible to separate the two requirements? For
        example, divides the work into three components: rdt-base,
	per task interface (current cgroup interface/IOCTL or something)
	and per CPU interface. The two interfaces are exclusive and
	selected at build time. One thing to reject this option would be
	even with per CPU partitioning, we still need per task partitioning,
	in that case we will go to option 1) again.

Thanks,
Chao
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | linux.kernel

csiph-web