Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1244201 > unrolled thread

Re: [PATCH V15 00/11] x86: Intel Cache Allocation Technology Support

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH V15 00/11] x86: Intel Cache Allocation Technology  Support Thomas Gleixner <tglx@linutronix.de> - 2015-10-11 22:00 +0200
    RE: [PATCH V15 00/11] x86: Intel Cache Allocation Technology Support "Yu, Fenghua" <fenghua.yu@intel.com> - 2015-10-12 21:00 +0200
      RE: [PATCH V15 00/11] x86: Intel Cache Allocation Technology  Support Thomas Gleixner <tglx@linutronix.de> - 2015-10-12 22:00 +0200
      Re: [PATCH V15 00/11] x86: Intel Cache Allocation Technology Support Marcelo Tosatti <mtosatti@redhat.com> - 2015-10-15 00:40 +0200
        Re: [PATCH V15 00/11] x86: Intel Cache Allocation Technology Support Peter Zijlstra <peterz@infradead.org> - 2015-10-15 13:40 +0200
          Re: [PATCH V15 00/11] x86: Intel Cache Allocation Technology Support Marcelo Tosatti <mtosatti@redhat.com> - 2015-10-16 03:50 +0200
            Re: [PATCH V15 00/11] x86: Intel Cache Allocation Technology Support Peter Zijlstra <peterz@infradead.org> - 2015-10-16 11:50 +0200
    Re: [PATCH V15 00/11] x86: Intel Cache Allocation Technology Support Marcelo Tosatti <mtosatti@redhat.com> - 2015-10-15 00:40 +0200
      Re: [PATCH V15 00/11] x86: Intel Cache Allocation Technology Support Peter Zijlstra <peterz@infradead.org> - 2015-10-15 13:40 +0200
        Re: [PATCH V15 00/11] x86: Intel Cache Allocation Technology Support Marcelo Tosatti <mtosatti@redhat.com> - 2015-10-16 04:30 +0200
          Re: [PATCH V15 00/11] x86: Intel Cache Allocation Technology Support Peter Zijlstra <peterz@infradead.org> - 2015-10-16 12:00 +0200

#1244201 — Re: [PATCH V15 00/11] x86: Intel Cache Allocation Technology Support

Fenghua,

On Thu, 1 Oct 2015, Fenghua Yu wrote:

+Cc: Marcelo

> This series has some preparatory patches and Intel cache allocation
> support.

<snip>

> Changes in v15:
>  - Add a global IPI to update the closid on CPUs for current tasks.
>  - Other minor changes where I remove the updating of clos_cbm_table to be
>  set to all 1s during init.
>  - Fix a few compilation warnings.
>  - Port the patches to 4.3-rc.
 
What's the state of the interface discussion? I have not yet seen any
agreement on that, unless I missed the important mail.

Aside of that, the patches miss a proper

From: Vikas ...

in the patch body at least for those which are untouched by you.

Thanks,

	tglx

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1245025 — RE: [PATCH V15 00/11] x86: Intel Cache Allocation Technology Support

> From: Thomas Gleixner [mailto:tglx@linutronix.de]
> Sent: Sunday, October 11, 2015 12:50 PM
> To: Yu, Fenghua
> Cc: H Peter Anvin; Ingo Molnar; Peter Zijlstra; linux-kernel; x86; Vikas
> Shivappa; Marcelo Tosatti
> Subject: Re: [PATCH V15 00/11] x86: Intel Cache Allocation Technology
> Support
> 
> Fenghua,
> 
> On Thu, 1 Oct 2015, Fenghua Yu wrote:
> 
> +Cc: Marcelo
> 
> > This series has some preparatory patches and Intel cache allocation
> > support.
> 
> <snip>
> 
> > Changes in v15:
> >  - Add a global IPI to update the closid on CPUs for current tasks.
> >  - Other minor changes where I remove the updating of clos_cbm_table
> > to be  set to all 1s during init.
> >  - Fix a few compilation warnings.
> >  - Port the patches to 4.3-rc.
> 
> What's the state of the interface discussion? I have not yet seen any
> agreement on that, unless I missed the important mail.

Peter Anvin will discuss the interface with Tejun during the Kernel Summit. Hopefully Tejun will agree with the current cgroup interface in the patch set.

> 
> Aside of that, the patches miss a proper
> 
> From: Vikas ...
> 
> in the patch body at least for those which are untouched by you.

I'll send patch set v16 today with "From: Vikas Shivappa <vikas.shivappa@linux.intel.com>" in the patches.

Vikas is on sabbatical until Dec. I'm covering him during his leave.

Thanks.

-Fenghua

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1245081

On Mon, 12 Oct 2015, Yu, Fenghua wrote:
> > What's the state of the interface discussion? I have not yet seen any
> > agreement on that, unless I missed the important mail.
> 
> Peter Anvin will discuss the interface with Tejun during the Kernel
> Summit. Hopefully Tejun will agree with the current cgroup interface
> in the patch set.

I hope Marcelo is there as well. He had very good arguments and actual
use cases.

> I'll send patch set v16 today with "From: Vikas Shivappa
> <vikas.shivappa@linux.intel.com>" in the patches.

Please don't before the interface is sorted out.

Thanks,

	tglx
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1247230 — Re: [PATCH V15 00/11] x86: Intel Cache Allocation Technology Support

On Mon, Oct 12, 2015 at 06:52:49PM +0000, Yu, Fenghua wrote:
> > From: Thomas Gleixner [mailto:tglx@linutronix.de]
> > Sent: Sunday, October 11, 2015 12:50 PM
> > To: Yu, Fenghua
> > Cc: H Peter Anvin; Ingo Molnar; Peter Zijlstra; linux-kernel; x86; Vikas
> > Shivappa; Marcelo Tosatti
> > Subject: Re: [PATCH V15 00/11] x86: Intel Cache Allocation Technology
> > Support
> > 
> > Fenghua,
> > 
> > On Thu, 1 Oct 2015, Fenghua Yu wrote:
> > 
> > +Cc: Marcelo
> > 
> > > This series has some preparatory patches and Intel cache allocation
> > > support.
> > 
> > <snip>
> > 
> > > Changes in v15:
> > >  - Add a global IPI to update the closid on CPUs for current tasks.
> > >  - Other minor changes where I remove the updating of clos_cbm_table
> > > to be  set to all 1s during init.
> > >  - Fix a few compilation warnings.
> > >  - Port the patches to 4.3-rc.
> > 
> > What's the state of the interface discussion? I have not yet seen any
> > agreement on that, unless I missed the important mail.
> 
> Peter Anvin will discuss the interface with Tejun during the Kernel Summit. Hopefully Tejun will agree with the current cgroup interface in the patch set.

How can you fix the issue of sockets with different reserved cache
regions with hw in the cgroup interface?

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1247703 — Re: [PATCH V15 00/11] x86: Intel Cache Allocation Technology Support

On Tue, Oct 13, 2015 at 07:40:58PM -0300, Marcelo Tosatti wrote:
> How can you fix the issue of sockets with different reserved cache
> regions with hw in the cgroup interface?

No idea what you're referring to. But IOCTLs blow.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1248294 — Re: [PATCH V15 00/11] x86: Intel Cache Allocation Technology Support

On Thu, Oct 15, 2015 at 01:37:02PM +0200, Peter Zijlstra wrote:
> On Tue, Oct 13, 2015 at 07:40:58PM -0300, Marcelo Tosatti wrote:
> > How can you fix the issue of sockets with different reserved cache
> > regions with hw in the cgroup interface?
> 
> No idea what you're referring to. But IOCTLs blow.

Tejun brought up syscalls. Syscalls seem too generic.
So ioctls were chosen instead.

It is necessary to perform the following operations:

1) create cache reservation (params = size, type).
2) delete cache reservation.
3) attach cache reservation (params = cache reservation id, pid).
4) detach cache reservation (params = cache reservation id, pid).

Can it done via cgroups? If so, works for me.

A list of problems with the cgroup interface has been written,
in the thread... and we found another problem.


List of problems with cgroup interface:

1) Global IPI on CBM <---> task change does not scale.

 * cbm_update_all() - Update the cache bit mask for all packages.
 */
static inline void cbm_update_all(u32 closid)
{
       on_each_cpu_mask(&rdt_cpumask, cbm_cpu_update, (void *)closid,
1);
}

Consider a machine with 32 sockets.

2) Syscall interface specification is in kbytes, not
cache ways (which is what must be recorded by the OS
to allow migration of the OS between different
hardware systems).

3) Compilers are able to configure cache optimally for
given ranges of code inside applications, easily,
if desired.

4) Problem-2: The decision to allocate cache is tied to application
initialization / destruction, and application initialization is
essentially random from the POV of the system (the events which trigger
the execution of the application are not visible from the system).

Think of a server running two different servers: one database
with requests that are received with poisson distribution, average 30
requests per hour, and every request takes 1 minute.

One httpd server with nearly constant load.

Without cache reservations, database requests takes 2 minutes.
That is not acceptable for the database clients.
But with cache reservation, database requests takes 1 minute.

You want to maximize performance of httpd and database requests
What you do? You allow the database server to perform cache
reservation once a request comes in, and to undo the reservation
once the request is finished.

Its impossible to perform this with a centralized interface.

5) Modify scenario 2 above as follows: each database request
is handled by two newly created threads, and they share a certain
percentage
of data cache, and a certain percentage of code cache.

So the dispatcher thread, on arrival of request, has to:

        - create data cache reservation = tcrid-A.
        - create code cache reservation = tcrid-B.
        - create thread-1.
        - assign tcird-A and B to thread-1.
        - create thread-2.
        - assign tcird-A and B to thread-2.

6) Create reservations in such a way that the sum is larger than
total amount of cache, and CPU pinning (example from Karen Noel):

VM-1 on socket-1 with 80% of reservation.
VM-2 on socket-2 with 80% of reservation.
VM-1 pinned to socket-1.
VM-2 pinned to socket-2.

Cgroups interface attempts to set a cache mask globally. This is the
problem the "expand" proposal solves:
https://lkml.org/lkml/2015/7/29/682

7) Consider two sockets with different region of L3 cache
shared with HW:

— CPUID.(EAX=10H, ECX=1):EBX[31:0] reports a bit mask. Each set bit
within the length of the CBM
indicates the corresponding unit of the L3 allocation may be used by
other entities in the platform (e.g. an
integrated graphics engine or hardware units outside the processor core
and have direct access to L3).
Each cleared bit within the length of the CBM indicates the
corresponding allocation unit can be configured
to implement a priority-based allocation scheme chosen by an OS/VMM
without interference with other
hardware agents in the system. Bits outside the length of the CBM are
reserved.

You want the kernel to maintain different bitmasks in the CBM:

        socket1 [range-A]
        socket2 [range-B]

And the kernel will automatically switch from range A to range B
when the thread switches sockets.

---------------------

Problems 6, 7 and 2 are fatal for us. If you can fix them in the cgroup
interface, we can use it (please understand these problems, you seem to 
ignore them for some reason).

Problems 1 4 and 5 seem to come from Tejun.

Problem 3 could be a possibility.


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1248528 — Re: [PATCH V15 00/11] x86: Intel Cache Allocation Technology Support

On Thu, Oct 15, 2015 at 09:17:16PM -0300, Marcelo Tosatti wrote:
> On Thu, Oct 15, 2015 at 01:37:02PM +0200, Peter Zijlstra wrote:
> > On Tue, Oct 13, 2015 at 07:40:58PM -0300, Marcelo Tosatti wrote:
> > > How can you fix the issue of sockets with different reserved cache
> > > regions with hw in the cgroup interface?
> > 
> > No idea what you're referring to. But IOCTLs blow.
> 
> Tejun brought up syscalls. Syscalls seem too generic.
> So ioctls were chosen instead.
> 
> It is necessary to perform the following operations:
> 
> 1) create cache reservation (params = size, type).

mkdir

> 2) delete cache reservation.

rmdir

> 3) attach cache reservation (params = cache reservation id, pid).
> 4) detach cache reservation (params = cache reservation id, pid).

echo $pid > tasks

> Can it done via cgroups? If so, works for me.

Trivially.

> A list of problems with the cgroup interface has been written,
> in the thread... and we found another problem.

Which was endless and tiresome so I stopped reading.

> List of problems with cgroup interface:
> 
> 1) Global IPI on CBM <---> task change does not scale.
> 
>  * cbm_update_all() - Update the cache bit mask for all packages.
>  */
> static inline void cbm_update_all(u32 closid)
> {
>        on_each_cpu_mask(&rdt_cpumask, cbm_cpu_update, (void *)closid,
> 1);
> }

There is no way around that, the moment you view the CBM as a global
resource; ie. a CBM is configured the same on all sockets; you need to
do this for a task using that CBM might run on any CPU at any time.

This is not because of the cgroup interface at all. This is because you
want CBMs to be the same machine wide.

The only way to actually change that is to _be_ a cgroup and co-mount
with cpusets and be incestuous and look at the cpusets state and
discover disjoint groups.

> 2) Syscall interface specification is in kbytes, not
> cache ways (which is what must be recorded by the OS
> to allow migration of the OS between different
> hardware systems).

Meh, that again is nothing fundamental. The cgroup interface could do
bytes just the same.

> 3) Compilers are able to configure cache optimally for
> given ranges of code inside applications, easily,
> if desired.

Yeah, so? Every SKU has a different cache size, so once you're down to
that level you're pretty hard set in your configuration and it really
doesn't matter if you give bytes or ways, you _KNOW_ what your
configuration will be.

> 4) Problem-2: The decision to allocate cache is tied to application
> initialization / destruction, and application initialization is
> essentially random from the POV of the system (the events which trigger
> the execution of the application are not visible from the system).
> 
> Think of a server running two different servers: one database
> with requests that are received with poisson distribution, average 30
> requests per hour, and every request takes 1 minute.
> 
> One httpd server with nearly constant load.
> 
> Without cache reservations, database requests takes 2 minutes.
> That is not acceptable for the database clients.
> But with cache reservation, database requests takes 1 minute.
> 
> You want to maximize performance of httpd and database requests
> What you do? You allow the database server to perform cache
> reservation once a request comes in, and to undo the reservation
> once the request is finished.

> Its impossible to perform this with a centralized interface.

Not so; just a wee bit more fragile that desired. But, this is a
pre-existing problem with cgroups and needs to be solved, not using
cgroups because of this is silly.

Every cgroup that can work on tasks suffers this and arguably a few
more.

> 5) Modify scenario 2 above as follows: each database request
> is handled by two newly created threads, and they share a certain
> percentage
> of data cache, and a certain percentage of code cache.
> 
> So the dispatcher thread, on arrival of request, has to:
> 
>         - create data cache reservation = tcrid-A.
>         - create code cache reservation = tcrid-B.
>         - create thread-1.
>         - assign tcird-A and B to thread-1.
>         - create thread-2.
>         - assign tcird-A and B to thread-2.
> 
> 6) Create reservations in such a way that the sum is larger than
> total amount of cache, and CPU pinning (example from Karen Noel):
> 
> VM-1 on socket-1 with 80% of reservation.
> VM-2 on socket-2 with 80% of reservation.
> VM-1 pinned to socket-1.
> VM-2 pinned to socket-2.
> 
> Cgroups interface attempts to set a cache mask globally. This is the
> problem the "expand" proposal solves:
> https://lkml.org/lkml/2015/7/29/682

That email is unparsable. But the only way to sanely do so it do closely
intertwine oneself with cpusets, doing that with anything other than
another cgroup controller absolutely full on insane.

> 7) Consider two sockets with different region of L3 cache
> shared with HW:
> 
> — CPUID.(EAX=10H, ECX=1):EBX[31:0] reports a bit mask. Each set bit
> within the length of the CBM
> indicates the corresponding unit of the L3 allocation may be used by
> other entities in the platform (e.g. an
> integrated graphics engine or hardware units outside the processor core
> and have direct access to L3).
> Each cleared bit within the length of the CBM indicates the
> corresponding allocation unit can be configured
> to implement a priority-based allocation scheme chosen by an OS/VMM
> without interference with other
> hardware agents in the system. Bits outside the length of the CBM are
> reserved.
> 
> You want the kernel to maintain different bitmasks in the CBM:
> 
>         socket1 [range-A]
>         socket2 [range-B]
> 
> And the kernel will automatically switch from range A to range B
> when the thread switches sockets.

This is firmly in the insane range of things.. not going to happen full
stop.

It a thread can freely schedule between two CPUs its configuration on
those two CPUs had better bloody be the same.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1247232 — Re: [PATCH V15 00/11] x86: Intel Cache Allocation Technology Support

On Sun, Oct 11, 2015 at 09:50:12PM +0200, Thomas Gleixner wrote:
> Fenghua,
> 
> On Thu, 1 Oct 2015, Fenghua Yu wrote:
> 
> +Cc: Marcelo
> 
> > This series has some preparatory patches and Intel cache allocation
> > support.
> 
> <snip>
> 
> > Changes in v15:
> >  - Add a global IPI to update the closid on CPUs for current tasks.
> >  - Other minor changes where I remove the updating of clos_cbm_table to be
> >  set to all 1s during init.
> >  - Fix a few compilation warnings.
> >  - Port the patches to 4.3-rc.
>  
> What's the state of the interface discussion? I have not yet seen any
> agreement on that, unless I missed the important mail.
> 
> Aside of that, the patches miss a proper
> 
> From: Vikas ...
> 
> in the patch body at least for those which are untouched by you.
> 
> Thanks,
> 
> 	tglx

There are a number of problems with the patches as discussed in the
thread.

I am rewriting the interface with ioctls, with commands similar to the
syscall interface proposed.


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1247707 — Re: [PATCH V15 00/11] x86: Intel Cache Allocation Technology Support

On Tue, Oct 13, 2015 at 06:31:27PM -0300, Marcelo Tosatti wrote:
> I am rewriting the interface with ioctls, with commands similar to the
> syscall interface proposed.

Which is horrible for other use cases. I really don't see the problem
with the cgroup stuff.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1248308 — Re: [PATCH V15 00/11] x86: Intel Cache Allocation Technology Support

On Thu, Oct 15, 2015 at 01:36:14PM +0200, Peter Zijlstra wrote:
> On Tue, Oct 13, 2015 at 06:31:27PM -0300, Marcelo Tosatti wrote:
> > I am rewriting the interface with ioctls, with commands similar to the
> > syscall interface proposed.
> 
> Which is horrible for other use cases. I really don't see the problem
> with the cgroup stuff.

Can you detail what "horrible" means? 

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1248537 — Re: [PATCH V15 00/11] x86: Intel Cache Allocation Technology Support

On Thu, Oct 15, 2015 at 11:28:52PM -0300, Marcelo Tosatti wrote:
> On Thu, Oct 15, 2015 at 01:36:14PM +0200, Peter Zijlstra wrote:
> > On Tue, Oct 13, 2015 at 06:31:27PM -0300, Marcelo Tosatti wrote:
> > > I am rewriting the interface with ioctls, with commands similar to the
> > > syscall interface proposed.
> > 
> > Which is horrible for other use cases. I really don't see the problem
> > with the cgroup stuff.
> 
> Can you detail what "horrible" means? 

Say an RT scenario; you set up your machine with cgroups. You create a
cpuset with is disjoint from the others, you frob around with the cpu
cgroup, etc..

So once you're all done, you start your RT app into a cgroup.

But oh, fail, now you have to go muck about with ioctl()s to get the
cache allocation cruft to work.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web