Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1594599 > unrolled thread

RE: [PATCH 1/1] x86/cqm: Cqm requirements

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  RE: [PATCH 1/1] x86/cqm: Cqm requirements "Luck, Tony" <tony.luck@intel.com> - 2017-03-07 21:10 +0100
    RE: [PATCH 1/1] x86/cqm: Cqm requirements Thomas Gleixner <tglx@linutronix.de> - 2017-03-07 21:50 +0100
    Re: [PATCH 1/1] x86/cqm: Cqm requirements Stephane Eranian <eranian@google.com> - 2017-03-08 00:30 +0100
      Re: [PATCH 1/1] x86/cqm: Cqm requirements Shivappa Vikas <vikas.shivappa@intel.com> - 2017-03-08 01:20 +0100
      Re: [PATCH 1/1] x86/cqm: Cqm requirements Thomas Gleixner <tglx@linutronix.de> - 2017-03-08 10:50 +0100
        Re: [PATCH 1/1] x86/cqm: Cqm requirements David Carrillo-Cisneros <davidcc@google.com> - 2017-03-08 19:00 +0100
          Re: [PATCH 1/1] x86/cqm: Cqm requirements Thomas Gleixner <tglx@linutronix.de> - 2017-03-09 12:10 +0100
            Re: [PATCH 1/1] x86/cqm: Cqm requirements David Carrillo-Cisneros <davidcc@google.com> - 2017-03-09 19:10 +0100
              Re: [PATCH 1/1] x86/cqm: Cqm requirements Thomas Gleixner <tglx@linutronix.de> - 2017-03-10 16:00 +0100
                Re: [PATCH 1/1] x86/cqm: Cqm requirements David Carrillo-Cisneros <davidcc@google.com> - 2017-03-11 03:00 +0100
                  Re: [PATCH 1/1] x86/cqm: Cqm requirements Thomas Gleixner <tglx@linutronix.de> - 2017-03-13 20:20 +0100
                    Re: [PATCH 1/1] x86/cqm: Cqm requirements David Carrillo-Cisneros <davidcc@google.com> - 2017-03-13 21:10 +0100
                      Re: [PATCH 1/1] x86/cqm: Cqm requirements Thomas Gleixner <tglx@linutronix.de> - 2017-03-13 21:30 +0100

#1594599 — RE: [PATCH 1/1] x86/cqm: Cqm requirements

> That's all nice and good, but I still have no coherent explanation why
> measuring across allocation domains makes sense.

Is this in reaction to this one?

>> 5)      Put multiple threads into a single measurement group

If we fix it to say "threads from the same CAT group" does it fix things?

We'd like to have measurement groups use a single RMID ... if we
allowed tasks from different CAT groups in the same measurement
group we wouldn't be able to split the numbers back to report the
right overall total for each of the CAT groups.

-Tony

[toc] | [next] | [standalone]

#1594627

On Tue, 7 Mar 2017, Luck, Tony wrote:

> > That's all nice and good, but I still have no coherent explanation why
> > measuring across allocation domains makes sense.
> 
> Is this in reaction to this one?
> 
> >> 5)      Put multiple threads into a single measurement group
> 
> If we fix it to say "threads from the same CAT group" does it fix things?
> 
> We'd like to have measurement groups use a single RMID ... if we
> allowed tasks from different CAT groups in the same measurement
> group we wouldn't be able to split the numbers back to report the
> right overall total for each of the CAT groups.

Right. And the same applies to CPU measurements. If we have threads from 3
CAT groups running on a CPU then the aggregate value (except for bandwidth
which can be computed by software) is useless.

Thanks,

	tglx

[toc] | [prev] | [next] | [standalone]

#1594721

Hi,

On Tue, Mar 7, 2017 at 12:04 PM, Luck, Tony <tony.luck@intel.com> wrote:
>> That's all nice and good, but I still have no coherent explanation why
>> measuring across allocation domains makes sense.
>
> Is this in reaction to this one?
>
>>> 5)      Put multiple threads into a single measurement group
>
> If we fix it to say "threads from the same CAT group" does it fix things?
>
Inside a CAT partition, there may be multiple tasks split into
different cgroups.
We need the ability to monitor groups of tasks individually within that CAT
partition. I think this is what this bullet is about.


> We'd like to have measurement groups use a single RMID ... if we
> allowed tasks from different CAT groups in the same measurement
> group we wouldn't be able to split the numbers back to report the
> right overall total for each of the CAT groups.
>
> -Tony

[toc] | [prev] | [next] | [standalone]

#1594742

On Tue, 7 Mar 2017, Stephane Eranian wrote:

> Hi,
>
> On Tue, Mar 7, 2017 at 12:04 PM, Luck, Tony <tony.luck@intel.com> wrote:
>>> That's all nice and good, but I still have no coherent explanation why
>>> measuring across allocation domains makes sense.
>>
>> Is this in reaction to this one?
>>
>>>> 5)      Put multiple threads into a single measurement group
>>
>> If we fix it to say "threads from the same CAT group" does it fix things?
>>
> Inside a CAT partition, there may be multiple tasks split into
> different cgroups.
> We need the ability to monitor groups of tasks individually within that CAT
> partition. I think this is what this bullet is about.
>

The #8 covers that I think (or what we intended for 5..) ?

8)      Can get measurements for subsets of tasks in a CAT group (to find the
threads hogging the resources).

Thanks,
Vikas

>
>> We'd like to have measurement groups use a single RMID ... if we
>> allowed tasks from different CAT groups in the same measurement
>> group we wouldn't be able to split the numbers back to report the
>> right overall total for each of the CAT groups.
>>
>> -Tony
>

[toc] | [prev] | [next] | [standalone]

#1594986

Stephane,

On Tue, 7 Mar 2017, Stephane Eranian wrote:
> On Tue, Mar 7, 2017 at 12:04 PM, Luck, Tony <tony.luck@intel.com> wrote:
> >> That's all nice and good, but I still have no coherent explanation why
> >> measuring across allocation domains makes sense.
> >
> > Is this in reaction to this one?
> >
> >>> 5)      Put multiple threads into a single measurement group
> >
> > If we fix it to say "threads from the same CAT group" does it fix things?
> >
> Inside a CAT partition, there may be multiple tasks split into different
> cgroups.  We need the ability to monitor groups of tasks individually
> within that CAT partition. I think this is what this bullet is about.

I completely understand that. That's fine and I never debated that one, but
the requirements list is too vague about what you want to measure.

> >>> 5)      Put multiple threads into a single measurement group

That can be:

     A) threads within a CAT group

     B) threads which belong to different CAT groups

A) is fine. B) does not make any sense to me

Same applies for per CPU measurements.

Thanks,

	tglx

[toc] | [prev] | [next] | [standalone]

#1595418

On Wed, Mar 8, 2017 at 12:30 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
> Stephane,
>
> On Tue, 7 Mar 2017, Stephane Eranian wrote:
>> On Tue, Mar 7, 2017 at 12:04 PM, Luck, Tony <tony.luck@intel.com> wrote:
>> >> That's all nice and good, but I still have no coherent explanation why
>> >> measuring across allocation domains makes sense.
>> >
>> > Is this in reaction to this one?
>> >
>> >>> 5)      Put multiple threads into a single measurement group
>> >
>> > If we fix it to say "threads from the same CAT group" does it fix things?
>> >
>> Inside a CAT partition, there may be multiple tasks split into different
>> cgroups.  We need the ability to monitor groups of tasks individually
>> within that CAT partition. I think this is what this bullet is about.
>
> I completely understand that. That's fine and I never debated that one, but
> the requirements list is too vague about what you want to measure.
>
>> >>> 5)      Put multiple threads into a single measurement group
>
> That can be:
>
>      A) threads within a CAT group
>
>      B) threads which belong to different CAT groups
>
> A) is fine. B) does not make any sense to me

It's A). As Tony suggested in a previous email, we can rephrase it to:

5) Put a subset of threads from the same CAT group into a single
measurement group.

>
> Same applies for per CPU measurements.

For CPU measurements. We need perf-like CPU filtering to support tools
that perform low overhead monitoring by polling CPU events. These
tools approximate per-cgroup/task events by reconciling CPU events
with logs of what job run when in what CPU.

>
> Thanks,
>
>         tglx

[toc] | [prev] | [next] | [standalone]

#1595934

On Wed, 8 Mar 2017, David Carrillo-Cisneros wrote:
> On Wed, Mar 8, 2017 at 12:30 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
> > Same applies for per CPU measurements.
> 
> For CPU measurements. We need perf-like CPU filtering to support tools
> that perform low overhead monitoring by polling CPU events. These
> tools approximate per-cgroup/task events by reconciling CPU events
> with logs of what job run when in what CPU.

Sorry, but for CQM that's just voodoo analysis. Lets look at an example:

CPU default is CAT group 0	(20% of cache)
T1 belongs to CAT group 1	(40% of cache)
T2 belongs to CAT group 2	(40% of cache)

Now you do low overhead samples of the CPU (all groups accounted) with 1
second period.

Lets assume that T1 runs 50% and T2 runs 20% the rest of the time is
utilized by random other things and the kernel itself (using CAT group 0).

What is the accumulated value telling you?

How do you approximate that back to T1/T2 and the rest?

How do you do that when the tasks are switching between the samples several
times?

I really have idea how that should work and what the value of this would
be.

Thanks,

	tglx

[toc] | [prev] | [next] | [standalone]

#1596274

On Thu, Mar 9, 2017 at 3:01 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
> On Wed, 8 Mar 2017, David Carrillo-Cisneros wrote:
>> On Wed, Mar 8, 2017 at 12:30 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
>> > Same applies for per CPU measurements.
>>
>> For CPU measurements. We need perf-like CPU filtering to support tools
>> that perform low overhead monitoring by polling CPU events. These
>> tools approximate per-cgroup/task events by reconciling CPU events
>> with logs of what job run when in what CPU.
>
> Sorry, but for CQM that's just voodoo analysis.

I'll argue that. Yet, perf-like CPU is also needed for MBM, a less
contentious scenario, I believe.


>
> CPU default is CAT group 0      (20% of cache)
> T1 belongs to CAT group 1       (40% of cache)
> T2 belongs to CAT group 2       (40% of cache)
>
> Now you do low overhead samples of the CPU (all groups accounted) with 1
> second period.
>
> Lets assume that T1 runs 50% and T2 runs 20% the rest of the time is
> utilized by random other things and the kernel itself (using CAT group 0).
>
> What is the accumulated value telling you?

In this single example not much, only the sum of occupancies. But
assume I have T1...T10000 different jobs, and I randomly select a pair
of those jobs to run together in a machine, (they become the T1 and T2
in your example). Then I repeat that hundreds of thousands of times.

I can collect all data with (tasks run, time run, occupancy) and build
a simple regression to estimate the expected occupancy (and some
confidence interval). That inaccurate but approximate value is very
useful to feed into a job scheduler. Furthermore, it can be correlated
with values of other events that are currently sampled this way.

>
> How do you approximate that back to T1/T2 and the rest?

Described above for large numbers and random samples. More
sophisticated (voodo?) statistic techniques are employed in practice
to account for almost all issues I could think of (selection bias,
missing values, interaction between tasks, etc). They seem to work
fine.

>
> How do you do that when the tasks are switching between the samples several
> times?

It does not work well for a single run (your example). But for the
example I gave, one can just rely on Random Sampling, Law of Large
Numbers, and Central Limit Theorem.

Thanks,
David

[toc] | [prev] | [next] | [standalone]

#1597903

On Thu, 9 Mar 2017, David Carrillo-Cisneros wrote:
> On Thu, Mar 9, 2017 at 3:01 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
> > On Wed, 8 Mar 2017, David Carrillo-Cisneros wrote:
> >> On Wed, Mar 8, 2017 at 12:30 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
> >> > Same applies for per CPU measurements.
> >>
> >> For CPU measurements. We need perf-like CPU filtering to support tools
> >> that perform low overhead monitoring by polling CPU events. These
> >> tools approximate per-cgroup/task events by reconciling CPU events
> >> with logs of what job run when in what CPU.
> >
> > Sorry, but for CQM that's just voodoo analysis.
> 
> I'll argue that. Yet, perf-like CPU is also needed for MBM, a less
> contentious scenario, I believe.

MBM is a different playground (albeit related due to the RMID stuff).

> It does not work well for a single run (your example). But for the
> example I gave, one can just rely on Random Sampling, Law of Large
> Numbers, and Central Limit Theorem.

Fine. So we need this for ONE particular use case. And if that is not well
documented including the underlying mechanics to analyze the data then this
will be a nice source of confusion for Joe User.

I still think that this can be done differently while keeping the overhead
small.

You look at this from the existing perf mechanics which require high
overhead context switching machinery. But that's just wrong because that's
not how the cache and bandwidth monitoring works.

Contrary to the other perf counters, CQM and MBM are based on a context
selectable set of counters which do not require readout and reconfiguration
when the switch happens.

Especially with CAT in play, the context switch overhead is there already
when CAT partitions need to be switched. So switching the RMID at the same
time is basically free, if we are smart enough to do an equivalent to the
CLOSID context switch mechanics and ideally combine both into a single MSR
write.

With that the low overhead periodic sampling can read N counters which are
related to the monitored set and provide N separate results. For bandwidth
the aggregation is a simple ADD and for cache residency it's pointless.

Just because perf was designed with the regular performance counters in
mind (way before that CQM/MBM stuff came around) does not mean that we
cannot change/extend that if it makes sense.

And looking at the way Cache/Bandwidth allocation and monitoring works, it
makes a lot of sense. Definitely more than shoving it into the current mode
of operandi with duct tape just because we can.

Thanks,

	tglx

[toc] | [prev] | [next] | [standalone]

#1598232

>
> Fine. So we need this for ONE particular use case. And if that is not well
> documented including the underlying mechanics to analyze the data then this
> will be a nice source of confusion for Joe User.
>
> I still think that this can be done differently while keeping the overhead
> small.
>
> You look at this from the existing perf mechanics which require high
> overhead context switching machinery. But that's just wrong because that's
> not how the cache and bandwidth monitoring works.
>
> Contrary to the other perf counters, CQM and MBM are based on a context
> selectable set of counters which do not require readout and reconfiguration
> when the switch happens.
>
> Especially with CAT in play, the context switch overhead is there already
> when CAT partitions need to be switched. So switching the RMID at the same
> time is basically free, if we are smart enough to do an equivalent to the
> CLOSID context switch mechanics and ideally combine both into a single MSR
> write.
>
> With that the low overhead periodic sampling can read N counters which are
> related to the monitored set and provide N separate results. For bandwidth
> the aggregation is a simple ADD and for cache residency it's pointless.
>
> Just because perf was designed with the regular performance counters in
> mind (way before that CQM/MBM stuff came around) does not mean that we
> cannot change/extend that if it makes sense.
>
> And looking at the way Cache/Bandwidth allocation and monitoring works, it
> makes a lot of sense. Definitely more than shoving it into the current mode
> of operandi with duct tape just because we can.
>

You made a point. The use case I described can be better served with
the low overhead monitoring groups that Fenghua is working on. Then
that info can be merged with the per-CPU profile collected for non-RDT
events.

I am ok removing the perf-like CPU filtering from the requirements.

Thanks,
David

[toc] | [prev] | [next] | [standalone]

#1599709

On Fri, 10 Mar 2017, David Carrillo-Cisneros wrote:
> > Fine. So we need this for ONE particular use case. And if that is not well
> > documented including the underlying mechanics to analyze the data then this
> > will be a nice source of confusion for Joe User.
> >
> > I still think that this can be done differently while keeping the overhead
> > small.
> >
> > You look at this from the existing perf mechanics which require high
> > overhead context switching machinery. But that's just wrong because that's
> > not how the cache and bandwidth monitoring works.
> >
> > Contrary to the other perf counters, CQM and MBM are based on a context
> > selectable set of counters which do not require readout and reconfiguration
> > when the switch happens.
> >
> > Especially with CAT in play, the context switch overhead is there already
> > when CAT partitions need to be switched. So switching the RMID at the same
> > time is basically free, if we are smart enough to do an equivalent to the
> > CLOSID context switch mechanics and ideally combine both into a single MSR
> > write.
> >
> > With that the low overhead periodic sampling can read N counters which are
> > related to the monitored set and provide N separate results. For bandwidth
> > the aggregation is a simple ADD and for cache residency it's pointless.
> >
> > Just because perf was designed with the regular performance counters in
> > mind (way before that CQM/MBM stuff came around) does not mean that we
> > cannot change/extend that if it makes sense.
> >
> > And looking at the way Cache/Bandwidth allocation and monitoring works, it
> > makes a lot of sense. Definitely more than shoving it into the current mode
> > of operandi with duct tape just because we can.
> >
> 
> You made a point. The use case I described can be better served with
> the low overhead monitoring groups that Fenghua is working on. Then
> that info can be merged with the per-CPU profile collected for non-RDT
> events.
>
> I am ok removing the perf-like CPU filtering from the requirements.

So if I'm not missing something then ALL remaining requirements can be
solved with the RDT integrated monitoring mechanics, right?

Thanks,

	tglx

[toc] | [prev] | [next] | [standalone]

#1599752

>> I am ok removing the perf-like CPU filtering from the requirements.
>
> So if I'm not missing something then ALL remaining requirements can be
> solved with the RDT integrated monitoring mechanics, right?
>

Right.

[toc] | [prev] | [next] | [standalone]

#1599785

On Mon, 13 Mar 2017, David Carrillo-Cisneros wrote:

> >> I am ok removing the perf-like CPU filtering from the requirements.
> >
> > So if I'm not missing something then ALL remaining requirements can be
> > solved with the RDT integrated monitoring mechanics, right?
> >
> 
> Right.

Excellent.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web