Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1310430 > unrolled thread

Re: [RFC PATCH v2 0/4] CPUs capacity information for heterogeneous systems

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [RFC PATCH v2 0/4] CPUs capacity information for heterogeneous  systems Steve Muckle <steve.muckle@linaro.org> - 2016-01-15 21:00 +0100
    Re: [RFC PATCH v2 0/4] CPUs capacity information for heterogeneous  systems Juri Lelli <juri.lelli@arm.com> - 2016-01-18 16:20 +0100
      Re: [RFC PATCH v2 0/4] CPUs capacity information for heterogeneous systems Vincent Guittot <vincent.guittot@linaro.org> - 2016-01-18 17:20 +0100
        Re: [RFC PATCH v2 0/4] CPUs capacity information for heterogeneous  systems Juri Lelli <juri.lelli@arm.com> - 2016-01-18 17:40 +0100
          Re: [RFC PATCH v2 0/4] CPUs capacity information for heterogeneous systems Vincent Guittot <vincent.guittot@linaro.org> - 2016-01-18 17:50 +0100
            Re: [RFC PATCH v2 0/4] CPUs capacity information for heterogeneous  systems Juri Lelli <juri.lelli@arm.com> - 2016-01-18 18:10 +0100
              Re: [RFC PATCH v2 0/4] CPUs capacity information for heterogeneous systems Vincent Guittot <vincent.guittot@linaro.org> - 2016-01-18 18:30 +0100
            Re: [RFC PATCH v2 0/4] CPUs capacity information for heterogeneous  systems Catalin Marinas <catalin.marinas@arm.com> - 2016-01-19 12:10 +0100
              Re: [RFC PATCH v2 0/4] CPUs capacity information for heterogeneous  systems Juri Lelli <juri.lelli@arm.com> - 2016-01-19 12:30 +0100
                Re: [RFC PATCH v2 0/4] CPUs capacity information for heterogeneous  systems Juri Lelli <juri.lelli@arm.com> - 2016-01-19 15:30 +0100
                  Re: [RFC PATCH v2 0/4] CPUs capacity information for heterogeneous  systems Steve Muckle <steve.muckle@linaro.org> - 2016-01-19 20:50 +0100
                    Re: [RFC PATCH v2 0/4] CPUs capacity information for heterogeneous  systems Mark Brown <broonie@kernel.org> - 2016-01-19 22:20 +0100
                      Re: [RFC PATCH v2 0/4] CPUs capacity information for heterogeneous  systems Juri Lelli <juri.lelli@arm.com> - 2016-01-20 11:30 +0100
      Re: [RFC PATCH v2 0/4] CPUs capacity information for heterogeneous  systems Steve Muckle <steve.muckle@linaro.org> - 2016-01-18 20:30 +0100

#1310430 — Re: [RFC PATCH v2 0/4] CPUs capacity information for heterogeneous systems

On 01/08/2016 06:09 AM, Juri Lelli wrote:
>  2. Dynamic profiling at boot (v2)
> 
>     pros: - does not require a standardized definition of capacity
>           - cannot be incorrectly tuned (once benchmark is fixed)
>           - does not require user/integrator work
> 
>     cons: - not easy to come up with a clean solution, as it seems interaction
>             with several subsystems (e.g., cpufreq) is required
>           - not easy to agree upon a single benchmark (that has to be both
>             representative and simple enough to run at boot)
>           - numbers might (and do) vary from boot to boot

An important additional con that was mentioned earlier IIRC was the
additional boot time required for the benchmark. Perhaps there could be
a kernel command line argument to bypass the benchmark if it is known
that predetermined values will be provided via sysfs later?

Though there may be another issue with that as mentioned below.

>  3. sysfs (v1)
> 
>     pros: - clean and super easy to implement
>           - values don't require to be physical properties, defining them is
>             probably easier
> 
>     cons: - CPUs capacity have to be provided after boot (by some init script?)
>           - API is modified, still some discussion/review is needed
>           - values can still be incorrectly used for runtime tuning purposes

Initializing the values via userspace init will cause more of the boot
process to run with incorrect CPU capacity values. Boot times may be
increased with tasks running on suboptimal CPUs. Such increases may also
not be deterministic.

Extending the kernel command line idea above, perhaps capacity values
could be provided there as well, similar to the lpj parameter? That has
scalability issues though if there's a huge highly heterogeneous platform...

DT solves these issues and would be the perfect place for this - we are
defining the compute capacity of a CPU which is a property of the
hardware. However there are a couple things forcing us to compromise.
One is that the amount and detail of information required to adequately
capture the computational abilities of a CPU across all possible
workloads seem onerous to collect and enumerate. The second is that even
if we were willing to undertake that, CPU vendors probably won't be
forthcoming with that information.

Despite this DT still seems to me like the best way to go. At their
heart these are properties of the hardware, even if we can't specify
them as such per se because of the problems above. The capacity would
have to be defined as a relative value among CPUs. And while it's true
it may be abused for tuning purposes, that's true of any strategy.
Certainly the sysfs strategy and even if only a dynamic option is
provided, it is guaranteed to be hacked by platform vendors.

thanks,
Steve

[toc] | [next] | [standalone]

#1311608

Hi Steve,

On 15/01/16 11:50, Steve Muckle wrote:
> On 01/08/2016 06:09 AM, Juri Lelli wrote:
> >  2. Dynamic profiling at boot (v2)
> > 
> >     pros: - does not require a standardized definition of capacity
> >           - cannot be incorrectly tuned (once benchmark is fixed)
> >           - does not require user/integrator work
> > 
> >     cons: - not easy to come up with a clean solution, as it seems interaction
> >             with several subsystems (e.g., cpufreq) is required
> >           - not easy to agree upon a single benchmark (that has to be both
> >             representative and simple enough to run at boot)
> >           - numbers might (and do) vary from boot to boot
> 
> An important additional con that was mentioned earlier IIRC was the
> additional boot time required for the benchmark.

Right. I forgot about that.

> Perhaps there could be
> a kernel command line argument to bypass the benchmark if it is known
> that predetermined values will be provided via sysfs later?
> 

This might work, yes.

> Though there may be another issue with that as mentioned below.
> 
> >  3. sysfs (v1)
> > 
> >     pros: - clean and super easy to implement
> >           - values don't require to be physical properties, defining them is
> >             probably easier
> > 
> >     cons: - CPUs capacity have to be provided after boot (by some init script?)
> >           - API is modified, still some discussion/review is needed
> >           - values can still be incorrectly used for runtime tuning purposes
> 
> Initializing the values via userspace init will cause more of the boot
> process to run with incorrect CPU capacity values. Boot times may be
> increased with tasks running on suboptimal CPUs. Such increases may also
> not be deterministic.
> 
> Extending the kernel command line idea above, perhaps capacity values
> could be provided there as well, similar to the lpj parameter? That has
> scalability issues though if there's a huge highly heterogeneous platform...
> 

Yeah, adding such option is not difficult, but I'm also a bit concerned
about the scalability of such a thing.

> DT solves these issues and would be the perfect place for this - we are
> defining the compute capacity of a CPU which is a property of the
> hardware. However there are a couple things forcing us to compromise.
> One is that the amount and detail of information required to adequately
> capture the computational abilities of a CPU across all possible
> workloads seem onerous to collect and enumerate. The second is that even
> if we were willing to undertake that, CPU vendors probably won't be
> forthcoming with that information.
> 

You mean because they won't publish performance data of their hw?

But we already use per platform normalized values (as you are proposing
below). So that a platform to platform comparison doesn't make sense.

> Despite this DT still seems to me like the best way to go. At their
> heart these are properties of the hardware, even if we can't specify
> them as such per se because of the problems above. The capacity would
> have to be defined as a relative value among CPUs. And while it's true
> it may be abused for tuning purposes, that's true of any strategy.
> Certainly the sysfs strategy and even if only a dynamic option is
> provided, it is guaranteed to be hacked by platform vendors.

I also like the DT approach and consider the sysfs option as something
that can go together with any solution we want to adopt.

Best,

- Juri

[toc] | [prev] | [next] | [standalone]

#1311653 — Re: [RFC PATCH v2 0/4] CPUs capacity information for heterogeneous systems

On 18 January 2016 at 16:13, Juri Lelli <juri.lelli@arm.com> wrote:
> Hi Steve,
>
> On 15/01/16 11:50, Steve Muckle wrote:
>> On 01/08/2016 06:09 AM, Juri Lelli wrote:
>> >  2. Dynamic profiling at boot (v2)
>> >
>> >     pros: - does not require a standardized definition of capacity
>> >           - cannot be incorrectly tuned (once benchmark is fixed)
>> >           - does not require user/integrator work
>> >
>> >     cons: - not easy to come up with a clean solution, as it seems interaction
>> >             with several subsystems (e.g., cpufreq) is required
>> >           - not easy to agree upon a single benchmark (that has to be both
>> >             representative and simple enough to run at boot)
>> >           - numbers might (and do) vary from boot to boot
>>
>> An important additional con that was mentioned earlier IIRC was the
>> additional boot time required for the benchmark.
>
> Right. I forgot about that.
>
>> Perhaps there could be
>> a kernel command line argument to bypass the benchmark if it is known
>> that predetermined values will be provided via sysfs later?
>>
>
> This might work, yes.

Instead of command line, I prefer to use DT.

Can't we use something similar to what is currently done in arm arch
for the early stage of the boot ? We don't have to provide performance
value for which it's difficult to find a consensus on how to define it
and which benchmark should be used. We use the micro arch and the
frequency of the core to define a relative capacity. This give us a
relatively good idea of the capacity of each core.
Then, the dynamic profiling can update it with a more accurate value
during the boot.

>
>> Though there may be another issue with that as mentioned below.
>>
>> >  3. sysfs (v1)
>> >
>> >     pros: - clean and super easy to implement
>> >           - values don't require to be physical properties, defining them is
>> >             probably easier
>> >
>> >     cons: - CPUs capacity have to be provided after boot (by some init script?)
>> >           - API is modified, still some discussion/review is needed
>> >           - values can still be incorrectly used for runtime tuning purposes
>>
>> Initializing the values via userspace init will cause more of the boot
>> process to run with incorrect CPU capacity values. Boot times may be
>> increased with tasks running on suboptimal CPUs. Such increases may also
>> not be deterministic.
>>
>> Extending the kernel command line idea above, perhaps capacity values
>> could be provided there as well, similar to the lpj parameter? That has
>> scalability issues though if there's a huge highly heterogeneous platform...
>>
>
> Yeah, adding such option is not difficult, but I'm also a bit concerned
> about the scalability of such a thing.
>
>> DT solves these issues and would be the perfect place for this - we are
>> defining the compute capacity of a CPU which is a property of the
>> hardware. However there are a couple things forcing us to compromise.
>> One is that the amount and detail of information required to adequately
>> capture the computational abilities of a CPU across all possible
>> workloads seem onerous to collect and enumerate. The second is that even
>> if we were willing to undertake that, CPU vendors probably won't be
>> forthcoming with that information.
>>
>
> You mean because they won't publish performance data of their hw?
>
> But we already use per platform normalized values (as you are proposing
> below). So that a platform to platform comparison doesn't make sense.
>
>> Despite this DT still seems to me like the best way to go. At their
>> heart these are properties of the hardware, even if we can't specify
>> them as such per se because of the problems above. The capacity would
>> have to be defined as a relative value among CPUs. And while it's true
>> it may be abused for tuning purposes, that's true of any strategy.
>> Certainly the sysfs strategy and even if only a dynamic option is
>> provided, it is guaranteed to be hacked by platform vendors.
>
> I also like the DT approach and consider the sysfs option as something
> that can go together with any solution we want to adopt.

I'm not sure that we should consider sysfs as an option because of all
the concerned that as already been put forward.
I would prefer a debugfs if we have play with these capacity values in
order to test their accuracy.

Regards,
Vincent
>
> Best,
>
> - Juri

[toc] | [prev] | [next] | [standalone]

#1311664

Hi Vincent,

On 18/01/16 17:13, Vincent Guittot wrote:
> On 18 January 2016 at 16:13, Juri Lelli <juri.lelli@arm.com> wrote:
> > Hi Steve,
> >
> > On 15/01/16 11:50, Steve Muckle wrote:
> >> On 01/08/2016 06:09 AM, Juri Lelli wrote:
> >> >  2. Dynamic profiling at boot (v2)
> >> >
> >> >     pros: - does not require a standardized definition of capacity
> >> >           - cannot be incorrectly tuned (once benchmark is fixed)
> >> >           - does not require user/integrator work
> >> >
> >> >     cons: - not easy to come up with a clean solution, as it seems interaction
> >> >             with several subsystems (e.g., cpufreq) is required
> >> >           - not easy to agree upon a single benchmark (that has to be both
> >> >             representative and simple enough to run at boot)
> >> >           - numbers might (and do) vary from boot to boot
> >>
> >> An important additional con that was mentioned earlier IIRC was the
> >> additional boot time required for the benchmark.
> >
> > Right. I forgot about that.
> >
> >> Perhaps there could be
> >> a kernel command line argument to bypass the benchmark if it is known
> >> that predetermined values will be provided via sysfs later?
> >>
> >
> > This might work, yes.
> 
> Instead of command line, I prefer to use DT.
> 
> Can't we use something similar to what is currently done in arm arch
> for the early stage of the boot ? We don't have to provide performance
> value for which it's difficult to find a consensus on how to define it
> and which benchmark should be used. We use the micro arch and the
> frequency of the core to define a relative capacity. This give us a
> relatively good idea of the capacity of each core.

I'm not sure I understand what you are proposing. arm arch is currently
based on having static hardcoded data (efficiency values). But, this has
already been NACKed for arm64 during last review of this RFC.

Are you proposing something different?

Thanks,

- Juri

> Then, the dynamic profiling can update it with a more accurate value
> during the boot.
> 
> >
> >> Though there may be another issue with that as mentioned below.
> >>
> >> >  3. sysfs (v1)
> >> >
> >> >     pros: - clean and super easy to implement
> >> >           - values don't require to be physical properties, defining them is
> >> >             probably easier
> >> >
> >> >     cons: - CPUs capacity have to be provided after boot (by some init script?)
> >> >           - API is modified, still some discussion/review is needed
> >> >           - values can still be incorrectly used for runtime tuning purposes
> >>
> >> Initializing the values via userspace init will cause more of the boot
> >> process to run with incorrect CPU capacity values. Boot times may be
> >> increased with tasks running on suboptimal CPUs. Such increases may also
> >> not be deterministic.
> >>
> >> Extending the kernel command line idea above, perhaps capacity values
> >> could be provided there as well, similar to the lpj parameter? That has
> >> scalability issues though if there's a huge highly heterogeneous platform...
> >>
> >
> > Yeah, adding such option is not difficult, but I'm also a bit concerned
> > about the scalability of such a thing.
> >
> >> DT solves these issues and would be the perfect place for this - we are
> >> defining the compute capacity of a CPU which is a property of the
> >> hardware. However there are a couple things forcing us to compromise.
> >> One is that the amount and detail of information required to adequately
> >> capture the computational abilities of a CPU across all possible
> >> workloads seem onerous to collect and enumerate. The second is that even
> >> if we were willing to undertake that, CPU vendors probably won't be
> >> forthcoming with that information.
> >>
> >
> > You mean because they won't publish performance data of their hw?
> >
> > But we already use per platform normalized values (as you are proposing
> > below). So that a platform to platform comparison doesn't make sense.
> >
> >> Despite this DT still seems to me like the best way to go. At their
> >> heart these are properties of the hardware, even if we can't specify
> >> them as such per se because of the problems above. The capacity would
> >> have to be defined as a relative value among CPUs. And while it's true
> >> it may be abused for tuning purposes, that's true of any strategy.
> >> Certainly the sysfs strategy and even if only a dynamic option is
> >> provided, it is guaranteed to be hacked by platform vendors.
> >
> > I also like the DT approach and consider the sysfs option as something
> > that can go together with any solution we want to adopt.
> 
> I'm not sure that we should consider sysfs as an option because of all
> the concerned that as already been put forward.
> I would prefer a debugfs if we have play with these capacity values in
> order to test their accuracy.
> 
> Regards,
> Vincent
> >
> > Best,
> >
> > - Juri
> 

[toc] | [prev] | [next] | [standalone]

#1311666 — Re: [RFC PATCH v2 0/4] CPUs capacity information for heterogeneous systems

On 18 January 2016 at 17:30, Juri Lelli <juri.lelli@arm.com> wrote:
> Hi Vincent,
>
> On 18/01/16 17:13, Vincent Guittot wrote:
>> On 18 January 2016 at 16:13, Juri Lelli <juri.lelli@arm.com> wrote:
>> > Hi Steve,
>> >
>> > On 15/01/16 11:50, Steve Muckle wrote:
>> >> On 01/08/2016 06:09 AM, Juri Lelli wrote:
>> >> >  2. Dynamic profiling at boot (v2)
>> >> >
>> >> >     pros: - does not require a standardized definition of capacity
>> >> >           - cannot be incorrectly tuned (once benchmark is fixed)
>> >> >           - does not require user/integrator work
>> >> >
>> >> >     cons: - not easy to come up with a clean solution, as it seems interaction
>> >> >             with several subsystems (e.g., cpufreq) is required
>> >> >           - not easy to agree upon a single benchmark (that has to be both
>> >> >             representative and simple enough to run at boot)
>> >> >           - numbers might (and do) vary from boot to boot
>> >>
>> >> An important additional con that was mentioned earlier IIRC was the
>> >> additional boot time required for the benchmark.
>> >
>> > Right. I forgot about that.
>> >
>> >> Perhaps there could be
>> >> a kernel command line argument to bypass the benchmark if it is known
>> >> that predetermined values will be provided via sysfs later?
>> >>
>> >
>> > This might work, yes.
>>
>> Instead of command line, I prefer to use DT.
>>
>> Can't we use something similar to what is currently done in arm arch
>> for the early stage of the boot ? We don't have to provide performance
>> value for which it's difficult to find a consensus on how to define it
>> and which benchmark should be used. We use the micro arch and the
>> frequency of the core to define a relative capacity. This give us a
>> relatively good idea of the capacity of each core.
>
> I'm not sure I understand what you are proposing. arm arch is currently
> based on having static hardcoded data (efficiency values). But, this has
> already been NACKed for arm64 during last review of this RFC.
>
> Are you proposing something different?

No, i'm proposing to use it at boot time until the dynamic profiling
gives better value.
We don't have to set any new properties.
IIRC, It was nacked because it was of static hardcoded value that was
not always reflecting the best accurate capacity of a system. IMHO,
it's not that far from reality so can't this be used as an
intermediate step while waiting for dynamic profiling ?

Vincent

>
> Thanks,
>
> - Juri
>
>> Then, the dynamic profiling can update it with a more accurate value
>> during the boot.
>>
>> >
>> >> Though there may be another issue with that as mentioned below.
>> >>
>> >> >  3. sysfs (v1)
>> >> >
>> >> >     pros: - clean and super easy to implement
>> >> >           - values don't require to be physical properties, defining them is
>> >> >             probably easier
>> >> >
>> >> >     cons: - CPUs capacity have to be provided after boot (by some init script?)
>> >> >           - API is modified, still some discussion/review is needed
>> >> >           - values can still be incorrectly used for runtime tuning purposes
>> >>
>> >> Initializing the values via userspace init will cause more of the boot
>> >> process to run with incorrect CPU capacity values. Boot times may be
>> >> increased with tasks running on suboptimal CPUs. Such increases may also
>> >> not be deterministic.
>> >>
>> >> Extending the kernel command line idea above, perhaps capacity values
>> >> could be provided there as well, similar to the lpj parameter? That has
>> >> scalability issues though if there's a huge highly heterogeneous platform...
>> >>
>> >
>> > Yeah, adding such option is not difficult, but I'm also a bit concerned
>> > about the scalability of such a thing.
>> >
>> >> DT solves these issues and would be the perfect place for this - we are
>> >> defining the compute capacity of a CPU which is a property of the
>> >> hardware. However there are a couple things forcing us to compromise.
>> >> One is that the amount and detail of information required to adequately
>> >> capture the computational abilities of a CPU across all possible
>> >> workloads seem onerous to collect and enumerate. The second is that even
>> >> if we were willing to undertake that, CPU vendors probably won't be
>> >> forthcoming with that information.
>> >>
>> >
>> > You mean because they won't publish performance data of their hw?
>> >
>> > But we already use per platform normalized values (as you are proposing
>> > below). So that a platform to platform comparison doesn't make sense.
>> >
>> >> Despite this DT still seems to me like the best way to go. At their
>> >> heart these are properties of the hardware, even if we can't specify
>> >> them as such per se because of the problems above. The capacity would
>> >> have to be defined as a relative value among CPUs. And while it's true
>> >> it may be abused for tuning purposes, that's true of any strategy.
>> >> Certainly the sysfs strategy and even if only a dynamic option is
>> >> provided, it is guaranteed to be hacked by platform vendors.
>> >
>> > I also like the DT approach and consider the sysfs option as something
>> > that can go together with any solution we want to adopt.
>>
>> I'm not sure that we should consider sysfs as an option because of all
>> the concerned that as already been put forward.
>> I would prefer a debugfs if we have play with these capacity values in
>> order to test their accuracy.
>>
>> Regards,
>> Vincent
>> >
>> > Best,
>> >
>> > - Juri
>>

[toc] | [prev] | [next] | [standalone]

#1311681

On 18/01/16 17:42, Vincent Guittot wrote:
> On 18 January 2016 at 17:30, Juri Lelli <juri.lelli@arm.com> wrote:
> > Hi Vincent,
> >
> > On 18/01/16 17:13, Vincent Guittot wrote:
> >> On 18 January 2016 at 16:13, Juri Lelli <juri.lelli@arm.com> wrote:
> >> > Hi Steve,
> >> >
> >> > On 15/01/16 11:50, Steve Muckle wrote:
> >> >> On 01/08/2016 06:09 AM, Juri Lelli wrote:
> >> >> >  2. Dynamic profiling at boot (v2)
> >> >> >
> >> >> >     pros: - does not require a standardized definition of capacity
> >> >> >           - cannot be incorrectly tuned (once benchmark is fixed)
> >> >> >           - does not require user/integrator work
> >> >> >
> >> >> >     cons: - not easy to come up with a clean solution, as it seems interaction
> >> >> >             with several subsystems (e.g., cpufreq) is required
> >> >> >           - not easy to agree upon a single benchmark (that has to be both
> >> >> >             representative and simple enough to run at boot)
> >> >> >           - numbers might (and do) vary from boot to boot
> >> >>
> >> >> An important additional con that was mentioned earlier IIRC was the
> >> >> additional boot time required for the benchmark.
> >> >
> >> > Right. I forgot about that.
> >> >
> >> >> Perhaps there could be
> >> >> a kernel command line argument to bypass the benchmark if it is known
> >> >> that predetermined values will be provided via sysfs later?
> >> >>
> >> >
> >> > This might work, yes.
> >>
> >> Instead of command line, I prefer to use DT.
> >>
> >> Can't we use something similar to what is currently done in arm arch
> >> for the early stage of the boot ? We don't have to provide performance
> >> value for which it's difficult to find a consensus on how to define it
> >> and which benchmark should be used. We use the micro arch and the
> >> frequency of the core to define a relative capacity. This give us a
> >> relatively good idea of the capacity of each core.
> >
> > I'm not sure I understand what you are proposing. arm arch is currently
> > based on having static hardcoded data (efficiency values). But, this has
> > already been NACKed for arm64 during last review of this RFC.
> >
> > Are you proposing something different?
> 
> No, i'm proposing to use it at boot time until the dynamic profiling
> gives better value.
> We don't have to set any new properties.
> IIRC, It was nacked because it was of static hardcoded value that was
> not always reflecting the best accurate capacity of a system. IMHO,
> it's not that far from reality so can't this be used as an
> intermediate step while waiting for dynamic profiling ?

It seems to me that we will only make things more complicated than
needed, without gaining much. Either we will have these values until the
profile happens (do we really think we will speed up pre-profile boot
time much?) or we will have them as defaults, and every concern that
brought this approach to be nacked will apply.

Thanks,

- Juri

> >
> > Thanks,
> >
> > - Juri
> >
> >> Then, the dynamic profiling can update it with a more accurate value
> >> during the boot.
> >>
> >> >
> >> >> Though there may be another issue with that as mentioned below.
> >> >>
> >> >> >  3. sysfs (v1)
> >> >> >
> >> >> >     pros: - clean and super easy to implement
> >> >> >           - values don't require to be physical properties, defining them is
> >> >> >             probably easier
> >> >> >
> >> >> >     cons: - CPUs capacity have to be provided after boot (by some init script?)
> >> >> >           - API is modified, still some discussion/review is needed
> >> >> >           - values can still be incorrectly used for runtime tuning purposes
> >> >>
> >> >> Initializing the values via userspace init will cause more of the boot
> >> >> process to run with incorrect CPU capacity values. Boot times may be
> >> >> increased with tasks running on suboptimal CPUs. Such increases may also
> >> >> not be deterministic.
> >> >>
> >> >> Extending the kernel command line idea above, perhaps capacity values
> >> >> could be provided there as well, similar to the lpj parameter? That has
> >> >> scalability issues though if there's a huge highly heterogeneous platform...
> >> >>
> >> >
> >> > Yeah, adding such option is not difficult, but I'm also a bit concerned
> >> > about the scalability of such a thing.
> >> >
> >> >> DT solves these issues and would be the perfect place for this - we are
> >> >> defining the compute capacity of a CPU which is a property of the
> >> >> hardware. However there are a couple things forcing us to compromise.
> >> >> One is that the amount and detail of information required to adequately
> >> >> capture the computational abilities of a CPU across all possible
> >> >> workloads seem onerous to collect and enumerate. The second is that even
> >> >> if we were willing to undertake that, CPU vendors probably won't be
> >> >> forthcoming with that information.
> >> >>
> >> >
> >> > You mean because they won't publish performance data of their hw?
> >> >
> >> > But we already use per platform normalized values (as you are proposing
> >> > below). So that a platform to platform comparison doesn't make sense.
> >> >
> >> >> Despite this DT still seems to me like the best way to go. At their
> >> >> heart these are properties of the hardware, even if we can't specify
> >> >> them as such per se because of the problems above. The capacity would
> >> >> have to be defined as a relative value among CPUs. And while it's true
> >> >> it may be abused for tuning purposes, that's true of any strategy.
> >> >> Certainly the sysfs strategy and even if only a dynamic option is
> >> >> provided, it is guaranteed to be hacked by platform vendors.
> >> >
> >> > I also like the DT approach and consider the sysfs option as something
> >> > that can go together with any solution we want to adopt.
> >>
> >> I'm not sure that we should consider sysfs as an option because of all
> >> the concerned that as already been put forward.
> >> I would prefer a debugfs if we have play with these capacity values in
> >> order to test their accuracy.
> >>
> >> Regards,
> >> Vincent
> >> >
> >> > Best,
> >> >
> >> > - Juri
> >>
> 

[toc] | [prev] | [next] | [standalone]

#1311698 — Re: [RFC PATCH v2 0/4] CPUs capacity information for heterogeneous systems

On 18 January 2016 at 18:08, Juri Lelli <juri.lelli@arm.com> wrote:
> On 18/01/16 17:42, Vincent Guittot wrote:
>> On 18 January 2016 at 17:30, Juri Lelli <juri.lelli@arm.com> wrote:
>> > Hi Vincent,
>> >
>> > On 18/01/16 17:13, Vincent Guittot wrote:
>> >> On 18 January 2016 at 16:13, Juri Lelli <juri.lelli@arm.com> wrote:
>> >> > Hi Steve,
>> >> >
>> >> > On 15/01/16 11:50, Steve Muckle wrote:
>> >> >> On 01/08/2016 06:09 AM, Juri Lelli wrote:
>> >> >> >  2. Dynamic profiling at boot (v2)
>> >> >> >
>> >> >> >     pros: - does not require a standardized definition of capacity
>> >> >> >           - cannot be incorrectly tuned (once benchmark is fixed)
>> >> >> >           - does not require user/integrator work
>> >> >> >
>> >> >> >     cons: - not easy to come up with a clean solution, as it seems interaction
>> >> >> >             with several subsystems (e.g., cpufreq) is required
>> >> >> >           - not easy to agree upon a single benchmark (that has to be both
>> >> >> >             representative and simple enough to run at boot)
>> >> >> >           - numbers might (and do) vary from boot to boot
>> >> >>
>> >> >> An important additional con that was mentioned earlier IIRC was the
>> >> >> additional boot time required for the benchmark.
>> >> >
>> >> > Right. I forgot about that.
>> >> >
>> >> >> Perhaps there could be
>> >> >> a kernel command line argument to bypass the benchmark if it is known
>> >> >> that predetermined values will be provided via sysfs later?
>> >> >>
>> >> >
>> >> > This might work, yes.
>> >>
>> >> Instead of command line, I prefer to use DT.
>> >>
>> >> Can't we use something similar to what is currently done in arm arch
>> >> for the early stage of the boot ? We don't have to provide performance
>> >> value for which it's difficult to find a consensus on how to define it
>> >> and which benchmark should be used. We use the micro arch and the
>> >> frequency of the core to define a relative capacity. This give us a
>> >> relatively good idea of the capacity of each core.
>> >
>> > I'm not sure I understand what you are proposing. arm arch is currently
>> > based on having static hardcoded data (efficiency values). But, this has
>> > already been NACKed for arm64 during last review of this RFC.
>> >
>> > Are you proposing something different?
>>
>> No, i'm proposing to use it at boot time until the dynamic profiling
>> gives better value.
>> We don't have to set any new properties.
>> IIRC, It was nacked because it was of static hardcoded value that was
>> not always reflecting the best accurate capacity of a system. IMHO,
>> it's not that far from reality so can't this be used as an
>> intermediate step while waiting for dynamic profiling ?
>
> It seems to me that we will only make things more complicated than
> needed, without gaining much. Either we will have these values until the

Not sure that this will complicate thing as it doesn't need any
specific inputs from DT as it uses properties already available.
Now, if we consider that the potential impact on the boot time of
using default value is not a issue, i agree that it doesn't worth
adding this step

Vincent

> profile happens (do we really think we will speed up pre-profile boot
> time much?) or we will have them as defaults, and every concern that
> brought this approach to be nacked will apply.
>
> Thanks,
>
> - Juri
>
>> >
>> > Thanks,
>> >
>> > - Juri
>> >
>> >> Then, the dynamic profiling can update it with a more accurate value
>> >> during the boot.
>> >>
>> >> >
>> >> >> Though there may be another issue with that as mentioned below.
>> >> >>
>> >> >> >  3. sysfs (v1)
>> >> >> >
>> >> >> >     pros: - clean and super easy to implement
>> >> >> >           - values don't require to be physical properties, defining them is
>> >> >> >             probably easier
>> >> >> >
>> >> >> >     cons: - CPUs capacity have to be provided after boot (by some init script?)
>> >> >> >           - API is modified, still some discussion/review is needed
>> >> >> >           - values can still be incorrectly used for runtime tuning purposes
>> >> >>
>> >> >> Initializing the values via userspace init will cause more of the boot
>> >> >> process to run with incorrect CPU capacity values. Boot times may be
>> >> >> increased with tasks running on suboptimal CPUs. Such increases may also
>> >> >> not be deterministic.
>> >> >>
>> >> >> Extending the kernel command line idea above, perhaps capacity values
>> >> >> could be provided there as well, similar to the lpj parameter? That has
>> >> >> scalability issues though if there's a huge highly heterogeneous platform...
>> >> >>
>> >> >
>> >> > Yeah, adding such option is not difficult, but I'm also a bit concerned
>> >> > about the scalability of such a thing.
>> >> >
>> >> >> DT solves these issues and would be the perfect place for this - we are
>> >> >> defining the compute capacity of a CPU which is a property of the
>> >> >> hardware. However there are a couple things forcing us to compromise.
>> >> >> One is that the amount and detail of information required to adequately
>> >> >> capture the computational abilities of a CPU across all possible
>> >> >> workloads seem onerous to collect and enumerate. The second is that even
>> >> >> if we were willing to undertake that, CPU vendors probably won't be
>> >> >> forthcoming with that information.
>> >> >>
>> >> >
>> >> > You mean because they won't publish performance data of their hw?
>> >> >
>> >> > But we already use per platform normalized values (as you are proposing
>> >> > below). So that a platform to platform comparison doesn't make sense.
>> >> >
>> >> >> Despite this DT still seems to me like the best way to go. At their
>> >> >> heart these are properties of the hardware, even if we can't specify
>> >> >> them as such per se because of the problems above. The capacity would
>> >> >> have to be defined as a relative value among CPUs. And while it's true
>> >> >> it may be abused for tuning purposes, that's true of any strategy.
>> >> >> Certainly the sysfs strategy and even if only a dynamic option is
>> >> >> provided, it is guaranteed to be hacked by platform vendors.
>> >> >
>> >> > I also like the DT approach and consider the sysfs option as something
>> >> > that can go together with any solution we want to adopt.
>> >>
>> >> I'm not sure that we should consider sysfs as an option because of all
>> >> the concerned that as already been put forward.
>> >> I would prefer a debugfs if we have play with these capacity values in
>> >> order to test their accuracy.
>> >>
>> >> Regards,
>> >> Vincent
>> >> >
>> >> > Best,
>> >> >
>> >> > - Juri
>> >>
>>

[toc] | [prev] | [next] | [standalone]

#1312119

On Mon, Jan 18, 2016 at 05:42:58PM +0100, Vincent Guittot wrote:
> On 18 January 2016 at 17:30, Juri Lelli <juri.lelli@arm.com> wrote:
> > On 18/01/16 17:13, Vincent Guittot wrote:
> >> On 18 January 2016 at 16:13, Juri Lelli <juri.lelli@arm.com> wrote:
> >> > On 15/01/16 11:50, Steve Muckle wrote:
> >> >> On 01/08/2016 06:09 AM, Juri Lelli wrote:
> >> >> >  2. Dynamic profiling at boot (v2)
> >> >> >
> >> >> >     pros: - does not require a standardized definition of capacity
> >> >> >           - cannot be incorrectly tuned (once benchmark is fixed)
> >> >> >           - does not require user/integrator work
> >> >> >
> >> >> >     cons: - not easy to come up with a clean solution, as it seems interaction
> >> >> >             with several subsystems (e.g., cpufreq) is required
> >> >> >           - not easy to agree upon a single benchmark (that has to be both
> >> >> >             representative and simple enough to run at boot)
> >> >> >           - numbers might (and do) vary from boot to boot
> >> >>
> >> >> An important additional con that was mentioned earlier IIRC was the
> >> >> additional boot time required for the benchmark.
> >> >
> >> > Right. I forgot about that.
> >> >
> >> >> Perhaps there could be
> >> >> a kernel command line argument to bypass the benchmark if it is known
> >> >> that predetermined values will be provided via sysfs later?
> >> >>
> >> >
> >> > This might work, yes.
> >>
> >> Instead of command line, I prefer to use DT.

I fully agree. Command line doesn't scale with multiple CPUs, at most an
option to bypass the benchmark (though we could just skip it when the DT
values are present).

> >> Can't we use something similar to what is currently done in arm arch
> >> for the early stage of the boot ? We don't have to provide performance
> >> value for which it's difficult to find a consensus on how to define it
> >> and which benchmark should be used. We use the micro arch and the
> >> frequency of the core to define a relative capacity. This give us a
> >> relatively good idea of the capacity of each core.
> >
> > I'm not sure I understand what you are proposing. arm arch is currently
> > based on having static hardcoded data (efficiency values). But, this has
> > already been NACKed for arm64 during last review of this RFC.
> >
> > Are you proposing something different?
> 
> No, i'm proposing to use it at boot time until the dynamic profiling
> gives better value.
> We don't have to set any new properties.
> IIRC, It was nacked because it was of static hardcoded value that was
> not always reflecting the best accurate capacity of a system. IMHO,
> it's not that far from reality so can't this be used as an
> intermediate step while waiting for dynamic profiling ?

My nack for hard-coded values still stands since this is not just about
the microarchitecture (MIDR) but how the CPUs are integrated with the
SoC, additional caches, memory latency, maximum clock frequency (or you
rely on DT again to get this information and scale the initial CPU
capacity/efficiency accordingly). MIDR does not capture SoC details.

Two questions:

1. How is the boot time affected by the benchmark?
2. How is the boot time affected by considering all the CPUs the same?

My preference is for DT and sysfs (especially useful for
development/tuning) but I'm not opposed to a boot-time benchmark if
people insist on it. If the answer to point 2 is "insignificant", we
could as well defer the capacity setting to user space (sysfs).

-- 
Catalin

[toc] | [prev] | [next] | [standalone]

#1312135

Hi Catalin,

On 19/01/16 10:59, Catalin Marinas wrote:
> On Mon, Jan 18, 2016 at 05:42:58PM +0100, Vincent Guittot wrote:
> > On 18 January 2016 at 17:30, Juri Lelli <juri.lelli@arm.com> wrote:
> > > On 18/01/16 17:13, Vincent Guittot wrote:
> > >> On 18 January 2016 at 16:13, Juri Lelli <juri.lelli@arm.com> wrote:
> > >> > On 15/01/16 11:50, Steve Muckle wrote:
> > >> >> On 01/08/2016 06:09 AM, Juri Lelli wrote:
> > >> >> >  2. Dynamic profiling at boot (v2)
> > >> >> >
> > >> >> >     pros: - does not require a standardized definition of capacity
> > >> >> >           - cannot be incorrectly tuned (once benchmark is fixed)
> > >> >> >           - does not require user/integrator work
> > >> >> >
> > >> >> >     cons: - not easy to come up with a clean solution, as it seems interaction
> > >> >> >             with several subsystems (e.g., cpufreq) is required
> > >> >> >           - not easy to agree upon a single benchmark (that has to be both
> > >> >> >             representative and simple enough to run at boot)
> > >> >> >           - numbers might (and do) vary from boot to boot
> > >> >>
> > >> >> An important additional con that was mentioned earlier IIRC was the
> > >> >> additional boot time required for the benchmark.
> > >> >
> > >> > Right. I forgot about that.
> > >> >
> > >> >> Perhaps there could be
> > >> >> a kernel command line argument to bypass the benchmark if it is known
> > >> >> that predetermined values will be provided via sysfs later?
> > >> >>
> > >> >
> > >> > This might work, yes.
> > >>
> > >> Instead of command line, I prefer to use DT.
> 
> I fully agree. Command line doesn't scale with multiple CPUs, at most an
> option to bypass the benchmark (though we could just skip it when the DT
> values are present).
> 
> > >> Can't we use something similar to what is currently done in arm arch
> > >> for the early stage of the boot ? We don't have to provide performance
> > >> value for which it's difficult to find a consensus on how to define it
> > >> and which benchmark should be used. We use the micro arch and the
> > >> frequency of the core to define a relative capacity. This give us a
> > >> relatively good idea of the capacity of each core.
> > >
> > > I'm not sure I understand what you are proposing. arm arch is currently
> > > based on having static hardcoded data (efficiency values). But, this has
> > > already been NACKed for arm64 during last review of this RFC.
> > >
> > > Are you proposing something different?
> > 
> > No, i'm proposing to use it at boot time until the dynamic profiling
> > gives better value.
> > We don't have to set any new properties.
> > IIRC, It was nacked because it was of static hardcoded value that was
> > not always reflecting the best accurate capacity of a system. IMHO,
> > it's not that far from reality so can't this be used as an
> > intermediate step while waiting for dynamic profiling ?
> 
> My nack for hard-coded values still stands since this is not just about
> the microarchitecture (MIDR) but how the CPUs are integrated with the
> SoC, additional caches, memory latency, maximum clock frequency (or you
> rely on DT again to get this information and scale the initial CPU
> capacity/efficiency accordingly). MIDR does not capture SoC details.
> 
> Two questions:
> 
> 1. How is the boot time affected by the benchmark?
> 2. How is the boot time affected by considering all the CPUs the same?
> 
> My preference is for DT and sysfs (especially useful for
> development/tuning) but I'm not opposed to a boot-time benchmark if
> people insist on it. If the answer to point 2 is "insignificant", we
> could as well defer the capacity setting to user space (sysfs).
> 

Given that we are not targeting boot time with this, but rather better
performance afterwards, I don't expect significant differences; but,
I'll get numbers :).

Thanks,

- Juri

[toc] | [prev] | [next] | [standalone]

#1312294

On 19/01/16 11:23, Juri Lelli wrote:
> Hi Catalin,
> 
> On 19/01/16 10:59, Catalin Marinas wrote:
> > On Mon, Jan 18, 2016 at 05:42:58PM +0100, Vincent Guittot wrote:
> > > On 18 January 2016 at 17:30, Juri Lelli <juri.lelli@arm.com> wrote:
> > > > On 18/01/16 17:13, Vincent Guittot wrote:
> > > >> On 18 January 2016 at 16:13, Juri Lelli <juri.lelli@arm.com> wrote:
> > > >> > On 15/01/16 11:50, Steve Muckle wrote:
> > > >> >> On 01/08/2016 06:09 AM, Juri Lelli wrote:

[...]

> > 
> > Two questions:
> > 
> > 1. How is the boot time affected by the benchmark?
> > 2. How is the boot time affected by considering all the CPUs the same?
> > 
> > My preference is for DT and sysfs (especially useful for
> > development/tuning) but I'm not opposed to a boot-time benchmark if
> > people insist on it. If the answer to point 2 is "insignificant", we
> > could as well defer the capacity setting to user space (sysfs).
> > 
> 
> Given that we are not targeting boot time with this, but rather better
> performance afterwards, I don't expect significant differences; but,
> I'll get numbers :).
> 

I've got some boot time numbers on TC2 and Juno based on timestamps.
They are of course not accurate and maybe not so representative of
products, but I guess still ballpark right.

I'm generally seeing ~1sec increase in boot time for 1 and practically
no difference for 2 (even after having added patches that provide
runtime performance improvements).

Best,

- Juri

[toc] | [prev] | [next] | [standalone]

#1312471

On 01/19/2016 06:29 AM, Juri Lelli wrote:
>>> Two questions:
>>> > > 
>>> > > 1. How is the boot time affected by the benchmark?
>>> > > 2. How is the boot time affected by considering all the CPUs the same?
>>> > > 
>>> > > My preference is for DT and sysfs (especially useful for
>>> > > development/tuning) but I'm not opposed to a boot-time benchmark if
>>> > > people insist on it. If the answer to point 2 is "insignificant", we
>>> > > could as well defer the capacity setting to user space (sysfs).
>>> > > 
>> > 
>> > Given that we are not targeting boot time with this, but rather better
>> > performance afterwards, I don't expect significant differences; but,
>> > I'll get numbers :).
>> > 
> I've got some boot time numbers on TC2 and Juno based on timestamps.
> They are of course not accurate and maybe not so representative of
> products, but I guess still ballpark right.
> 
> I'm generally seeing ~1sec increase in boot time for 1 and practically
> no difference for 2 (even after having added patches that provide
> runtime performance improvements).

One second is considerable IMO. Aside from the general desire to have
shorter boot times on any platform there are environments like
automotive where boot time is critical.

How are the CPUs numbered on TC2 and Juno? When all CPUs are considered
the same, is work running on the big CPUs because of the way they are
numbered?

thanks,
Steve

[toc] | [prev] | [next] | [standalone]

#1312518

[Multipart message — attachments visible in raw view] — view raw

On Tue, Jan 19, 2016 at 11:48:15AM -0800, Steve Muckle wrote:
> On 01/19/2016 06:29 AM, Juri Lelli wrote:

> > I'm generally seeing ~1sec increase in boot time for 1 and practically
> > no difference for 2 (even after having added patches that provide
> > runtime performance improvements).

> One second is considerable IMO. Aside from the general desire to have
> shorter boot times on any platform there are environments like
> automotive where boot time is critical.

Yeah, definitely.  Is this actually blocking boot and if so can we
arrange to do this in parallel with other activity (with likely knock on
effects on reproducibility...)?

[toc] | [prev] | [next] | [standalone]

#1313080

On 19/01/16 21:10, Mark Brown wrote:
> On Tue, Jan 19, 2016 at 11:48:15AM -0800, Steve Muckle wrote:
> > On 01/19/2016 06:29 AM, Juri Lelli wrote:
> 
> > > I'm generally seeing ~1sec increase in boot time for 1 and practically
> > > no difference for 2 (even after having added patches that provide
> > > runtime performance improvements).
> 
> > One second is considerable IMO. Aside from the general desire to have
> > shorter boot times on any platform there are environments like
> > automotive where boot time is critical.
> 
> Yeah, definitely.  Is this actually blocking boot and if so can we
> arrange to do this in parallel with other activity (with likely knock on
> effects on reproducibility...)?

No, this goes in parallel. That's also showed by the fact that the
benchmarking thing itself usually takes more that 1 sec, but it seems to
impact for that amount of time only.

[toc] | [prev] | [next] | [standalone]

#1311762

On 01/18/2016 07:13 AM, Juri Lelli wrote:
>> DT solves these issues and would be the perfect place for this - we are
>> > defining the compute capacity of a CPU which is a property of the
>> > hardware. However there are a couple things forcing us to compromise.
>> > One is that the amount and detail of information required to adequately
>> > capture the computational abilities of a CPU across all possible
>> > workloads seem onerous to collect and enumerate. The second is that even
>> > if we were willing to undertake that, CPU vendors probably won't be
>> > forthcoming with that information.
>> > 
>
> You mean because they won't publish performance data of their hw?

More specific things like IPC and other architectural details that could
comprise a precise physical definition of a CPU that would meet the
ideal goals of a device tree definition.

> But we already use per platform normalized values (as you are proposing
> below). So that a platform to platform comparison doesn't make sense.

Yeah I'm just advocating for that strategy here.

cheers,
Steve

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web