Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1221695 > unrolled thread

Re: [RFC 08/14] sched/tune: add detailed documentation

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [RFC 08/14] sched/tune: add detailed documentation Steve Muckle <steve.muckle@linaro.org> - 2015-09-09 22:20 +0200
    Re: [RFC 08/14] sched/tune: add detailed documentation Patrick Bellasi <patrick.bellasi@arm.com> - 2015-09-11 13:20 +0200
      Re: [RFC 08/14] sched/tune: add detailed documentation Steve Muckle <steve.muckle@linaro.org> - 2015-09-14 22:10 +0200
        Re: [RFC 08/14] sched/tune: add detailed documentation Patrick Bellasi <patrick.bellasi@arm.com> - 2015-09-15 17:10 +0200
          Re: [RFC 08/14] sched/tune: add detailed documentation Peter Zijlstra <peterz@infradead.org> - 2015-09-15 17:30 +0200
            Re: [RFC 08/14] sched/tune: add detailed documentation Steve Muckle <steve.muckle@linaro.org> - 2015-09-16 02:40 +0200
              Re: [RFC 08/14] sched/tune: add detailed documentation Ingo Molnar <mingo@kernel.org> - 2015-09-16 09:50 +0200
          Re: [RFC 08/14] sched/tune: add detailed documentation Steve Muckle <steve.muckle@linaro.org> - 2015-09-16 02:00 +0200
            Re: [RFC 08/14] sched/tune: add detailed documentation Juri Lelli <juri.lelli@arm.com> - 2015-09-16 11:30 +0200
              Re: [RFC 08/14] sched/tune: add detailed documentation Vincent Guittot <vincent.guittot@linaro.org> - 2015-09-16 15:50 +0200
            Re: [RFC 08/14] sched/tune: add detailed documentation Patrick Bellasi <patrick.bellasi@arm.com> - 2015-09-16 12:10 +0200

#1221695 — Re: [RFC 08/14] sched/tune: add detailed documentation

Hi Patrick,

On 09/03/2015 02:18 AM, Patrick Bellasi wrote:
> In my view, one of the main goals of sched-DVFS is actually that to be
> a solid and generic replacement of different CPUFreq governors.
> Being driven by the scheduler, sched-DVFS can exploit information on
> CPU demand of active tasks in order to select the optimal Operating
> Performance Point (OPP) using a "proactive" approach instead of the
> "reactive" approach commonly used by existing governors.

I'd agree that with knowledge of CPU demand on a per-task basis, rather
than the aggregate per-CPU demand that cpufreq governors use today, it
is possible to proactively address changes in CPU demand which result
from task migrations, task creation and exit, etc.

That said I believe setting the OPP based on a particular given
historical profile of task load still relies on a heuristic algorithm of
some sort where there is no single right answer. I am concerned about
whether sched-dvfs and SchedTune, as currently proposed, will support
enough of a range of possible heuristics/policies to effectively replace
the existing cpufreq governors.

The two most popular governors for normal operation in the mobile world:

* ondemand: Samples periodically, CPU usage calculated as simple busy
fraction of last X ms window of time. Goes straight to fmax when load
exceeds up_threshold tunable %, otherwise scales frequency
proportionally with load. Can stay at fmax longer if requested before
re-evaluating by configuring the sampling_down_factor tunable.

* interactive: Samples periodically, CPU usage calculated as simple busy
fraction of last Xms window of time. Goes to an intermediate tunable
freq (hispeed_freq) when load exceeds a user definable threshold
(go_hispeed_load). Otherwise strives to maintain the CPU usage set by
the user in the "target_loads" array. Other knobs that affect behavior
include min_sample_time (min time to spend at a freq before slowing
down) and above_hispeed_delay (allows various delays to further raise
speed above hispeed freq).

It's also worth noting that mobile vendors typically add all sorts of
hacks on top of the existing cpufreq governors which further complicate
policy.

The current proposal:

* sched-dvfs/schedtune: Event driven, CPU usage calculated using
exponential moving average. AFAICS tries to maintain some % of idle
headroom, but if that headroom doesn't exist at task_tick_fair(), goes
to max frequency. Schedtune provides a way to boost/inflate the demand
of individual tasks or overall system demand.

This looks a bit like ondemand to me but without the
sampling_down_factor functionality and using per-entity load tracking
instead of a simple window-based aggregate CPU usage. The interactive
functionality would require additional knobs. I don't think schedtune
will allow for tuning the latency of CPU frequency changes
(min_sample_time, above_hispeed_delay, etc).

A separate but related concern - in the (IMO likely, given the above)
case that folks want to tinker with that policy, it now means they're
hacking the scheduler as opposed to a self-contained frequency policy
plugin.

Another issue with policy (but not specific to this proposal) is that
putting a bunch of it in the CPU frequency selection may derail the
efforts of the EAS algorithm, which I'm still working on digesting.
Perhaps a unified sched/cpufreq policy could go there.

thanks,
Steve

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1222721

On Wed, Sep 09, 2015 at 09:16:10PM +0100, Steve Muckle wrote:
> Hi Patrick,

Hi Steve,
 
> On 09/03/2015 02:18 AM, Patrick Bellasi wrote:
> > In my view, one of the main goals of sched-DVFS is actually that to be
> > a solid and generic replacement of different CPUFreq governors.
> > Being driven by the scheduler, sched-DVFS can exploit information on
> > CPU demand of active tasks in order to select the optimal Operating
> > Performance Point (OPP) using a "proactive" approach instead of the
> > "reactive" approach commonly used by existing governors.
> 
> I'd agree that with knowledge of CPU demand on a per-task basis, rather
> than the aggregate per-CPU demand that cpufreq governors use today, it
> is possible to proactively address changes in CPU demand which result
> from task migrations, task creation and exit, etc.
> 
> That said I believe setting the OPP based on a particular given
> historical profile of task load still relies on a heuristic algorithm of
> some sort where there is no single right answer. I am concerned about
> whether sched-dvfs and SchedTune, as currently proposed, will support
> enough of a range of possible heuristics/policies to effectively replace
> the existing cpufreq governors.
> 
> The two most popular governors for normal operation in the mobile world:
> 
> * ondemand: Samples periodically, CPU usage calculated as simple busy
> fraction of last X ms window of time. Goes straight to fmax when load
> exceeds up_threshold tunable %, otherwise scales frequency
> proportionally with load. Can stay at fmax longer if requested before
> re-evaluating by configuring the sampling_down_factor tunable.
> 
> * interactive: Samples periodically, CPU usage calculated as simple busy
> fraction of last Xms window of time. Goes to an intermediate tunable
> freq (hispeed_freq) when load exceeds a user definable threshold
> (go_hispeed_load). Otherwise strives to maintain the CPU usage set by
> the user in the "target_loads" array. Other knobs that affect behavior
> include min_sample_time (min time to spend at a freq before slowing
> down) and above_hispeed_delay (allows various delays to further raise
> speed above hispeed freq).
> 
> It's also worth noting that mobile vendors typically add all sorts of
> hacks on top of the existing cpufreq governors which further complicate
> policy.

Could it be that many of the hacks introduced by vendors are just
there to implement a kind of "scenario based" tuning of governors?
I mean, depending on the specific use-case they try to refine the
value of exposed tunables to improve either performance,
responsiveness or power consumption?

If this is the case, it means that the currently available governors
are missing an important bit of information: what are the best
tunables values for a specific (set of) tasks?

> The current proposal:
> 
> * sched-dvfs/schedtune: Event driven, CPU usage calculated using
> exponential moving average. AFAICS tries to maintain some % of idle
> headroom, but if that headroom doesn't exist at task_tick_fair(), goes
> to max frequency. Schedtune provides a way to boost/inflate the demand
> of individual tasks or overall system demand.

That's quite of a good description. One small correction is that, at
least in the implementation presented by this RFC, SchedTune is not
boosting individual tasks but just the CPU usage.
The link with tasks is just that SchedTune knows how much to boost a
CPU usage by keeping track of which tasks are runnable on that CPU.
However, the utilization signal of each task is not actually modified
from the scheduler standpoint.

> This looks a bit like ondemand to me but without the
> sampling_down_factor functionality and using per-entity load tracking
> instead of a simple window-based aggregate CPU usage.

I agree in principle.
An important difference worth to notice is that we use an "event
based" approach. This means that an enqueue/dequeue can trigger
an immediate OPP change.
If you consider that commonly ondemand uses a 20ms sample rate while
an OPP switch never requires (quite likely) more than 1 or 2 ms, this
means that sched-DVFS can be much more reactive on adapting to
variable loads.

> The interactive functionality would require additional knobs. I
> don't think schedtune will allow for tuning the latency of CPU
> frequency changes (min_sample_time, above_hispeed_delay, etc).

Well, there can be certainly some limitations in the current
implementation. Indeed, the goal of this RFC is to trigger the
discussion and verify if the overall idea make sense and how we
can improve it.

However, regarding specifically the latency on OPP changes, there are
a couple of extension we was thinking about:
1. link the SchedTune boost value with the % of idle headroom which
   triggers an OPP increase
2. use the SchedTune boost value to defined the high frequency to jump
   at when a CPU crosses the % of idle headroom

These are tunables which allows to parameterize the way the PELT
signal for CPU usage is interpreted by the sched-DVFS governor.

How such tunables should be exposed and tuned is to be discussed.
Indeed, one of the main goals of the sched-DVFS and SchedTune
specifically, is to simplify the tuning of a platform by exposing to
userspace a reduced number of tunables, preferably just one.
 
> A separate but related concern - in the (IMO likely, given the above)
> case that folks want to tinker with that policy, it now means they're
> hacking the scheduler as opposed to a self-contained frequency policy
> plugin.

I do not agree on that point. SchedTune, as well as sched-DVFS, are
framework quit well separated from the scheduler.
They are "consumers" of signals usually used by the scheduler, but
they are not directly affecting scheduler decisions (at least in the
implementation proposed by this RFC).

Side effects are possible, of course. For example the selection of an
OPP instead of another can affect the residency of a task on a CPU,
thus somehow biasing some scheduler decisions. However, I think that
this kind of side effects can be produced by current governors as
well.

Eventually, I agree with you if you mean that one can have the
impression of hacking the scheduler because the main compilation unit
of SchedTune is a file under kernel/sched. If this can be a problem,
for example from a maintenance perspective, perhaps we can find a
better location for that code.

> Another issue with policy (but not specific to this proposal) is that
> putting a bunch of it in the CPU frequency selection may derail the
> efforts of the EAS algorithm, which I'm still working on digesting.
> Perhaps a unified sched/cpufreq policy could go there.

We have an internal extension of SchedTune which is proposing an
integration with EAS. We have not included it on that RFC to keep
things simple by exposing at first instance only generic bits which
can extend sched-DVFS features.

However, one of the main goals of this proposal is to respond to a
couple of long lasting demands (e.g. [1,2]) for:
1. a better integration of CPUFreq with the scheduler, which has all
   the required knowledge about workloads demands to target both
   performances and energy efficiency
2. a simple approach to configure a system to care more about
   performance or energy-efficiency

SchedTune addresses mainly the second point. Once SchedTune is
integrated with EAS it will provide a support to decide, in an
energy-efficient way, how much we want to reduce power or boost
performances.

> thanks,
> Steve

Thanks for the interesting feedbacks, this is actually the kind of
discussion we would like to have around this initial proposal.

Cheers Patrick

[1] https://lkml.org/lkml/2012/5/18/91
[2] http://lwn.net/Articles/552889/

-- 
#include <best/regards.h>

Patrick Bellasi

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1224435

Hi Patrick,

On 09/11/2015 04:09 AM, Patrick Bellasi wrote:
>> It's also worth noting that mobile vendors typically add all sorts of
>> hacks on top of the existing cpufreq governors which further complicate
>> policy.
> 
> Could it be that many of the hacks introduced by vendors are just
> there to implement a kind of "scenario based" tuning of governors?
> I mean, depending on the specific use-case they try to refine the
> value of exposed tunables to improve either performance,
> responsiveness or power consumption?

From what I've seen I think it's both scenario based tuning (add
functionality to detect and improve power/perf for say web browsing or
mp3 playback usecases specifically), as well as tailoring general case
behavior. Some of these are actually new features in the governor though
as opposed to just tweaks of existing tunables.

> If this is the case, it means that the currently available governors
> are missing an important bit of information: what are the best
> tunables values for a specific (set of) tasks?

Agreed, though I also think those tunable values might also change for a
given set of tasks in different circumstances.

> 
>> The current proposal:
>>
>> * sched-dvfs/schedtune: Event driven, CPU usage calculated using
>> exponential moving average. AFAICS tries to maintain some % of idle
>> headroom, but if that headroom doesn't exist at task_tick_fair(), goes
>> to max frequency. Schedtune provides a way to boost/inflate the demand
>> of individual tasks or overall system demand.
> 
> That's quite of a good description. One small correction is that, at
> least in the implementation presented by this RFC, SchedTune is not
> boosting individual tasks but just the CPU usage.
> The link with tasks is just that SchedTune knows how much to boost a
> CPU usage by keeping track of which tasks are runnable on that CPU.
> However, the utilization signal of each task is not actually modified
> from the scheduler standpoint.

Ah yes I see what you mean. I was thinking of the cgroup stuff but I see
that max per-task boost is tracked per-CPU and that CPU's aggregate
usage is boosted accordingly.

>> This looks a bit like ondemand to me but without the
>> sampling_down_factor functionality and using per-entity load tracking
>> instead of a simple window-based aggregate CPU usage.
> 
> I agree in principle.
> An important difference worth to notice is that we use an "event
> based" approach. This means that an enqueue/dequeue can trigger
> an immediate OPP change.
> If you consider that commonly ondemand uses a 20ms sample rate while
> an OPP switch never requires (quite likely) more than 1 or 2 ms, this
> means that sched-DVFS can be much more reactive on adapting to
> variable loads.

"Can be" are the important words to me there... it'd be nice to be able
to control that. Aggressive frequency changes may not be desirable for
power or performance, even if the transition can be quickly completed.
The configuration values of min_sample_time and above_hispeed_delay in
the interactive governor on some recent devices may give clues as to
whether latency is being intentionally increased on various platforms.

The latency/reactiveness of CPU frequency changes are also IMO a product
of two things - the CPUfreq/sched-dvfs policy, and the task load
tracking algorithm. I don't have enough experience with the mainline
task load tracking algorithm yet to know how it will compare with the
window-based aggregate CPU usage metric used by mainline cpufreq
governors. But I would imagine it will smooth out some of the aggressive
nature of sched-dvfs' event-driven approach. The hardcoded values in the
task load tracking algorithm seem concerning though from a tuning
standpoint.

>> The interactive functionality would require additional knobs. I
...
> However, regarding specifically the latency on OPP changes, there are
> a couple of extension we was thinking about:
> 1. link the SchedTune boost value with the % of idle headroom which
>    triggers an OPP increase
> 2. use the SchedTune boost value to defined the high frequency to jump
>    at when a CPU crosses the % of idle headroom

Hmmm... This may be useful (only testing/profiling would tell) though it
may be nice to be able to tune these values.

> These are tunables which allows to parameterize the way the PELT
> signal for CPU usage is interpreted by the sched-DVFS governor.
> 
> How such tunables should be exposed and tuned is to be discussed.
> Indeed, one of the main goals of the sched-DVFS and SchedTune
> specifically, is to simplify the tuning of a platform by exposing to
> userspace a reduced number of tunables, preferably just one.

This last point (the desire for a single tunable) is perhaps at the root
of my main concern. There are users/vendors for whom the current
tunables are insufficient, resulting in their hacking the governors to
add more tunables or features in the policy.

Consolidating CPU frequency and idle management in the scheduler will
clean things up and probably make things more effective, but I don't
think it will remove the need for a highly configurable policy.

I'm curious about the drive for one tunable. Is that something there's
specifically been a broad call for? Don't get me wrong, I'm all for
simplification and cleanup, if the flexibility and used features can be
retained.

>> A separate but related concern - in the (IMO likely, given the above)
>> case that folks want to tinker with that policy, it now means they're
>> hacking the scheduler as opposed to a self-contained frequency policy
>> plugin.
> 
> I do not agree on that point. SchedTune, as well as sched-DVFS, are
> framework quit well separated from the scheduler.
> They are "consumers" of signals usually used by the scheduler, but
> they are not directly affecting scheduler decisions (at least in the
> implementation proposed by this RFC).

Agreed it's not affecting scheduler decision making (not directly). It's
more just the mixing of the policy into the same code, as margin is
added in enqueue_task_fair()/task_tick_fair() etc. That one in
particular would probably be easy to solve. A more difficult one is if
someone wants to make adjustments to the load tracking algorithm because
it is driving CPU frequency.

> Side effects are possible, of course. For example the selection of an
...
> However, one of the main goals of this proposal is to respond to a
> couple of long lasting demands (e.g. [1,2]) for:
> 1. a better integration of CPUFreq with the scheduler, which has all
>    the required knowledge about workloads demands to target both
>    performances and energy efficiency
> 2. a simple approach to configure a system to care more about
>    performance or energy-efficiency
> 
> SchedTune addresses mainly the second point. Once SchedTune is
> integrated with EAS it will provide a support to decide, in an
> energy-efficient way, how much we want to reduce power or boost
> performances.

The provided links definitely establish the need for (1) but I am still
wondering about the motivation for (2), because I don't think it's going
to be possible to boil everything down to a single slider tunable
without losing flexibility/functionality.

cheers,
Steve

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1225281

On Mon, Sep 14, 2015 at 09:00:51PM +0100, Steve Muckle wrote:
> Hi Patrick,
> 
> On 09/11/2015 04:09 AM, Patrick Bellasi wrote:
> >> It's also worth noting that mobile vendors typically add all sorts of
> >> hacks on top of the existing cpufreq governors which further complicate
> >> policy.
> > 
> > Could it be that many of the hacks introduced by vendors are just
> > there to implement a kind of "scenario based" tuning of governors?
> > I mean, depending on the specific use-case they try to refine the
> > value of exposed tunables to improve either performance,
> > responsiveness or power consumption?
> 
> From what I've seen I think it's both scenario based tuning (add
> functionality to detect and improve power/perf for say web browsing or
> mp3 playback usecases specifically), as well as tailoring general case
> behavior. Some of these are actually new features in the governor though
> as opposed to just tweaks of existing tunables.
> 
> > If this is the case, it means that the currently available governors
> > are missing an important bit of information: what are the best
> > tunables values for a specific (set of) tasks?
> 
> Agreed, though I also think those tunable values might also change for a
> given set of tasks in different circumstances.

Could you provide an example?

In my view the per-task support should be exploited just for quite
specialized tasks, which are usually not subject to many different
phases during their execution.

For example, in a graphics rendering pipeline usually we have a host
"controller" task and a set set of "worker" tasks running on the
processing elements of the GPU.
Since the controller task is usually low intensity, it does not
generate on the CPU a load big enough to trigger the selection of an
higher OPP. The main issue in this case is that running this task on a
lower OPP could have sensible effects on latency affecting the
performance of the whole graphics pipeline.

For example, on Intel machines I was able to verify that running two
OpenCL workloads concurrently on the same GPU gives better FPS than
just running a single workload. And that it's mainly due to the
selection of an higher OPP on the CPU side when two instances are
running instead of just one.

In these scenarios, the boosting of the CPU OPP when a specific task
is runnable can help on getting better performance.

> >> The current proposal:
> >>
> >> * sched-dvfs/schedtune: Event driven, CPU usage calculated using
> >> exponential moving average. AFAICS tries to maintain some % of idle
> >> headroom, but if that headroom doesn't exist at task_tick_fair(), goes
> >> to max frequency. Schedtune provides a way to boost/inflate the demand
> >> of individual tasks or overall system demand.
> > 
> > That's quite of a good description. One small correction is that, at
> > least in the implementation presented by this RFC, SchedTune is not
> > boosting individual tasks but just the CPU usage.
> > The link with tasks is just that SchedTune knows how much to boost a
> > CPU usage by keeping track of which tasks are runnable on that CPU.
> > However, the utilization signal of each task is not actually modified
> > from the scheduler standpoint.
> 
> Ah yes I see what you mean. I was thinking of the cgroup stuff but I see
> that max per-task boost is tracked per-CPU and that CPU's aggregate
> usage is boosted accordingly.

Right, the idea is to have a sort of "boosting inheritance" mechanism.
While two tasks, with two different boosting values, are concurrently
runnable on a CPU, that CPU is boosted according to the max boost
value for these two tasks.

> >> This looks a bit like ondemand to me but without the
> >> sampling_down_factor functionality and using per-entity load tracking
> >> instead of a simple window-based aggregate CPU usage.
> > 
> > I agree in principle.
> > An important difference worth to notice is that we use an "event
> > based" approach. This means that an enqueue/dequeue can trigger
> > an immediate OPP change.
> > If you consider that commonly ondemand uses a 20ms sample rate while
> > an OPP switch never requires (quite likely) more than 1 or 2 ms, this
> > means that sched-DVFS can be much more reactive on adapting to
> > variable loads.
> 
> "Can be" are the important words to me there... it'd be nice to be able
> to control that. Aggressive frequency changes may not be desirable for
> power or performance, even if the transition can be quickly completed.
> The configuration values of min_sample_time and above_hispeed_delay in
> the interactive governor on some recent devices may give clues as to
> whether latency is being intentionally increased on various platforms.

IMO these knobs are more like fixes for a too "coarse grained" solution.
The main limitation of the current CPUFreq governors are:
1. use a single set of knobs to track many different tasks
2. use a system-wide view to control all tasks

The solution we get is working but, of course, it is an "average"
solution which satisfy on "average" the requirement of different
tasks.

With SchedTune we would like to get a similar result to the one you
describe using min_sample_time and above_hispeed_delay by linking
somehow the "interpretation" of the PELT signal with the boost value.

Right now we have in sched-DVFS an idle % headroom which is hardcoded
to be ~20% of the current OPP capacity. When we cross that boundary
that threshold with the CPU usage, we switch straight to the max OPP.
If we could figure out a proper mechanism to link the boost signal to
both the idle % headroom and the target OPP, I think we could achieve
quite similar results than what you can get with the knobs offered by
the interactive governor.
The more you boost a task the bigger is the idle % headroom and
the higher is the OPP you will jump.

> The latency/reactiveness of CPU frequency changes are also IMO a product
> of two things - the CPUfreq/sched-dvfs policy, and the task load
> tracking algorithm. I don't have enough experience with the mainline
> task load tracking algorithm yet to know how it will compare with the
> window-based aggregate CPU usage metric used by mainline cpufreq
> governors. But I would imagine it will smooth out some of the aggressive
> nature of sched-dvfs' event-driven approach.

That's right, somehow the PELT signal has a dynamic which is well
defined by the time constants it uses. Task enqueue/dequeue events
could happen with a higher frequency dynamic, however these are only
"check points" where the most updated value of a PELT signal could be
used to take a decision.

> The hardcoded values in the
> task load tracking algorithm seem concerning though from a tuning
> standpoint.

I agree, that's why we are thinking about the solution described
before. Exploit the boost value to replace the hardcoded thresholds
should allow to get more flexibility while being per-task defined.
Hopefully, tuning per task can be more easy and effective than
selection a single value fitting all needs.

> 
> >> The interactive functionality would require additional knobs. I
> ...
> > However, regarding specifically the latency on OPP changes, there are
> > a couple of extension we was thinking about:
> > 1. link the SchedTune boost value with the % of idle headroom which
> >    triggers an OPP increase
> > 2. use the SchedTune boost value to defined the high frequency to jump
> >    at when a CPU crosses the % of idle headroom
> 
> Hmmm... This may be useful (only testing/profiling would tell) though it
> may be nice to be able to tune these values.

Again, in my view the tuning should be per task with a single knob.
The value of the knob should than be properly mapped on other internal
values to obtain a well defined behavior driven by information shared
with the scheduler, i.e. a PELT signal.

> > These are tunables which allows to parameterize the way the PELT
> > signal for CPU usage is interpreted by the sched-DVFS governor.
> > 
> > How such tunables should be exposed and tuned is to be discussed.
> > Indeed, one of the main goals of the sched-DVFS and SchedTune
> > specifically, is to simplify the tuning of a platform by exposing to
> > userspace a reduced number of tunables, preferably just one.
> 
> This last point (the desire for a single tunable) is perhaps at the root
> of my main concern. There are users/vendors for whom the current
> tunables are insufficient, resulting in their hacking the governors to
> add more tunables or features in the policy.

We should also consider that we are proposing not only a single
tunable but also a completely different standpoint. Not more a "blind"
system-wide view on the average system behaviors, but instead a more
detailed view on tasks behaviors. A single tunable used to "tag" tasks
maybe it's not such a limited solution in this design.

> Consolidating CPU frequency and idle management in the scheduler will
> clean things up and probably make things more effective, but I don't
> think it will remove the need for a highly configurable policy.

This can be verified only by starting to use sched-DVFS + SchedTune on
real/synthetic setup to verify which features are eventually missing,
or specific use-cases not properly managed.
If we are able to setup these experiments perhaps we will be able to
identify a better design for a scheduler driver solution.

> I'm curious about the drive for one tunable. Is that something there's
> specifically been a broad call for? Don't get me wrong, I'm all for
> simplification and cleanup, if the flexibility and used features can be
> retained.

All this thread [1] was somehow calling out for a solution which goes
in the direction of a single tunable.

The main idea is to exploit the current effort around EAS.
While we are redesign some parts of the scheduler to be energy-ware it
is convenient also to include in that design a knob which allows to
configure how much we want to optimize for reduced power consumption
or increased performance.

> >> A separate but related concern - in the (IMO likely, given the above)
> >> case that folks want to tinker with that policy, it now means they're
> >> hacking the scheduler as opposed to a self-contained frequency policy
> >> plugin.
> > 
> > I do not agree on that point. SchedTune, as well as sched-DVFS, are
> > framework quit well separated from the scheduler.
> > They are "consumers" of signals usually used by the scheduler, but
> > they are not directly affecting scheduler decisions (at least in the
> > implementation proposed by this RFC).
> 
> Agreed it's not affecting scheduler decision making (not directly). It's
> more just the mixing of the policy into the same code, as margin is
> added in enqueue_task_fair()/task_tick_fair() etc. That one in
> particular would probably be easy to solve. A more difficult one is if
> someone wants to make adjustments to the load tracking algorithm because
> it is driving CPU frequency.

That's not so straightforward.

We have plenty of experience, collected on the past years, on CPUFreq
governors and customer specific mods.
Don't you think we can exploit that experience to reason around a
fresh new design that allows to satisfy all requirements while
providing possibly a simpler interface?

I agree with you that all the current scenarios must be supported by
the new proposal. We should probably start by listing them and come
out with a set of test cases that allow to verify where we are wrt
the state of the art.

Tools and benchmarks to verify the proposals and measure the
regress/progress should become more and more used.
This is an even more important requirement to setup a common
language and aims at objective evaluations.
Moreover, it has been already required by scheduler maintainers in the
past.

> > Side effects are possible, of course. For example the selection of an
> ...
> > However, one of the main goals of this proposal is to respond to a
> > couple of long lasting demands (e.g. [1,2]) for:
> > 1. a better integration of CPUFreq with the scheduler, which has all
> >    the required knowledge about workloads demands to target both
> >    performances and energy efficiency
> > 2. a simple approach to configure a system to care more about
> >    performance or energy-efficiency
> > 
> > SchedTune addresses mainly the second point. Once SchedTune is
> > integrated with EAS it will provide a support to decide, in an
> > energy-efficient way, how much we want to reduce power or boost
> > performances.
> 
> The provided links definitely establish the need for (1) but I am still
> wondering about the motivation for (2), because I don't think it's going
> to be possible to boil everything down to a single slider tunable
> without losing flexibility/functionality.

I see and understand your concerns, still I'm on the idea that we
should try to evaluate a different solution which possibly allows to
simplify the user-space interface as well as to reduce the tuning
effort.
All that without scarifying the (measurable) efficiency of the final
result.

> cheers,
> Steve
> 

Thanks for this interesting discussion.

Patrick

[1] http://thread.gmane.org/gmane.linux.kernel/1236846/focus=1237796

-- 
#include <best/regards.h>

Patrick Bellasi

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1225313

On Tue, Sep 15, 2015 at 04:00:45PM +0100, Patrick Bellasi wrote:

> > I'm curious about the drive for one tunable. Is that something there's
> > specifically been a broad call for? Don't get me wrong, I'm all for
> > simplification and cleanup, if the flexibility and used features can be
> > retained.
> 
> All this thread [1] was somehow calling out for a solution which goes
> in the direction of a single tunable.
> 
> The main idea is to exploit the current effort around EAS.
> While we are redesign some parts of the scheduler to be energy-ware it
> is convenient also to include in that design a knob which allows to
> configure how much we want to optimize for reduced power consumption
> or increased performance.

Please flip the argument around; providing lots of knobs for vendors to
do $magic with is _NOT_ a good thing.

The whole out-of-tree cpufreq governor hack fest Android thing is a
complete and utter fail on all levels. Its the embedded, ship, forget,
not contribute cycle all over again.

Making that harder is a _GOOD_ thing.

Esp. now that we get hardware which has multiple frequency domains on
the CPU cores, this is going to be really important.


> > Agreed it's not affecting scheduler decision making (not directly). It's
> > more just the mixing of the policy into the same code, as margin is
> > added in enqueue_task_fair()/task_tick_fair() etc. That one in
> > particular would probably be easy to solve. A more difficult one is if
> > someone wants to make adjustments to the load tracking algorithm because
> > it is driving CPU frequency.
> 
> That's not so straightforward.
> 
> We have plenty of experience, collected on the past years, on CPUFreq
> governors and customer specific mods.
> Don't you think we can exploit that experience to reason around a
> fresh new design that allows to satisfy all requirements while
> providing possibly a simpler interface?
> 
> I agree with you that all the current scenarios must be supported by
> the new proposal. We should probably start by listing them and come
> out with a set of test cases that allow to verify where we are wrt
> the state of the art.
> 
> Tools and benchmarks to verify the proposals and measure the
> regress/progress should become more and more used.
> This is an even more important requirement to setup a common
> language and aims at objective evaluations.
> Moreover, it has been already required by scheduler maintainers in the
> past.

This.

And if $vendor feels their use case doesn't perform well, have them
contribute a benchmark for it. They must have one anyway -- how else are
they going to evaluate the current cpufreq hackery?


Do not encourage vendors to add 'features' in magic warts. Strive to
improve Linux for everyone.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1225630

On 09/15/2015 08:19 AM, Peter Zijlstra wrote:
> Please flip the argument around; providing lots of knobs for vendors to
> do $magic with is _NOT_ a good thing.
> 
> The whole out-of-tree cpufreq governor hack fest Android thing is a
> complete and utter fail on all levels. Its the embedded, ship, forget,
> not contribute cycle all over again.
> 
> Making that harder is a _GOOD_ thing.

I get why the plugin-like governor interface may encourage out of tree
development, but why would providing lots of policy knobs/tunables from
the scheduler be bad?

Shouldn't that hopefully reduce the likelihood that someone feels the
need to roll their own stack of kernel modifications which never make it
upstream?

cheers,
Steve
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1225802

* Steve Muckle <steve.muckle@linaro.org> wrote:

> On 09/15/2015 08:19 AM, Peter Zijlstra wrote:
> > Please flip the argument around; providing lots of knobs for vendors to
> > do $magic with is _NOT_ a good thing.
> > 
> > The whole out-of-tree cpufreq governor hack fest Android thing is a
> > complete and utter fail on all levels. Its the embedded, ship, forget,
> > not contribute cycle all over again.
> > 
> > Making that harder is a _GOOD_ thing.
> 
> I get why the plugin-like governor interface may encourage out of tree
> development, but why would providing lots of policy knobs/tunables from
> the scheduler be bad?

There's many disadvantages:

 - People/vendors learn to rely on their knob based hacks and start making noise 
   if one of the knobs changes or goes away, reporting regressions, not always 
   reporting that they have tunables twiddled.

 - If distros start using knobs it's easy to get into a situation where different 
   distros have different knobs, and people tune them further. For every single 
   bugreport we'd have to first make 100% sure what the knobs are.

 - Workloads can sometimes be improved by twiddling knobs - while breaking lots of 
   other workloads. We'd like people who contribute to the scheduler to think 
   about the hard problems and improve the whole picture, not just a small part of 
   it. There's a rich set of 'knobs' to prevent the scheduler from doing things 
   (the various resource affinity system calls and facilities), so it's not like
   user-space does not have the flexibility.

 - Having too much configuration space makes upgrades to newer kernels generally 
   harder - and we want to have the opposite effects.

> Shouldn't that hopefully reduce the likelihood that someone feels the need to 
> roll their own stack of kernel modifications which never make it upstream?

If a scheduler bug or inefficiency can be kludged around with a knob then that 
reduces the likelihood of the scheduler getting improved.

Otherwise I'd like to encourage people to change the source if they want to debug 
a problem or improve things - so having to roll your own patches isn't an 
unconditional negative - it's what this whole OSS thing is about.

Thanks,

	Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1225602

On 09/15/2015 08:00 AM, Patrick Bellasi wrote:
>> Agreed, though I also think those tunable values might also change for a
>> given set of tasks in different circumstances.
> 
> Could you provide an example?
>
> In my view the per-task support should be exploited just for quite
> specialized tasks, which are usually not subject to many different
> phases during their execution.

The surfaceflinger task in Android is a possible example. It can have
the same issue as the graphics controller task you mentioned - needing
to finish quickly so the overall display pipeline can meet its deadline,
but often not exerting enough CPU demand by itself to raise the
frequency high enough.

Since mobile platforms are so power sensitive though, it won't be
possible to boost surfaceflinger all the time. Perhaps the
surfaceflinger boost could be managed by some sort of userspace daemon
monitoring the sort of usecase running and/or whether display deadlines
are being missed, and updating a schedtune boost cgroup.

> For example, in a graphics rendering pipeline usually we have a host
...
> With SchedTune we would like to get a similar result to the one you
> describe using min_sample_time and above_hispeed_delay by linking
> somehow the "interpretation" of the PELT signal with the boost value.
> 
> Right now we have in sched-DVFS an idle % headroom which is hardcoded
> to be ~20% of the current OPP capacity. When we cross that boundary
> that threshold with the CPU usage, we switch straight to the max OPP.
> If we could figure out a proper mechanism to link the boost signal to
> both the idle % headroom and the target OPP, I think we could achieve
> quite similar results than what you can get with the knobs offered by
> the interactive governor.
> The more you boost a task the bigger is the idle % headroom and
> the higher is the OPP you will jump.

Let's say I have a system with one task (to set aside the per-task vs.
global policy issue temporarily) and I want to define a policy which

 - quickly goes to 1.2GHz when the current frequency is less than
   that and demand exceeds capacity

 - waits at least 40ms (or just "a longer time") before increasing the
   frequency if the current frequency is 1.2GHz or higher

This is similar to (though a simplification of) what interactive is
often configured to do on mobile platforms. AFAIK it's a fairly common
strategy due to the power-perf curves and OPPs available on CPUs, and at
the same time striving to maintain decent UI responsiveness.

Even with the proposed modification to link boost with idle % and target
OPP I don't think there'd currently be a way to express this policy,
which goes beyond the linear scaling of the magnitude of CPU demand
requested by a task, idle headroom or target OPP.

> 
...
>> The hardcoded values in the
>> task load tracking algorithm seem concerning though from a tuning
>> standpoint.
> 
> I agree, that's why we are thinking about the solution described
> before. Exploit the boost value to replace the hardcoded thresholds
> should allow to get more flexibility while being per-task defined.
> Hopefully, tuning per task can be more easy and effective than
> selection a single value fitting all needs.
> 
>>
>>>> The interactive functionality would require additional knobs. I
>> ...
>>> However, regarding specifically the latency on OPP changes, there are
>>> a couple of extension we was thinking about:
>>> 1. link the SchedTune boost value with the % of idle headroom which
>>>    triggers an OPP increase
>>> 2. use the SchedTune boost value to defined the high frequency to jump
>>>    at when a CPU crosses the % of idle headroom
>>
>> Hmmm... This may be useful (only testing/profiling would tell) though it
>> may be nice to be able to tune these values.
> 
> Again, in my view the tuning should be per task with a single knob.
> The value of the knob should than be properly mapped on other internal
> values to obtain a well defined behavior driven by information shared
> with the scheduler, i.e. a PELT signal.
> 
>>> These are tunables which allows to parameterize the way the PELT
>>> signal for CPU usage is interpreted by the sched-DVFS governor.
>>>
>>> How such tunables should be exposed and tuned is to be discussed.
>>> Indeed, one of the main goals of the sched-DVFS and SchedTune
>>> specifically, is to simplify the tuning of a platform by exposing to
>>> userspace a reduced number of tunables, preferably just one.
>>
>> This last point (the desire for a single tunable) is perhaps at the root
>> of my main concern. There are users/vendors for whom the current
>> tunables are insufficient, resulting in their hacking the governors to
>> add more tunables or features in the policy.
> 
> We should also consider that we are proposing not only a single
> tunable but also a completely different standpoint. Not more a "blind"
> system-wide view on the average system behaviors, but instead a more
> detailed view on tasks behaviors. A single tunable used to "tag" tasks
> maybe it's not such a limited solution in this design.

I think the algorithm is still fairly blind. There still has to be a
heuristic for future CPU usage, it's now just per-task and in the
scheduler (PELT), whereas it used to be per-CPU and in the governor.

This allows for good features like adjusting frequency right away on
task migration/creation/exit or per task boosting etc., but I think
policy will still be important. Tasks change their behavior all the
time, at least in the mobile usecases I've seen.

>> Consolidating CPU frequency and idle management in the scheduler will
>> clean things up and probably make things more effective, but I don't
>> think it will remove the need for a highly configurable policy.
> 
> This can be verified only by starting to use sched-DVFS + SchedTune on
> real/synthetic setup to verify which features are eventually missing,
> or specific use-cases not properly managed.
> If we are able to setup these experiments perhaps we will be able to
> identify a better design for a scheduler driver solution.

Agree. I hope to be able to run some of these experiments to help.

>> I'm curious about the drive for one tunable. Is that something there's
...
> We have plenty of experience, collected on the past years, on CPUFreq
> governors and customer specific mods.
> Don't you think we can exploit that experience to reason around a
> fresh new design that allows to satisfy all requirements while
> providing possibly a simpler interface?

Sure. I'm just communicating requirements I've seen :) .

> I agree with you that all the current scenarios must be supported by
> the new proposal. We should probably start by listing them and come
> out with a set of test cases that allow to verify where we are wrt
> the state of the art.

Sounds like a good plan to me... Perhaps we could discuss some mobile
usecases next week at Linaro Connect?

cheers,
Steve

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1225885

Hi Steve,

thanks a lot for this interesting discussion.

On 16/09/15 00:55, Steve Muckle wrote:
> On 09/15/2015 08:00 AM, Patrick Bellasi wrote:
>>> Agreed, though I also think those tunable values might also change for a
>>> given set of tasks in different circumstances.
>>
>> Could you provide an example?
>>
>> In my view the per-task support should be exploited just for quite
>> specialized tasks, which are usually not subject to many different
>> phases during their execution.
> 
> The surfaceflinger task in Android is a possible example. It can have
> the same issue as the graphics controller task you mentioned - needing
> to finish quickly so the overall display pipeline can meet its deadline,
> but often not exerting enough CPU demand by itself to raise the
> frequency high enough.
>

SurfaceFlinger timeliness requirements, and maybe AudioFlinger's and
others' as well, might be better expressed by using other scheduling
classes, IMHO. SCHED_DEADLINE, for example, has built-in explicit
deadlines awareness and might work better with this kind of activities.
Not to mention that Android has already started using SCHED_FIFO for
some of its time sensitive tasks. It seems to me that the long run goal
should be to give the scheduler more information about what is going on
and then use such information to do more informed decisions (scheduling,
OPP selection, etc.).

> Since mobile platforms are so power sensitive though, it won't be
> possible to boost surfaceflinger all the time. Perhaps the
> surfaceflinger boost could be managed by some sort of userspace daemon
> monitoring the sort of usecase running and/or whether display deadlines
> are being missed, and updating a schedtune boost cgroup.
> 

I'd say you would like to "boost" just enough to meet a certain quality
of service in the end.

>> For example, in a graphics rendering pipeline usually we have a host
> ...
>> With SchedTune we would like to get a similar result to the one you
>> describe using min_sample_time and above_hispeed_delay by linking
>> somehow the "interpretation" of the PELT signal with the boost value.
>>
>> Right now we have in sched-DVFS an idle % headroom which is hardcoded
>> to be ~20% of the current OPP capacity. When we cross that boundary
>> that threshold with the CPU usage, we switch straight to the max OPP.
>> If we could figure out a proper mechanism to link the boost signal to
>> both the idle % headroom and the target OPP, I think we could achieve
>> quite similar results than what you can get with the knobs offered by
>> the interactive governor.
>> The more you boost a task the bigger is the idle % headroom and
>> the higher is the OPP you will jump.
> 
> Let's say I have a system with one task (to set aside the per-task vs.
> global policy issue temporarily) and I want to define a policy which
> 
>  - quickly goes to 1.2GHz when the current frequency is less than
>    that and demand exceeds capacity
> 
>  - waits at least 40ms (or just "a longer time") before increasing the
>    frequency if the current frequency is 1.2GHz or higher
> 
> This is similar to (though a simplification of) what interactive is
> often configured to do on mobile platforms. AFAIK it's a fairly common
> strategy due to the power-perf curves and OPPs available on CPUs, and at
> the same time striving to maintain decent UI responsiveness.
> 

Not that this is already in place, but, once we'll have an energy model
of the platform available to the scheduler (the EAS idea), shouldn't
this kind of considerations be possible without any explicit
configuration? I mean, it seems to me that you start reasoning about
trade-offs after you obtained power-perf curves for your platform; but,
once this data will be available to the scheduler, don't you think we
could put a bit more intelligence there to make the same kind of
decisions you would configure a governor to do?

> Even with the proposed modification to link boost with idle % and target
> OPP I don't think there'd currently be a way to express this policy,
> which goes beyond the linear scaling of the magnitude of CPU demand
> requested by a task, idle headroom or target OPP.
> 
>>
> ...
>>> The hardcoded values in the
>>> task load tracking algorithm seem concerning though from a tuning
>>> standpoint.
>>
>> I agree, that's why we are thinking about the solution described
>> before. Exploit the boost value to replace the hardcoded thresholds
>> should allow to get more flexibility while being per-task defined.
>> Hopefully, tuning per task can be more easy and effective than
>> selection a single value fitting all needs.
>>
>>>
>>>>> The interactive functionality would require additional knobs. I
>>> ...
>>>> However, regarding specifically the latency on OPP changes, there are
>>>> a couple of extension we was thinking about:
>>>> 1. link the SchedTune boost value with the % of idle headroom which
>>>>    triggers an OPP increase
>>>> 2. use the SchedTune boost value to defined the high frequency to jump
>>>>    at when a CPU crosses the % of idle headroom
>>>
>>> Hmmm... This may be useful (only testing/profiling would tell) though it
>>> may be nice to be able to tune these values.
>>
>> Again, in my view the tuning should be per task with a single knob.
>> The value of the knob should than be properly mapped on other internal
>> values to obtain a well defined behavior driven by information shared
>> with the scheduler, i.e. a PELT signal.
>>
>>>> These are tunables which allows to parameterize the way the PELT
>>>> signal for CPU usage is interpreted by the sched-DVFS governor.
>>>>
>>>> How such tunables should be exposed and tuned is to be discussed.
>>>> Indeed, one of the main goals of the sched-DVFS and SchedTune
>>>> specifically, is to simplify the tuning of a platform by exposing to
>>>> userspace a reduced number of tunables, preferably just one.
>>>
>>> This last point (the desire for a single tunable) is perhaps at the root
>>> of my main concern. There are users/vendors for whom the current
>>> tunables are insufficient, resulting in their hacking the governors to
>>> add more tunables or features in the policy.
>>
>> We should also consider that we are proposing not only a single
>> tunable but also a completely different standpoint. Not more a "blind"
>> system-wide view on the average system behaviors, but instead a more
>> detailed view on tasks behaviors. A single tunable used to "tag" tasks
>> maybe it's not such a limited solution in this design.
> 
> I think the algorithm is still fairly blind. There still has to be a
> heuristic for future CPU usage, it's now just per-task and in the
> scheduler (PELT), whereas it used to be per-CPU and in the governor.
> 
> This allows for good features like adjusting frequency right away on
> task migration/creation/exit or per task boosting etc., but I think
> policy will still be important. Tasks change their behavior all the
> time, at least in the mobile usecases I've seen.
> 
>>> Consolidating CPU frequency and idle management in the scheduler will
>>> clean things up and probably make things more effective, but I don't
>>> think it will remove the need for a highly configurable policy.
>>
>> This can be verified only by starting to use sched-DVFS + SchedTune on
>> real/synthetic setup to verify which features are eventually missing,
>> or specific use-cases not properly managed.
>> If we are able to setup these experiments perhaps we will be able to
>> identify a better design for a scheduler driver solution.
> 
> Agree. I hope to be able to run some of these experiments to help.
> 
>>> I'm curious about the drive for one tunable. Is that something there's
> ...
>> We have plenty of experience, collected on the past years, on CPUFreq
>> governors and customer specific mods.
>> Don't you think we can exploit that experience to reason around a
>> fresh new design that allows to satisfy all requirements while
>> providing possibly a simpler interface?
> 
> Sure. I'm just communicating requirements I've seen :) .
> 

And that's great! :-)

>> I agree with you that all the current scenarios must be supported by
>> the new proposal. We should probably start by listing them and come
>> out with a set of test cases that allow to verify where we are wrt
>> the state of the art.
> 
> Sounds like a good plan to me... Perhaps we could discuss some mobile
> usecases next week at Linaro Connect?
> 

I'm up for it!

Best,

- Juri

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1226093

On 16 September 2015 at 11:26, Juri Lelli <juri.lelli@arm.com> wrote:
>
> Hi Steve,
>
> thanks a lot for this interesting discussion.
>
> On 16/09/15 00:55, Steve Muckle wrote:
> > On 09/15/2015 08:00 AM, Patrick Bellasi wrote:
> >>> Agreed, though I also think those tunable values might also change for a
> >>> given set of tasks in different circumstances.
> >>
> >> Could you provide an example?
> >>
> >> In my view the per-task support should be exploited just for quite
> >> specialized tasks, which are usually not subject to many different
> >> phases during their execution.
> >
> > The surfaceflinger task in Android is a possible example. It can have
> > the same issue as the graphics controller task you mentioned - needing
> > to finish quickly so the overall display pipeline can meet its deadline,
> > but often not exerting enough CPU demand by itself to raise the
> > frequency high enough.
> >
>
> SurfaceFlinger timeliness requirements, and maybe AudioFlinger's and
> others' as well, might be better expressed by using other scheduling
> classes, IMHO. SCHED_DEADLINE, for example, has built-in explicit

I fully agree on this point that we must be sure to not create knob to
solve some latency/perf/power issue in a sched class whereas it can be
easily solved with a more appropriate sched class.
Surface flinger and sched_deadline is a good example for this kind of
"critical" task that can accept a limited amount of latency

Vincent

>
> deadlines awareness and might work better with this kind of activities.
> Not to mention that Android has already started using SCHED_FIFO for
> some of its time sensitive tasks. It seems to me that the long run goal
> should be to give the scheduler more information about what is going on
> and then use such information to do more informed decisions (scheduling,
> OPP selection, etc.).
>
> > Since mobile platforms are so power sensitive though, it won't be
> > possible to boost surfaceflinger all the time. Perhaps the
> > surfaceflinger boost could be managed by some sort of userspace daemon
> > monitoring the sort of usecase running and/or whether display deadlines
> > are being missed, and updating a schedtune boost cgroup.
> >
>
> I'd say you would like to "boost" just enough to meet a certain quality
> of service in the end.
>
> >> For example, in a graphics rendering pipeline usually we have a host
> > ...
> >> With SchedTune we would like to get a similar result to the one you
> >> describe using min_sample_time and above_hispeed_delay by linking
> >> somehow the "interpretation" of the PELT signal with the boost value.
> >>
> >> Right now we have in sched-DVFS an idle % headroom which is hardcoded
> >> to be ~20% of the current OPP capacity. When we cross that boundary
> >> that threshold with the CPU usage, we switch straight to the max OPP.
> >> If we could figure out a proper mechanism to link the boost signal to
> >> both the idle % headroom and the target OPP, I think we could achieve
> >> quite similar results than what you can get with the knobs offered by
> >> the interactive governor.
> >> The more you boost a task the bigger is the idle % headroom and
> >> the higher is the OPP you will jump.
> >
> > Let's say I have a system with one task (to set aside the per-task vs.
> > global policy issue temporarily) and I want to define a policy which
> >
> >  - quickly goes to 1.2GHz when the current frequency is less than
> >    that and demand exceeds capacity
> >
> >  - waits at least 40ms (or just "a longer time") before increasing the
> >    frequency if the current frequency is 1.2GHz or higher
> >
> > This is similar to (though a simplification of) what interactive is
> > often configured to do on mobile platforms. AFAIK it's a fairly common
> > strategy due to the power-perf curves and OPPs available on CPUs, and at
> > the same time striving to maintain decent UI responsiveness.
> >
>
> Not that this is already in place, but, once we'll have an energy model
> of the platform available to the scheduler (the EAS idea), shouldn't
> this kind of considerations be possible without any explicit
> configuration? I mean, it seems to me that you start reasoning about
> trade-offs after you obtained power-perf curves for your platform; but,
> once this data will be available to the scheduler, don't you think we
> could put a bit more intelligence there to make the same kind of
> decisions you would configure a governor to do?
>
> > Even with the proposed modification to link boost with idle % and target
> > OPP I don't think there'd currently be a way to express this policy,
> > which goes beyond the linear scaling of the magnitude of CPU demand
> > requested by a task, idle headroom or target OPP.
> >
> >>
> > ...
> >>> The hardcoded values in the
> >>> task load tracking algorithm seem concerning though from a tuning
> >>> standpoint.
> >>
> >> I agree, that's why we are thinking about the solution described
> >> before. Exploit the boost value to replace the hardcoded thresholds
> >> should allow to get more flexibility while being per-task defined.
> >> Hopefully, tuning per task can be more easy and effective than
> >> selection a single value fitting all needs.
> >>
> >>>
> >>>>> The interactive functionality would require additional knobs. I
> >>> ...
> >>>> However, regarding specifically the latency on OPP changes, there are
> >>>> a couple of extension we was thinking about:
> >>>> 1. link the SchedTune boost value with the % of idle headroom which
> >>>>    triggers an OPP increase
> >>>> 2. use the SchedTune boost value to defined the high frequency to jump
> >>>>    at when a CPU crosses the % of idle headroom
> >>>
> >>> Hmmm... This may be useful (only testing/profiling would tell) though it
> >>> may be nice to be able to tune these values.
> >>
> >> Again, in my view the tuning should be per task with a single knob.
> >> The value of the knob should than be properly mapped on other internal
> >> values to obtain a well defined behavior driven by information shared
> >> with the scheduler, i.e. a PELT signal.
> >>
> >>>> These are tunables which allows to parameterize the way the PELT
> >>>> signal for CPU usage is interpreted by the sched-DVFS governor.
> >>>>
> >>>> How such tunables should be exposed and tuned is to be discussed.
> >>>> Indeed, one of the main goals of the sched-DVFS and SchedTune
> >>>> specifically, is to simplify the tuning of a platform by exposing to
> >>>> userspace a reduced number of tunables, preferably just one.
> >>>
> >>> This last point (the desire for a single tunable) is perhaps at the root
> >>> of my main concern. There are users/vendors for whom the current
> >>> tunables are insufficient, resulting in their hacking the governors to
> >>> add more tunables or features in the policy.
> >>
> >> We should also consider that we are proposing not only a single
> >> tunable but also a completely different standpoint. Not more a "blind"
> >> system-wide view on the average system behaviors, but instead a more
> >> detailed view on tasks behaviors. A single tunable used to "tag" tasks
> >> maybe it's not such a limited solution in this design.
> >
> > I think the algorithm is still fairly blind. There still has to be a
> > heuristic for future CPU usage, it's now just per-task and in the
> > scheduler (PELT), whereas it used to be per-CPU and in the governor.
> >
> > This allows for good features like adjusting frequency right away on
> > task migration/creation/exit or per task boosting etc., but I think
> > policy will still be important. Tasks change their behavior all the
> > time, at least in the mobile usecases I've seen.
> >
> >>> Consolidating CPU frequency and idle management in the scheduler will
> >>> clean things up and probably make things more effective, but I don't
> >>> think it will remove the need for a highly configurable policy.
> >>
> >> This can be verified only by starting to use sched-DVFS + SchedTune on
> >> real/synthetic setup to verify which features are eventually missing,
> >> or specific use-cases not properly managed.
> >> If we are able to setup these experiments perhaps we will be able to
> >> identify a better design for a scheduler driver solution.
> >
> > Agree. I hope to be able to run some of these experiments to help.
> >
> >>> I'm curious about the drive for one tunable. Is that something there's
> > ...
> >> We have plenty of experience, collected on the past years, on CPUFreq
> >> governors and customer specific mods.
> >> Don't you think we can exploit that experience to reason around a
> >> fresh new design that allows to satisfy all requirements while
> >> providing possibly a simpler interface?
> >
> > Sure. I'm just communicating requirements I've seen :) .
> >
>
> And that's great! :-)
>
> >> I agree with you that all the current scenarios must be supported by
> >> the new proposal. We should probably start by listing them and come
> >> out with a set of test cases that allow to verify where we are wrt
> >> the state of the art.
> >
> > Sounds like a good plan to me... Perhaps we could discuss some mobile
> > usecases next week at Linaro Connect?
> >
>
> I'm up for it!
>
> Best,
>
> - Juri
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-pm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1225918

On Wed, Sep 16, 2015 at 12:55:12AM +0100, Steve Muckle wrote:
> On 09/15/2015 08:00 AM, Patrick Bellasi wrote:
> >> Agreed, though I also think those tunable values might also change for a
> >> given set of tasks in different circumstances.
> > 
> > Could you provide an example?
> >
> > In my view the per-task support should be exploited just for quite
> > specialized tasks, which are usually not subject to many different
> > phases during their execution.
> 
> The surfaceflinger task in Android is a possible example. It can have
> the same issue as the graphics controller task you mentioned - needing
> to finish quickly so the overall display pipeline can meet its deadline,
> but often not exerting enough CPU demand by itself to raise the
> frequency high enough.

Right, that's actually a really good example which can be an
interesting starting point to experiment and get plots and
performance numbers to compare interactive with the new proposal.

> Since mobile platforms are so power sensitive though, it won't be
> possible to boost surfaceflinger all the time. Perhaps the
> surfaceflinger boost could be managed by some sort of userspace daemon
> monitoring the sort of usecase running and/or whether display deadlines
> are being missed, and updating a schedtune boost cgroup.

That's a really good example for the need to expose a simple yet
effective interface. In the mobile space, middleware like Android or
Chrome (I'm thinking about ChromeOS devises) can provide valuable
input to the scheduler. IMHO, the more the scheduler know about a task
(set of tasks) the more we can aim at improving it to give a standard
and well tested solution which targets both energy efficiency and
performance boosting.

I'm still not completely convinced that CGroup could be a suitable
interface, especially considering the running discussion on the
restructuring of the cpuset controller. However, the idea to provide a
per-task/per-process tunable interface is still sound to me.

> > For example, in a graphics rendering pipeline usually we have a host
> ...
> > With SchedTune we would like to get a similar result to the one you
> > describe using min_sample_time and above_hispeed_delay by linking
> > somehow the "interpretation" of the PELT signal with the boost value.
> > 
> > Right now we have in sched-DVFS an idle % headroom which is hardcoded
> > to be ~20% of the current OPP capacity. When we cross that boundary
> > that threshold with the CPU usage, we switch straight to the max OPP.
> > If we could figure out a proper mechanism to link the boost signal to
> > both the idle % headroom and the target OPP, I think we could achieve
> > quite similar results than what you can get with the knobs offered by
> > the interactive governor.
> > The more you boost a task the bigger is the idle % headroom and
> > the higher is the OPP you will jump.
> 
> Let's say I have a system with one task (to set aside the per-task vs.
> global policy issue temporarily) and I want to define a policy which
> 
>  - quickly goes to 1.2GHz when the current frequency is less than
>    that and demand exceeds capacity
> 
>  - waits at least 40ms (or just "a longer time") before increasing the
>    frequency if the current frequency is 1.2GHz or higher
> 
> This is similar to (though a simplification of) what interactive is
> often configured to do on mobile platforms. AFAIK it's a fairly common
> strategy due to the power-perf curves and OPPs available on CPUs, and at
> the same time striving to maintain decent UI responsiveness.

In the proposal presented with this RFC there is just one "signal
boosting strategy" named "Signal Proportional Compensation" (SPC).
Actually, internally we are evaluating other boosting policies as well
that we decided to not post to keep it simple at the beginning.

What you are proposing make sense and it's similar to another policy
we was considering, which is however just a slightly variation of the
SPC. The idea is to use a parameter to define the compensation
boundary. Right now that boundary is set to be SCHED_LOAD_SCALE (i.e.
1024) which it the maximum capacity available on a system.

The same SPC works fine if we use a different value (lower than
SCHED_LOAD_SCALE) which could be configured for example to match a
specific feature of the OPP curve or, in case of a big.LITTLE system,
the max capacity of a LITTLE cluster.

> Even with the proposed modification to link boost with idle % and target
> OPP I don't think there'd currently be a way to express this policy,
> which goes beyond the linear scaling of the magnitude of CPU demand
> requested by a task, idle headroom or target OPP.

In you example, a 100% compensation could be configured to select
right the 1.2GHz OPP. From that point, any further increase of OPP
will be driven just by the original (i.e. not boosted) task
utilization. This should allow for example on a big.LITTLE system to
boost a small task to the max OPP of a LITTLE cluster while making it
eligible for a migration on the big cluster only if its real
utilization (after a while) becomes bigger than the LITTLE capacity.

It's worth to notice that in this case "a longer time" is not
something defined once for all the tasks in a system, but instead it
is a time frame more closely related to the specific nature of tasks
running on a CPU.

We do not spend time trying to tune a system to match on average all
the tasks of the system but instead we provide a valuable information
to the scheduler (and sched-DVFS) to understand when it's worth to
switch OPP according to the information it has about the tasks it's
managing.


> ...
> >> The hardcoded values in the
> >> task load tracking algorithm seem concerning though from a tuning
> >> standpoint.
> > 
> > I agree, that's why we are thinking about the solution described
> > before. Exploit the boost value to replace the hardcoded thresholds
> > should allow to get more flexibility while being per-task defined.
> > Hopefully, tuning per task can be more easy and effective than
> > selection a single value fitting all needs.
> > 
> >>
> >>>> The interactive functionality would require additional knobs. I
> >> ...
> >>> However, regarding specifically the latency on OPP changes, there are
> >>> a couple of extension we was thinking about:
> >>> 1. link the SchedTune boost value with the % of idle headroom which
> >>>    triggers an OPP increase
> >>> 2. use the SchedTune boost value to defined the high frequency to jump
> >>>    at when a CPU crosses the % of idle headroom
> >>
> >> Hmmm... This may be useful (only testing/profiling would tell) though it
> >> may be nice to be able to tune these values.
> > 
> > Again, in my view the tuning should be per task with a single knob.
> > The value of the knob should than be properly mapped on other internal
> > values to obtain a well defined behavior driven by information shared
> > with the scheduler, i.e. a PELT signal.
> > 
> >>> These are tunables which allows to parameterize the way the PELT
> >>> signal for CPU usage is interpreted by the sched-DVFS governor.
> >>>
> >>> How such tunables should be exposed and tuned is to be discussed.
> >>> Indeed, one of the main goals of the sched-DVFS and SchedTune
> >>> specifically, is to simplify the tuning of a platform by exposing to
> >>> userspace a reduced number of tunables, preferably just one.
> >>
> >> This last point (the desire for a single tunable) is perhaps at the root
> >> of my main concern. There are users/vendors for whom the current
> >> tunables are insufficient, resulting in their hacking the governors to
> >> add more tunables or features in the policy.
> > 
> > We should also consider that we are proposing not only a single
> > tunable but also a completely different standpoint. Not more a "blind"
> > system-wide view on the average system behaviors, but instead a more
> > detailed view on tasks behaviors. A single tunable used to "tag" tasks
> > maybe it's not such a limited solution in this design.
> 
> I think the algorithm is still fairly blind. There still has to be a
> heuristic for future CPU usage, it's now just per-task and in the
> scheduler (PELT), whereas it used to be per-CPU and in the governor.

Forecasting the future is a tough task, especially if you do not have
sensible information from informed entities. The main risk with
heuristics decoupled from sensible information is that you get just an
"average good" result at the cost of a long and painful tuning
activity. If your workload mix changes, than the tuning risks to be
broken.

IMO a more valuable approach is to provide effective interfaces to
collect sensible information. Than, underneath, a well defined design
can be found to correlate and exploit all these information to take
"good enough" decisions.

> This allows for good features like adjusting frequency right away on
> task migration/creation/exit or per task boosting etc., but I think
> policy will still be important. Tasks change their behavior all the
> time, at least in the mobile usecases I've seen.

That's where a middleware (possibly) should have a simple and well
defined interface to update the hits given to the scheduler for a
specific task.

> >> Consolidating CPU frequency and idle management in the scheduler will
> >> clean things up and probably make things more effective, but I don't
> >> think it will remove the need for a highly configurable policy.
> > 
> > This can be verified only by starting to use sched-DVFS + SchedTune on
> > real/synthetic setup to verify which features are eventually missing,
> > or specific use-cases not properly managed.
> > If we are able to setup these experiments perhaps we will be able to
> > identify a better design for a scheduler driver solution.
> 
> Agree. I hope to be able to run some of these experiments to help.

Good, actually we should discuss also about an effective way to run
experiments and collect/share results. We have some tools and ideas
about that... we can discuss better about that next week at the Linaro
Connect.

> >> I'm curious about the drive for one tunable. Is that something there's
> ...
> > We have plenty of experience, collected on the past years, on CPUFreq
> > governors and customer specific mods.
> > Don't you think we can exploit that experience to reason around a
> > fresh new design that allows to satisfy all requirements while
> > providing possibly a simpler interface?
> 
> Sure. I'm just communicating requirements I've seen :) .

That's exactly what we need for this initial stage.
I think we are on the right direction to setup a fruitful discussion.

> > I agree with you that all the current scenarios must be supported by
> > the new proposal. We should probably start by listing them and come
> > out with a set of test cases that allow to verify where we are wrt
> > the state of the art.
> 
> Sounds like a good plan to me... Perhaps we could discuss some mobile
> usecases next week at Linaro Connect?

Absolutely yes!

> 
> cheers,
> Steve
> 

Cheers Patrick

-- 
#include <best/regards.h>

Patrick Bellasi

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web