Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1217357 > unrolled thread

Re: [RFC,08/14] sched/tune: add detailed documentation

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [RFC,08/14] sched/tune: add detailed documentation Ricky Liang <jcliang@chromium.org> - 2015-09-02 09:00 +0200
    Re: [RFC 08/14] sched/tune: add detailed documentation Patrick Bellasi <patrick.bellasi@arm.com> - 2015-09-03 11:20 +0200
      Re: [RFC 08/14] sched/tune: add detailed documentation Ricky Liang <jcliang@chromium.org> - 2015-09-04 10:10 +0200

#1217357 — Re: [RFC,08/14] sched/tune: add detailed documentation

Hi Patrick,

I wonder if this can replace the boost function in the interactive governor [0],
which is widely used in both Android and ChromeOS kernels.

My understanding is that the boost in interactive governor is to simply raise
the OPP on selected cores. The SchedTune boost works by adding a margin to the
original load of a task which makes the kernel think that the task is more
demanding than it actually is. My intuition was that they work differently and
could cause different reaction in the kernel.

I feel that the per-task cgroup ScheTune boost should work as expected as it
only boosts a set of tasks and make them appear relatively high demanding
comparing to other tasks. But if the ScheTune boost is applied globally to boost
all the tasks in the system, will it cause unnecessary task migrations as all
the tasks appear to be high demanding to the kernel?

Specifically, my questions is: When the global SchedTune boost is enabled in a
on-demand manner, is it possible that a light task gets migrated to the big
core, and in turn kicks out a heavy task originally on that core? I'm wondering
whether global SchedTune boost could result in a "priority inversion" causing
the heavy task to run on the little core and the light task to run on the big
core.

[0]: https://android.googlesource.com/kernel/common.git/+/android-3.18/drivers/cpufreq/cpufreq_interactive.c

Thanks,
Ricky

On Wed, Aug 19, 2015 at 07:47:18PM +0100, Patrick Bellasi wrote:
> The topic of a single simple power-performance tunable, that is wholly
> scheduler centric, and has well defined and predictable properties has
> come up on several occasions in the past. With techniques such as a
> scheduler driven DVFS, we now have a good framework for implementing
> such a tunable.
> 
> This patch provides a detailed description of the motivations and design
> decisions behind the implementation of the SchedTune.
> 
> cc: Jonathan Corbet <corbet@lwn.net>
> cc: linux-doc@vger.kernel.org
> Signed-off-by: Patrick Bellasi <patrick.bellasi@arm.com>
> 
> ---
> Documentation/scheduler/sched-tune.txt | 367 +++++++++++++++++++++++++++++++++
>  1 file changed, 367 insertions(+)
>  create mode 100644 Documentation/scheduler/sched-tune.txt
> 
> diff --git a/Documentation/scheduler/sched-tune.txt b/Documentation/scheduler/sched-tune.txt
> new file mode 100644
> index 0000000..cb795e6
> --- /dev/null
> +++ b/Documentation/scheduler/sched-tune.txt
> @@ -0,0 +1,367 @@
> +             Central, scheduler-driven, power-performance control
> +                               (EXPERIMENTAL)
> +
> +Abstract
> +========
> +
> +The topic of a single simple power-performance tunable, that is wholly
> +scheduler centric, and has well defined and predictable properties has come up
> +on several occasions in the past [1,2]. With techniques such as a scheduler
> +driven DVFS [3], we now have a good framework for implementing such a tunable.
> +This document describes the overall ideas behind its design and implementation.
> +
> +
> +Table of Contents
> +=================
> +
> +1. Motivation
> +2. Introduction
> +3. Signal Boosting Strategy
> +4. OPP selection using boosted CPU utilization
> +5. Per task group boosting
> +6. Question and Answers
> +   - What about "auto" mode?
> +   - What about boosting on a congested system?
> +   - How CPUs are boosted when we have tasks with multiple boost values?
> +7. References
> +
> +
> +1. Motivation
> +=============
> +
> +Sched-DVFS [3] is a new event-driven cpufreq governor which allows the
> +scheduler to select the optimal DVFS operating point (OPP) for running a task
> +allocated to a CPU. The introduction of sched-DVFS enables running workloads at
> +the most energy efficient OPPs.
> +
> +However, sometimes it may be desired to intentionally boost the performance of
> +a workload even if that could imply a reasonable increase in energy
> +consumption. For example, in order to reduce the response time of a task, we
> +may want to run the task at a higher OPP than the one that is actually required
> +by it's CPU bandwidth demand.
> +
> +This last requirement is especially important if we consider that one of the
> +main goals of the sched-DVFS component is to replace all currently available
> +CPUFreq policies. Since sched-DVFS is event based, as opposed to the sampling
> +driven governors we currently have, it is already more responsive at selecting
> +the optimal OPP to run tasks allocated to a CPU. However, just tracking the
> +actual task load demand may not be enough from a performance standpoint.  For
> +example, it is not possible to get behaviors similar to those provided by the
> +"performance" and "interactive" CPUFreq governors.
> +
> +This document describes an implementation of a tunable, stacked on top of the
> +sched-DVFS which extends its functionality to support task performance
> +boosting.
> +
> +By "performance boosting" we mean the reduction of the time required to
> +complete a task activation, i.e. the time elapsed from a task wakeup to its
> +next deactivation (e.g. because it goes back to sleep or it terminates).  For
> +example, if we consider a simple periodic task which executes the same workload
> +for 5[s] every 20[s] while running at a certain OPP, a boosted execution of
> +that task must complete each of its activations in less than 5[s].
> +
> +A previous attempt [5] to introduce such a boosting feature has not been
> +successful mainly because of the complexity of the proposed solution.  The
> +approach described in this document exposes a single simple interface to
> +user-space.  This single tunable knob allows the tuning of system wide
> +scheduler behaviours ranging from energy efficiency at one end through to
> +incremental performance boosting at the other end.  This first tunable affects
> +all tasks. However, a more advanced extension of the concept is also provided
> +which uses CGroups to boost the performance of only selected tasks while using
> +the energy efficient default for all others.
> +
> +The rest of this document introduces in more details the proposed solution
> +which has been named SchedTune.
> +
> +
> +2. Introduction
> +===============
> +
> +SchedTune exposes a simple user-space interface with a single power-performance
> +tunable:
> +
> +  /proc/sys/kernel/sched_cfs_boost
> +
> +This permits expressing a boost value as an integer in the range [0..100].
> +
> +A value of 0 (default) configures the CFS scheduler for maximum energy
> +efficiency. This means that sched-DVFS runs the tasks at the minimum OPP
> +required to satisfy their workload demand.
> +A value of 100 configures scheduler for maximum performance, which translates
> +to the selection of the maximum OPP on that CPU.
> +
> +The range between 0 and 100 can be set to satisfy other scenarios suitably. For
> +example to satisfy interactive response or depending on other system events
> +(battery level etc).
> +
> +A CGroup based extension is also provided, which permits further user-space
> +defined task classification to tune the scheduler for different goals depending
> +on the specific nature of the task, e.g. background vs interactive vs
> +low-priority.
> +
> +The overall design of the SchedTune module is built on top of "Per-Entity Load
> +Tracking" (PELT) signals and sched-DVFS by introducing a bias on the Operating
> +Performance Point (OPP) selection.
> +Each time a task is allocated on a CPU, sched-DVFS has the opportunity to tune
> +the operating frequency of that CPU to better match the workload demand. The
> +selection of the actual OPP being activated is influenced by the global boost
> +value, or the boost value for the task CGroup when in use.
> +
> +This simple biasing approach leverages existing frameworks, which means minimal
> +modifications to the scheduler, and yet it allows to achieve a range of
> +different behaviours all from a single simple tunable knob.
> +The only new concept introduced is that of signal boosting.
> +
> +
> +3. Signal Boosting Strategy
> +===========================
> +
> +The whole PELT machinery works based on the value of a few load tracking signals
> +which basically track the CPU bandwidth requirements for tasks and the capacity
> +of CPUs. The basic idea behind the SchedTune knob is to artificially inflate
> +some of these load tracking signals to make a task or RQ appears more demanding
> +that it actually is.
> +
> +Which signals have to be inflated depends on the specific "consumer".  However,
> +independently from the specific (signal, consumer) pair, it is important to
> +define a simple and possibly consistent strategy for the concept of boosting a
> +signal.
> +
> +A boosting strategy defines how the "abstract" user-space defined
> +sched_cfs_boost value is translated into an internal "margin" value to be added
> +to a signal to get its inflated value:
> +
> +  margin         := boosting_strategy(sched_cfs_boost, signal)
> +  boosted_signal := signal + margin
> +
> +Different boosting strategies were identified and analyzed before selecting the
> +one found to be most effective.
> +
> +Signal Proportional Compensation (SPC)
> +--------------------------------------
> +
> +In this boosting strategy the sched_cfs_boost value is used to compute a
> +margin which is proportional to the complement of the original signal.
> +When a signal has a maximum possible value, its complement is defined as
> +the delta from the actual value and its possible maximum.
> +
> +Since the tunable implementation uses signals which have SCHED_LOAD_SCALE as
> +the maximum possible value, the margin becomes:
> +
> +	margin := sched_cfs_boost * (SCHED_LOAD_SCALE - signal)
> +
> +Using this boosting strategy:
> +- a 100% sched_cfs_boost means that the signal is scaled to the maximum value
> +- each value in the range of sched_cfs_boost effectively inflates the signal in
> +  question by a quantity which is proportional to the maximum value.
> +
> +For example, by applying the SPC boosting strategy to the selection of the OPP
> +to run a task it is possible to achieve these behaviors:
> +
> +-   0% boosting: run the task at the minimum OPP required by its workload
> +- 100% boosting: run the task at the maximum OPP available for the CPU
> +-  50% boosting: run at the half-way OPP between minimum and maximum
> +
> +Which means that, at 50% boosting, a task will be scheduled to run at half of
> +the maximum theoretically achievable performance on the specific target
> +platform.
> +
> +A graphical representation of an SPC boosted signal is represented in the
> +following figure where:
> + a) "-" represents the original signal
> + b) "b" represents a  50% boosted signal
> + c) "p" represents a 100% boosted signal
> +
> +
> +   ^
> +   |  SCHED_LOAD_SCALE
> +   +-----------------------------------------------------------------+
> +   |pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp
> +   |
> +   |                                             boosted_signal
> +   |                                          bbbbbbbbbbbbbbbbbbbbbbbb
> +   |
> +   |                                            original signal
> +   |                  bbbbbbbbbbbbbbbbbbbbbbbb+----------------------+
> +   |                                          |
> +   |bbbbbbbbbbbbbbbbbb                        |
> +   |                                          |
> +   |                                          |
> +   |                                          |
> +   |                  +-----------------------+
> +   |                  |
> +   |                  |
> +   |                  |
> +   |------------------+
> +   |
> +   |
> +   +----------------------------------------------------------------------->
> +
> +The plot above shows a ramped load signal (titled 'original_signal') and it's
> +boosted equivalent. For each step of the original signal the boosted signal
> +corresponding to a 50% boost is midway from the original signal and the upper
> +bound. Boosting by 100% generates a boosted signal which is always saturated to
> +the upper bound.
> +
> +
> +4. OPP selection using boosted CPU utilization
> +==============================================
> +
> +It is worth calling out that the implementation does not introduce any new load
> +signals. Instead, it provides an API to tune existing signals. This tuning is
> +done on demand and only in scheduler code paths where it is sensible to do so.
> +The new API calls are defined to return either the default signal or a boosted
> +one, depending on the value of sched_cfs_boost. This is a clean an non invasive
> +modification of the existing existing code paths.
> +
> +The signal representing a CPU's utilization is boosted according to the
> +previously described SPC boosting strategy. To sched-DVFS, this allows a CPU
> +(ie CFS run-queue) to appear more used then it actually is.
> +
> +Thus, with the sched_cfs_boost enabled we have the following main functions to
> +get the current utilization of a CPU:
> +
> +  cpu_util()
> +  boosted_cpu_util()
> +
> +The new boosted_cpu_util() is similar to the first but returns a boosted
> +utilization signal which is a function of the sched_cfs_boost value.
> +
> +This function is used in the CFS scheduler code paths where sched-DVFS needs to
> +decide the OPP to run a CPU at.
> +For example, this allows selecting the highest OPP for a CPU which has
> +the boost value set to 100%.
> +
> +
> +5. Per task group boosting
> +==========================
> +
> +The availability of a single knob which is used to boost all tasks in the
> +system is certainly a simple solution but it quite likely doesn't fit many
> +utilization scenarios, especially in the mobile device space.
> +
> +For example, on battery powered devices there usually are many background
> +services which are long running and need energy efficient scheduling. On the
> +other hand, some applications are more performance sensitive and require an
> +interactive response and/or maximum performance, regardless of the energy cost.
> +To better service such scenarios, the SchedTune implementation has an extension
> +that provides a more fine grained boosting interface.
> +
> +A new CGroup controller, namely "schedtune", could be enabled which allows to
> +defined and configure task groups with different boosting values.
> +Tasks that require special performance can be put into separate CGroups.
> +The value of the boost associated with the tasks in this group can be specified
> +using a single knob exposed by the CGroup controller:
> +
> +   schedtune.boost
> +
> +This knob allows the definition of a boost value that is to be used for
> +SPC boosting of all tasks attached to this group.
> +
> +The current schedtune controller implementation is really simple and has these
> +main characteristics:
> +
> +  1) It is only possible to create 1 level depth hierarchies
> +
> +     The root control groups define the system-wide boost value to be applied
> +     by default to all tasks. Its direct subgroups are named "boost groups" and
> +     they define the boost value for specific set of tasks.
> +     Further nested subgroups are not allowed since they do not have a sensible
> +     meaning from a user-space standpoint.
> +
> +  2) It is possible to define only a limited number of "boost groups"
> +
> +     This number is defined at compile time and by default configured to 16.
> +     This is a design decision motivated by two main reasons:
> +     a) In a real system we do not expect utilization scenarios with more then few
> +	boost groups. For example, a reasonable collection of groups could be
> +        just "background", "interactive" and "performance".
> +     b) It simplifies the implementation considerably, especially for the code
> +	which has to compute the per CPU boosting once there are multiple
> +        RUNNABLE tasks with different boost values.
> +
> +Such a simple design should allow servicing the main utilization scenarios identified
> +so far. It provides a simple interface which can be used to manage the
> +power-performance of all tasks or only selected tasks.
> +Moreover, this interface can be easily integrated by user-space run-times (e.g.
> +Android, ChromeOS) to implement a QoS solution for task boosting based on tasks
> +classification, which has been a long standing requirement.
> +
> +Setup and usage
> +---------------
> +
> +0. Use a kernel with CGROUP_SCHEDTUNE support enabled
> +
> +1. Check that the "schedtune" CGroup controller is available:
> +
> +   root@linaro-nano:~# cat /proc/cgroups
> +   #subsys_name	hierarchy	num_cgroups	enabled
> +   cpuset  	0		1		1
> +   cpu     	0		1		1
> +   schedtune	0		1		1
> +
> +2. Mount a tmpfs to create the CGroups mount point (Optional)
> +
> +   root@linaro-nano:~# sudo mount -t tmpfs cgroups /sys/fs/cgroup
> +
> +3. Mount the "schedtune" controller
> +
> +   root@linaro-nano:~# mkdir /sys/fs/cgroup/stune
> +   root@linaro-nano:~# sudo mount -t cgroup -o schedtune stune /sys/fs/cgroup/stune
> +
> +4. Setup the system-wide boost value (Optional)
> +
> +   If not configured the root control group has a 0% boost value, which
> +   basically disables boosting for all tasks in the system thus running in
> +   an energy-efficient mode.
> +
> +   root@linaro-nano:~# echo $SYSBOOST > /sys/fs/cgroup/stune/schedtune.boost
> +
> +5. Create task groups and configure their specific boost value (Optional)
> +
> +   For example here we create a "performance" boost group configure to boost
> +   all its tasks to 100%
> +
> +   root@linaro-nano:~# mkdir /sys/fs/cgroup/stune/performance
> +   root@linaro-nano:~# echo 100 > /sys/fs/cgroup/stune/performance/schedtune.boost
> +
> +6. Move tasks into the boost group
> +
> +   For example, the following moves the tasks with PID $TASKPID (and all its
> +   threads) into the "performance" boost group.
> +
> +   root@linaro-nano:~# echo "TASKPID > /sys/fs/cgroup/stune/performance/cgroup.procs
> +
> +This simple configuration allows only the threads of the $TASKPID task to run,
> +when needed, at the highest OPP in the most capable CPU of the system.
> +
> +
> +6. Question and Answers
> +=======================
> +
> +What about "auto" mode?
> +-----------------------
> +
> +The 'auto' mode as described in [5] can be implemented by interfacing SchedTune
> +with some suitable user-space element. This element could use the exposed
> +system-wide or cgroup based interface.
> +
> +How are multiple groups of tasks with different boost values managed?
> +---------------------------------------------------------------------
> +
> +The current SchedTune implementation keeps track of the boosted RUNNABLE tasks
> +on a CPU. Once sched-DVFS selects the OPP to run a CPU at, the CPU utilization
> +is boosted with a value which is the maximum of the boost values of the
> +currently RUNNABLE tasks in its RQ.
> +
> +This allows sched-DVFS to boost a CPU only while there are boosted tasks ready
> +to run and switch back to the energy efficient mode as soon as the last boosted
> +task is dequeued.
> +
> +
> +7. References
> +=============
> +[1] http://lwn.net/Articles/552889
> +[2] http://lkml.org/lkml/2012/5/18/91
> +[3] http://lkml.org/lkml/2015/6/26/620
> +
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1218113 — Re: [RFC 08/14] sched/tune: add detailed documentation

On Wed, Sep 02, 2015 at 07:49:58AM +0100, Ricky Liang wrote:
> Hi Patrick,

Hi Ricky,

> I wonder if this can replace the boost function in the interactive
> governor [0], which is widely used in both Android and ChromeOS
> kernels.

In my view, one of the main goals of sched-DVFS is actually that to be
a solid and generic replacement of different CPUFreq governors.
Being driven by the scheduler, sched-DVFS can exploit information on
CPU demand of active tasks in order to select the optimal Operating
Performance Point (OPP) using a "proactive" approach instead of the
"reactive" approach commonly used by existing governors.

In the current implementation proposed by this RFC, SchedTune is just
a simple mechanism on top of sched-DVFS to bias the selection of the
OPP.
In case a task is running for a limited amount of time at each of its
(sporadic) activation, it does not contribute a CPU load which selects
an higher OPP. Thus, the actual performance (i.e. time to completion)
of that task depends on which other tasks are co-scheduled with it.
If it has the chance to be scheduled on a loaded CPU it will run fast,
to the contrary it will be slower when scheduled alone on a CPU
running at the lowest OPP.

If this task is (for whatever reason) "important" and should always
complete an activation as soon as possible, the current situation is:
a) we use the "performance" governor when we know that the task could
   be active, thus running the whole system in "race-to-idle" mode
b) we use the "interactive" governor (if possible, since it is not not
   in mainline) and ensure that this task pokes the "boost" attribute
   when it is active

Notice that, for both these solutions:
1) unless we pin the task on a specific set of CPUs, we must enable
   this governor for all the frequency domains since we do not know on
   which CPU the scheduler will end up to run the task
2) the tuning for a single task is likely to affect the whole system
   once the task as been started all the tasks are going to be boosted
   even when this task is not runnable

SchedTune provides a "global tunable" which allows to get the same
results as a) and b) with the main advantage that only the specific
frequency domain where the task is RUNNING is boosted. Since we do not
need to pin the task to get this result this can simplify (eventually)
the modification required in user-space while still getting optimal
performances for the task without compromising overall system
consumption.

AFAIU, regarding specifically the boost modes supported by the
Interactive governor:

1) the "boost" tunable is substantially similar to setting to 100% the
   SchedTune boost value. Userspace is in charge to trigger the start
   and end of a boost period

2) the "boostpulse" tunable triggers a 100% boost.

   The main difference is that the Interactive governor resets the
   boost after a configurable time (usually 80ms) while in SchedTune
   the boost value is asserted until release by userspace.

   This has advantages and disadvantages. By using SchedTune the
   userspace has to release the boost explicitly. With the Interactive
   governor this is automatic but still the userspace has to defined a
   suitable timeout. However, this can be different for different
   tasks.

3) the "boost_input" tunable is just an hook exposed to kernel drivers
   which can generate input events expected to impact on user the
   experience.
   The actual implementation is just similar to the previous knob.

IMHO the "boostpulse/input_pulse" tunables are a simple solution to
the problem of running fast to get better UI interactive response.
Indeed, the driver/task which generates the input event is not
necessary the actual target of the load and/or user perceived
response.
Moreover, it boosts all the frequency domains independently from where
the actual UI related workload is running.

By exploiting scheduler information on the actual workload demand of
some tasks, we could aim at a more effective solution which boost just
the required CPUs and only when the task affecting the UI experience
is actually running. This is what the "per-task" SchedTune boosting is
trying to enable.

I'm wondering if you could provide some example to better describe
when the "boostpulse" tunables are used in ChromiumOS.
Maybe that by starting from the description of some use-case we could
better understand if the tunables provided by the Interactive governor
are really required of if we can figure out a possible better even if
different approach to be implemented in SchedTune.

> My understanding is that the boost in interactive governor is to
> simply raise the OPP on selected cores.

AFAIU the "boost" of the Interactive governor affects all the (online)
CPUs. Thus if you have a multi frequency domain system (e.g.
big.LITTLE), the Interactive governor switch to performance mode for
all the CPUs. This makes sense since that boosting is triggered by an
event but does not exploit any information on which tasks really need
boosting and where they are executed by the scheduler.

> The SchedTune boost works by adding a margin to the original load of
> a task which makes the kernel think that the task is more demanding
> than it actually is. My intuition was that they work differently and
> could cause different reaction in the kernel.

That's absolutely true, they works differently. However it is worth to
notice that the SchedTune boost value is "consumed" just by
sched-DVFS, when it has to select an OPP.
There are not other links with the scheduler and/or signals "consumed"
by the scheduler. Specifically, all the task/RQ specific signals used
by the scheduler are not affected by the SchedTune value.

This is what happens in the SchedTune version presented by this
RFC. Internally we are working on an extension which integrates the
Energy-Aware scheduler (EAS).
In that case you are right, the boost value could affect some decision
of the EAS scheduler. For example, boosted tasks could end up being
moved into a more capable CPU of a big.LITTLE system even if they are
not generating a big utilization.

> I feel that the per-task cgroup ScheTune boost should work as
> expected as it only boosts a set of tasks and make them appear
> relatively high demanding comparing to other tasks. But if the
> ScheTune boost is applied globally to boost all the tasks in the
> system, will it cause unnecessary task migrations as all the tasks
> appear to be high demanding to the kernel?

IMHO the best usage of SchedTune is via "per-task" boosting, where it
is more easy to control when the system must work at higher OPPs.
However, this will probably require more efforts in the user-space
middleware layers to feed the scheduler with sensible information
about tasks demands.

Meanwhile, the current solutions are based on system-wide tuning, and
that's why SchedTune has been proposed with a support for "global"
boosting.

When we are boosting globally the only information we are providing to
the kernel is that we are in a rush and everything is important. Thus
yes, small tasks could eventually end up being moved into a more
capable CPU.

However, how SchedTune is going to bias tasks allocation is part of our
internal developments targeting its integration with EAS.

> Specifically, my questions is: When the global SchedTune boost is
> enabled in a on-demand manner, is it possible that a light task gets
> migrated to the big core, and in turn kicks out a heavy task
> originally on that core?

In this RFC we presented just the initial idea of task boosting with a
solution which is generic enough to possibly replace some of the most
commonly used CPUFreq governors (e.g. Performance, Ondemand and
Interactive) while still being completely unrelated from the scheduler
decisions on tasks allocation.

We think that the approach of posting small and self-contained updates
can be more effective on creating consensus by working together on
designing and building a solution which fits many different needs.

> I'm wondering whether global SchedTune boost could result in a
> "priority inversion" causing the heavy task to run on the little
> core and the light task to run on the big core.

That's an interesting point we should keep into consideration for the
design of the complete solution.
I would prefer to post-pone this discussion on the list once we will
present the next extension of SchedTune which integrates into EAS.


> [0]: https://android.googlesource.com/kernel/common.git/+/android-3.18/drivers/cpufreq/cpufreq_interactive.c
> 
> Thanks,
> Ricky

Cheers,
Patrick

-- 
#include <best/regards.h>

Patrick Bellasi

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1218721 — Re: [RFC 08/14] sched/tune: add detailed documentation

Hi Patrick,

Please find my replies inline.

On Thu, Sep 3, 2015 at 5:18 PM, Patrick Bellasi <patrick.bellasi@arm.com> wrote:
> On Wed, Sep 02, 2015 at 07:49:58AM +0100, Ricky Liang wrote:
>> Hi Patrick,
>
> Hi Ricky,
>
>> I wonder if this can replace the boost function in the interactive
>> governor [0], which is widely used in both Android and ChromeOS
>> kernels.
>
> In my view, one of the main goals of sched-DVFS is actually that to be
> a solid and generic replacement of different CPUFreq governors.
> Being driven by the scheduler, sched-DVFS can exploit information on
> CPU demand of active tasks in order to select the optimal Operating
> Performance Point (OPP) using a "proactive" approach instead of the
> "reactive" approach commonly used by existing governors.
>
> In the current implementation proposed by this RFC, SchedTune is just
> a simple mechanism on top of sched-DVFS to bias the selection of the
> OPP.
> In case a task is running for a limited amount of time at each of its
> (sporadic) activation, it does not contribute a CPU load which selects
> an higher OPP. Thus, the actual performance (i.e. time to completion)
> of that task depends on which other tasks are co-scheduled with it.
> If it has the chance to be scheduled on a loaded CPU it will run fast,
> to the contrary it will be slower when scheduled alone on a CPU
> running at the lowest OPP.
>
> If this task is (for whatever reason) "important" and should always
> complete an activation as soon as possible, the current situation is:
> a) we use the "performance" governor when we know that the task could
>    be active, thus running the whole system in "race-to-idle" mode
> b) we use the "interactive" governor (if possible, since it is not not
>    in mainline) and ensure that this task pokes the "boost" attribute
>    when it is active
>
> Notice that, for both these solutions:
> 1) unless we pin the task on a specific set of CPUs, we must enable
>    this governor for all the frequency domains since we do not know on
>    which CPU the scheduler will end up to run the task
> 2) the tuning for a single task is likely to affect the whole system
>    once the task as been started all the tasks are going to be boosted
>    even when this task is not runnable
>
> SchedTune provides a "global tunable" which allows to get the same
> results as a) and b) with the main advantage that only the specific
> frequency domain where the task is RUNNING is boosted. Since we do not
> need to pin the task to get this result this can simplify (eventually)
> the modification required in user-space while still getting optimal
> performances for the task without compromising overall system
> consumption.
>
> AFAIU, regarding specifically the boost modes supported by the
> Interactive governor:
>
> 1) the "boost" tunable is substantially similar to setting to 100% the
>    SchedTune boost value. Userspace is in charge to trigger the start
>    and end of a boost period
>
> 2) the "boostpulse" tunable triggers a 100% boost.
>
>    The main difference is that the Interactive governor resets the
>    boost after a configurable time (usually 80ms) while in SchedTune
>    the boost value is asserted until release by userspace.
>
>    This has advantages and disadvantages. By using SchedTune the
>    userspace has to release the boost explicitly. With the Interactive
>    governor this is automatic but still the userspace has to defined a
>    suitable timeout. However, this can be different for different
>    tasks.
>
> 3) the "boost_input" tunable is just an hook exposed to kernel drivers
>    which can generate input events expected to impact on user the
>    experience.
>    The actual implementation is just similar to the previous knob.
>
> IMHO the "boostpulse/input_pulse" tunables are a simple solution to
> the problem of running fast to get better UI interactive response.
> Indeed, the driver/task which generates the input event is not
> necessary the actual target of the load and/or user perceived
> response.
> Moreover, it boosts all the frequency domains independently from where
> the actual UI related workload is running.
>
> By exploiting scheduler information on the actual workload demand of
> some tasks, we could aim at a more effective solution which boost just
> the required CPUs and only when the task affecting the UI experience
> is actually running. This is what the "per-task" SchedTune boosting is
> trying to enable.
>
> I'm wondering if you could provide some example to better describe
> when the "boostpulse" tunables are used in ChromiumOS.
> Maybe that by starting from the description of some use-case we could
> better understand if the tunables provided by the Interactive governor
> are really required of if we can figure out a possible better even if
> different approach to be implemented in SchedTune.
>

In addition to the "boost" or "boost pulse" that are triggered by user space, in
ChromiumOS we register a input event handler in the interactive governor
which triggers interactive boost upon receiving any input events. The handler
causes the interactive governor to boost all CPUs and the boost lasts until the
CPUs go idle - in other words the boost lasts until there's no work for the
CPUs to do. Sometimes it's not trivial to tell which processes are crucial to
interactive response, so we are doing a global boost.

This is a use case specific to ChromiumOS, so it's probably not suitable to
be included in the mainline kernel. However, there are probably other similar
use cases out there so it's interesting to explore how SchedTune could
support this use case.

>> My understanding is that the boost in interactive governor is to
>> simply raise the OPP on selected cores.
>
> AFAIU the "boost" of the Interactive governor affects all the (online)
> CPUs. Thus if you have a multi frequency domain system (e.g.
> big.LITTLE), the Interactive governor switch to performance mode for
> all the CPUs. This makes sense since that boosting is triggered by an
> event but does not exploit any information on which tasks really need
> boosting and where they are executed by the scheduler.
>

You can also boost specific CPU in user space. The boost can be enabled
in a per-policy granularity. In any case you are right, the interactive governor
doesn't have context about tasks so SchedTune can be more effective.

>> The SchedTune boost works by adding a margin to the original load of
>> a task which makes the kernel think that the task is more demanding
>> than it actually is. My intuition was that they work differently and
>> could cause different reaction in the kernel.
>
> That's absolutely true, they works differently. However it is worth to
> notice that the SchedTune boost value is "consumed" just by
> sched-DVFS, when it has to select an OPP.
> There are not other links with the scheduler and/or signals "consumed"
> by the scheduler. Specifically, all the task/RQ specific signals used
> by the scheduler are not affected by the SchedTune value.
>
> This is what happens in the SchedTune version presented by this
> RFC. Internally we are working on an extension which integrates the
> Energy-Aware scheduler (EAS).
> In that case you are right, the boost value could affect some decision
> of the EAS scheduler. For example, boosted tasks could end up being
> moved into a more capable CPU of a big.LITTLE system even if they are
> not generating a big utilization.
>
>> I feel that the per-task cgroup ScheTune boost should work as
>> expected as it only boosts a set of tasks and make them appear
>> relatively high demanding comparing to other tasks. But if the
>> ScheTune boost is applied globally to boost all the tasks in the
>> system, will it cause unnecessary task migrations as all the tasks
>> appear to be high demanding to the kernel?
>
> IMHO the best usage of SchedTune is via "per-task" boosting, where it
> is more easy to control when the system must work at higher OPPs.
> However, this will probably require more efforts in the user-space
> middleware layers to feed the scheduler with sensible information
> about tasks demands.
>
> Meanwhile, the current solutions are based on system-wide tuning, and
> that's why SchedTune has been proposed with a support for "global"
> boosting.
>
> When we are boosting globally the only information we are providing to
> the kernel is that we are in a rush and everything is important. Thus
> yes, small tasks could eventually end up being moved into a more
> capable CPU.
>
> However, how SchedTune is going to bias tasks allocation is part of our
> internal developments targeting its integration with EAS.
>
>> Specifically, my questions is: When the global SchedTune boost is
>> enabled in a on-demand manner, is it possible that a light task gets
>> migrated to the big core, and in turn kicks out a heavy task
>> originally on that core?
>
> In this RFC we presented just the initial idea of task boosting with a
> solution which is generic enough to possibly replace some of the most
> commonly used CPUFreq governors (e.g. Performance, Ondemand and
> Interactive) while still being completely unrelated from the scheduler
> decisions on tasks allocation.
>
> We think that the approach of posting small and self-contained updates
> can be more effective on creating consensus by working together on
> designing and building a solution which fits many different needs.
>
>> I'm wondering whether global SchedTune boost could result in a
>> "priority inversion" causing the heavy task to run on the little
>> core and the light task to run on the big core.
>
> That's an interesting point we should keep into consideration for the
> design of the complete solution.
> I would prefer to post-pone this discussion on the list once we will
> present the next extension of SchedTune which integrates into EAS.
>
>
>> [0]: https://android.googlesource.com/kernel/common.git/+/android-3.18/drivers/cpufreq/cpufreq_interactive.c
>>
>> Thanks,
>> Ricky
>
> Cheers,
> Patrick
>
> --
> #include <best/regards.h>
>
> Patrick Bellasi
>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web