Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1271935 > unrolled thread

Re: [PATCH 3/4] sched: introduce synchronized idle injection

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH 3/4] sched: introduce synchronized idle injection Ingo Molnar <mingo@kernel.org> - 2015-11-18 09:40 +0100
    Re: [PATCH 3/4] sched: introduce synchronized idle injection Peter Zijlstra <peterz@infradead.org> - 2015-11-18 11:40 +0100
      Re: [PATCH 3/4] sched: introduce synchronized idle injection Morten Rasmussen <morten.rasmussen@arm.com> - 2015-11-18 13:30 +0100
        Re: [PATCH 3/4] sched: introduce synchronized idle injection Peter Zijlstra <peterz@infradead.org> - 2015-11-18 14:00 +0100
          Re: [PATCH 3/4] sched: introduce synchronized idle injection Morten Rasmussen <morten.rasmussen@arm.com> - 2015-11-18 15:10 +0100
            Re: [PATCH 3/4] sched: introduce synchronized idle injection Jacob Pan <jacob.jun.pan@linux.intel.com> - 2015-11-18 16:00 +0100
              Re: [PATCH 3/4] sched: introduce synchronized idle injection Morten Rasmussen <morten.rasmussen@arm.com> - 2015-11-18 16:10 +0100
                Re: [PATCH 3/4] sched: introduce synchronized idle injection Jacob Pan <jacob.jun.pan@linux.intel.com> - 2015-11-18 16:20 +0100
                  Re: [PATCH 3/4] sched: introduce synchronized idle injection Thomas Gleixner <tglx@linutronix.de> - 2015-11-18 16:30 +0100
                    Re: [PATCH 3/4] sched: introduce synchronized idle injection Jacob Pan <jacob.jun.pan@linux.intel.com> - 2015-11-18 20:00 +0100
                  Re: [PATCH 3/4] sched: introduce synchronized idle injection Morten Rasmussen <morten.rasmussen@arm.com> - 2015-11-18 17:10 +0100
      Re: [PATCH 3/4] sched: introduce synchronized idle injection Ingo Molnar <mingo@kernel.org> - 2015-11-27 10:20 +0100
    Re: [PATCH 3/4] sched: introduce synchronized idle injection Jacob Pan <jacob.jun.pan@linux.intel.com> - 2015-11-18 15:20 +0100
      Re: [PATCH 3/4] sched: introduce synchronized idle injection Ingo Molnar <mingo@kernel.org> - 2015-11-27 10:20 +0100
    Re: [PATCH 3/4] sched: introduce synchronized idle injection Arjan van de Ven <arjan@linux.intel.com> - 2015-11-18 15:30 +0100

#1271935 — Re: [PATCH 3/4] sched: introduce synchronized idle injection

* Jacob Pan <jacob.jun.pan@linux.intel.com> wrote:

> With increasingly constrained power and thermal budget, it's often necessary to 
> cap power via throttling. Throttling individual CPUs or devices at random times 
> can help power capping but may not be optimal in terms of energy efficiency. 
> Frequency scaling is also limited by certain range before losing energy 
> efficiency.
> 
> In general, the optimal solution in terms of energy efficiency is to align idle 
> periods such that more shared circuits can be power gated to enter lower power 
> states. Combined with energy efficient frequency point, idle injection provides 
> a way to scale power and performance efficiently.
> 
> This patch introduces a scheduler based idle injection method, it works by 
> blocking CFS runqueue synchronously and periodically. The actions on all online 
> CPUs are orchestrated by per CPU hrtimers.
> 
> Two sysctl knobs are given to the userspace for selecting the
> percentage of idle time as well as the forced idle duration for each
> idle period injected.

What's the purpose of these knobs? Just testing, or will some user-space daemon 
set them dynamically?

I.e. what mechanism will drive the throttling in the typical case?

> Since only CFS class is targeted, other high priority tasks are not affected, 
> such as EDF and RT tasks as well as softirq and interrupts.
> 
> Hotpath in CFS pick_next_task is optimized by Peter Zijlstra, where a new 
> runnable flag is introduced to combine forced idle and nr_running.

> +config CFS_IDLE_INJECT
> +	bool "Synchronized CFS idle injection"
> +	depends on NO_HZ_IDLE && HIGH_RES_TIMERS
> +	default n
> +	help
> +	  This feature let scheduler inject synchronized idle time across all online
> +	  CPUs. Idle injection affects normal tasks only, yeilds to RT and interrupts.
> +	  Effecitvely, CPUs can be duty cycled between running at the most power
> +	  efficient performance state and deep idle states.

So there are 3 typos in this single paragraph alone ...

I also think that naming it 'idle injection' is pretty euphemistic: this is forced 
idling, right? So why not name it CFS_FORCED_IDLE?

What will such throttling do to latencies, as observed by user-space tasks? What's 
the typical expected frequency of the throttling frequency that you are targeting?

Thanks,

	Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1272085

On Wed, Nov 18, 2015 at 09:36:22AM +0100, Ingo Molnar wrote:
> What will such throttling do to latencies, as observed by user-space tasks? What's 
> the typical expected frequency of the throttling frequency that you are targeting?

The default has 5ms (iirc) of forced idle, so depending on what you do,
noticeable to outright painful.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1272120

On Wed, Nov 18, 2015 at 11:35:41AM +0100, Peter Zijlstra wrote:
> On Wed, Nov 18, 2015 at 09:36:22AM +0100, Ingo Molnar wrote:
> > What will such throttling do to latencies, as observed by user-space tasks? What's 
> > the typical expected frequency of the throttling frequency that you are targeting?
> 
> The default has 5ms (iirc) of forced idle, so depending on what you do,
> noticeable to outright painful.

IIUC, it is 5 ticks, not ms.

Which raises the question, doesn't that mean that we get disturbed four
times on each cpu during the forced idle period? So idle injection only
makes sense if the platform has package states with a target residency
less than a jiffy. Or, do we enter NOHZ idle? I haven't looked closely
enough to figure out yet.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1272132

On Wed, Nov 18, 2015 at 12:27:04PM +0000, Morten Rasmussen wrote:
> On Wed, Nov 18, 2015 at 11:35:41AM +0100, Peter Zijlstra wrote:
> > On Wed, Nov 18, 2015 at 09:36:22AM +0100, Ingo Molnar wrote:
> > > What will such throttling do to latencies, as observed by user-space tasks? What's 
> > > the typical expected frequency of the throttling frequency that you are targeting?
> > 
> > The default has 5ms (iirc) of forced idle, so depending on what you do,
> > noticeable to outright painful.
> 
> IIUC, it is 5 ticks, not ms.

The code uses hrtimers (badly), this means there _should_ not be a tick
dependency.

> Which raises the question, doesn't that mean that we get disturbed four
> times on each cpu during the forced idle period? So idle injection only
> makes sense if the platform has package states with a target residency
> less than a jiffy. Or, do we enter NOHZ idle? I haven't looked closely
> enough to figure out yet.

The idea is to hit NOHZ.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1272193

On Wed, Nov 18, 2015 at 01:49:46PM +0100, Peter Zijlstra wrote:
> On Wed, Nov 18, 2015 at 12:27:04PM +0000, Morten Rasmussen wrote:
> > On Wed, Nov 18, 2015 at 11:35:41AM +0100, Peter Zijlstra wrote:
> > > On Wed, Nov 18, 2015 at 09:36:22AM +0100, Ingo Molnar wrote:
> > > > What will such throttling do to latencies, as observed by user-space tasks? What's 
> > > > the typical expected frequency of the throttling frequency that you are targeting?
> > > 
> > > The default has 5ms (iirc) of forced idle, so depending on what you do,
> > > noticeable to outright painful.
> > 
> > IIUC, it is 5 ticks, not ms.
> 
> The code uses hrtimers (badly), this means there _should_ not be a tick
> dependency.

Then I'm confused :-/

I see the hrtimers, but the actual idle duration appears to be in ticks
rather than ms and then converted later.

+/* Duration of idle time in ticks of each injection period */                                                                               
+unsigned int sysctl_sched_cfs_idle_inject_duration = 5UL;

...and futher down we have:

+       duration_msec = jiffies_to_msecs(sysctl_sched_cfs_idle_inject_duration);

I will go back and look harder.

> 
> > Which raises the question, doesn't that mean that we get disturbed four
> > times on each cpu during the forced idle period? So idle injection only
> > makes sense if the platform has package states with a target residency
> > less than a jiffy. Or, do we enter NOHZ idle? I haven't looked closely
> > enough to figure out yet.
> 
> The idea is to hit NOHZ.

Nice!
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1272267

On Wed, 18 Nov 2015 14:04:41 +0000
Morten Rasmussen <morten.rasmussen@arm.com> wrote:

> Then I'm confused :-/
> 
> I see the hrtimers, but the actual idle duration appears to be in
> ticks rather than ms and then converted later.
> 
> +/* Duration of idle time in ticks of each injection period
> */ +unsigned int sysctl_sched_cfs_idle_inject_duration = 5UL;
> 
> ...and futher down we have:
> 
> +       duration_msec =
> jiffies_to_msecs(sysctl_sched_cfs_idle_inject_duration);
I used hrtimers but still round around tick boundaries of ktime.
It is more reliable than jiffies in terms of getting timely updates.

Jacob
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1272279

On Wed, Nov 18, 2015 at 06:52:33AM -0800, Jacob Pan wrote:
> On Wed, 18 Nov 2015 14:04:41 +0000
> Morten Rasmussen <morten.rasmussen@arm.com> wrote:
> 
> > Then I'm confused :-/
> > 
> > I see the hrtimers, but the actual idle duration appears to be in
> > ticks rather than ms and then converted later.
> > 
> > +/* Duration of idle time in ticks of each injection period
> > */ +unsigned int sysctl_sched_cfs_idle_inject_duration = 5UL;
> > 
> > ...and futher down we have:
> > 
> > +       duration_msec =
> > jiffies_to_msecs(sysctl_sched_cfs_idle_inject_duration);
> I used hrtimers but still round around tick boundaries of ktime.
> It is more reliable than jiffies in terms of getting timely updates.

Okay. But it does mean that the defeault idle period is 5 ticks (50ms @
HZ=100) and not 5 ms.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1272289

On Wed, 18 Nov 2015 15:09:44 +0000
Morten Rasmussen <morten.rasmussen@arm.com> wrote:

> Okay. But it does mean that the defeault idle period is 5 ticks (50ms
> @ HZ=100) and not 5 ms
correct. my reason is to scale with various HZ values.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1272293

On Wed, 18 Nov 2015, Jacob Pan wrote:
> On Wed, 18 Nov 2015 15:09:44 +0000
> Morten Rasmussen <morten.rasmussen@arm.com> wrote:
> 
> > Okay. But it does mean that the defeault idle period is 5 ticks (50ms
> > @ HZ=100) and not 5 ms
> correct. my reason is to scale with various HZ values.

So for smaller HZ values we get longer disruption. That's well thought
out scaling.

Thanks,

	tglx

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1272473

On Wed, 18 Nov 2015 16:21:27 +0100 (CET)
Thomas Gleixner <tglx@linutronix.de> wrote:

> >   
> > > Okay. But it does mean that the defeault idle period is 5 ticks
> > > (50ms @ HZ=100) and not 5 ms  
> > correct. my reason is to scale with various HZ values.  
> 
> So for smaller HZ values we get longer disruption. That's well thought
> out scaling.
well it might be too long for embedded systems who uses 100HZ. Is there
a better way to scale in sub tick level?

My original thought was for smaller HZ value, I assume they care less
about latency, so the idle injection period is proportional to what
they set for HZ.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1272330

On Wed, Nov 18, 2015 at 07:11:31AM -0800, Jacob Pan wrote:
> On Wed, 18 Nov 2015 15:09:44 +0000
> Morten Rasmussen <morten.rasmussen@arm.com> wrote:
> 
> > Okay. But it does mean that the defeault idle period is 5 ticks (50ms
> > @ HZ=100) and not 5 ms
> correct. my reason is to scale with various HZ values.

50ms seems quite long for embedded applications. OTOH, it has to be long
enough to enter a package idle state and stay there for a while. You
would surely notice it when it kicks in on an Android device.

Do you intend to use it only as a last option? One might want to keep
using inefficient frequencies (P-states) if those don't cause UI lag
then.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1278663

* Peter Zijlstra <peterz@infradead.org> wrote:

> On Wed, Nov 18, 2015 at 09:36:22AM +0100, Ingo Molnar wrote:
> > What will such throttling do to latencies, as observed by user-space tasks? What's 
> > the typical expected frequency of the throttling frequency that you are targeting?
> 
> The default has 5ms (iirc) of forced idle, so depending on what you do,
> noticeable to outright painful.

That's absolutely painful!

Thanks,

	Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1272207

On Wed, 18 Nov 2015 09:36:22 +0100
Ingo Molnar <mingo@kernel.org> wrote:

> > 
> > This patch introduces a scheduler based idle injection method, it
> > works by blocking CFS runqueue synchronously and periodically. The
> > actions on all online CPUs are orchestrated by per CPU hrtimers.
> > 
> > Two sysctl knobs are given to the userspace for selecting the
> > percentage of idle time as well as the forced idle duration for each
> > idle period injected.  
> 
> What's the purpose of these knobs? Just testing, or will some
> user-space daemon set them dynamically?
> 
yes, it is to be used by userspace daemon such as thermal daemon.
Though there are interests from in kernel thermal governor but that is
another story.

> I also think that naming it 'idle injection' is pretty euphemistic:
> this is forced idling, right? So why not name it CFS_FORCED_IDLE?
yes, it is forced idle. sounds good.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1278664

* Jacob Pan <jacob.jun.pan@linux.intel.com> wrote:

> On Wed, 18 Nov 2015 09:36:22 +0100
> Ingo Molnar <mingo@kernel.org> wrote:
> 
> > > 
> > > This patch introduces a scheduler based idle injection method, it
> > > works by blocking CFS runqueue synchronously and periodically. The
> > > actions on all online CPUs are orchestrated by per CPU hrtimers.
> > > 
> > > Two sysctl knobs are given to the userspace for selecting the
> > > percentage of idle time as well as the forced idle duration for each
> > > idle period injected.  
> > 
> > What's the purpose of these knobs? Just testing, or will some
> > user-space daemon set them dynamically?
> > 
> yes, it is to be used by userspace daemon such as thermal daemon.
> Though there are interests from in kernel thermal governor but that is
> another story.

Yeah, so let me make this very clear: for a kernel scheduling feature to be 
self-sufficient is not 'another story', but a must-have aspect for this feature
to become upstream acceptable.

We don't add scheduler features that rely on pushing 'policy' to user-space. 
That's poor design with many disadvantages. This feature should offer a reasonable 
and automatic in-kernel default behavior with numbers that prove that it works.

Keeping an essential part of the feature in user-space earns a NAK from me.

Thanks,

	Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1272216

On 11/18/2015 12:36 AM, Ingo Molnar wrote:
>
> What will such throttling do to latencies, as observed by user-space tasks? What's
> the typical expected frequency of the throttling frequency that you are targeting?

for this to meaningfully reduce power consumption, deep system power states need to be reached,
and for those to pay of, several milliseconds of idle are required.

for hard realtime stuff that is obviously insane, but I would assume that for those cases
your system is thermally fine. This only kicks in at the end of a "we're in thermal
problems" path, which can happen both on clients (thin devices) as well as servers
(airconditioning issues). The objective for this is to kick in before the
hardware built-in protections kick in (which are power off/reboot depending on a bunch of things).
The frequency of how often these 5 msec get injected depend on how deep the system
is in trouble; and is zero if the system is not in trouble.

The idea is that for the user it is better to inject several 5 msec intervals than it is
to inject one longer period.


You can compare this method to other ways of reducing thermal issues (like lowering cpu frequency),
and in a typical setup, this is done after the benign of those methods are exhausted.
Lowering frequency even lower is usually of a low efficiency (you need to lower the frequency a LOT
to gain a little bit of power in the bottom parts of the frequency ranges), while this idle will
not only put the CPU in low power, but will also put the system memory in low power and usually
a big chunk of the rest of the SOC. In many client systems, memory power consumption is higher than CPU
power consumption (and in big servers, it's also quite sizable), so there is a pretty hard
limit of how much you can do on thermals if you're not also kicking some of the memory power savings.
This means that to achieve a certain amount of reduction, the performance is impacted a lot less than
the more drastic methods you would need on the cpu side, if possible at all.
(stepping your 2 Ghz cpu down to 50Mhz may sound less evil than injecting 5msec of idle time, but in reality
that is impacting user tasks a heck of a lot more than 5msec of not being scheduled)


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web