Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1683892 > unrolled thread
| Started by | Aubrey Li <aubrey.li@intel.com> |
|---|---|
| First post | 2017-07-10 03:50 +0200 |
| Last post | 2017-07-11 11:10 +0200 |
| Articles | 20 on this page of 109 — 11 participants |
Back to article view | Back to linux.kernel
[RFC PATCH v1 00/11] Create fast idle path for short idle periods Aubrey Li <aubrey.li@intel.com> - 2017-07-10 03:50 +0200
[RFC PATCH v1 04/11] sched/idle: make the fast idle path for short idle periods Aubrey Li <aubrey.li@intel.com> - 2017-07-10 03:50 +0200
Re: [RFC PATCH v1 04/11] sched/idle: make the fast idle path for short idle periods "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-07-11 15:00 +0200
Re: [RFC PATCH v1 04/11] sched/idle: make the fast idle path for short idle periods Frederic Weisbecker <fweisbec@gmail.com> - 2017-07-11 18:40 +0200
Re: [RFC PATCH v1 04/11] sched/idle: make the fast idle path for short idle periods "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-07-11 20:20 +0200
Re: [RFC PATCH v1 04/11] sched/idle: make the fast idle path for short idle periods "Li, Aubrey" <aubrey.li@linux.intel.com> - 2017-07-12 05:30 +0200
Re: [RFC PATCH v1 04/11] sched/idle: make the fast idle path for short idle periods "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-07-12 07:10 +0200
Re: [RFC PATCH v1 04/11] sched/idle: make the fast idle path for short idle periods "Li, Aubrey" <aubrey.li@linux.intel.com> - 2017-07-12 07:30 +0200
Re: [RFC PATCH v1 04/11] sched/idle: make the fast idle path for short idle periods Peter Zijlstra <peterz@infradead.org> - 2017-07-12 14:30 +0200
[RFC PATCH v1 05/11] cpuidle: update idle statistics before cpuidle governor Aubrey Li <aubrey.li@intel.com> - 2017-07-10 03:50 +0200
[RFC PATCH v1 08/11] cpuidle: menu: remove reduplicative implementation Aubrey Li <aubrey.li@intel.com> - 2017-07-10 04:00 +0200
[RFC PATCH v1 07/11] cpuidle: make idle residency update more generic Aubrey Li <aubrey.li@intel.com> - 2017-07-10 04:00 +0200
[RFC PATCH v1 03/11] cpuidle: introduce cpuidle governor for idle prediction Aubrey Li <aubrey.li@intel.com> - 2017-07-10 04:00 +0200
Re: [RFC PATCH v1 03/11] cpuidle: introduce cpuidle governor for idle prediction Peter Zijlstra <peterz@infradead.org> - 2017-07-12 14:20 +0200
[RFC PATCH v1 09/11] cpuidle: menu: feed cpuidle prediction to menu governor Aubrey Li <aubrey.li@intel.com> - 2017-07-10 04:00 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Peter Zijlstra <peterz@infradead.org> - 2017-07-10 10:50 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Wanpeng Li <kernellwp@gmail.com> - 2017-07-10 11:40 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods "Li, Aubrey" <aubrey.li@linux.intel.com> - 2017-07-10 16:00 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Andi Kleen <ak@linux.intel.com> - 2017-07-10 16:50 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Peter Zijlstra <peterz@infradead.org> - 2017-07-10 18:50 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Andi Kleen <ak@linux.intel.com> - 2017-07-10 19:30 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods "Li, Aubrey" <aubrey.li@linux.intel.com> - 2017-07-11 06:50 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Peter Zijlstra <peterz@infradead.org> - 2017-07-11 11:50 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Frederic Weisbecker <fweisbec@gmail.com> - 2017-07-11 18:10 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Peter Zijlstra <peterz@infradead.org> - 2017-07-11 18:40 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-07-11 20:10 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Peter Zijlstra <peterz@infradead.org> - 2017-07-12 14:00 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-07-12 18:00 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Peter Zijlstra <peterz@infradead.org> - 2017-07-12 19:50 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-07-12 21:00 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-07-12 21:10 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Frederic Weisbecker <fweisbec@gmail.com> - 2017-07-19 15:50 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-07-19 17:00 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Peter Zijlstra <peterz@infradead.org> - 2017-07-12 14:30 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-07-12 18:00 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Peter Zijlstra <peterz@infradead.org> - 2017-07-12 19:20 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Peter Zijlstra <peterz@infradead.org> - 2017-07-12 20:00 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-07-12 21:00 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-07-12 20:50 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Peter Zijlstra <peterz@infradead.org> - 2017-07-13 10:40 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods "Li, Aubrey" <aubrey.li@linux.intel.com> - 2017-07-12 06:20 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Peter Zijlstra <peterz@infradead.org> - 2017-07-12 10:40 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Andi Kleen <ak@linux.intel.com> - 2017-07-12 23:40 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Peter Zijlstra <peterz@infradead.org> - 2017-07-13 10:40 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods "Li, Aubrey" <aubrey.li@linux.intel.com> - 2017-07-13 16:50 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Peter Zijlstra <peterz@infradead.org> - 2017-07-13 17:00 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods "Li, Aubrey" <aubrey.li@linux.intel.com> - 2017-07-13 17:20 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Peter Zijlstra <peterz@infradead.org> - 2017-07-13 20:30 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods "Li, Aubrey" <aubrey.li@linux.intel.com> - 2017-07-14 06:00 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Peter Zijlstra <peterz@infradead.org> - 2017-07-14 17:40 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Peter Zijlstra <peterz@infradead.org> - 2017-07-14 18:00 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Andi Kleen <ak@linux.intel.com> - 2017-07-14 18:10 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Peter Zijlstra <peterz@infradead.org> - 2017-07-17 11:30 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods "Li, Aubrey" <aubrey.li@linux.intel.com> - 2017-07-17 15:50 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Andi Kleen <ak@linux.intel.com> - 2017-07-14 18:00 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Peter Zijlstra <peterz@infradead.org> - 2017-07-14 18:10 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Andi Kleen <ak@linux.intel.com> - 2017-07-14 18:30 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Peter Zijlstra <peterz@infradead.org> - 2017-07-17 21:30 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Arjan van de Ven <arjan@linux.intel.com> - 2017-07-17 21:30 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Thomas Gleixner <tglx@linutronix.de> - 2017-07-17 21:50 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Arjan van de Ven <arjan@linux.intel.com> - 2017-07-17 22:00 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Thomas Gleixner <tglx@linutronix.de> - 2017-07-17 22:10 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Arjan van de Ven <arjan@linux.intel.com> - 2017-07-17 21:50 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Thomas Gleixner <tglx@linutronix.de> - 2017-07-17 22:00 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Arjan van de Ven <arjan@linux.intel.com> - 2017-07-17 22:00 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods "Li, Aubrey" <aubrey.li@linux.intel.com> - 2017-07-18 05:30 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Peter Zijlstra <peterz@infradead.org> - 2017-07-18 21:00 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods "Li, Aubrey" <aubrey.li@linux.intel.com> - 2017-07-18 05:20 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Andi Kleen <ak@linux.intel.com> - 2017-07-18 06:50 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Thomas Gleixner <tglx@linutronix.de> - 2017-07-18 08:50 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Andi Kleen <ak@linux.intel.com> - 2017-07-18 09:00 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Thomas Gleixner <tglx@linutronix.de> - 2017-07-18 09:20 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods "Li, Aubrey" <aubrey.li@linux.intel.com> - 2017-07-19 08:20 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Thomas Gleixner <tglx@linutronix.de> - 2017-07-19 10:00 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods "Li, Aubrey" <aubrey.li@linux.intel.com> - 2017-07-20 04:00 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Thomas Gleixner <tglx@linutronix.de> - 2017-07-20 10:20 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Arjan van de Ven <arjan@linux.intel.com> - 2017-07-20 15:50 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Thomas Gleixner <tglx@linutronix.de> - 2017-07-18 09:30 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods "Li, Aubrey" <aubrey.li@linux.intel.com> - 2017-07-18 09:00 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-07-18 17:30 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Arjan van de Ven <arjan@linux.intel.com> - 2017-07-18 17:30 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Arjan van de Ven <arjan@linux.intel.com> - 2017-07-18 18:40 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Peter Zijlstra <peterz@infradead.org> - 2017-07-18 19:10 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Peter Zijlstra <peterz@infradead.org> - 2017-07-18 18:40 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods "Li, Aubrey" <aubrey.li@linux.intel.com> - 2017-07-19 07:50 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-07-19 16:50 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Christopher Lameter <cl@linux.com> - 2017-07-19 17:10 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-07-19 19:00 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods "Li, Aubrey" <aubrey.li@linux.intel.com> - 2017-07-20 03:50 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-07-20 15:00 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Arjan van de Ven <arjan@linux.intel.com> - 2017-07-20 15:50 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Peter Zijlstra <peterz@infradead.org> - 2017-07-20 16:30 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-07-20 18:10 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Peter Zijlstra <peterz@infradead.org> - 2017-07-18 18:50 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Peter Zijlstra <peterz@infradead.org> - 2017-07-18 15:30 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Peter Zijlstra <peterz@infradead.org> - 2017-07-19 15:50 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Arjan van de Ven <arjan@linux.intel.com> - 2017-07-14 18:00 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-07-13 17:30 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods "Li, Aubrey" <aubrey.li@linux.intel.com> - 2017-07-14 05:50 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-07-14 06:10 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods "Li, Aubrey" <aubrey.li@linux.intel.com> - 2017-07-17 15:30 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Peter Zijlstra <peterz@infradead.org> - 2017-07-17 16:00 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods "Li, Aubrey" <aubrey.li@linux.intel.com> - 2017-07-17 16:10 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Peter Zijlstra <peterz@infradead.org> - 2017-07-12 14:20 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Christoph Lameter <cl@linux.com> - 2017-07-11 20:00 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods "Li, Aubrey" <aubrey.li@linux.intel.com> - 2017-07-12 04:10 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods "Li, Aubrey" <aubrey.li@linux.intel.com> - 2017-07-12 04:40 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Peter Zijlstra <peterz@infradead.org> - 2017-07-12 20:20 +0200
Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Peter Zijlstra <peterz@infradead.org> - 2017-07-11 11:10 +0200
Page 5 of 6 — ← Prev page 1 2 3 4 [5] 6 Next page →
| From | Arjan van de Ven <arjan@linux.intel.com> |
|---|---|
| Date | 2017-07-18 17:30 +0200 |
| Message-ID | <u4CBH-45e-13@gated-at.bofh.it> |
| In reply to | #1690369 |
On 7/18/2017 8:20 AM, Paul E. McKenney wrote: > 3.2) how to determine if the idle is short or long. My current proposal is to > use a tunable value via /sys, while Peter prefers an auto-adjust mechanism. I > didn't get the details of an auto-adjust mechanism yet the most obvious way to do this (for me, maybe I'm naive) is to add another C state, lets call it "C1-lite" with its own thresholds and power levels etc, and just let that be picked naturally based on the heuristics. (if we want to improve the heuristics, that's fine and always welcome but that is completely orthogonal in my mind) this C1-lite would then skip some of the idle steps like the nohz logic. How we plumb that ... might end up being a flag or whatever, we'll figure that out easily. as long as "real C1" has a break even time that is appropriate compared to C1-lite, we'll only pick C1-lite for very very short idles like is desired... but we don't end up creating a parallel infra for picking states, that part just does not make sense to me tbh.... I have yet to see any reason why C1-lite couldn't be just another C-state for everything except the actual place where we do the "go idle" last bit of logic. (Also note that for extreme short idles, today we just spinloop (C0), so by this argument we should also do a C0-lite.. or make this C0 always the lite variant)
[toc] | [prev] | [next] | [standalone]
| From | Arjan van de Ven <arjan@linux.intel.com> |
|---|---|
| Date | 2017-07-18 18:40 +0200 |
| Message-ID | <u4DHr-4J6-1@gated-at.bofh.it> |
| In reply to | #1690371 |
On 7/18/2017 9:36 AM, Peter Zijlstra wrote: > On Tue, Jul 18, 2017 at 08:29:40AM -0700, Arjan van de Ven wrote: >> >> the most obvious way to do this (for me, maybe I'm naive) is to add another >> C state, lets call it "C1-lite" with its own thresholds and power levels etc, >> and just let that be picked naturally based on the heuristics. >> (if we want to improve the heuristics, that's fine and always welcome but that >> is completely orthogonal in my mind) > > C1-lite would then have a threshold < C1, whereas I understood the > desire to be for the fast-idle crud to have a larger threshold than C1 > currently has. > > That is, from what I understood, they want C1 selected *longer*. that's just a matter of fixing the C1 and later thresholds to line up right. shrug that's the most trivial thing to do, it's a number in a table. some distros do those tunings anyway when they don't like the upstream tunings
[toc] | [prev] | [next] | [standalone]
| From | Peter Zijlstra <peterz@infradead.org> |
|---|---|
| Date | 2017-07-18 19:10 +0200 |
| Message-ID | <u4Eav-5aT-43@gated-at.bofh.it> |
| In reply to | #1690430 |
On Tue, Jul 18, 2017 at 09:37:57AM -0700, Arjan van de Ven wrote: > that's just a matter of fixing the C1 and later thresholds to line up right. > shrug that's the most trivial thing to do, it's a number in a table. Well, they represent a physical measure, namely the break-even-time. If you go muck with them you don't have anything left. This is tinkering of the worst possible kind. Fix the estimator if you want behavioural changes. > some distros do those tunings anyway when they don't like the upstream tunings *shudder*
[toc] | [prev] | [next] | [standalone]
| From | Peter Zijlstra <peterz@infradead.org> |
|---|---|
| Date | 2017-07-18 18:40 +0200 |
| Message-ID | <u4DHs-4J6-3@gated-at.bofh.it> |
| In reply to | #1690371 |
On Tue, Jul 18, 2017 at 08:29:40AM -0700, Arjan van de Ven wrote: > > the most obvious way to do this (for me, maybe I'm naive) is to add another > C state, lets call it "C1-lite" with its own thresholds and power levels etc, > and just let that be picked naturally based on the heuristics. > (if we want to improve the heuristics, that's fine and always welcome but that > is completely orthogonal in my mind) C1-lite would then have a threshold < C1, whereas I understood the desire to be for the fast-idle crud to have a larger threshold than C1 currently has. That is, from what I understood, they want C1 selected *longer*.
[toc] | [prev] | [next] | [standalone]
| From | "Li, Aubrey" <aubrey.li@linux.intel.com> |
|---|---|
| Date | 2017-07-19 07:50 +0200 |
| Message-ID | <u4Q1X-49w-3@gated-at.bofh.it> |
| In reply to | #1690369 |
On 2017/7/18 23:20, Paul E. McKenney wrote: >> 2) for rcu idle enter/exit, I measured the details which Paul provided, and >> the result matches with what I have measured before, nothing notable found. >> But it still makes more sense if we can make rcu idle enter/exit hooked with >> tick off. (it's possible other workloads behave differently) > > Again, assuming that RCU is informed of CPUs in the kernel, regardless > of whether or not the tick is on that that point in time. > Yeah, I see, no problem for a normal idle. But for a short idle, we want to return to the task ASAP. Even though RCU cost is not notable, it would still be better for me if we can save some cycles in idle entry and idle exit. Do we have any problem if we skip RCU idle enter/exit under a fast idle scenario? My understanding is, if tick is not stopped, then we don't need inform RCU in idle path, it can be informed in irq exit. Thanks, -Aubrey
[toc] | [prev] | [next] | [standalone]
| From | "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> |
|---|---|
| Date | 2017-07-19 16:50 +0200 |
| Message-ID | <u4Ysy-1t8-9@gated-at.bofh.it> |
| In reply to | #1690985 |
On Wed, Jul 19, 2017 at 01:44:06PM +0800, Li, Aubrey wrote: > On 2017/7/18 23:20, Paul E. McKenney wrote: > > >> 2) for rcu idle enter/exit, I measured the details which Paul provided, and > >> the result matches with what I have measured before, nothing notable found. > >> But it still makes more sense if we can make rcu idle enter/exit hooked with > >> tick off. (it's possible other workloads behave differently) > > > > Again, assuming that RCU is informed of CPUs in the kernel, regardless > > of whether or not the tick is on that that point in time. > > > Yeah, I see, no problem for a normal idle. > > But for a short idle, we want to return to the task ASAP. Even though RCU cost > is not notable, it would still be better for me if we can save some cycles in > idle entry and idle exit. > > Do we have any problem if we skip RCU idle enter/exit under a fast idle scenario? > My understanding is, if tick is not stopped, then we don't need inform RCU in > idle path, it can be informed in irq exit. Indeed, the problem arises when the tick is stopped. Thanx, Paul
[toc] | [prev] | [next] | [standalone]
| From | Christopher Lameter <cl@linux.com> |
|---|---|
| Date | 2017-07-19 17:10 +0200 |
| Subject | Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods |
| Message-ID | <u4YLU-1Q5-9@gated-at.bofh.it> |
| In reply to | #1691802 |
On Wed, 19 Jul 2017, Paul E. McKenney wrote: > > Do we have any problem if we skip RCU idle enter/exit under a fast idle scenario? > > My understanding is, if tick is not stopped, then we don't need inform RCU in > > idle path, it can be informed in irq exit. > > Indeed, the problem arises when the tick is stopped. Well is there a boundary when you would want the notification calls? I would think that even an idle period of a couple of seconds does not necessarily require a callback to rcu. Had some brokenness here where RCU calls did not occur for hours or so. At some point the system ran out of memory but thats far off.
[toc] | [prev] | [next] | [standalone]
| From | "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> |
|---|---|
| Date | 2017-07-19 19:00 +0200 |
| Message-ID | <u50um-2TW-17@gated-at.bofh.it> |
| In reply to | #1691826 |
On Wed, Jul 19, 2017 at 10:03:22AM -0500, Christopher Lameter wrote:
> On Wed, 19 Jul 2017, Paul E. McKenney wrote:
>
> > > Do we have any problem if we skip RCU idle enter/exit under a fast idle scenario?
> > > My understanding is, if tick is not stopped, then we don't need inform RCU in
> > > idle path, it can be informed in irq exit.
> >
> > Indeed, the problem arises when the tick is stopped.
>
> Well is there a boundary when you would want the notification calls? I
> would think that even an idle period of a couple of seconds does not
> necessarily require a callback to rcu. Had some brokenness here where RCU
> calls did not occur for hours or so. At some point the system ran out of
> memory but thats far off.
Yeah, I should spell this out more completely, shouldn't I? And get
it into the documentation if it isn't already there... Where it is
currently at best hinted at. :-/
1. If a CPU is either idle or executing in usermode, and RCU believes
it is non-idle, the scheduling-clock tick had better be running.
Otherwise, you will get RCU CPU stall warnings. Or at best,
very long (11-second) grace periods, with a pointless IPI waking
the CPU each time.
2. If a CPU is in a portion of the kernel that executes RCU read-side
critical sections, and RCU believes this CPU to be idle, you can get
random memory corruption. DON'T DO THIS!!!
This is one reason to test with lockdep, which will complain
about this sort of thing.
3. If a CPU is in a portion of the kernel that is absolutely
positively no-joking guaranteed to never execute any RCU read-side
critical sections, and RCU believes this CPU to to be idle,
no problem. This sort of thing is used by some architectures
for light-weight exception handlers, which can then avoid the
overhead of rcu_irq_enter() and rcu_irq_exit(). Some go further
and avoid the entireties of irq_enter() and irq_exit().
Just make very sure you are running some of your tests with
CONFIG_PROVE_RCU=y.
4. If a CPU is executing in the kernel with the scheduling-clock
interrupt disabled and RCU believes this CPU to be non-idle,
and if the CPU goes idle (from an RCU perspective) every few
jiffies, no problem. It is usually OK for there to be the
occasional gap between idle periods of up to a second or so.
If the gap grows too long, you get RCU CPU stall warnings.
5. If a CPU is either idle or executing in usermode, and RCU believes
it to be idle, of course no problem.
6. If a CPU is executing in the kernel, the kernel code
path is passing through quiescent states at a reasonable
frequency (preferably about once per few jiffies, but the
occasional excursion to a second or so is usually OK) and the
scheduling-clock interrupt is enabled, of course no problem.
If the gap between a successive pair of quiescent states grows
too long, you get RCU CPU stall warnings.
For purposes of comparison, the default time until RCU CPU stall warning
in mainline is 21 seconds. A number of distros set this to 60 seconds.
Back in the 90s, the analogous timeout for DYNIX/ptx was 1.5 seconds. :-/
Hence "a second or so" instead of your "a couple of seconds". ;-)
Please see below for the corresponding patch to RCU's requirements.
Thoughts?
Thanx, Paul
------------------------------------------------------------------------
commit 693c9bfa43f92570dd362d8834440b418bbb994a
Author: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Date: Wed Jul 19 09:52:58 2017 -0700
doc: Set down RCU's scheduling-clock-interrupt needs
This commit documents the situations in which RCU needs the
scheduling-clock interrupt to be enabled, along with the consequences
of failing to meet RCU's needs in this area.
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
diff --git a/Documentation/RCU/Design/Requirements/Requirements.html b/Documentation/RCU/Design/Requirements/Requirements.html
index 95b30fa25d56..7980bee5607f 100644
--- a/Documentation/RCU/Design/Requirements/Requirements.html
+++ b/Documentation/RCU/Design/Requirements/Requirements.html
@@ -2080,6 +2080,8 @@ Some of the relevant points of interest are as follows:
<li> <a href="#Scheduler and RCU">Scheduler and RCU</a>.
<li> <a href="#Tracing and RCU">Tracing and RCU</a>.
<li> <a href="#Energy Efficiency">Energy Efficiency</a>.
+<li> <a href="#Scheduling-Clock Interrupts and RCU">
+ Scheduling-Clock Interrupts and RCU</a>.
<li> <a href="#Memory Efficiency">Memory Efficiency</a>.
<li> <a href="#Performance, Scalability, Response Time, and Reliability">
Performance, Scalability, Response Time, and Reliability</a>.
@@ -2532,6 +2534,113 @@ I learned of many of these requirements via angry phone calls:
Flaming me on the Linux-kernel mailing list was apparently not
sufficient to fully vent their ire at RCU's energy-efficiency bugs!
+<h3><a name="Scheduling-Clock Interrupts and RCU">
+Scheduling-Clock Interrupts and RCU</a></h3>
+
+<p>
+The kernel transitions between in-kernel non-idle execution, userspace
+execution, and the idle loop.
+Depending on kernel configuration, RCU handles these states differently:
+
+<table border=3>
+<tr><th><tt>HZ</tt> Kconfig</th>
+ <th>In-Kernel</th>
+ <th>Usermode</th>
+ <th>Idle</th></tr>
+<tr><th align="left"><tt>HZ_PERIODIC</tt></th>
+ <td>Can rely on scheduling-clock interrupt.</td>
+ <td>Can rely on scheduling-clock interrupt and its
+ detection of interrupt from usermode.</td>
+ <td>Can rely on RCU's dyntick-idle detection.</td></tr>
+<tr><th align="left"><tt>NO_HZ_IDLE</tt></th>
+ <td>Can rely on scheduling-clock interrupt.</td>
+ <td>Can rely on scheduling-clock interrupt and its
+ detection of interrupt from usermode.</td>
+ <td>Can rely on RCU's dyntick-idle detection.</td></tr>
+<tr><th align="left"><tt>NO_HZ_FULL</tt></th>
+ <td>Can only sometimes rely on scheduling-clock interrupt.
+ In other cases, it is necessary to bound kernel execution
+ times and/or use IPIs.</td>
+ <td>Can rely on RCU's dyntick-idle detection.</td>
+ <td>Can rely on RCU's dyntick-idle detection.</td></tr>
+</table>
+
+<table>
+<tr><th> </th></tr>
+<tr><th align="left">Quick Quiz:</th></tr>
+<tr><td>
+ Why can't <tt>NO_HZ_FULL</tt> in-kernel execution rely on the
+ scheduling-clock interrupt, just like <tt>HZ_PERIODIC</tt>
+ and <tt>NO_HZ_IDLE</tt> do?
+</td></tr>
+<tr><th align="left">Answer:</th></tr>
+<tr><td bgcolor="#ffffff"><font color="ffffff">
+ Because, as a performance optimization, <tt>NO_HZ_FULL</tt>
+ does not necessarily re-enable the scheduling-clock interrupt
+ on entry to each and every system call.
+</font></td></tr>
+<tr><td> </td></tr>
+</table>
+
+<p>
+However, RCU must be reliably informed as to whether any given
+CPU is currently in the idle loop, and, for <tt>NO_HZ_FULL</tt>,
+also whether that CPU is executing in usermode, as discussed
+<a href="#Energy Efficiency">earlier</a>.
+It also requires that the scheduling-clock interrupt be enabled when
+RCU needs it to be:
+
+<ol>
+<li> If a CPU is either idle or executing in usermode, and RCU believes
+ it is non-idle, the scheduling-clock tick had better be running.
+ Otherwise, you will get RCU CPU stall warnings. Or at best,
+ very long (11-second) grace periods, with a pointless IPI waking
+ the CPU from time to time.
+<li> If a CPU is in a portion of the kernel that executes RCU read-side
+ critical sections, and RCU believes this CPU to be idle, you will get
+ random memory corruption. <b>DON'T DO THIS!!!</b>
+
+ <br>This is one reason to test with lockdep, which will complain
+ about this sort of thing.
+<li> If a CPU is in a portion of the kernel that is absolutely
+ positively no-joking guaranteed to never execute any RCU read-side
+ critical sections, and RCU believes this CPU to to be idle,
+ no problem. This sort of thing is used by some architectures
+ for light-weight exception handlers, which can then avoid the
+ overhead of <tt>rcu_irq_enter()</tt> and <tt>rcu_irq_exit()</tt>
+ at exception entry and exit, respectively.
+ Some go further and avoid the entireties of <tt>irq_enter()</tt>
+ and <tt>irq_exit()</tt>.
+
+ <br>Just make very sure you are running some of your tests with
+ <tt>CONFIG_PROVE_RCU=y</tt>, just in case one of your code paths
+ was in fact joking about not doing RCU read-side critical sections.
+<li> If a CPU is executing in the kernel with the scheduling-clock
+ interrupt disabled and RCU believes this CPU to be non-idle,
+ and if the CPU goes idle (from an RCU perspective) every few
+ jiffies, no problem. It is usually OK for there to be the
+ occasional gap between idle periods of up to a second or so.
+
+ <br>If the gap grows too long, you get RCU CPU stall warnings.
+<li> If a CPU is either idle or executing in usermode, and RCU believes
+ it to be idle, of course no problem.
+<li> If a CPU is executing in the kernel, the kernel code
+ path is passing through quiescent states at a reasonable
+ frequency (preferably about once per few jiffies, but the
+ occasional excursion to a second or so is usually OK) and the
+ scheduling-clock interrupt is enabled, of course no problem.
+
+ <br>If the gap between a successive pair of quiescent states grows
+ too long, you get RCU CPU stall warnings.
+</ol>
+
+<p>
+But as long as RCU is properly informed of kernel state transitions between
+in-kernel execution, usermode execution, and idle, and as long as the
+scheduling-clock interrupt is enabled when RCU needs it to be, you
+can rest assured that the bugs you encounter will be in some other
+part of RCU or some other part of the kernel!
+
<h3><a name="Memory Efficiency">Memory Efficiency</a></h3>
<p>
[toc] | [prev] | [next] | [standalone]
| From | "Li, Aubrey" <aubrey.li@linux.intel.com> |
|---|---|
| Date | 2017-07-20 03:50 +0200 |
| Message-ID | <u58Lg-cj-3@gated-at.bofh.it> |
| In reply to | #1691802 |
On 2017/7/19 22:48, Paul E. McKenney wrote:
> On Wed, Jul 19, 2017 at 01:44:06PM +0800, Li, Aubrey wrote:
>> On 2017/7/18 23:20, Paul E. McKenney wrote:
>>
>>>> 2) for rcu idle enter/exit, I measured the details which Paul provided, and
>>>> the result matches with what I have measured before, nothing notable found.
>>>> But it still makes more sense if we can make rcu idle enter/exit hooked with
>>>> tick off. (it's possible other workloads behave differently)
>>>
>>> Again, assuming that RCU is informed of CPUs in the kernel, regardless
>>> of whether or not the tick is on that that point in time.
>>>
>> Yeah, I see, no problem for a normal idle.
>>
>> But for a short idle, we want to return to the task ASAP. Even though RCU cost
>> is not notable, it would still be better for me if we can save some cycles in
>> idle entry and idle exit.
>>
>> Do we have any problem if we skip RCU idle enter/exit under a fast idle scenario?
>> My understanding is, if tick is not stopped, then we don't need inform RCU in
>> idle path, it can be informed in irq exit.
>
> Indeed, the problem arises when the tick is stopped.
My question is, does problem arise when the tick is *not* stopped (skipping nohz idle)?
instead of
static void cpuidle_idle_call()
{
rcu_idle_enter()
......
rcu_idle_exit()
}
I want
static void cpuidle_idle_call()
{
if (tick stopped)
rcu_idle_enter()
......
if (tick stopped)
rcu_idle_exit()
}
Or checking tick stop can be put into rcu_idle_enter/exit
Thanks,
-Aubrey
[toc] | [prev] | [next] | [standalone]
| From | "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> |
|---|---|
| Date | 2017-07-20 15:00 +0200 |
| Message-ID | <u5jdE-7zI-15@gated-at.bofh.it> |
| In reply to | #1692338 |
On Thu, Jul 20, 2017 at 09:40:49AM +0800, Li, Aubrey wrote:
> On 2017/7/19 22:48, Paul E. McKenney wrote:
> > On Wed, Jul 19, 2017 at 01:44:06PM +0800, Li, Aubrey wrote:
> >> On 2017/7/18 23:20, Paul E. McKenney wrote:
> >>
> >>>> 2) for rcu idle enter/exit, I measured the details which Paul provided, and
> >>>> the result matches with what I have measured before, nothing notable found.
> >>>> But it still makes more sense if we can make rcu idle enter/exit hooked with
> >>>> tick off. (it's possible other workloads behave differently)
> >>>
> >>> Again, assuming that RCU is informed of CPUs in the kernel, regardless
> >>> of whether or not the tick is on that that point in time.
> >>>
> >> Yeah, I see, no problem for a normal idle.
> >>
> >> But for a short idle, we want to return to the task ASAP. Even though RCU cost
> >> is not notable, it would still be better for me if we can save some cycles in
> >> idle entry and idle exit.
> >>
> >> Do we have any problem if we skip RCU idle enter/exit under a fast idle scenario?
> >> My understanding is, if tick is not stopped, then we don't need inform RCU in
> >> idle path, it can be informed in irq exit.
> >
> > Indeed, the problem arises when the tick is stopped.
>
> My question is, does problem arise when the tick is *not* stopped (skipping nohz idle)?
>
> instead of
>
> static void cpuidle_idle_call()
> {
> rcu_idle_enter()
> ......
> rcu_idle_exit()
> }
>
> I want
>
> static void cpuidle_idle_call()
> {
> if (tick stopped)
> rcu_idle_enter()
> ......
> if (tick stopped)
> rcu_idle_exit()
> }
>
> Or checking tick stop can be put into rcu_idle_enter/exit
The answer is the traditional "it depends".
If the above change was all that you did, that would be a bug in the
case where the predicted short idle time turned out to in reality be an
indefinite idle time. RCU would indefinitely believe that the CPU was
non-idle, and would wait for it to report a quiescent state, which it
would not, even given the scheduling-clock interrupt (it is idle, so it
knows RCU already knows that it is idle, you see). RCU would eventually
send it a resched IPI, which would probably get things going again,
but the best you could hope for was extra resched IPIs and excessively
long grace periods.
To make this work reasonably, you would also need some way to check for
the case where the prediction idle time is short but the real idle time
is very long.
Thanx, Paul
[toc] | [prev] | [next] | [standalone]
| From | Arjan van de Ven <arjan@linux.intel.com> |
|---|---|
| Date | 2017-07-20 15:50 +0200 |
| Message-ID | <u5k02-8cN-21@gated-at.bofh.it> |
| In reply to | #1692899 |
On 7/20/2017 5:50 AM, Paul E. McKenney wrote: > To make this work reasonably, you would also need some way to check for > the case where the prediction idle time is short but the real idle time > is very long. so the case where you predict very short but is actually "indefinite", the real solution likely is that we set a timer some time in the future (say 100msec, or some other value that is long but not indefinite) where we wake up the system and make a new prediction, since clearly we were insanely wrong in the prediction and should try again. that or we turn the prediction from a single value into a range of (expected, upper bound) where upper bound is likely the next timer or other going-to-happen events.
[toc] | [prev] | [next] | [standalone]
| From | Peter Zijlstra <peterz@infradead.org> |
|---|---|
| Date | 2017-07-20 16:30 +0200 |
| Message-ID | <u5kCJ-ew-9@gated-at.bofh.it> |
| In reply to | #1692899 |
On Thu, Jul 20, 2017 at 05:50:54AM -0700, Paul E. McKenney wrote:
> >
> > static void cpuidle_idle_call()
> > {
> > rcu_idle_enter()
> > ......
> > rcu_idle_exit()
> > }
> >
> > I want
> >
> > static void cpuidle_idle_call()
> > {
> > if (tick stopped)
> > rcu_idle_enter()
> > ......
> > if (tick stopped)
> > rcu_idle_exit()
> > }
> >
> > Or checking tick stop can be put into rcu_idle_enter/exit
>
> The answer is the traditional "it depends".
>
> If the above change was all that you did, that would be a bug in the
> case where the predicted short idle time turned out to in reality be an
> indefinite idle time.
Can't be, you didn't disable the tick after all, so you're guaranteed to
get interrupted by the tick and try again.
[toc] | [prev] | [next] | [standalone]
| From | "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> |
|---|---|
| Date | 2017-07-20 18:10 +0200 |
| Message-ID | <u5mbw-1jG-29@gated-at.bofh.it> |
| In reply to | #1692978 |
On Thu, Jul 20, 2017 at 04:19:40PM +0200, Peter Zijlstra wrote:
> On Thu, Jul 20, 2017 at 05:50:54AM -0700, Paul E. McKenney wrote:
> > >
> > > static void cpuidle_idle_call()
> > > {
> > > rcu_idle_enter()
> > > ......
> > > rcu_idle_exit()
> > > }
> > >
> > > I want
> > >
> > > static void cpuidle_idle_call()
> > > {
> > > if (tick stopped)
> > > rcu_idle_enter()
> > > ......
> > > if (tick stopped)
> > > rcu_idle_exit()
> > > }
> > >
> > > Or checking tick stop can be put into rcu_idle_enter/exit
> >
> > The answer is the traditional "it depends".
> >
> > If the above change was all that you did, that would be a bug in the
> > case where the predicted short idle time turned out to in reality be an
> > indefinite idle time.
>
> Can't be, you didn't disable the tick after all, so you're guaranteed to
> get interrupted by the tick and try again.
I will reserve judgment on that until I see the patch. But to your point,
I would indeed hope that it works that way. ;-)
Thanx, Paul
[toc] | [prev] | [next] | [standalone]
| From | Peter Zijlstra <peterz@infradead.org> |
|---|---|
| Date | 2017-07-18 18:50 +0200 |
| Message-ID | <u4DR8-4Ny-15@gated-at.bofh.it> |
| In reply to | #1689788 |
On Tue, Jul 18, 2017 at 02:56:47PM +0800, Li, Aubrey wrote:
> 3) for tick nohz idle, we want to skip if the coming idle is short. If we can
> skip the tick nohz idle, we then skip all the items depending on it. But, there
> are two hard points:
>
> 3.1) how to compute the period of the coming idle. My current proposal is to
> use two factors in the current idle menu governor. There are two possible
> options from Peter and Thomas, the one is to use scheduler idle estimate, which
> is task activity based, the other is to use the statistics generated from irq
> timings work.
>
> 3.2) how to determine if the idle is short or long. My current proposal is to
> use a tunable value via /sys, while Peter prefers an auto-adjust mechanism. I
> didn't get the details of an auto-adjust mechanism yet
So the problem is that the cost of NOHZ enter/exit are for a large part
determined by (micro) architecture costs of programming timer hardware.
A single static threshold will never be the right value across all the
various machines we run Linux on.
So my suggestion was simply timing the cost of doing those functions
ever time we do them and keeping a running average of their cost. Then
use that measured cost as a basis for selecting when to skip them. For
example if the estimated idle period (by whatever estimate we end up
using) is less than 4 times the cost of doing NOHZ, don't do NOHZ.
Note how both tick_nohz_idle_{enter,exit}() already take a timestamp at
entry; all you need to do is take another one at exit and subtract.
[toc] | [prev] | [next] | [standalone]
| From | Peter Zijlstra <peterz@infradead.org> |
|---|---|
| Date | 2017-07-18 15:30 +0200 |
| Message-ID | <u4AJB-2VN-35@gated-at.bofh.it> |
| In reply to | #1689628 |
On Tue, Jul 18, 2017 at 11:14:57AM +0800, Li, Aubrey wrote:
> On 2017/7/18 3:23, Peter Zijlstra wrote:
> > On Fri, Jul 14, 2017 at 09:26:19AM -0700, Andi Kleen wrote:
> >>> And as said; Daniel has been working on a better predictor -- now he's
> >>> probably not used it on the network workload you're looking at, so that
> >>> might be something to consider.
> >>
> >> Deriving a better idle predictor is a bit orthogonal to fast idle.
> >
> > No. If you want a different C state selected we need to fix the current
> > C state selector. We're not going to tinker.
> >
> > And the predictor is probably the most fundamental part of the whole C
> > state selection logic.
> >
> > Now I think the problem is that the current predictor goes for an
> > average idle duration. This means that we, on average, get it wrong 50%
> > of the time. For performance that's bad.
> >
> > If you want to improve the worst case, we need to consider a cumulative
> > distribution function, and staying with the Gaussian assumption already
> > present, that would mean using:
> >
> > 1 x - mu
> > CDF(x) = - [ 1 + erf(-------------) ]
> > 2 sigma sqrt(2)
> >
> > Where, per the normal convention mu is the average and sigma^2 the
> > variance. See also:
> >
> > https://en.wikipedia.org/wiki/Normal_distribution
> >
> > We then solve CDF(x) = n% to find the x for which we get it wrong n% of
> > the time (IIRC something like: 'mu - 2sigma' ends up being 5% or so).
> >
> > This conceptually gets us better exit latency for the cases where we got
> > it wrong before, and practically pushes down the estimate which gets us
> > C1 longer.
> >
> > Of course, this all assumes a Gaussian distribution to begin with, if we
> > get bimodal (or worse) distributions we can still get it wrong. To fix
> > that, we'd need to do something better than what we currently have.
> >
> Maybe you are talking about applying some machine learning algorithm online
> to fit a multivariate normal distribution, :)
Nah, nothing that fancy..
Something that _could_ work and deals with arbitrary distributions is
buckets divided on the actual C-state selection boundaries and a
(cyclic) array of the N most recent idle times.
Something like so:
struct est_data {
u8 array[64]
u8 *dist;
u8 idx;
}
DEFINE_PER_CPU(struct est_data, est_data);
void est_init(void)
{
int size = drv->state_count;
int cpu;
for_each_possible_cpu(cpu) {
per_cpu(est_data, cpu).dist = kzalloc(size);
// handle errors
}
}
u8 est_duration_2_state(u64 duration)
{
for (i=0; i<drv->state_count; i++) {
if (duration/1024 < drv->state[i].target_residency)
return i;
}
return i-1;
}
void est_contemplate(u64 duration)
{
struct est_data *ed = this_cpu_ptr(&est_data);
int state = est_duration_2_state(duration);
int idx = (ed->idx++ % ARRAY_SIZE(ed->array);
ed->dist[ed->array[idx]]--;
ed->array[idx] = state;
ed->dist[ed->array[idx]]++;
}
int est_state(int pct)
{
struct est_data *ed = this_cpu_ptr(&est_data);
int limit = pct * ARRAY_SIZE(ed->array) / 100; /* XXX move div out of line */
int cnt, last = 0;
/* CDF */
for (i=0; i<drv->state_count; i++) {
cnt += ed->dist[i];
if (cnt > limit)
break;
last = i;
}
return last;
}
> Well, back to the problem, when the scheduler picks up idle thread, it does
> not look at the history, nor make the prediction. So it's possible it has
> to switch back a task ASAP when it's going into idle(very common under some
> workloads).
>
> That is, (idle_entry + idle_exit) > idle. If the system has multiple
> hardware idle states, then:
>
> (idle_entry + idle_exit + HW_entry + HW_exit) > HW_sleep
>
> So we eventually want the idle path lighter than what we currently have.
> A complex predictor may have high accuracy, but the cost could be high as well.
I never suggested anything complex. The current menu thing uses an
average, all I said is if instead of the average you use something less,
say 'avg - 2*stdev' (it already computes the stdev) you get something,
which assuming Gaussian, is less than ~5 wrong on exit latency.
The above, also simple thing, uses a generic distribution function,
which works because it uses the exact boundaries we're limited to
anyway.
Of course, the above needs to be augmented with IRQ bits etc..
[toc] | [prev] | [next] | [standalone]
| From | Peter Zijlstra <peterz@infradead.org> |
|---|---|
| Date | 2017-07-19 15:50 +0200 |
| Message-ID | <u4Xwt-P7-7@gated-at.bofh.it> |
| In reply to | #1689628 |
On Tue, Jul 18, 2017 at 11:14:57AM +0800, Li, Aubrey wrote: > On 2017/7/18 3:23, Peter Zijlstra wrote: > > On Fri, Jul 14, 2017 at 09:26:19AM -0700, Andi Kleen wrote: > >>> And as said; Daniel has been working on a better predictor -- now he's > >>> probably not used it on the network workload you're looking at, so that > >>> might be something to consider. > >> > >> Deriving a better idle predictor is a bit orthogonal to fast idle. > > > > No. If you want a different C state selected we need to fix the current > > C state selector. We're not going to tinker. > > > > And the predictor is probably the most fundamental part of the whole C > > state selection logic. > > > > Now I think the problem is that the current predictor goes for an > > average idle duration. This means that we, on average, get it wrong 50% > > of the time. For performance that's bad. > > > > If you want to improve the worst case, we need to consider a cumulative > > distribution function, and staying with the Gaussian assumption already > > present, that would mean using: > > > > 1 x - mu > > CDF(x) = - [ 1 + erf(-------------) ] > > 2 sigma sqrt(2) > > > > Where, per the normal convention mu is the average and sigma^2 the > > variance. See also: > > > > https://en.wikipedia.org/wiki/Normal_distribution > > > > We then solve CDF(x) = n% to find the x for which we get it wrong n% of > > the time (IIRC something like: 'mu - 2sigma' ends up being 5% or so). > > > > This conceptually gets us better exit latency for the cases where we got > > it wrong before, and practically pushes down the estimate which gets us > > C1 longer. > > > > Of course, this all assumes a Gaussian distribution to begin with, if we > > get bimodal (or worse) distributions we can still get it wrong. To fix > > that, we'd need to do something better than what we currently have. > > > Maybe you are talking about applying some machine learning algorithm online > to fit a multivariate normal distribution, :) See here for an implementation of what I spoke about above: https://lkml.kernel.org/r/20170719133940.uytsixvfgpmo3ane@hirez.programming.kicks-ass.net Very much statistics 101.
[toc] | [prev] | [next] | [standalone]
| From | Arjan van de Ven <arjan@linux.intel.com> |
|---|---|
| Date | 2017-07-14 18:00 +0200 |
| Message-ID | <u3bax-5vu-3@gated-at.bofh.it> |
| In reply to | #1687517 |
On 7/14/2017 8:38 AM, Peter Zijlstra wrote: > No, that's wrong. We want to fix the normal C state selection process to > pick the right C state. > > The fast-idle criteria could cut off a whole bunch of available C > states. We need to understand why our current C state pick is wrong and > amend the algorithm to do better. Not just bolt something on the side. I can see a fast path through selection if you know the upper bound of any selection is just 1 state. But also, how much of this is about "C1 be fast" versus "selecting C1 is slow" a lot of the patches in the thread seem to be about making a lighter/faster C1, which is reasonable (you can even argue we might end up with 2 C1s, one fast one full feature)
[toc] | [prev] | [next] | [standalone]
| From | "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> |
|---|---|
| Date | 2017-07-13 17:30 +0200 |
| Message-ID | <u2OdX-7kO-7@gated-at.bofh.it> |
| In reply to | #1686601 |
On Thu, Jul 13, 2017 at 04:53:11PM +0200, Peter Zijlstra wrote: > On Thu, Jul 13, 2017 at 10:48:55PM +0800, Li, Aubrey wrote: > > > - totally from arch_cpu_idle_enter entry to arch_cpu_idle_exit return costs > > 9122ns - 15318ns. > > ---- In this period(arch idle), rcu_idle_enter costs 1985ns - 2262ns, rcu_idle_exit > > costs 1813ns - 3507ns > > > > Besides RCU, > > So Paul wants more details on where RCU hurts so we can try to fix. More specifically: rcu_needs_cpu(), rcu_prepare_for_idle(), rcu_cleanup_after_idle(), rcu_eqs_enter(), rcu_eqs_enter_common(), rcu_dynticks_eqs_enter(), do_nocb_deferred_wakeup(), rcu_dynticks_task_enter(), rcu_eqs_exit(), rcu_eqs_exit_common(), rcu_dynticks_task_exit(), rcu_dynticks_eqs_exit(). The first three (rcu_needs_cpu(), rcu_prepare_for_idle(), and rcu_cleanup_after_idle()) should not be significant unless you have CONFIG_RCU_FAST_NO_HZ=y. If you do, it would be interesting to learn how often invoke_rcu_core() is invoked from rcu_prepare_for_idle() and rcu_cleanup_after_idle(), as this can raise softirq. Also rcu_accelerate_cbs() and rcu_try_advance_all_cbs(). Knowing which of these is causing the most trouble might help me reduce the overhead in the current idle path. Also, how big is this system? If you can say, about what is the cost of a cache miss to some other CPU's cache? Thanx, Paul > > the period includes c-state selection on X86, a few timestamp updates > > and a few computations in menu governor. Also, deep HW-cstate latency can be up > > to 100+ microseconds, even if the system is very busy, CPU still has chance to enter > > deep cstate, which I guess some outburst workloads are not happy with it. > > > > That's my major concern without a fast idle path. > > Fixing C-state selection by creating an alternative idle path sounds so > very wrong. >
[toc] | [prev] | [next] | [standalone]
| From | "Li, Aubrey" <aubrey.li@linux.intel.com> |
|---|---|
| Date | 2017-07-14 05:50 +0200 |
| Message-ID | <u2ZM5-66v-1@gated-at.bofh.it> |
| In reply to | #1686628 |
On 2017/7/13 23:20, Paul E. McKenney wrote: > On Thu, Jul 13, 2017 at 04:53:11PM +0200, Peter Zijlstra wrote: >> On Thu, Jul 13, 2017 at 10:48:55PM +0800, Li, Aubrey wrote: >> >>> - totally from arch_cpu_idle_enter entry to arch_cpu_idle_exit return costs >>> 9122ns - 15318ns. >>> ---- In this period(arch idle), rcu_idle_enter costs 1985ns - 2262ns, rcu_idle_exit >>> costs 1813ns - 3507ns >>> >>> Besides RCU, >> >> So Paul wants more details on where RCU hurts so we can try to fix. > > More specifically: rcu_needs_cpu(), rcu_prepare_for_idle(), > rcu_cleanup_after_idle(), rcu_eqs_enter(), rcu_eqs_enter_common(), > rcu_dynticks_eqs_enter(), do_nocb_deferred_wakeup(), > rcu_dynticks_task_enter(), rcu_eqs_exit(), rcu_eqs_exit_common(), > rcu_dynticks_task_exit(), rcu_dynticks_eqs_exit(). > > The first three (rcu_needs_cpu(), rcu_prepare_for_idle(), and > rcu_cleanup_after_idle()) should not be significant unless you have > CONFIG_RCU_FAST_NO_HZ=y. If you do, it would be interesting to learn > how often invoke_rcu_core() is invoked from rcu_prepare_for_idle() > and rcu_cleanup_after_idle(), as this can raise softirq. Also > rcu_accelerate_cbs() and rcu_try_advance_all_cbs(). > > Knowing which of these is causing the most trouble might help me > reduce the overhead in the current idle path. > I don't have details of these functions, I can measure if you want. Do you have preferred workload for the measurement? > Also, how big is this system? If you can say, about what is the cost > of a cache miss to some other CPU's cache? > The system has two NUMA nodes. nproc returns 104. local memory access is ~100 ns and remote memory access is ~200ns, reported by mgen. Does this address your question? Thanks, -Aubrey
[toc] | [prev] | [next] | [standalone]
| From | "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> |
|---|---|
| Date | 2017-07-14 06:10 +0200 |
| Message-ID | <u305s-6t9-5@gated-at.bofh.it> |
| In reply to | #1687051 |
On Fri, Jul 14, 2017 at 11:47:32AM +0800, Li, Aubrey wrote: > On 2017/7/13 23:20, Paul E. McKenney wrote: > > On Thu, Jul 13, 2017 at 04:53:11PM +0200, Peter Zijlstra wrote: > >> On Thu, Jul 13, 2017 at 10:48:55PM +0800, Li, Aubrey wrote: > >> > >>> - totally from arch_cpu_idle_enter entry to arch_cpu_idle_exit return costs > >>> 9122ns - 15318ns. > >>> ---- In this period(arch idle), rcu_idle_enter costs 1985ns - 2262ns, rcu_idle_exit > >>> costs 1813ns - 3507ns > >>> > >>> Besides RCU, > >> > >> So Paul wants more details on where RCU hurts so we can try to fix. > > > > More specifically: rcu_needs_cpu(), rcu_prepare_for_idle(), > > rcu_cleanup_after_idle(), rcu_eqs_enter(), rcu_eqs_enter_common(), > > rcu_dynticks_eqs_enter(), do_nocb_deferred_wakeup(), > > rcu_dynticks_task_enter(), rcu_eqs_exit(), rcu_eqs_exit_common(), > > rcu_dynticks_task_exit(), rcu_dynticks_eqs_exit(). > > > > The first three (rcu_needs_cpu(), rcu_prepare_for_idle(), and > > rcu_cleanup_after_idle()) should not be significant unless you have > > CONFIG_RCU_FAST_NO_HZ=y. If you do, it would be interesting to learn > > how often invoke_rcu_core() is invoked from rcu_prepare_for_idle() > > and rcu_cleanup_after_idle(), as this can raise softirq. Also > > rcu_accelerate_cbs() and rcu_try_advance_all_cbs(). > > > > Knowing which of these is causing the most trouble might help me > > reduce the overhead in the current idle path. > > > I don't have details of these functions, I can measure if you want. > Do you have preferred workload for the measurement? I do not have a specific workload in mind. Could you please choose one with very frequent transitions to and from idle? > > Also, how big is this system? If you can say, about what is the cost > > of a cache miss to some other CPU's cache? > > > The system has two NUMA nodes. nproc returns 104. local memory access is > ~100 ns and remote memory access is ~200ns, reported by mgen. Does this > address your question? Very much so, thank you! This will allow me to correctly interpret time spent in the above functions. Thanx, Paul
[toc] | [prev] | [next] | [standalone]
Page 5 of 6 — ← Prev page 1 2 3 4 [5] 6 Next page →
Back to top | Article view | linux.kernel
csiph-web