Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1270272 > unrolled thread

Re: [PATCH 2/4] timer: relax tick stop in idle entry

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH 2/4] timer: relax tick stop in idle entry Thomas Gleixner <tglx@linutronix.de> - 2015-11-16 16:10 +0100
    Re: [PATCH 2/4] timer: relax tick stop in idle entry Jacob Pan <jacob.jun.pan@linux.intel.com> - 2015-11-16 23:00 +0100
      Re: [PATCH 2/4] timer: relax tick stop in idle entry Thomas Gleixner <tglx@linutronix.de> - 2015-11-16 23:10 +0100
        Re: [PATCH 2/4] timer: relax tick stop in idle entry Jacob Pan <jacob.jun.pan@linux.intel.com> - 2015-11-17 01:20 +0100
          Re: [PATCH 2/4] timer: relax tick stop in idle entry Jacob Pan <jacob.jun.pan@linux.intel.com> - 2015-11-19 18:50 +0100
          Re: [PATCH 2/4] timer: relax tick stop in idle entry Peter Zijlstra <peterz@infradead.org> - 2015-11-19 20:10 +0100
            Re: [PATCH 2/4] timer: relax tick stop in idle entry Jacob Pan <jacob.jun.pan@linux.intel.com> - 2015-11-19 20:30 +0100
              Re: [PATCH 2/4] timer: relax tick stop in idle entry Peter Zijlstra <peterz@infradead.org> - 2015-11-19 21:00 +0100
                Re: [PATCH 2/4] timer: relax tick stop in idle entry Jacob Pan <jacob.jun.pan@linux.intel.com> - 2015-11-20 00:50 +0100
      Re: [PATCH 2/4] timer: relax tick stop in idle entry "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-11-16 23:40 +0100
        Re: [PATCH 2/4] timer: relax tick stop in idle entry Jacob Pan <jacob.jun.pan@linux.intel.com> - 2015-11-17 00:10 +0100
        Re: [PATCH 2/4] timer: relax tick stop in idle entry Jacob Pan <jacob.jun.pan@linux.intel.com> - 2015-11-17 00:20 +0100
          Re: [PATCH 2/4] timer: relax tick stop in idle entry "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-11-17 00:30 +0100
            Re: [PATCH 2/4] timer: relax tick stop in idle entry Arjan van de Ven <arjan@linux.intel.com> - 2015-11-17 00:40 +0100
              Re: [PATCH 2/4] timer: relax tick stop in idle entry Jacob Pan <jacob.jun.pan@linux.intel.com> - 2015-11-17 00:50 +0100
                Re: [PATCH 2/4] timer: relax tick stop in idle entry "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-11-17 01:10 +0100
      Re: [PATCH 2/4] timer: relax tick stop in idle entry Josh Triplett <josh@joshtriplett.org> - 2015-11-16 23:40 +0100
        Re: [PATCH 2/4] timer: relax tick stop in idle entry "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-11-17 00:30 +0100
          Re: [PATCH 2/4] timer: relax tick stop in idle entry Josh Triplett <josh@joshtriplett.org> - 2015-11-17 02:50 +0100
            Re: [PATCH 2/4] timer: relax tick stop in idle entry Arjan van de Ven <arjan@linux.intel.com> - 2015-11-17 04:00 +0100
              Re: [PATCH 2/4] timer: relax tick stop in idle entry "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-11-17 06:10 +0100
                Re: [PATCH 2/4] timer: relax tick stop in idle entry Peter Zijlstra <peterz@infradead.org> - 2015-11-17 11:30 +0100
                  Re: [PATCH 2/4] timer: relax tick stop in idle entry Jacob Pan <jacob.jun.pan@linux.intel.com> - 2015-11-17 14:00 +0100
                    Re: [PATCH 2/4] timer: relax tick stop in idle entry "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-11-17 14:50 +0100
            Re: [PATCH 2/4] timer: relax tick stop in idle entry "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-11-17 04:00 +0100

Page 1 of 2  [1] 2  Next page →

#1270272 — Re: [PATCH 2/4] timer: relax tick stop in idle entry

On Fri, 13 Nov 2015, Jacob Pan wrote:
>           <idle>-0     [000]    30.093473: bprint:               __tick_nohz_idle_enter: JPAN: __tick_nohz_idle_enter 803
>           <idle>-0     [000]    30.093473: bprint:               __tick_nohz_idle_enter: JPAN: can_stop_idle_tick 743
> [JP] can_stop_idle_tick() checks ok to stop tick
> 
>           <idle>-0     [000]    30.093474: bprint:               __tick_nohz_idle_enter: JPAN: tick_nohz_stop_sched_tick 609 delta 1000000
> [JP] but sees delta is exactly 1 tick away. didn't stop tick.

If the delta is 1 tick then it is not supposed to stop it. Did you
ever try to figure out WHY it is 1 tick?

There are two code pathes which can set it to basemono + TICK_NSEC:

        if (rcu_needs_cpu(basemono, &next_rcu) ||
            arch_needs_cpu() || irq_work_needs_cpu()) {
                next_tick = basemono + TICK_NSEC;
        } else {
                next_tmr = get_next_timer_interrupt(basejiff, basemono);
                ts->next_timer = next_tmr;
                /* Take the next rcu event into account */
                next_tick = next_rcu < next_tmr ? next_rcu : next_tmr;
        }

Can you please figure out WHY the tick is requested to continue
instead of blindly wreckaging the logic in that code?

Thanks,

	tglx
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1270629

On Mon, 16 Nov 2015 16:06:57 +0100 (CET)
Thomas Gleixner <tglx@linutronix.de> wrote:

> >           <idle>-0     [000]    30.093474: bprint:
> > __tick_nohz_idle_enter: JPAN: tick_nohz_stop_sched_tick 609 delta
> > 1000000 [JP] but sees delta is exactly 1 tick away. didn't stop
> > tick.  
> 
> If the delta is 1 tick then it is not supposed to stop it. Did you
> ever try to figure out WHY it is 1 tick?
> 
> There are two code pathes which can set it to basemono + TICK_NSEC:
> 
>         if (rcu_needs_cpu(basemono, &next_rcu) ||
>             arch_needs_cpu() || irq_work_needs_cpu()) {
>                 next_tick = basemono + TICK_NSEC;
>         } else {
>                 next_tmr = get_next_timer_interrupt(basejiff,
> basemono); ts->next_timer = next_tmr;
>                 /* Take the next rcu event into account */
>                 next_tick = next_rcu < next_tmr ? next_rcu : next_tmr;
>         }
> 
> Can you please figure out WHY the tick is requested to continue
> instead of blindly wreckaging the logic in that code?

Looks like the it hits in both cases during forced idle.
+ Josh
+ Paul

For the first case, it is always related to RCU. I found there are two
CONFIG options to avoid this undesired tick in idle loop.
1. enable CONFIG_RCU_NOCB_CPU_ALL, offload to orcu kthreads
2. or enable CONFIG_RCU_FAST_NO_HZ (enter dytick idle w/ rcu callback)

Either one works but my concern is that users may not realize the
intricate CONFIG_ options and how they translate into energy savings.
Consulted with Josh, it seems we could add a check here to recognize
the forced idle state and relax rcu_needs_cpu() to return false even it
has callbacks. Since we are blocking everybody for a short time (5 ticks
default). It should not impact synchronize and kfree rcu.

For the second case, which is much more rare, I think we do have next
timer exactly one tick away. Just don't know why tick will continue into
idle loop.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1270639

On Mon, 16 Nov 2015, Jacob Pan wrote:

> For the second case, which is much more rare, I think we do have next
> timer exactly one tick away. Just don't know why tick will continue into
> idle loop.

Well, it should not be hard to figure that out. There are not so many
checks involved when tick_nohz_irq_exit() is called.

Thanks,

	tglx
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1270715

On Mon, 16 Nov 2015 23:01:12 +0100 (CET)
Thomas Gleixner <tglx@linutronix.de> wrote:

> > For the second case, which is much more rare, I think we do have
> > next timer exactly one tick away. Just don't know why tick will
> > continue into idle loop.  
> 
> Well, it should not be hard to figure that out. There are not so many
> checks involved when tick_nohz_irq_exit() is called.
Thanks for the tip, I found the cause is in 
int idle_cpu(int cpu)
{
	if (rq->nr_running)
		return 0;

Since we only take into account of cfs_rq runnable taking over
cfs_rq->nr_running when forced_idle is set.


Jacob
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1273306

On Mon, 16 Nov 2015 16:09:10 -0800
Jacob Pan <jacob.jun.pan@linux.intel.com> wrote:

> > > For the second case, which is much more rare, I think we do have
> > > next timer exactly one tick away. Just don't know why tick will
> > > continue into idle loop.    
> > 
> > Well, it should not be hard to figure that out. There are not so
> > many checks involved when tick_nohz_irq_exit() is called.  
> Thanks for the tip, I found the cause is in 
> int idle_cpu(int cpu)
> {
> 	if (rq->nr_running)
> 		return 0;
> 
> Since we only take into account of cfs_rq runnable taking over
> cfs_rq->nr_running when forced_idle is set.
I am not sure what is the best solution. It seems I can add additional
checks like this.
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -3520,9 +3520,14 @@ int idle_cpu(int cpu)
        if (rq->curr != rq->idle)
                return 0;
 
-       if (rq->nr_running)
-               return 0;
-
+       if (rq->nr_running) {
+               /* if cfs_rq is in forced idle, nr_running could be
nonzero but still in idle */
+               if ((rq->nr_running != rq->cfs.h_nr_running) ||
+                       cfs_rq_runnable(&rq->cfs))
+                       return 0;
+       }

To recap the problem statement.
1. When entering idle loop tick_nohz_stop_sched_tick() checks if the
next timer interrupt is exactly one tick away. if so, it will not stop
it to avoid threshing timer disable and enable.
2. so it relies on the next round tick_nohz_irq_exit() to have another
chance to stop the tick
3. with idle injection rq->nr_running could be nonzero when in idle
4. tick_nohz_irq_exit() will not be called if !idle_cpu()

However, idle_cpu() is used by many other callers, e.g. load balance.
Do we want to consider forced idle in those cases? Or we can forgo this
case and ignore it?

Jacob
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1273408

On Mon, Nov 16, 2015 at 04:09:10PM -0800, Jacob Pan wrote:
> On Mon, 16 Nov 2015 23:01:12 +0100 (CET)
> Thomas Gleixner <tglx@linutronix.de> wrote:
> 
> > > For the second case, which is much more rare, I think we do have
> > > next timer exactly one tick away. Just don't know why tick will
> > > continue into idle loop.  
> > 
> > Well, it should not be hard to figure that out. There are not so many
> > checks involved when tick_nohz_irq_exit() is called.

> Thanks for the tip, I found the cause is in 
> int idle_cpu(int cpu)

Which one? That is, I cannot find a idle_cpu() call from
tick_nohz_irq_exit().
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1273434

On Thu, 19 Nov 2015 20:06:30 +0100
Peter Zijlstra <peterz@infradead.org> wrote:

> Which one? That is, I cannot find a idle_cpu() call from
> tick_nohz_irq_exit().
it is  tick_irq_exit(void), i will not call tick_nohz_irq_exit() if
!idle_cpu()

please see my other email for complete story. pasted below.

I am not sure what is the best solution. It seems I can add additional
checks like this.
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -3520,9 +3520,14 @@ int idle_cpu(int cpu)
        if (rq->curr != rq->idle)
                return 0;
 
-       if (rq->nr_running)
-               return 0;
-
+       if (rq->nr_running) {
+               /* if cfs_rq is in forced idle, nr_running could be
nonzero but still in idle */
+               if ((rq->nr_running != rq->cfs.h_nr_running) ||
+                       cfs_rq_runnable(&rq->cfs))
+                       return 0;
+       }

To recap the problem statement.
1. When entering idle loop tick_nohz_stop_sched_tick() checks if the
next timer interrupt is exactly one tick away. if so, it will not stop
it to avoid threshing timer disable and enable.
2. so it relies on the next round tick_nohz_irq_exit() to have another
chance to stop the tick
3. with idle injection rq->nr_running could be nonzero when in idle
4. tick_nohz_irq_exit() will not be called if !idle_cpu()

However, idle_cpu() is used by many other callers, e.g. load balance.
Do we want to consider forced idle in those cases? Or we can forgo this
case and ignore it?

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1273442

On Thu, Nov 19, 2015 at 11:21:25AM -0800, Jacob Pan wrote:
> On Thu, 19 Nov 2015 20:06:30 +0100
> Peter Zijlstra <peterz@infradead.org> wrote:
> 
> > Which one? That is, I cannot find a idle_cpu() call from
> > tick_nohz_irq_exit().
> it is  tick_irq_exit(void), i will not call tick_nohz_irq_exit() if
> !idle_cpu()

Ah, but I think we really only need to test for the idle task there, the
&& need_resched() part of that function takes care of the rest.

Once we have need_resched() set, we'll be on our way to
tick_nohz_idle_exit() to restart the tick again.

---
 kernel/softirq.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/softirq.c b/kernel/softirq.c
index 479e4436f787..3bd30404f2ee 100644
--- a/kernel/softirq.c
+++ b/kernel/softirq.c
@@ -367,7 +367,7 @@ static inline void tick_irq_exit(void)
 	int cpu = smp_processor_id();
 
 	/* Make sure that timer wheel updates are propagated */
-	if ((idle_cpu(cpu) && !need_resched()) || tick_nohz_full_cpu(cpu)) {
+	if ((is_idle_task(current) && !need_resched()) || tick_nohz_full_cpu(cpu)) {
 		if (!in_interrupt())
 			tick_nohz_irq_exit();
 	}
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1273606

On Thu, 19 Nov 2015 20:59:05 +0100
Peter Zijlstra <peterz@infradead.org> wrote:

> On Thu, Nov 19, 2015 at 11:21:25AM -0800, Jacob Pan wrote:
> > On Thu, 19 Nov 2015 20:06:30 +0100
> > Peter Zijlstra <peterz@infradead.org> wrote:
> >   
> > > Which one? That is, I cannot find a idle_cpu() call from
> > > tick_nohz_irq_exit().  
> > it is  tick_irq_exit(void), i will not call tick_nohz_irq_exit() if
> > !idle_cpu()  
> 
> Ah, but I think we really only need to test for the idle task there,
> the && need_resched() part of that function takes care of the rest.
> 
> Once we have need_resched() set, we'll be on our way to
> tick_nohz_idle_exit() to restart the tick again.
Tested, it did the trick also. has less impact than changing idle_cpu().

My other point is the callers of idle_cpu() will see it returns false
but in forced idle. Can't see anything bad since we skip load balance.

Thanks, 

Jacob
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1270656

On Mon, Nov 16, 2015 at 01:51:26PM -0800, Jacob Pan wrote:
> On Mon, 16 Nov 2015 16:06:57 +0100 (CET)
> Thomas Gleixner <tglx@linutronix.de> wrote:
> 
> > >           <idle>-0     [000]    30.093474: bprint:
> > > __tick_nohz_idle_enter: JPAN: tick_nohz_stop_sched_tick 609 delta
> > > 1000000 [JP] but sees delta is exactly 1 tick away. didn't stop
> > > tick.  
> > 
> > If the delta is 1 tick then it is not supposed to stop it. Did you
> > ever try to figure out WHY it is 1 tick?
> > 
> > There are two code pathes which can set it to basemono + TICK_NSEC:
> > 
> >         if (rcu_needs_cpu(basemono, &next_rcu) ||
> >             arch_needs_cpu() || irq_work_needs_cpu()) {
> >                 next_tick = basemono + TICK_NSEC;
> >         } else {
> >                 next_tmr = get_next_timer_interrupt(basejiff,
> > basemono); ts->next_timer = next_tmr;
> >                 /* Take the next rcu event into account */
> >                 next_tick = next_rcu < next_tmr ? next_rcu : next_tmr;
> >         }
> > 
> > Can you please figure out WHY the tick is requested to continue
> > instead of blindly wreckaging the logic in that code?
> 
> Looks like the it hits in both cases during forced idle.
> + Josh
> + Paul
> 
> For the first case, it is always related to RCU. I found there are two
> CONFIG options to avoid this undesired tick in idle loop.
> 1. enable CONFIG_RCU_NOCB_CPU_ALL, offload to orcu kthreads
> 2. or enable CONFIG_RCU_FAST_NO_HZ (enter dytick idle w/ rcu callback)
> 
> Either one works but my concern is that users may not realize the
> intricate CONFIG_ options and how they translate into energy savings.
> Consulted with Josh, it seems we could add a check here to recognize
> the forced idle state and relax rcu_needs_cpu() to return false even it
> has callbacks. Since we are blocking everybody for a short time (5 ticks
> default). It should not impact synchronize and kfree rcu.

Or we could just set things up so that whatever Kconfig you are using
to enable this state causes CONFIG_RCU_NOCB_CPU_ALL to also be enabled.
Or that causes CONFIG_RCU_FAST_NO_HZ to also be enabled, if that works
better for you.

Just out of curiosity, what is the purpose of the forced idle state?
Thermal control or some such?

							Thanx, Paul

> For the second case, which is much more rare, I think we do have next
> timer exactly one tick away. Just don't know why tick will continue into
> idle loop.
> 

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1270679

On Mon, 16 Nov 2015 14:31:17 -0800
"Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:

> Just out of curiosity, what is the purpose of the forced idle state?
> Thermal control or some such?
yes. for thermal control or power capping. I have some description in
cover letter.
https://lwn.net/Articles/664405/

Thanks,
Jacob
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1270682

On Mon, 16 Nov 2015 14:31:17 -0800
"Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:

> > Either one works but my concern is that users may not realize the
> > intricate CONFIG_ options and how they translate into energy
> > savings. Consulted with Josh, it seems we could add a check here to
> > recognize the forced idle state and relax rcu_needs_cpu() to return
> > false even it has callbacks. Since we are blocking everybody for a
> > short time (5 ticks default). It should not impact synchronize and
> > kfree rcu.  
> 
> Or we could just set things up so that whatever Kconfig you are using
> to enable this state causes CONFIG_RCU_NOCB_CPU_ALL to also be
> enabled. Or that causes CONFIG_RCU_FAST_NO_HZ to also be enabled, if
> that works better for you.
That would be great, we can work this out once the patch is
finalized. This is not a hard dependency in that it only affects the
efficiency of idle injection.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1270688

On Mon, Nov 16, 2015 at 03:15:03PM -0800, Jacob Pan wrote:
> On Mon, 16 Nov 2015 14:31:17 -0800
> "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
> 
> > > Either one works but my concern is that users may not realize the
> > > intricate CONFIG_ options and how they translate into energy
> > > savings. Consulted with Josh, it seems we could add a check here to
> > > recognize the forced idle state and relax rcu_needs_cpu() to return
> > > false even it has callbacks. Since we are blocking everybody for a
> > > short time (5 ticks default). It should not impact synchronize and
> > > kfree rcu.  
> > 
> > Or we could just set things up so that whatever Kconfig you are using
> > to enable this state causes CONFIG_RCU_NOCB_CPU_ALL to also be
> > enabled. Or that causes CONFIG_RCU_FAST_NO_HZ to also be enabled, if
> > that works better for you.
> 
> That would be great, we can work this out once the patch is
> finalized. This is not a hard dependency in that it only affects the
> efficiency of idle injection.

Is this mostly an special-purpose embedded thing, or do you expect distros
to be enabling this?  If the former, I suggest CONFIG_RCU_NOCB_CPU_ALL,
but if distros are doing this for general-purpose workloads, I instead
suggest CONFIG_RCU_FAST_NO_HZ.

But as you say, we can work this out later.  Figured I should ask now,
though, just to get people thinking about it.

							Thanx, Paul

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1270696

On 11/16/2015 3:28 PM, Paul E. McKenney wrote:

> Is this mostly an special-purpose embedded thing, or do you expect distros
> to be enabling this?  If the former, I suggest CONFIG_RCU_NOCB_CPU_ALL,
> but if distros are doing this for general-purpose workloads, I instead
> suggest CONFIG_RCU_FAST_NO_HZ.

thermal overload happens a lot on small devices, but sadly also in big datacenters
where it is not uncommon to underprovision cooling capacity by a bit
(it's one of those "99% of the time you only need THIS much, the 1% you need 30% more"
and that more is expensive or even impractical)
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1270700

On Mon, 16 Nov 2015 15:32:38 -0800
Arjan van de Ven <arjan@linux.intel.com> wrote:

> On 11/16/2015 3:28 PM, Paul E. McKenney wrote:
> 
> > Is this mostly an special-purpose embedded thing, or do you expect
> > distros to be enabling this?  If the former, I suggest
> > , but if distros are doing this for
> > general-purpose workloads, I instead suggest CONFIG_RCU_FAST_NO_HZ.
> 
> thermal overload happens a lot on small devices, but sadly also in
> big datacenters where it is not uncommon to underprovision cooling
> capacity by a bit (it's one of those "99% of the time you only need
> THIS much, the 1% you need 30% more" and that more is expensive or
> even impractical)

CONFIG_RCU_FAST_NO_HZ is more inline with idle injection in that both
are targeted energy efficiency.
Checking ubuntu and fedora, seems both have CONFIG_RCU_NOCB_CPU_ALL=y.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1270711

On Mon, Nov 16, 2015 at 03:40:46PM -0800, Jacob Pan wrote:
> On Mon, 16 Nov 2015 15:32:38 -0800
> Arjan van de Ven <arjan@linux.intel.com> wrote:
> 
> > On 11/16/2015 3:28 PM, Paul E. McKenney wrote:
> > 
> > > Is this mostly an special-purpose embedded thing, or do you expect
> > > distros to be enabling this?  If the former, I suggest
> > > , but if distros are doing this for
> > > general-purpose workloads, I instead suggest CONFIG_RCU_FAST_NO_HZ.
> > 
> > thermal overload happens a lot on small devices, but sadly also in
> > big datacenters where it is not uncommon to underprovision cooling
> > capacity by a bit (it's one of those "99% of the time you only need
> > THIS much, the 1% you need 30% more" and that more is expensive or
> > even impractical)

Then CONFIG_RCU_FAST_NO_HZ it is.

> CONFIG_RCU_FAST_NO_HZ is more inline with idle injection in that both
> are targeted energy efficiency.
> Checking ubuntu and fedora, seems both have CONFIG_RCU_NOCB_CPU_ALL=y.

Ah, that is right -- they both do CONFIG_NO_HZ_FULL=y, which does imply
CONFIG_RCU_NOCB_CPU_ALL=y.  But SUSE does not, last I knew.

But it should be easy to set up Kconfig for this.  Make your new Kconfig
option select CONFIG_RCU_FAST_NO_HZ unless CONFIG_RCU_NOCB_CPU_ALL is
already set.  Alternatively, CONFIG_RCU_FAST_NO_HZ could be set up
something like the following:

	depends on (NO_HZ_COMMON && SMP && RCU_EXPERT) || CONFIG_THERM_THROT
	default CONFIG_THERM_THROT

But the "select RCU_FAST_NO_HZ if !RCU_NOCB_CPU_ALL" is probably cleaner.

Anyway, again, the details can be settled later.

							Thanx, Paul

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1270666

On Mon, Nov 16, 2015 at 01:51:26PM -0800, Jacob Pan wrote:
> On Mon, 16 Nov 2015 16:06:57 +0100 (CET)
> Thomas Gleixner <tglx@linutronix.de> wrote:
> 
> > >           <idle>-0     [000]    30.093474: bprint:
> > > __tick_nohz_idle_enter: JPAN: tick_nohz_stop_sched_tick 609 delta
> > > 1000000 [JP] but sees delta is exactly 1 tick away. didn't stop
> > > tick.  
> > 
> > If the delta is 1 tick then it is not supposed to stop it. Did you
> > ever try to figure out WHY it is 1 tick?
> > 
> > There are two code pathes which can set it to basemono + TICK_NSEC:
> > 
> >         if (rcu_needs_cpu(basemono, &next_rcu) ||
> >             arch_needs_cpu() || irq_work_needs_cpu()) {
> >                 next_tick = basemono + TICK_NSEC;
> >         } else {
> >                 next_tmr = get_next_timer_interrupt(basejiff,
> > basemono); ts->next_timer = next_tmr;
> >                 /* Take the next rcu event into account */
> >                 next_tick = next_rcu < next_tmr ? next_rcu : next_tmr;
> >         }
> > 
> > Can you please figure out WHY the tick is requested to continue
> > instead of blindly wreckaging the logic in that code?
> 
> Looks like the it hits in both cases during forced idle.
> + Josh
> + Paul
> 
> For the first case, it is always related to RCU. I found there are two
> CONFIG options to avoid this undesired tick in idle loop.
> 1. enable CONFIG_RCU_NOCB_CPU_ALL, offload to orcu kthreads
> 2. or enable CONFIG_RCU_FAST_NO_HZ (enter dytick idle w/ rcu callback)
> 
> Either one works but my concern is that users may not realize the
> intricate CONFIG_ options and how they translate into energy savings.
> Consulted with Josh, it seems we could add a check here to recognize
> the forced idle state and relax rcu_needs_cpu() to return false even it
> has callbacks. Since we are blocking everybody for a short time (5 ticks
> default). It should not impact synchronize and kfree rcu.

Right; as long as you're blocking *everybody*, and RCU priority boosting
doesn't come into play (meaning a real-time task is waiting on RCU
callbacks), then I don't see any harm in blocking RCU callbacks for a
while.  You'd block completion of synchronize_rcu() and similar, as well
as memory reclamation, but since you've blocked *every* CPU systemwide
then that doesn't cause a problem.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1270692

On Mon, Nov 16, 2015 at 02:32:11PM -0800, Josh Triplett wrote:
> On Mon, Nov 16, 2015 at 01:51:26PM -0800, Jacob Pan wrote:
> > On Mon, 16 Nov 2015 16:06:57 +0100 (CET)
> > Thomas Gleixner <tglx@linutronix.de> wrote:
> > 
> > > >           <idle>-0     [000]    30.093474: bprint:
> > > > __tick_nohz_idle_enter: JPAN: tick_nohz_stop_sched_tick 609 delta
> > > > 1000000 [JP] but sees delta is exactly 1 tick away. didn't stop
> > > > tick.  
> > > 
> > > If the delta is 1 tick then it is not supposed to stop it. Did you
> > > ever try to figure out WHY it is 1 tick?
> > > 
> > > There are two code pathes which can set it to basemono + TICK_NSEC:
> > > 
> > >         if (rcu_needs_cpu(basemono, &next_rcu) ||
> > >             arch_needs_cpu() || irq_work_needs_cpu()) {
> > >                 next_tick = basemono + TICK_NSEC;
> > >         } else {
> > >                 next_tmr = get_next_timer_interrupt(basejiff,
> > > basemono); ts->next_timer = next_tmr;
> > >                 /* Take the next rcu event into account */
> > >                 next_tick = next_rcu < next_tmr ? next_rcu : next_tmr;
> > >         }
> > > 
> > > Can you please figure out WHY the tick is requested to continue
> > > instead of blindly wreckaging the logic in that code?
> > 
> > Looks like the it hits in both cases during forced idle.
> > + Josh
> > + Paul
> > 
> > For the first case, it is always related to RCU. I found there are two
> > CONFIG options to avoid this undesired tick in idle loop.
> > 1. enable CONFIG_RCU_NOCB_CPU_ALL, offload to orcu kthreads
> > 2. or enable CONFIG_RCU_FAST_NO_HZ (enter dytick idle w/ rcu callback)
> > 
> > Either one works but my concern is that users may not realize the
> > intricate CONFIG_ options and how they translate into energy savings.
> > Consulted with Josh, it seems we could add a check here to recognize
> > the forced idle state and relax rcu_needs_cpu() to return false even it
> > has callbacks. Since we are blocking everybody for a short time (5 ticks
> > default). It should not impact synchronize and kfree rcu.
> 
> Right; as long as you're blocking *everybody*, and RCU priority boosting
> doesn't come into play (meaning a real-time task is waiting on RCU
> callbacks), then I don't see any harm in blocking RCU callbacks for a
> while.  You'd block completion of synchronize_rcu() and similar, as well
> as memory reclamation, but since you've blocked *every* CPU systemwide
> then that doesn't cause a problem.

True enough.  But how does RCU distinguish between this being a
normal idle cycle that might last indefinitely on the one hand and the
five-jiffy system-wide throttling on the other?  OK, maybe there is a
global variable that says that the just-now-starting idle period is
system-wide throttling.  But then what about the CPU that just went
idle 10 microseconds ago, and therefore left its timer tick running?
Fine and well, we could IPI it to wake it up and let it see that we
are now doing thermal throttling.  But then we presumably also have to
IPI it at the end of the thermal-throttling interval in order for it to
re-evaluate whether or not it should have the tick going.  :-/

On the one hand, I am sure that all of this can be made to work,
but simply having systems using thermal throttling enable either
CONFIG_RCU_NOCB_CPU_ALL or CONFIG_RCU_FAST_NO_HZ seems -way- simpler.
CONFIG_RCU_FAST_NO_HZ is probably the better choice for generic workloads,
but CONFIG_RCU_NOCB_CPU_ALL is the better choice for embedded workloads
where it is less likely that RCU callbacks will be posted with continuous
wild abandon.

Or am I missing something subtle here?

							Thanx, Paul

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1270756

On Mon, Nov 16, 2015 at 03:26:40PM -0800, Paul E. McKenney wrote:
> On Mon, Nov 16, 2015 at 02:32:11PM -0800, Josh Triplett wrote:
> > On Mon, Nov 16, 2015 at 01:51:26PM -0800, Jacob Pan wrote:
> > > On Mon, 16 Nov 2015 16:06:57 +0100 (CET)
> > > Thomas Gleixner <tglx@linutronix.de> wrote:
> > > 
> > > > >           <idle>-0     [000]    30.093474: bprint:
> > > > > __tick_nohz_idle_enter: JPAN: tick_nohz_stop_sched_tick 609 delta
> > > > > 1000000 [JP] but sees delta is exactly 1 tick away. didn't stop
> > > > > tick.  
> > > > 
> > > > If the delta is 1 tick then it is not supposed to stop it. Did you
> > > > ever try to figure out WHY it is 1 tick?
> > > > 
> > > > There are two code pathes which can set it to basemono + TICK_NSEC:
> > > > 
> > > >         if (rcu_needs_cpu(basemono, &next_rcu) ||
> > > >             arch_needs_cpu() || irq_work_needs_cpu()) {
> > > >                 next_tick = basemono + TICK_NSEC;
> > > >         } else {
> > > >                 next_tmr = get_next_timer_interrupt(basejiff,
> > > > basemono); ts->next_timer = next_tmr;
> > > >                 /* Take the next rcu event into account */
> > > >                 next_tick = next_rcu < next_tmr ? next_rcu : next_tmr;
> > > >         }
> > > > 
> > > > Can you please figure out WHY the tick is requested to continue
> > > > instead of blindly wreckaging the logic in that code?
> > > 
> > > Looks like the it hits in both cases during forced idle.
> > > + Josh
> > > + Paul
> > > 
> > > For the first case, it is always related to RCU. I found there are two
> > > CONFIG options to avoid this undesired tick in idle loop.
> > > 1. enable CONFIG_RCU_NOCB_CPU_ALL, offload to orcu kthreads
> > > 2. or enable CONFIG_RCU_FAST_NO_HZ (enter dytick idle w/ rcu callback)
> > > 
> > > Either one works but my concern is that users may not realize the
> > > intricate CONFIG_ options and how they translate into energy savings.
> > > Consulted with Josh, it seems we could add a check here to recognize
> > > the forced idle state and relax rcu_needs_cpu() to return false even it
> > > has callbacks. Since we are blocking everybody for a short time (5 ticks
> > > default). It should not impact synchronize and kfree rcu.
> > 
> > Right; as long as you're blocking *everybody*, and RCU priority boosting
> > doesn't come into play (meaning a real-time task is waiting on RCU
> > callbacks), then I don't see any harm in blocking RCU callbacks for a
> > while.  You'd block completion of synchronize_rcu() and similar, as well
> > as memory reclamation, but since you've blocked *every* CPU systemwide
> > then that doesn't cause a problem.
> 
> True enough.  But how does RCU distinguish between this being a
> normal idle cycle that might last indefinitely on the one hand and the
> five-jiffy system-wide throttling on the other?  OK, maybe there is a
> global variable that says that the just-now-starting idle period is
> system-wide throttling.  But then what about the CPU that just went
> idle 10 microseconds ago, and therefore left its timer tick running?
> Fine and well, we could IPI it to wake it up and let it see that we
> are now doing thermal throttling.  But then we presumably also have to
> IPI it at the end of the thermal-throttling interval in order for it to
> re-evaluate whether or not it should have the tick going.  :-/
> 
> On the one hand, I am sure that all of this can be made to work,
> but simply having systems using thermal throttling enable either
> CONFIG_RCU_NOCB_CPU_ALL or CONFIG_RCU_FAST_NO_HZ seems -way- simpler.
> CONFIG_RCU_FAST_NO_HZ is probably the better choice for generic workloads,
> but CONFIG_RCU_NOCB_CPU_ALL is the better choice for embedded workloads
> where it is less likely that RCU callbacks will be posted with continuous
> wild abandon.
> 
> Or am I missing something subtle here?

I agree that it seems preferable to make this require an existing RCU
solution rather than adding more complexity to the RCU idle path.  One
possible thing that may affect the choice of solution: this needs to
idle *every* CPU, without leaving any CPU awake to handle callbacks or
similar.

- Josh Triplett
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1270778

On 11/16/2015 6:53 PM, Paul E. McKenney wrote:
> Fair point.  When in the five-jiffy throttling state, what can wake up
> a CPU?  In an earlier version of this proposal, the answer was "nothing",
> but maybe that has changed.

device interrupts are likely to wake the cpus.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | linux.kernel

csiph-web