Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1308691 > unrolled thread

Re: [ANNOUNCE] 4.4-rc6-rt1

Started bySebastian Andrzej Siewior <bigeasy@linutronix.de>
First post2016-01-13 19:00 +0100
Last post2016-01-14 17:10 +0100
Articles 10 — 4 participants

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [ANNOUNCE] 4.4-rc6-rt1 Sebastian Andrzej Siewior <bigeasy@linutronix.de> - 2016-01-13 19:00 +0100
    Re: [ANNOUNCE] 4.4-rc6-rt1 Grygorii Strashko <grygorii.strashko@ti.com> - 2016-01-13 19:40 +0100
      Re: [ANNOUNCE] 4.4-rc6-rt1 Sebastian Andrzej Siewior <bigeasy@linutronix.de> - 2016-01-14 16:00 +0100
    Re: [ANNOUNCE] 4.4-rc6-rt1 Mike Galbraith <umgwanakikbuti@gmail.com> - 2016-01-14 10:40 +0100
    Re: [ANNOUNCE] 4.4-rc6-rt1 Sebastian Andrzej Siewior <bigeasy@linutronix.de> - 2016-01-14 15:20 +0100
      Re: [ANNOUNCE] 4.4-rc6-rt1 Mike Galbraith <umgwanakikbuti@gmail.com> - 2016-01-14 15:40 +0100
        Re: [ANNOUNCE] 4.4-rc6-rt1 Thomas Gleixner <tglx@linutronix.de> - 2016-01-14 15:40 +0100
          Re: [ANNOUNCE] 4.4-rc6-rt1 Mike Galbraith <umgwanakikbuti@gmail.com> - 2016-01-14 16:00 +0100
            Re: [ANNOUNCE] 4.4-rc6-rt1 Thomas Gleixner <tglx@linutronix.de> - 2016-01-14 16:10 +0100
              Re: [ANNOUNCE] 4.4-rc6-rt1 Mike Galbraith <umgwanakikbuti@gmail.com> - 2016-01-14 17:10 +0100

#1308691 — Re: [ANNOUNCE] 4.4-rc6-rt1

FromSebastian Andrzej Siewior <bigeasy@linutronix.de>
Date2016-01-13 19:00 +0100
SubjectRe: [ANNOUNCE] 4.4-rc6-rt1
Message-ID<qQxSb-37P-23@gated-at.bofh.it>
* Mike Galbraith | 2016-01-01 08:19:41 [+0100]:

>>   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                      
>>     3 root      -2   0       0      0      0 S  37.3  0.0 307:52.44 ksoftirqd/0                  
>>    32 root      -2   0       0      0      0 S  37.3  0.0 308:08.72 ksoftirqd/2                  
>>    42 root      -2   0       0      0      0 R  37.3  0.0 308:32.84 ksoftirqd/3                  
>>    22 root      -2   0       0      0      0 S  26.9  0.0 222:29.82 ksoftirqd/1                  
>>     1 root      20   0   46628   6980   4976 S   1.3  0.2   0:13.98 systemd                      
>> 22358 williams  20   0  159980   4552   3780 R   1.0  0.1   0:00.39 top       
>
>Heh, I didn't notice immediately because I throttle nohz, am seeing
>only tiny utilization (but nohz idle isn't working).  With throttle
>patch removed, box is screaming, expires=4294990471 pokes eyeball.

This is due to NO_HZ as far as I can tell. My AMD A10 in idle mode has
0.7% utilisation of ksoftirqd/ with CONFIG_HZ_PERIODIC and with
CONFIG_NO_HZ_FULL it shows about 25% on all CPU threads.

Sebastian

[toc] | [next] | [standalone]

#1308723

FromGrygorii Strashko <grygorii.strashko@ti.com>
Date2016-01-13 19:40 +0100
Message-ID<qQyuS-3Ce-25@gated-at.bofh.it>
In reply to#1308691
On 01/13/2016 07:58 PM, Sebastian Andrzej Siewior wrote:
> * Mike Galbraith | 2016-01-01 08:19:41 [+0100]:
>
>>>    PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
>>>      3 root      -2   0       0      0      0 S  37.3  0.0 307:52.44 ksoftirqd/0
>>>     32 root      -2   0       0      0      0 S  37.3  0.0 308:08.72 ksoftirqd/2
>>>     42 root      -2   0       0      0      0 R  37.3  0.0 308:32.84 ksoftirqd/3
>>>     22 root      -2   0       0      0      0 S  26.9  0.0 222:29.82 ksoftirqd/1
>>>      1 root      20   0   46628   6980   4976 S   1.3  0.2   0:13.98 systemd
>>> 22358 williams  20   0  159980   4552   3780 R   1.0  0.1   0:00.39 top
>>
>> Heh, I didn't notice immediately because I throttle nohz, am seeing
>> only tiny utilization (but nohz idle isn't working).  With throttle
>> patch removed, box is screaming, expires=4294990471 pokes eyeball.
>
> This is due to NO_HZ as far as I can tell. My AMD A10 in idle mode has
> 0.7% utilisation of ksoftirqd/ with CONFIG_HZ_PERIODIC and with
> CONFIG_NO_HZ_FULL it shows about 25% on all CPU threads.
>

I might be wrong completely, but could below two patches affect on
CPU utilization of ksoftirqd?

6047967 ksoftirqd: Use new cond_resched_rcu_qs() function
28423ad ksoftirqd: Enable IRQs and call cond_resched() before poking RCU

above two patches are not applied on -RT part of softirqs processing.

static void run_ksoftirqd(unsigned int cpu)
{
	local_irq_disable();
	current->softirq_nestcnt++;

	do_current_softirqs();
	current->softirq_nestcnt--;
	rcu_note_context_switch();

^^^ IRQs disabled
	local_irq_enable();
}

-- 
regards,
-grygorii

[toc] | [prev] | [next] | [standalone]

#1309349

FromSebastian Andrzej Siewior <bigeasy@linutronix.de>
Date2016-01-14 16:00 +0100
Message-ID<qQRxw-8vf-11@gated-at.bofh.it>
In reply to#1308723
* Grygorii Strashko | 2016-01-13 20:36:56 [+0200]:

>I might be wrong completely, but could below two patches affect on
>CPU utilization of ksoftirqd?
>
>6047967 ksoftirqd: Use new cond_resched_rcu_qs() function
>28423ad ksoftirqd: Enable IRQs and call cond_resched() before poking RCU
>
>above two patches are not applied on -RT part of softirqs processing.

this has been overseen, thanks.

Sebastian

[toc] | [prev] | [next] | [standalone]

#1309115

FromMike Galbraith <umgwanakikbuti@gmail.com>
Date2016-01-14 10:40 +0100
Message-ID<qQMxQ-54U-3@gated-at.bofh.it>
In reply to#1308691
On Wed, 2016-01-13 at 18:58 +0100, Sebastian Andrzej Siewior wrote:
> * Mike Galbraith | 2016-01-01 08:19:41 [+0100]:
> 
> > >   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM    
> > >  TIME+ COMMAND                      
> > >     3 root      -2   0       0      0      0 S  37.3  0.0
> > > 307:52.44 ksoftirqd/0                  
> > >    32 root      -2   0       0      0      0 S  37.3  0.0
> > > 308:08.72 ksoftirqd/2                  
> > >    42 root      -2   0       0      0      0 R  37.3  0.0
> > > 308:32.84 ksoftirqd/3                  
> > >    22 root      -2   0       0      0      0 S  26.9  0.0
> > > 222:29.82 ksoftirqd/1                  
> > >     1 root      20   0   46628   6980   4976 S   1.3  0.2  
> > >  0:13.98 systemd                      
> > > 22358 williams  20   0  159980   4552   3780 R   1.0  0.1  
> > >  0:00.39 top       
> > 
> > Heh, I didn't notice immediately because I throttle nohz, am seeing
> > only tiny utilization (but nohz idle isn't working).  With throttle
> > patch removed, box is screaming, expires=4294990471 pokes eyeball.
> 
> This is due to NO_HZ as far as I can tell. My AMD A10 in idle mode
> has
> 0.7% utilisation of ksoftirqd/ with CONFIG_HZ_PERIODIC and with
> CONFIG_NO_HZ_FULL it shows about 25% on all CPU threads.

Thomas said the hrtimer adjustments went a little out of round.  I
started rummaging, but then the world woke up from the holidays, so I
didn't get _to_ square one, much lest past it.  Hohum.

	-Mike

[toc] | [prev] | [next] | [standalone]

#1309303

FromSebastian Andrzej Siewior <bigeasy@linutronix.de>
Date2016-01-14 15:20 +0100
Message-ID<qQQUO-8fS-5@gated-at.bofh.it>
In reply to#1308691
* Sebastian Andrzej Siewior | 2016-01-13 18:58:45 [+0100]:

>This is due to NO_HZ as far as I can tell. My AMD A10 in idle mode has
>0.7% utilisation of ksoftirqd/ with CONFIG_HZ_PERIODIC and with
>CONFIG_NO_HZ_FULL it shows about 25% on all CPU threads.

This should fixed it:

--- a/kernel/time/timer.c
+++ b/kernel/time/timer.c
@@ -1453,7 +1453,7 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem)
 	 * the base lock to check when the next timer is pending and so
 	 * we assume the next jiffy.
 	 */
-	return basej;
+	return basem + TICK_NSEC;
 #endif
 	spin_lock(&base->lock);
 	if (base->active_timers) {

Sebastian

[toc] | [prev] | [next] | [standalone]

#1309332

FromMike Galbraith <umgwanakikbuti@gmail.com>
Date2016-01-14 15:40 +0100
Message-ID<qQRe9-8oq-17@gated-at.bofh.it>
In reply to#1309303
On Thu, 2016-01-14 at 15:17 +0100, Sebastian Andrzej Siewior wrote:
> * Sebastian Andrzej Siewior | 2016-01-13 18:58:45 [+0100]:
> 
> > This is due to NO_HZ as far as I can tell. My AMD A10 in idle mode
> > has
> > 0.7% utilisation of ksoftirqd/ with CONFIG_HZ_PERIODIC and with
> > CONFIG_NO_HZ_FULL it shows about 25% on all CPU threads.
> 
> This should fixed it:
> 
> --- a/kernel/time/timer.c
> +++ b/kernel/time/timer.c
> @@ -1453,7 +1453,7 @@ u64 get_next_timer_interrupt(unsigned long
> basej, u64 basem)
>  	 * the base lock to check when the next timer is pending and
> so
>  	 * we assume the next jiffy.
>  	 */
> -	return basej;
> +	return basem + TICK_NSEC;
>  #endif
>  	spin_lock(&base->lock);
>  	if (base->active_timers) {

That's what I had done to stop the screaming interrupt, but box still
behaved very badly.

	-Mike

[toc] | [prev] | [next] | [standalone]

#1309336

FromThomas Gleixner <tglx@linutronix.de>
Date2016-01-14 15:40 +0100
Message-ID<qQRea-8oq-31@gated-at.bofh.it>
In reply to#1309332
On Thu, 14 Jan 2016, Mike Galbraith wrote:

> On Thu, 2016-01-14 at 15:17 +0100, Sebastian Andrzej Siewior wrote:
> > * Sebastian Andrzej Siewior | 2016-01-13 18:58:45 [+0100]:
> > 
> > > This is due to NO_HZ as far as I can tell. My AMD A10 in idle mode
> > > has
> > > 0.7% utilisation of ksoftirqd/ with CONFIG_HZ_PERIODIC and with
> > > CONFIG_NO_HZ_FULL it shows about 25% on all CPU threads.
> > 
> > This should fixed it:
> > 
> > --- a/kernel/time/timer.c
> > +++ b/kernel/time/timer.c
> > @@ -1453,7 +1453,7 @@ u64 get_next_timer_interrupt(unsigned long
> > basej, u64 basem)
> >  	 * the base lock to check when the next timer is pending and
> > so
> >  	 * we assume the next jiffy.
> >  	 */
> > -	return basej;
> > +	return basem + TICK_NSEC;
> >  #endif
> >  	spin_lock(&base->lock);
> >  	if (base->active_timers) {
> 
> That's what I had done to stop the screaming interrupt, but box still
> behaved very badly.

If you turn off CONFIG_NO_HZ_FULL and switch to NO_HZ_IDLE is it still bad?

Thanks,

	tglx

[toc] | [prev] | [next] | [standalone]

#1309360

FromMike Galbraith <umgwanakikbuti@gmail.com>
Date2016-01-14 16:00 +0100
Message-ID<qQRxx-8vf-39@gated-at.bofh.it>
In reply to#1309336
On Thu, 2016-01-14 at 15:30 +0100, Thomas Gleixner wrote:
> On Thu, 14 Jan 2016, Mike Galbraith wrote:
> 
> > On Thu, 2016-01-14 at 15:17 +0100, Sebastian Andrzej Siewior wrote:
> > > * Sebastian Andrzej Siewior | 2016-01-13 18:58:45 [+0100]:
> > > 
> > > > This is due to NO_HZ as far as I can tell. My AMD A10 in idle
> mode
> > > > has
> > > > 0.7% utilisation of ksoftirqd/ with CONFIG_HZ_PERIODIC and with
> > > > CONFIG_NO_HZ_FULL it shows about 25% on all CPU threads.
> > > 
> > > This should fixed it:
> > > 
> > > --- a/kernel/time/timer.c
> > > +++ b/kernel/time/timer.c
> > > @@ -1453,7 +1453,7 @@ u64 get_next_timer_interrupt(unsigned long
> > > basej, u64 basem)
> > >      * the base lock to check when the next timer is pending and
> > > so
> > >      * we assume the next jiffy.
> > >      */
> > > -   return basej;
> > > +   return basem + TICK_NSEC;
> > >  #endif
> > >     spin_lock(&base->lock);
> > >     if (base->active_timers) {
> > 
> > That's what I had done to stop the screaming interrupt, but box
> still
> > behaved very badly.
> 
> If you turn off CONFIG_NO_HZ_FULL and switch to NO_HZ_IDLE is it
> still bad?

I didn't have CONFIG_NO_HZ_FULL enabled, it was CONFIG_NO_HZ_IDLE.

	-Mike

[toc] | [prev] | [next] | [standalone]

#1309364

FromThomas Gleixner <tglx@linutronix.de>
Date2016-01-14 16:10 +0100
Message-ID<qQRHc-nE-11@gated-at.bofh.it>
In reply to#1309360
On Thu, 14 Jan 2016, Mike Galbraith wrote:

> On Thu, 2016-01-14 at 15:30 +0100, Thomas Gleixner wrote:
> > On Thu, 14 Jan 2016, Mike Galbraith wrote:
> > 
> > > On Thu, 2016-01-14 at 15:17 +0100, Sebastian Andrzej Siewior wrote:
> > > > * Sebastian Andrzej Siewior | 2016-01-13 18:58:45 [+0100]:
> > > > 
> > > > > This is due to NO_HZ as far as I can tell. My AMD A10 in idle
> > mode
> > > > > has
> > > > > 0.7% utilisation of ksoftirqd/ with CONFIG_HZ_PERIODIC and with
> > > > > CONFIG_NO_HZ_FULL it shows about 25% on all CPU threads.
> > > > 
> > > > This should fixed it:
> > > > 
> > > > --- a/kernel/time/timer.c
> > > > +++ b/kernel/time/timer.c
> > > > @@ -1453,7 +1453,7 @@ u64 get_next_timer_interrupt(unsigned long
> > > > basej, u64 basem)
> > > >      * the base lock to check when the next timer is pending and
> > > > so
> > > >      * we assume the next jiffy.
> > > >      */
> > > > -   return basej;
> > > > +   return basem + TICK_NSEC;
> > > >  #endif
> > > >     spin_lock(&base->lock);
> > > >     if (base->active_timers) {
> > > 
> > > That's what I had done to stop the screaming interrupt, but box
> > still
> > > behaved very badly.
> > 
> > If you turn off CONFIG_NO_HZ_FULL and switch to NO_HZ_IDLE is it
> > still bad?
> 
> I didn't have CONFIG_NO_HZ_FULL enabled, it was CONFIG_NO_HZ_IDLE.

So with the above fix it still behaves badly. Can you provide your config and
a hint which workload/idle/whatever state results in bad behaviour.

Thanks,

	tglx

[toc] | [prev] | [next] | [standalone]

#1309450

FromMike Galbraith <umgwanakikbuti@gmail.com>
Date2016-01-14 17:10 +0100
Message-ID<qQSDg-10I-3@gated-at.bofh.it>
In reply to#1309364

[Multipart message — attachments visible in raw view] — view raw

On Thu, 2016-01-14 at 16:07 +0100, Thomas Gleixner wrote:

> > I didn't have CONFIG_NO_HZ_FULL enabled, it was CONFIG_NO_HZ_IDLE.
> 
> So with the above fix it still behaves badly. Can you provide your config and
> a hint which workload/idle/whatever state results in bad behaviour.

This is virgin -rt1 modulo fixlet applied to v4.4.0, built with the
.config from v4.4.0 that built it (modulo RT_FULL) rebuilding itself
via make -j8.

homer:/root # vmstat 10
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  1      0 14742988 179832 680792    0    0   634    25  326 1196  2  1 89  8  0
 8  0      0 14476084 179848 701632    0    0   814   525 2421 12314 11  1 87  1  0
 8  0      0 14483232 179864 712656    0    0   165  1628 2404 12320 11  1 87  1  0
 8  0      0 14493336 180008 727836    0    0   141   762 2328 11306 11  1 87  0  0
 8  0      0 14456436 180024 738356    0    0   159  1478 2336 11939 11  1 87  0  0

Way too idle, taking forever.

	-Mike

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web