Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1621630 > unrolled thread

There is a Tasks RCU stall warning

Started by"Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
First post2017-04-11 23:20 +0200
Last post2017-04-11 23:40 +0200
Articles 8 on this page of 28 — 2 participants

Back to article view | Back to linux.kernel

Contents

  There is a Tasks RCU stall warning "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-11 23:20 +0200
    Re: There is a Tasks RCU stall warning Steven Rostedt <rostedt@goodmis.org> - 2017-04-11 23:30 +0200
      Re: There is a Tasks RCU stall warning "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-11 23:40 +0200
    Re: There is a Tasks RCU stall warning Steven Rostedt <rostedt@goodmis.org> - 2017-04-11 23:40 +0200
      Re: There is a Tasks RCU stall warning Steven Rostedt <rostedt@goodmis.org> - 2017-04-11 23:40 +0200
        Re: There is a Tasks RCU stall warning "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-11 23:50 +0200
          Re: There is a Tasks RCU stall warning "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-12 00:00 +0200
            Re: There is a Tasks RCU stall warning Steven Rostedt <rostedt@goodmis.org> - 2017-04-12 00:20 +0200
              Re: There is a Tasks RCU stall warning "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-12 01:10 +0200
                Re: There is a Tasks RCU stall warning "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-12 01:10 +0200
                  Re: There is a Tasks RCU stall warning "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-12 01:20 +0200
                    Re: There is a Tasks RCU stall warning "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-12 05:30 +0200
                      Re: There is a Tasks RCU stall warning Steven Rostedt <rostedt@goodmis.org> - 2017-04-12 15:20 +0200
                        Re: There is a Tasks RCU stall warning "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-12 16:30 +0200
                          Re: There is a Tasks RCU stall warning Steven Rostedt <rostedt@goodmis.org> - 2017-04-12 16:50 +0200
                            Re: There is a Tasks RCU stall warning "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-12 17:20 +0200
                              Re: There is a Tasks RCU stall warning Steven Rostedt <rostedt@goodmis.org> - 2017-04-12 18:00 +0200
                                Re: There is a Tasks RCU stall warning "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-12 18:30 +0200
                                  Re: There is a Tasks RCU stall warning Steven Rostedt <rostedt@goodmis.org> - 2017-04-12 18:50 +0200
                    Re: There is a Tasks RCU stall warning "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-12 16:50 +0200
                      Re: There is a Tasks RCU stall warning Steven Rostedt <rostedt@goodmis.org> - 2017-04-12 17:00 +0200
                        Re: There is a Tasks RCU stall warning "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-12 18:30 +0200
                          Re: There is a Tasks RCU stall warning Steven Rostedt <rostedt@goodmis.org> - 2017-04-12 19:00 +0200
                            Re: There is a Tasks RCU stall warning "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-12 19:10 +0200
                              Re: There is a Tasks RCU stall warning Steven Rostedt <rostedt@goodmis.org> - 2017-04-12 19:20 +0200
                                Re: There is a Tasks RCU stall warning "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-12 22:10 +0200
          Re: There is a Tasks RCU stall warning Steven Rostedt <rostedt@goodmis.org> - 2017-04-12 00:00 +0200
    Re: There is a Tasks RCU stall warning Steven Rostedt <rostedt@goodmis.org> - 2017-04-11 23:40 +0200

Page 2 of 2 — ← Prev page 1 [2]

#1622211

FromSteven Rostedt <rostedt@goodmis.org>
Date2017-04-12 17:00 +0200
Message-ID<tvrUu-5IE-17@gated-at.bofh.it>
In reply to#1622201
On Wed, 12 Apr 2017 07:48:00 -0700
"Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:

> > > > Like this?  (Untested, but builds at least some of the time.)  
> > > 
> > > Not like that...  :-/  Update on its way.  
> > 
> > Perhaps more like this.  Started rcutorture on it, will see how it goes.  

I just love the above discussion with yourself ;-)

> 
> Do you need this patch?  If so, I should do some more work on it to
> eliminate the extra common-case branch on the scheduler fastpath.
> 

Do I still need this patch? Maybe. :-)

I changed my benchmark test to call cond_resched_rcu_qs() instead and
that appears to fix the issue. But I'm not sure if there's any other
kthread out there that just calls cond_resched() or schedule().

Actually, I think it is still a good idea to have it. I believe that it
will still allow synchronize_rcu_tasks() to progress even if there's a
kthread task that is constantly being woken up, and never sleeps when
it calls schedule(), as it may always have the R state.

-- Steve

[toc] | [prev] | [next] | [standalone]

#1622302

From"Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Date2017-04-12 18:30 +0200
Message-ID<tvtjA-6IX-31@gated-at.bofh.it>
In reply to#1622211
On Wed, Apr 12, 2017 at 10:59:37AM -0400, Steven Rostedt wrote:
> On Wed, 12 Apr 2017 07:48:00 -0700
> "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
> 
> > > > > Like this?  (Untested, but builds at least some of the time.)  
> > > > 
> > > > Not like that...  :-/  Update on its way.  
> > > 
> > > Perhaps more like this.  Started rcutorture on it, will see how it goes.  
> 
> I just love the above discussion with yourself ;-)

Talking to oneself used to cause passersby to get really bent out of
shape.  But one big benefit of ubiquitous cellphones is that now
people just assume that you are talking on a cellphone that they
cannot see.  ;-)

> > Do you need this patch?  If so, I should do some more work on it to
> > eliminate the extra common-case branch on the scheduler fastpath.
> 
> Do I still need this patch? Maybe. :-)
> 
> I changed my benchmark test to call cond_resched_rcu_qs() instead and
> that appears to fix the issue. But I'm not sure if there's any other
> kthread out there that just calls cond_resched() or schedule().
> 
> Actually, I think it is still a good idea to have it. I believe that it
> will still allow synchronize_rcu_tasks() to progress even if there's a
> kthread task that is constantly being woken up, and never sleeps when
> it calls schedule(), as it may always have the R state.

OK, will optimize it a bit.  When are you planning to get this in?

							Thanx, Paul

[toc] | [prev] | [next] | [standalone]

#1622356

FromSteven Rostedt <rostedt@goodmis.org>
Date2017-04-12 19:00 +0200
Message-ID<tvtMD-6Uu-33@gated-at.bofh.it>
In reply to#1622302
On Wed, 12 Apr 2017 09:27:54 -0700
"Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:

> On Wed, Apr 12, 2017 at 10:59:37AM -0400, Steven Rostedt wrote:
> > On Wed, 12 Apr 2017 07:48:00 -0700
> > "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
> >   
> > > > > > Like this?  (Untested, but builds at least some of the time.)    
> > > > > 
> > > > > Not like that...  :-/  Update on its way.    
> > > > 
> > > > Perhaps more like this.  Started rcutorture on it, will see how it goes.    
> > 
> > I just love the above discussion with yourself ;-)  
> 
> Talking to oneself used to cause passersby to get really bent out of
> shape.  But one big benefit of ubiquitous cellphones is that now
> people just assume that you are talking on a cellphone that they
> cannot see.  ;-)
> 
> > > Do you need this patch?  If so, I should do some more work on it to
> > > eliminate the extra common-case branch on the scheduler fastpath.  
> > 
> > Do I still need this patch? Maybe. :-)
> > 
> > I changed my benchmark test to call cond_resched_rcu_qs() instead and
> > that appears to fix the issue. But I'm not sure if there's any other
> > kthread out there that just calls cond_resched() or schedule().
> > 
> > Actually, I think it is still a good idea to have it. I believe that it
> > will still allow synchronize_rcu_tasks() to progress even if there's a
> > kthread task that is constantly being woken up, and never sleeps when
> > it calls schedule(), as it may always have the R state.  
> 
> OK, will optimize it a bit.  When are you planning to get this in?
> 

Well, I added the use case for synchronize_rcu_tasks() in my current
for-next. I'll have to make sure I get the schedule_idle() in as well
as my update to the event benchmark thread as well.

I don't think anything will truly break without it yet. But that's
assuming there's not another kernel thread somewhere that just spins
calling schedule.

And this patch will still speed up those that do call
synchronize_rcu_tasks(). But that's an optimization and not really a
fix.

-- Steve

[toc] | [prev] | [next] | [standalone]

#1622376

From"Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Date2017-04-12 19:10 +0200
Message-ID<tvtWh-7dn-9@gated-at.bofh.it>
In reply to#1622356
On Wed, Apr 12, 2017 at 12:57:22PM -0400, Steven Rostedt wrote:
> On Wed, 12 Apr 2017 09:27:54 -0700
> "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
> 
> > On Wed, Apr 12, 2017 at 10:59:37AM -0400, Steven Rostedt wrote:
> > > On Wed, 12 Apr 2017 07:48:00 -0700
> > > "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
> > >   
> > > > > > > Like this?  (Untested, but builds at least some of the time.)    
> > > > > > 
> > > > > > Not like that...  :-/  Update on its way.    
> > > > > 
> > > > > Perhaps more like this.  Started rcutorture on it, will see how it goes.    
> > > 
> > > I just love the above discussion with yourself ;-)  
> > 
> > Talking to oneself used to cause passersby to get really bent out of
> > shape.  But one big benefit of ubiquitous cellphones is that now
> > people just assume that you are talking on a cellphone that they
> > cannot see.  ;-)
> > 
> > > > Do you need this patch?  If so, I should do some more work on it to
> > > > eliminate the extra common-case branch on the scheduler fastpath.  
> > > 
> > > Do I still need this patch? Maybe. :-)
> > > 
> > > I changed my benchmark test to call cond_resched_rcu_qs() instead and
> > > that appears to fix the issue. But I'm not sure if there's any other
> > > kthread out there that just calls cond_resched() or schedule().
> > > 
> > > Actually, I think it is still a good idea to have it. I believe that it
> > > will still allow synchronize_rcu_tasks() to progress even if there's a
> > > kthread task that is constantly being woken up, and never sleeps when
> > > it calls schedule(), as it may always have the R state.  
> > 
> > OK, will optimize it a bit.  When are you planning to get this in?
> > 
> 
> Well, I added the use case for synchronize_rcu_tasks() in my current
> for-next. I'll have to make sure I get the schedule_idle() in as well
> as my update to the event benchmark thread as well.
> 
> I don't think anything will truly break without it yet. But that's
> assuming there's not another kernel thread somewhere that just spins
> calling schedule.
> 
> And this patch will still speed up those that do call
> synchronize_rcu_tasks(). But that's an optimization and not really a
> fix.

The upcoming v4.12 merge window, then?

							Thanx, Paul

[toc] | [prev] | [next] | [standalone]

#1622392

FromSteven Rostedt <rostedt@goodmis.org>
Date2017-04-12 19:20 +0200
Message-ID<tvu5Y-7gS-19@gated-at.bofh.it>
In reply to#1622376
On Wed, 12 Apr 2017 10:07:16 -0700
"Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:


> > > OK, will optimize it a bit.  When are you planning to get this in?
> > >   
> > 
> > Well, I added the use case for synchronize_rcu_tasks() in my current
> > for-next. I'll have to make sure I get the schedule_idle() in as well
> > as my update to the event benchmark thread as well.
> > 
> > I don't think anything will truly break without it yet. But that's
> > assuming there's not another kernel thread somewhere that just spins
> > calling schedule.
> > 
> > And this patch will still speed up those that do call
> > synchronize_rcu_tasks(). But that's an optimization and not really a
> > fix.  
> 
> The upcoming v4.12 merge window, then?

Yep.

-- Steve

[toc] | [prev] | [next] | [standalone]

#1622508

From"Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Date2017-04-12 22:10 +0200
Message-ID<tvwKu-Gu-13@gated-at.bofh.it>
In reply to#1622392
On Wed, Apr 12, 2017 at 01:13:25PM -0400, Steven Rostedt wrote:
> On Wed, 12 Apr 2017 10:07:16 -0700
> "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
> 
> 
> > > > OK, will optimize it a bit.  When are you planning to get this in?
> > > >   
> > > 
> > > Well, I added the use case for synchronize_rcu_tasks() in my current
> > > for-next. I'll have to make sure I get the schedule_idle() in as well
> > > as my update to the event benchmark thread as well.
> > > 
> > > I don't think anything will truly break without it yet. But that's
> > > assuming there's not another kernel thread somewhere that just spins
> > > calling schedule.
> > > 
> > > And this patch will still speed up those that do call
> > > synchronize_rcu_tasks(). But that's an optimization and not really a
> > > fix.  
> > 
> > The upcoming v4.12 merge window, then?
> 
> Yep.

Also, is the default 10-minute stall warning OK?  For purposes of
comparison, when running rcutorture, Tasks RCU grace periods take 2-3
seconds, but of course your mileage may vary.

For whatever it is worth, I set the default ratio of the Tasks RCU
stall-warning time to its average rcutorture grace period to be about
the same as the ratio for other RCU flavors.

And the kernel boot parameter rcupdate.rcu_task_stall_timeout allows
you to set whatever time you want (in jiffies) at boot time.  And also
modify it at runtime via sysfs.

							Thanx, Paul

[toc] | [prev] | [next] | [standalone]

#1621651

FromSteven Rostedt <rostedt@goodmis.org>
Date2017-04-12 00:00 +0200
Message-ID<tvbZn-4ay-3@gated-at.bofh.it>
In reply to#1621647
On Tue, 11 Apr 2017 14:44:43 -0700
"Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:

 
> Works for me!
> 
> Hopefully it will also work for your computer.  :-)
> 
> And whew!  Glad to see that the stall warnings worked!

Ah! but I think I found a bug in synchronize_rcu_tasks()!

Calling schedule isn't good enough. For rcu_tasks to continue, the task
needs to schedule out. With my updated code, I just triggered:

[  196.276868] INFO: rcu_tasks detected stalls on tasks:
[  196.284294] ffff8800c26f8040: .. nvcsw: 2/2 holdout: 1 idle_cpu: -1/1
[  196.293175] event_benchmark R  running task    30536  1127      2 0x10000000
[  196.302746] Call Trace:
[  196.307640]  ? _raw_spin_unlock_irq+0x1f/0x50
[  196.314453]  __schedule+0x222/0x1210
[  196.320476]  ? pci_mmcfg_check_reserved+0xc0/0xc0
[  196.327616]  ? preempt_count_add+0xb7/0xf0
[  196.334174]  ? __asan_store8+0x15/0x70
[  196.340384]  schedule+0x57/0xe0
[  196.345888]  benchmark_event_kthread+0x2e/0x3c0
[  196.352823]  kthread+0x178/0x1d0
[  196.358411]  ? trace_benchmark_reg+0x80/0x80
[  196.365073]  ? kthread_create_on_node+0xa0/0xa0
[  196.371999]  ret_from_fork+0x2e/0x40


And here my benchmark called schedule(), but nothing scheduled it out,
and it still fails on rcu_tasks.

-- Steve

[toc] | [prev] | [next] | [standalone]

#1621641

FromSteven Rostedt <rostedt@goodmis.org>
Date2017-04-11 23:40 +0200
Message-ID<tvbG1-43u-3@gated-at.bofh.it>
In reply to#1621630
On Tue, 11 Apr 2017 14:18:02 -0700
"Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:

> Hello, Steve,
> 
> Network connectivity issues...  :-/
> 
> There is already a Tasks RCU stall warning, but it waits for
> ten -minutes- before complaining.  You can change this with the
> rcupdate.rcu_task_stall_timeout kernel boot parameter, if you wish.
> 
> Or did you wait longer than ten minutes?
> 

Remember when I said I had a bunch of debugging turned on in this
kernel. Well, I also included my ftrace trace_event benchmark code.
Which appears to be what is triggering this.

[  205.159860] INFO: rcu_tasks detected stalls on tasks:
[  205.165042] ffff8800c40450c0: .. nvcsw: 2/2 holdout: 1 idle_cpu: -1/1
[  205.171627] event_benchmark R  running task    30224  1113      2 0x10000000
[  205.178829] Call Trace:
[  205.181379]  __schedule+0x574/0x1210
[  205.185061]  ? __schedule+0x574/0x1210
[  205.188917]  ? mark_held_locks+0x23/0xc0
[  205.192944]  ? mark_held_locks+0x23/0xc0
[  205.196964]  ? retint_kernel+0x2d/0x2d
[  205.200831]  ? retint_kernel+0x2d/0x2d
[  205.204679]  ? trace_hardirqs_on_caller+0x182/0x280
[  205.209660]  ? trace_hardirqs_on_thunk+0x1a/0x1c
[  205.214380]  ? trace_hardirqs_on_thunk+0x1a/0x1c
[  205.219097]  irq_exit+0x91/0x100
[  205.222418]  ? retint_kernel+0x2d/0x2d
[  205.226263]  ? retint_kernel+0x2d/0x2d
[  205.230113]  ? kthread_should_stop+0x3d/0x60
[  205.234486]  ? __asan_load8+0x11/0x70
[  205.238245]  ? ring_buffer_record_is_on+0x11/0x20
[  205.243052]  ? tracing_is_on+0x15/0x30
[  205.246895]  ? benchmark_event_kthread+0x4f/0x3c0
[  205.251708]  ? kthread+0x178/0x1d0
[  205.255200]  ? trace_benchmark_reg+0x80/0x80
[  205.259563]  ? kthread_create_on_node+0xa0/0xa0
[  205.264205]  ? ret_from_fork+0x2e/0x40

The thread gets created when I enable the benchmark tracepoint. It just
so happens that my test enables *all* tracepoints, which would of
course include this one as well.

I'll have to look at this code to see why it is getting missed.

-- Steve

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | linux.kernel

csiph-web