Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1697582

Re: [PATCH tip/core/rcu 02/15] rcu: Use timer as backstop for NOCB deferred wakeups

Show all headers | View raw

On Wed, Jul 26, 2017 at 05:18:01PM -0400, Steven Rostedt wrote:
> On Tue, 25 Jul 2017 17:05:40 -0700
> "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
> 
> > On Tue, Jul 25, 2017 at 06:17:10PM -0400, Steven Rostedt wrote:
> > > On Tue, 25 Jul 2017 12:18:14 -0700
> > > "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
> > >   
> > > > On Tue, Jul 25, 2017 at 02:12:20PM -0400, Steven Rostedt wrote:  
> > > > > On Mon, 24 Jul 2017 14:44:31 -0700
> > > > > "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
> > > > >     
> > > > > > The handling of RCU's no-CBs CPUs has a maintenance headache, namely
> > > > > > that if call_rcu() is invoked with interrupts disabled, the rcuo kthread
> > > > > > wakeup must be defered to a point where we can be sure that scheduler
> > > > > > locks are not held.  Of course, there are a lot of code paths leading
> > > > > > from an interrupts-disabled invocation of call_rcu(), and missing any
> > > > > > one of these can result in excessive callback-invocation latency, and
> > > > > > potentially even system hangs.    
> > > > > 
> > > > > What about using irq_work? That's what perf and ftrace use for such a
> > > > > case.    
> > > > 
> > > > I hadn't looked at irq_work before, thank you for the pointer!
> > > > 
> > > > I nevertheless believe that timers work better in this particular case
> > > > because they can be cancelled (which appears to be the common case), they  
> > > 
> > > Is the common case here that it doesn't trigger? That is, the
> > > del_timer() will be called?  
> > 
> > If you have lots of call_rcu() invocations, many of them will be invoked
> > with interrupts enabled, and a later one with interrupts enabled will
> > take care of things for the earlier ones.  So there can be workloads
> > where this is the case.
> 
> Note, only the first irq_work called will take action. The other
> callers will see that a irq_work is pending and will not reivoke one.

OK, that does make things a bit easier.

But suppose that an old irq_work has just done the wakeup on CPU 0,
but has not yet completed, and the rcuo kthead duly wakes up, does
some stuff on CPU 1 and goes to sleep, then CPU 2 gets a call_rcu()
with interrupts disabled, and therefore wants to do an irq_work again.
But the irq_work on CPU 0 is still running.

OK, this seems to be handled by clearing IRQ_WORK_PENDING before invoking
the irq_work handler.

> > > > normally are not at all time-critical, and because running in softirq
> > > > is just fine -- no need to run out of the scheduling-clock interrupt.  
> > > 
> > > irq_work doesn't always use the scheduling clock. IIRC, it will simply
> > > trigger a interrupt (if the arch supports it), and the work will be
> > > done when interrupts are enabled (the interrupt that will do the work
> > > will trigger)  
> > 
> > Ah, OK, so scheduling clock is just the backstop.  Still, softirq
> > is a bit nicer to manage than hardirq.
> 
> Still requires a hard interrupt (timer) (thinking of NOHZ FULL where
> this does matter).

But only assuming that there isn't an interrupts-enabled invocation of
call_rcu() before the timer would have gone off.  In this case, the
irq_work would still trigger, and if I didn't keep the "don't need it"
complexity of the current timer-based patch, could further result in
a spurious wakeup of the rcuo kthread, which could be just as much of
a problem for nohz_full CPUs.  (Yes, hopefully the rcuo kthread would
be placed to avoid nohz_full CPUs, but on the other hand, hopefully
code that caused call_rcu() to be invoked with interrupts disabled
would also be so placed.)

> > > > Seem reasonable?  
> > > 
> > > Don't know. With irq_work, you just call it and forget about it. No
> > > need to mod or del timers.  
> > 
> > But I could have a series of call_rcu() invocations with interrupts
> > disabled, so I would need to interact somehow with the irq_work handler.
> > Either that or dynamically allocate the needed data structure.
> > 
> > Or am I missing something here?
> 
> You treat it just like you are with the timer code. You have a irq_work
> struct attached to your rdp descriptor. And call irq_work_run() when
> interrupts are disabled. If it hasn't already been invoked it will
> invoke one. Then the irq_work handler will look at the rdp attached to
> the irq_work (container_of()), and then wake the associated thread.
> 
> It is much lighter weight than a timer setup.

How much lighter weight?  In other words, what fraction of the
timers have to avoid being cancelled for irq_work to break even?

							Thanx, Paul

Back to linux.kernel | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

[PATCH tip/core/rcu 0/15] General fixes "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-07-24 23:50 +0200
  [PATCH tip/core/rcu 04/15] rcu: Create reasonable API for do_exit() TASKS_RCU processing "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-07-24 23:50 +0200
  [PATCH tip/core/rcu 09/15] rcu: use idle versions of swait to make idle-hack clear "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-07-24 23:50 +0200
  [PATCH tip/core/rcu 02/15] rcu: Use timer as backstop for NOCB deferred wakeups "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-07-24 23:50 +0200
    Re: [PATCH tip/core/rcu 02/15] rcu: Use timer as backstop for NOCB  deferred wakeups Steven Rostedt <rostedt@goodmis.org> - 2017-07-25 20:20 +0200
      Re: [PATCH tip/core/rcu 02/15] rcu: Use timer as backstop for NOCB  deferred wakeups "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-07-25 21:20 +0200
        Re: [PATCH tip/core/rcu 02/15] rcu: Use timer as backstop for NOCB  deferred wakeups Steven Rostedt <rostedt@goodmis.org> - 2017-07-26 00:20 +0200
          Re: [PATCH tip/core/rcu 02/15] rcu: Use timer as backstop for NOCB  deferred wakeups "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-07-26 02:10 +0200
            Re: [PATCH tip/core/rcu 02/15] rcu: Use timer as backstop for NOCB  deferred wakeups Steven Rostedt <rostedt@goodmis.org> - 2017-07-26 23:20 +0200
              Re: [PATCH tip/core/rcu 02/15] rcu: Use timer as backstop for NOCB  deferred wakeups "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-07-26 23:50 +0200
                Re: [PATCH tip/core/rcu 02/15] rcu: Use timer as backstop for NOCB  deferred wakeups Steven Rostedt <rostedt@goodmis.org> - 2017-07-27 01:10 +0200
                Re: [PATCH tip/core/rcu 02/15] rcu: Use timer as backstop for NOCB  deferred wakeups "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-07-27 19:40 +0200
  [PATCH tip/core/rcu 10/15] rcu: Add TPS() protection for _rcu_barrier_trace strings "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-07-24 23:50 +0200
    Re: [PATCH tip/core/rcu 10/15] rcu: Add TPS() protection for  _rcu_barrier_trace strings Steven Rostedt <rostedt@goodmis.org> - 2017-07-28 03:50 +0200
  [PATCH tip/core/rcu 14/15] rcu: Add warning to rcu_idle_enter() for irqs enabled "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-07-24 23:50 +0200
  [PATCH tip/core/rcu 01/15] sched,rcu: Make cond_resched() provide RCU quiescent state "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-07-24 23:50 +0200
  [PATCH tip/core/rcu 08/15] swait: add idle variants which don't contribute to load average "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-07-24 23:50 +0200
  [PATCH tip/core/rcu 15/15] rcu: Remove exports from rcu_idle_exit() and rcu_idle_enter() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-07-24 23:50 +0200
  [PATCH tip/core/rcu 03/15] rcu: Drive TASKS_RCU directly off of PREEMPT "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-07-24 23:50 +0200
    Re: [PATCH tip/core/rcu 03/15] rcu: Drive TASKS_RCU directly off of  PREEMPT Steven Rostedt <rostedt@goodmis.org> - 2017-07-25 20:20 +0200
      Re: [PATCH tip/core/rcu 03/15] rcu: Drive TASKS_RCU directly off of  PREEMPT "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-07-25 21:20 +0200
  [PATCH tip/core/rcu 12/15] rcu: Add assertions verifying blocked-tasks list "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-07-24 23:50 +0200
  [PATCH tip/core/rcu 05/15] rcu: Add TPS() to event-traced strings "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-07-24 23:50 +0200
    Re: [PATCH tip/core/rcu 05/15] rcu: Add TPS() to event-traced  strings Steven Rostedt <rostedt@goodmis.org> - 2017-07-28 03:40 +0200
  [PATCH tip/core/rcu 06/15] rcu: Move rcu.h to new trivial-function style "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-07-24 23:50 +0200

csiph-web