Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1573090

Re: [tip:sched/core] sched/core: Add debugging code to catch missing update_rq_clock() calls

Show all headers | View raw

On Fri, Feb 03, 2017 at 01:59:34PM +0100, Mike Galbraith wrote:
> On Fri, 2017-02-03 at 09:53 +0100, Peter Zijlstra wrote:
> > On Fri, Feb 03, 2017 at 10:03:14AM +0530, Sachin Sant wrote:
> 
> > > I ran few cycles of cpu hot(un)plug tests. In most cases it works except one
> > > where I ran into rcu stall:
> > > 
> > > [  173.493453] INFO: rcu_sched detected stalls on CPUs/tasks:
> > > [  173.493473] > > 	> > 8-...: (2 GPs behind) idle=006/140000000000000/0 softirq=0/0 fqs=2996 
> > > [  173.493476] > > 	> > (detected by 0, t=6002 jiffies, g=885, c=884, q=6350)
> > 
> > Right, I actually saw that too, but I don't think that would be related
> > to my patch. I'll see if I can dig into this though, ought to get fixed
> > regardless.
> 
> FWIW, I'm not seeing stalls/hangs while beating hotplug up in tip. (so
> next grew a wart?)

I've seen it on tip. It looks like hot unplug goes really slow when
there's running tasks on the CPU being taken down.

What I did was something like:

  taskset -p $((1<<1)) $$
  for ((i=0; i<20; i++)) do while :; do :; done & done

  taskset -p $((1<<0)) $$
  echo 0 > /sys/devices/system/cpu/cpu1/online

And with those 20 tasks stuck sucking cycles on CPU1, the unplug goes
_really_ slow and the RCU stall triggers. What I suspect happens is that
hotplug stops participating in the RCU state machine early, but only
tells RCU about it really late, and in between it gets suspicious it
takes too long.

I've yet to dig through the RCU code to figure out the exact sequence of
events, but found the above to be fairly reliable in triggering the
issue.

Back to linux.kernel | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Re: [tip:sched/core] sched/core: Add debugging code to catch missing  update_rq_clock() calls Matt Fleming <matt@codeblueprint.co.uk> - 2017-01-30 22:40 +0100
  Re: [tip:sched/core] sched/core: Add debugging code to catch missing update_rq_clock() calls Michael Ellerman <mpe@ellerman.id.au> - 2017-01-31 09:40 +0100
  Re: [tip:sched/core] sched/core: Add debugging code to catch missing  update_rq_clock() calls Sachin Sant <sachinp@linux.vnet.ibm.com> - 2017-01-31 12:10 +0100
    Re: [tip:sched/core] sched/core: Add debugging code to catch  missing update_rq_clock() calls Mike Galbraith <efault@gmx.de> - 2017-01-31 12:50 +0100
      Re: [tip:sched/core] sched/core: Add debugging code to catch missing  update_rq_clock() calls Ross Zwisler <zwisler@gmail.com> - 2017-01-31 18:30 +0100
        Re: [tip:sched/core] sched/core: Add debugging code to catch missing  update_rq_clock() calls Peter Zijlstra <peterz@infradead.org> - 2017-02-02 17:00 +0100
          Re: [tip:sched/core] sched/core: Add debugging code to catch missing  update_rq_clock() calls Matt Fleming <matt@codeblueprint.co.uk> - 2017-02-02 23:10 +0100
          Re: [tip:sched/core] sched/core: Add debugging code to catch  missing update_rq_clock() calls Mike Galbraith <efault@gmx.de> - 2017-02-03 04:10 +0100
          Re: [tip:sched/core] sched/core: Add debugging code to catch missing  update_rq_clock() calls Sachin Sant <sachinp@linux.vnet.ibm.com> - 2017-02-03 05:40 +0100
            Re: [tip:sched/core] sched/core: Add debugging code to catch missing  update_rq_clock() calls Peter Zijlstra <peterz@infradead.org> - 2017-02-03 10:00 +0100
              Re: [tip:sched/core] sched/core: Add debugging code to catch  missing update_rq_clock() calls Mike Galbraith <efault@gmx.de> - 2017-02-03 14:10 +0100
                Re: [tip:sched/core] sched/core: Add debugging code to catch missing  update_rq_clock() calls Peter Zijlstra <peterz@infradead.org> - 2017-02-03 14:40 +0100
                Re: [tip:sched/core] sched/core: Add debugging code to catch  missing update_rq_clock() calls Mike Galbraith <efault@gmx.de> - 2017-02-03 15:00 +0100
                Re: [tip:sched/core] sched/core: Add debugging code to catch missing  update_rq_clock() calls "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-02-03 16:50 +0100
                Re: [tip:sched/core] sched/core: Add debugging code to catch missing  update_rq_clock() calls "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-02-03 17:00 +0100
                Re: [tip:sched/core] sched/core: Add debugging code to catch missing  update_rq_clock() calls Sachin Sant <sachinp@linux.vnet.ibm.com> - 2017-02-06 07:30 +0100
                Re: [tip:sched/core] sched/core: Add debugging code to catch missing  update_rq_clock() calls "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-02-06 16:20 +0100
                Re: [tip:sched/core] sched/core: Add debugging code to catch missing  update_rq_clock() calls "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-02-06 16:20 +0100
          Re: [tip:sched/core] sched/core: Add debugging code to catch missing  update_rq_clock() calls Borislav Petkov <bp@alien8.de> - 2017-02-03 14:10 +0100

csiph-web