Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1310012 > unrolled thread

Re: timers: HARDIRQ-safe -> HARDIRQ-unsafe lock order detected

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: timers: HARDIRQ-safe -> HARDIRQ-unsafe lock order detected Thomas Gleixner <tglx@linutronix.de> - 2016-01-15 11:10 +0100
    Re: timers: HARDIRQ-safe -> HARDIRQ-unsafe lock order detected "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-01-15 22:20 +0100
      Re: timers: HARDIRQ-safe -> HARDIRQ-unsafe lock order detected "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-01-15 23:20 +0100
        Re: timers: HARDIRQ-safe -> HARDIRQ-unsafe lock order detected "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-01-16 00:20 +0100

#1310012 — Re: timers: HARDIRQ-safe -> HARDIRQ-unsafe lock order detected

On Thu, 14 Jan 2016, Paul E. McKenney wrote:
> > Untested patch below.
> 
> One small fix to make it build below.  Started rcutorture, somewhat
> pointlessly given that the splat doesn't appear on my setup.

Well, at least it tells us whether the change explodes by itself.

[toc] | [next] | [standalone]

#1310469

On Fri, Jan 15, 2016 at 11:03:24AM +0100, Thomas Gleixner wrote:
> On Thu, 14 Jan 2016, Paul E. McKenney wrote:
> > > Untested patch below.
> > 
> > One small fix to make it build below.  Started rcutorture, somewhat
> > pointlessly given that the splat doesn't appear on my setup.
> 
> Well, at least it tells us whether the change explodes by itself.

Hmmm...

So this is a strange one.  I have been seeing increasing instability
in mainline over the past couple of releases, with the main symptom
being that the kernel decides that awakening RCU's grace-period kthreads
is an optional activity.  The usual situation is that the kthread is
blocked for tens of seconds in an wait_event_interruptible_timeout(),
despite having a three-jiffy timeout.  Doing periodic wakeups from
the scheduling-clock interrupt seems to clear things up, but such hacks
should not be necessary.

Normally, I have to run for for some hours to have a good chance of seeing
this happen.  This change triggered in a 30-minute run.  Not only that,
but in a .config scenario that is normally very hard to trigger.  This
scenario does involve CPU hotplug, and I am re-running with CPU hotplug
disabled.

That said, I am starting to hear reports of people hitting this without
CPU hotplug operations...

							Thanx, Paul

[toc] | [prev] | [next] | [standalone]

#1310506

On Fri, Jan 15, 2016 at 01:11:25PM -0800, Paul E. McKenney wrote:
> On Fri, Jan 15, 2016 at 11:03:24AM +0100, Thomas Gleixner wrote:
> > On Thu, 14 Jan 2016, Paul E. McKenney wrote:
> > > > Untested patch below.
> > > 
> > > One small fix to make it build below.  Started rcutorture, somewhat
> > > pointlessly given that the splat doesn't appear on my setup.
> > 
> > Well, at least it tells us whether the change explodes by itself.
> 
> Hmmm...
> 
> So this is a strange one.  I have been seeing increasing instability
> in mainline over the past couple of releases, with the main symptom
> being that the kernel decides that awakening RCU's grace-period kthreads
> is an optional activity.  The usual situation is that the kthread is
> blocked for tens of seconds in an wait_event_interruptible_timeout(),
> despite having a three-jiffy timeout.  Doing periodic wakeups from
> the scheduling-clock interrupt seems to clear things up, but such hacks
> should not be necessary.
> 
> Normally, I have to run for for some hours to have a good chance of seeing
> this happen.  This change triggered in a 30-minute run.  Not only that,
> but in a .config scenario that is normally very hard to trigger.  This
> scenario does involve CPU hotplug, and I am re-running with CPU hotplug
> disabled.
> 
> That said, I am starting to hear reports of people hitting this without
> CPU hotplug operations...

And without hotplug operations, instead of dying repeatedly in 30 minutes,
it goes four hours with no complaints.  Next trying wakeups.

							Thanx, Paul

[toc] | [prev] | [next] | [standalone]

#1310522

On Fri, Jan 15, 2016 at 02:10:45PM -0800, Paul E. McKenney wrote:
> On Fri, Jan 15, 2016 at 01:11:25PM -0800, Paul E. McKenney wrote:
> > On Fri, Jan 15, 2016 at 11:03:24AM +0100, Thomas Gleixner wrote:
> > > On Thu, 14 Jan 2016, Paul E. McKenney wrote:
> > > > > Untested patch below.
> > > > 
> > > > One small fix to make it build below.  Started rcutorture, somewhat
> > > > pointlessly given that the splat doesn't appear on my setup.
> > > 
> > > Well, at least it tells us whether the change explodes by itself.
> > 
> > Hmmm...
> > 
> > So this is a strange one.  I have been seeing increasing instability
> > in mainline over the past couple of releases, with the main symptom
> > being that the kernel decides that awakening RCU's grace-period kthreads
> > is an optional activity.  The usual situation is that the kthread is
> > blocked for tens of seconds in an wait_event_interruptible_timeout(),
> > despite having a three-jiffy timeout.  Doing periodic wakeups from
> > the scheduling-clock interrupt seems to clear things up, but such hacks
> > should not be necessary.
> > 
> > Normally, I have to run for for some hours to have a good chance of seeing
> > this happen.  This change triggered in a 30-minute run.  Not only that,
> > but in a .config scenario that is normally very hard to trigger.  This
> > scenario does involve CPU hotplug, and I am re-running with CPU hotplug
> > disabled.
> > 
> > That said, I am starting to hear reports of people hitting this without
> > CPU hotplug operations...
> 
> And without hotplug operations, instead of dying repeatedly in 30 minutes,
> it goes four hours with no complaints.  Next trying wakeups.

And if I make the scheduling-clock interrupt send extra wakeups to the RCU
grace-period kthread when needed, things work even with CPU hotplug going.

The "when needed" means any time that the RCU grace-period kthread has
been sleeping three times as long as the timeout interval.  If the first
wakeup does nothing, it does another wakeup once per second.

So it looks like this change makes an existing problem much worse, as
opposed to introducing a new problem.

							Thanx, Paul

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web