Re: [V9fs-developer] Hang triggered by udev coldplug, looks like a race

From	Peter Zijlstra <peterz@infradead.org>
Newsgroups	linux.kernel
Subject	Re: [V9fs-developer] Hang triggered by udev coldplug, looks like a race
Date	2016-01-04 20:30 +0100
Message-ID	<qNiZk-7GT-3@gated-at.bofh.it> (permalink)
References	(5 earlier) <qJbMK-5Gd-7@gated-at.bofh.it> <qLiK6-33r-11@gated-at.bofh.it> <qNfI6-5oL-9@gated-at.bofh.it> <qNgb7-5PP-3@gated-at.bofh.it> <qNiFY-7z5-25@gated-at.bofh.it>
Organization	linux.* mail to news gateway

Show all headers | View raw

On Mon, Jan 04, 2016 at 11:07:43AM -0800, Andy Lutomirski wrote:
> On Mon, Jan 4, 2016 at 8:09 AM, Dominique Martinet
> <dominique.martinet@cea.fr> wrote:
> > Peter Zijlstra wrote on Mon, Jan 04, 2016 at 04:59:15PM +0100:
> >> On Tue, Dec 29, 2015 at 10:43:26PM -0800, Andy Lutomirski wrote:
> >> > [add cc's]
> >> >
> >> > Hi scheduler people:
> >> >
> >> > This is relatively easy for me to reproduce.  Any hints for debugging
> >> > it?  Could we really have a bug in which processes that are
> >> > schedulable as a result of mutex unlock aren't always reliably
> >> > scheduled?
> >>
> >> I would expect that to cause wide-spread fail, then again, virt is known
> >> to tickle timing issues that are improbable on actual hardware so
> >> anything is possible.
> >>
> >> Does it reproduce with DEBUG_MUTEXES set? (I'm not seeing a .config
> >> here).
> >
> > The config has CONFIG_DEBUG_MUTEXES=y
> >
> > It got attached a while ago, reposting it here.
> >
> >> If its really easy you could start by tracing events/sched/sched_switch
> >> events/sched/sched_wakeup, those would be the actual scheduling events.
> >
> > I'm sure I've missed something in /Documentation but I'm not aware how
> > to trace these? (I'm happy to save Andy some precious time as I've got a
> > reproducer all set up now)
> 
> My reproducer, at least, would make this tricky -- the system ends up
> mostly hung, so I don't know how I'd read out the result.  Maybe I'd
> try to get something to dump the ftrace buffer to serial console after
> a delay and stick all that in initramfs where it wouldn't get stuck
> behind the same mutex as everything else.
> 
> Or is there a way to tell the kernel to do that for us?

If you can generate a core, I think crash knows how to read the ftrace
buffers from it.

  http://people.redhat.com/anderson/extensions/trace_help_trace.html

But yes, you can use one of the watchdog thingies to dump the buffers
over 'serial' too, but I suspect that will take a little longer, even
with virtual serial ports.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Back to linux.kernel | Previous | Next — Previous in thread | Find similar | Unroll thread

Thread

Re: [V9fs-developer] Hang triggered by udev coldplug, looks like a  race Peter Zijlstra <peterz@infradead.org> - 2016-01-04 17:00 +0100
  Re: [V9fs-developer] Hang triggered by udev coldplug, looks like a  race Dominique Martinet <dominique.martinet@cea.fr> - 2016-01-04 17:30 +0100
    Re: [V9fs-developer] Hang triggered by udev coldplug, looks like a race Andy Lutomirski <luto@amacapital.net> - 2016-01-04 20:10 +0100
      Re: [V9fs-developer] Hang triggered by udev coldplug, looks like a  race Peter Zijlstra <peterz@infradead.org> - 2016-01-04 20:30 +0100

csiph-web