Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1300843 > unrolled thread

Re: [V9fs-developer] Hang triggered by udev coldplug, looks like a race

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [V9fs-developer] Hang triggered by udev coldplug, looks like a  race Peter Zijlstra <peterz@infradead.org> - 2016-01-04 17:00 +0100
    Re: [V9fs-developer] Hang triggered by udev coldplug, looks like a  race Dominique Martinet <dominique.martinet@cea.fr> - 2016-01-04 17:30 +0100
      Re: [V9fs-developer] Hang triggered by udev coldplug, looks like a race Andy Lutomirski <luto@amacapital.net> - 2016-01-04 20:10 +0100
        Re: [V9fs-developer] Hang triggered by udev coldplug, looks like a  race Peter Zijlstra <peterz@infradead.org> - 2016-01-04 20:30 +0100

#1300843 — Re: [V9fs-developer] Hang triggered by udev coldplug, looks like a race

On Tue, Dec 29, 2015 at 10:43:26PM -0800, Andy Lutomirski wrote:
> [add cc's]
> 
> Hi scheduler people:
> 
> This is relatively easy for me to reproduce.  Any hints for debugging
> it?  Could we really have a bug in which processes that are
> schedulable as a result of mutex unlock aren't always reliably
> scheduled?

I would expect that to cause wide-spread fail, then again, virt is known
to tickle timing issues that are improbable on actual hardware so
anything is possible.

Does it reproduce with DEBUG_MUTEXES set? (I'm not seeing a .config
here).

If its really easy you could start by tracing events/sched/sched_switch
events/sched/sched_wakeup, those would be the actual scheduling events.

Without DEBUG_MUTEXES there's the MUTEX_SPIN_ON_OWNER code that could
still confuse things, but that's mutex internal and not scheduler
related.

If it ends up being the SPIN_ON_OWNER bits we'll have to cook up some
extra debug patches.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1300872

[Multipart message — attachments visible in raw view] — view raw

Peter Zijlstra wrote on Mon, Jan 04, 2016 at 04:59:15PM +0100:
> On Tue, Dec 29, 2015 at 10:43:26PM -0800, Andy Lutomirski wrote:
> > [add cc's]
> > 
> > Hi scheduler people:
> > 
> > This is relatively easy for me to reproduce.  Any hints for debugging
> > it?  Could we really have a bug in which processes that are
> > schedulable as a result of mutex unlock aren't always reliably
> > scheduled?
> 
> I would expect that to cause wide-spread fail, then again, virt is known
> to tickle timing issues that are improbable on actual hardware so
> anything is possible.
>
> Does it reproduce with DEBUG_MUTEXES set? (I'm not seeing a .config
> here).

The config has CONFIG_DEBUG_MUTEXES=y

It got attached a while ago, reposting it here.

> If its really easy you could start by tracing events/sched/sched_switch
> events/sched/sched_wakeup, those would be the actual scheduling events.

I'm sure I've missed something in /Documentation but I'm not aware how
to trace these? (I'm happy to save Andy some precious time as I've got a
reproducer all set up now)

> Without DEBUG_MUTEXES there's the MUTEX_SPIN_ON_OWNER code that could
> still confuse things, but that's mutex internal and not scheduler
> related.
> 
> If it ends up being the SPIN_ON_OWNER bits we'll have to cook up some
> extra debug patches.

-- 
Dominique

[toc] | [prev] | [next] | [standalone]

#1300976 — Re: [V9fs-developer] Hang triggered by udev coldplug, looks like a race

On Mon, Jan 4, 2016 at 8:09 AM, Dominique Martinet
<dominique.martinet@cea.fr> wrote:
> Peter Zijlstra wrote on Mon, Jan 04, 2016 at 04:59:15PM +0100:
>> On Tue, Dec 29, 2015 at 10:43:26PM -0800, Andy Lutomirski wrote:
>> > [add cc's]
>> >
>> > Hi scheduler people:
>> >
>> > This is relatively easy for me to reproduce.  Any hints for debugging
>> > it?  Could we really have a bug in which processes that are
>> > schedulable as a result of mutex unlock aren't always reliably
>> > scheduled?
>>
>> I would expect that to cause wide-spread fail, then again, virt is known
>> to tickle timing issues that are improbable on actual hardware so
>> anything is possible.
>>
>> Does it reproduce with DEBUG_MUTEXES set? (I'm not seeing a .config
>> here).
>
> The config has CONFIG_DEBUG_MUTEXES=y
>
> It got attached a while ago, reposting it here.
>
>> If its really easy you could start by tracing events/sched/sched_switch
>> events/sched/sched_wakeup, those would be the actual scheduling events.
>
> I'm sure I've missed something in /Documentation but I'm not aware how
> to trace these? (I'm happy to save Andy some precious time as I've got a
> reproducer all set up now)

My reproducer, at least, would make this tricky -- the system ends up
mostly hung, so I don't know how I'd read out the result.  Maybe I'd
try to get something to dump the ftrace buffer to serial console after
a delay and stick all that in initramfs where it wouldn't get stuck
behind the same mutex as everything else.

Or is there a way to tell the kernel to do that for us?

--Andy
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1300986

On Mon, Jan 04, 2016 at 11:07:43AM -0800, Andy Lutomirski wrote:
> On Mon, Jan 4, 2016 at 8:09 AM, Dominique Martinet
> <dominique.martinet@cea.fr> wrote:
> > Peter Zijlstra wrote on Mon, Jan 04, 2016 at 04:59:15PM +0100:
> >> On Tue, Dec 29, 2015 at 10:43:26PM -0800, Andy Lutomirski wrote:
> >> > [add cc's]
> >> >
> >> > Hi scheduler people:
> >> >
> >> > This is relatively easy for me to reproduce.  Any hints for debugging
> >> > it?  Could we really have a bug in which processes that are
> >> > schedulable as a result of mutex unlock aren't always reliably
> >> > scheduled?
> >>
> >> I would expect that to cause wide-spread fail, then again, virt is known
> >> to tickle timing issues that are improbable on actual hardware so
> >> anything is possible.
> >>
> >> Does it reproduce with DEBUG_MUTEXES set? (I'm not seeing a .config
> >> here).
> >
> > The config has CONFIG_DEBUG_MUTEXES=y
> >
> > It got attached a while ago, reposting it here.
> >
> >> If its really easy you could start by tracing events/sched/sched_switch
> >> events/sched/sched_wakeup, those would be the actual scheduling events.
> >
> > I'm sure I've missed something in /Documentation but I'm not aware how
> > to trace these? (I'm happy to save Andy some precious time as I've got a
> > reproducer all set up now)
> 
> My reproducer, at least, would make this tricky -- the system ends up
> mostly hung, so I don't know how I'd read out the result.  Maybe I'd
> try to get something to dump the ftrace buffer to serial console after
> a delay and stick all that in initramfs where it wouldn't get stuck
> behind the same mutex as everything else.
> 
> Or is there a way to tell the kernel to do that for us?

If you can generate a core, I think crash knows how to read the ftrace
buffers from it.

  http://people.redhat.com/anderson/extensions/trace_help_trace.html

But yes, you can use one of the watchdog thingies to dump the buffers
over 'serial' too, but I suspect that will take a little longer, even
with virtual serial ports.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web