Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1294269 > unrolled thread

Re: [V9fs-developer] Hang triggered by udev coldplug, looks like a race

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [V9fs-developer] Hang triggered by udev coldplug, looks like a race Andy Lutomirski <luto@amacapital.net> - 2015-12-17 22:10 +0100
    Re: [V9fs-developer] Hang triggered by udev coldplug, looks like a  race Dominique Martinet <dominique.martinet@cea.fr> - 2015-12-24 12:00 +0100
      Re: [V9fs-developer] Hang triggered by udev coldplug, looks like a race Andy Lutomirski <luto@amacapital.net> - 2015-12-30 07:50 +0100

#1294269 — Re: [V9fs-developer] Hang triggered by udev coldplug, looks like a race

On Tue, Dec 8, 2015 at 10:45 PM, Al Viro <viro@zeniv.linux.org.uk> wrote:
> On Wed, Dec 09, 2015 at 07:23:16AM +0100, Dominique Martinet wrote:
>> Andy Lutomirski wrote on Tue, Dec 08, 2015:
>> > Trace attached.  I don't see anything wrong, but I also don't know
>> > what I'm looking for.
>>
>> Actually doesn't look good, not sure if trace could be missing messages
>> but it looks like tags get reused...
>>
>> Quick and dirty parse script (attached output, it doesn't handle TFLUSH
>> but there are none in his message):
>
> There's also not a single RCLUNK in there, which looks like missing packets.
> AFAICS, all (apparent) tag reuses caught by your awk script are of that nature.
> And all RERROR are plain "no such file" in response to TWALK; all those
> TCLUNK in there get no reply whatsoever - neither success nor failure...

This could be QEMU's analysis script screwing up.  Is there a good way
for me to gather more info?

--Andy
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1297813 — Re: [V9fs-developer] Hang triggered by udev coldplug, looks like a race

Andy Lutomirski wrote on Thu, Dec 17, 2015:
> This could be QEMU's analysis script screwing up.  Is there a good way
> for me to gather more info?

I finally took some time to reproduce it (sorry for the delay)

Using your config, virtme commit (17363c2) and kernel tag v4.4-rc3 I was
able to reproduce it just fine with my qemu (2.4.90)

Now for the fun bit... I ran it with a gdb server, attaching gdb and
running cont always 'unblocks' it
Using the kernel gdb scripts (lx-ps) I see about 250 kworker threads
running, the backtraces all look the same:

[   20.273945]  [<ffffffff818c3910>] schedule+0x30/0x80
[   20.274644]  [<ffffffff818c3c39>] schedule_preempt_disabled+0x9/0x10
[   20.275539]  [<ffffffff818c6147>] __mutex_lock_slowpath+0x107/0x2f0
[   20.276421]  [<ffffffff811cf02e>] ? lookup_fast+0xbe/0x320
[   20.277195]  [<ffffffff818c6345>] mutex_lock+0x15/0x30
[   20.277916]  [<ffffffff811d0df7>] walk_component+0x1a7/0x270


so given it unblocks after hooking gdb + cont I'm actually thinking this
might be a pure scheduling issue? (e.g. thread is never re-scheduled or
something like that?)
I can't see any task not in schedule() in your sysrq dump task
transcript either.


Not sure how to go around debugging that, to be honest.
I've tried both default one virtual cpu and -smp 3 or 4 and both can
reproduce it; cpu usage on the host is always low so it doesn't look
like there's any busy-polling involved.. This is a pretty subtle bug we
have there..

-- 
Dominique Martinet
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1299256

[add cc's]

Hi scheduler people:

This is relatively easy for me to reproduce.  Any hints for debugging
it?  Could we really have a bug in which processes that are
schedulable as a result of mutex unlock aren't always reliably
scheduled?

--Andy

On Thu, Dec 24, 2015 at 2:51 AM, Dominique Martinet
<dominique.martinet@cea.fr> wrote:
> Andy Lutomirski wrote on Thu, Dec 17, 2015:
>> This could be QEMU's analysis script screwing up.  Is there a good way
>> for me to gather more info?
>
> I finally took some time to reproduce it (sorry for the delay)
>
> Using your config, virtme commit (17363c2) and kernel tag v4.4-rc3 I was
> able to reproduce it just fine with my qemu (2.4.90)
>
> Now for the fun bit... I ran it with a gdb server, attaching gdb and
> running cont always 'unblocks' it
> Using the kernel gdb scripts (lx-ps) I see about 250 kworker threads
> running, the backtraces all look the same:
>
> [   20.273945]  [<ffffffff818c3910>] schedule+0x30/0x80
> [   20.274644]  [<ffffffff818c3c39>] schedule_preempt_disabled+0x9/0x10
> [   20.275539]  [<ffffffff818c6147>] __mutex_lock_slowpath+0x107/0x2f0
> [   20.276421]  [<ffffffff811cf02e>] ? lookup_fast+0xbe/0x320
> [   20.277195]  [<ffffffff818c6345>] mutex_lock+0x15/0x30
> [   20.277916]  [<ffffffff811d0df7>] walk_component+0x1a7/0x270
>
>
> so given it unblocks after hooking gdb + cont I'm actually thinking this
> might be a pure scheduling issue? (e.g. thread is never re-scheduled or
> something like that?)
> I can't see any task not in schedule() in your sysrq dump task
> transcript either.
>
>
> Not sure how to go around debugging that, to be honest.
> I've tried both default one virtual cpu and -smp 3 or 4 and both can
> reproduce it; cpu usage on the host is always low so it doesn't look
> like there's any busy-polling involved.. This is a pretty subtle bug we
> have there..
>
> --
> Dominique Martinet



-- 
Andy Lutomirski
AMA Capital Management, LLC
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web