Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1484652 > unrolled thread

Crashes in next-20160915 (BUG at fs/notify/notification.c:66!)

Started byGuenter Roeck <linux@roeck-us.net>
First post2016-09-16 04:30 +0200
Last post2016-09-20 03:20 +0200
Articles 8 — 4 participants

Back to article view | Back to linux.kernel

Contents

  Crashes in next-20160915 (BUG at fs/notify/notification.c:66!) Guenter Roeck <linux@roeck-us.net> - 2016-09-16 04:30 +0200
    Re: Crashes in next-20160915 (BUG at fs/notify/notification.c:66!) Jan Kara <jack@suse.cz> - 2016-09-16 11:20 +0200
      Re: Crashes in next-20160915 (BUG at fs/notify/notification.c:66!) Guenter Roeck <linux@roeck-us.net> - 2016-09-16 13:20 +0200
        Re: Crashes in next-20160915 (BUG at fs/notify/notification.c:66!) Jan Kara <jack@suse.cz> - 2016-09-16 15:00 +0200
          Re: Crashes in next-20160915 (BUG at fs/notify/notification.c:66!) Andrew Morton <akpm@linux-foundation.org> - 2016-09-20 03:20 +0200
            Re: Crashes in next-20160915 (BUG at fs/notify/notification.c:66!) Michael Ellerman <mpe@ellerman.id.au> - 2016-09-20 14:10 +0200
              Re: Crashes in next-20160915 (BUG at fs/notify/notification.c:66!) Jan Kara <jack@suse.cz> - 2016-09-21 11:20 +0200
          Re: Crashes in next-20160915 (BUG at fs/notify/notification.c:66!) Michael Ellerman <mpe@ellerman.id.au> - 2016-09-20 03:20 +0200

#1484652 — Crashes in next-20160915 (BUG at fs/notify/notification.c:66!)

FromGuenter Roeck <linux@roeck-us.net>
Date2016-09-16 04:30 +0200
SubjectCrashes in next-20160915 (BUG at fs/notify/notification.c:66!)
Message-ID<shR4B-6UH-7@gated-at.bofh.it>
Hi,

I see various architectures crashing in -next with the following error.

------------[ cut here ]------------
kernel BUG at fs/notify/notification.c:66!
invalid opcode: 0000 [#1] PREEMPT
Modules linked in:
CPU: 0 PID: 110 Comm: udevd Not tainted 4.8.0-rc6-next-20160915-yocto-standard #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.9.1-0-gb3ef39f-prebuilt.qemu-project.org 04/01/2014
task: ffff88000dd58000 task.stack: ffffc90000250000
RIP: 0010:[<ffffffff811bb399>]  [<ffffffff811bb399>] fsnotify_notify_queue_is_empty+0x9/0x10
RSP: 0018:ffffc90000253e68  EFLAGS: 00000296
RAX: ffff88000dd58000 RBX: ffff88000dc9a9c0 RCX: 0000000000011ff0
RDX: ffffffff811bd52a RSI: ffff88000e3c1258 RDI: ffff88000dc9a9c0
RBP: ffffc90000253e68 R08: ffff88000ffd37e0 R09: ffff88000dcdd600
R10: ffff88000dcdd600 R11: 0000000000000000 R12: 0000000000000001
R13: ffff88000dd28c00 R14: ffff88000dc0a380 R15: ffff88000dc0a398
FS:  00007ff5bea85740(0000) GS:ffffffff81c31000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff5bea79000 CR3: 000000000dd63000 CR4: 00000000000406f0
Stack:
  ffffc90000253e88 ffffffff811bd532 0000000000000000 ffff88000dd28c00
  ffffc90000253f48 ffffffff811bfeba ffffffff00000006 0000000000000000
  ffff88000dcdd600 ffff88000dcdd600 0000001900253f58 0000040000080002
Call Trace:
  [<ffffffff811bd532>] inotify_poll+0x42/0x70
  [<ffffffff811bfeba>] SyS_epoll_ctl+0x84a/0xf60
  [<ffffffff811be4a0>] ? ep_send_events_proc+0x180/0x180
  [<ffffffff8176be98>] entry_SYSCALL_64_fastpath+0x13/0x8f
Code: 90 90 0f 1f 44 00 00 55 b8 01 00 00 00 48 89 e5 0f c1 05 bb e4 d4 00 83 c0 01 5d c3 66 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 e5 <0f> 0b 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 e5 48 83 ec 10 48
RIP  [<ffffffff811bb399>] fsnotify_notify_queue_is_empty+0x9/0x10
  RSP <ffffc90000253e68>
---[ end trace 7dc4a27003f0b575 ]---

I didn't bisect, but I would guess the culprit is one of the new patches in the
affected file.

22e9cf146d3b fanotify: fix possible false warning when freeing events
ced89591817c fsnotify: convert notification_mutex to a spinlock
f82fa3d0e7f5 fsnotify: drop notification_mutex before destroying event
782fbc7e8685 fanotify: fix list corruption in fanotify_get_response()
56cf1c8a1b35 fsnotify: add a way to stop queueing events on group shutdown

Guenter

[toc] | [next] | [standalone]

#1484829

FromJan Kara <jack@suse.cz>
Date2016-09-16 11:20 +0200
Message-ID<shXtn-2IP-1@gated-at.bofh.it>
In reply to#1484652
Hi,

On Thu 15-09-16 19:20:10, Guenter Roeck wrote:
> I see various architectures crashing in -next with the following error.
> 
> ------------[ cut here ]------------
> kernel BUG at fs/notify/notification.c:66!

Thanks for report! 

...

> Call Trace:
>  [<ffffffff811bd532>] inotify_poll+0x42/0x70
>  [<ffffffff811bfeba>] SyS_epoll_ctl+0x84a/0xf60
>  [<ffffffff811be4a0>] ? ep_send_events_proc+0x180/0x180
>  [<ffffffff8176be98>] entry_SYSCALL_64_fastpath+0x13/0x8f
> Code: 90 90 0f 1f 44 00 00 55 b8 01 00 00 00 48 89 e5 0f c1 05 bb e4 d4 00 83 c0 01 5d c3 66 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 e5 <0f> 0b 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 e5 48 83 ec 10 48
> RIP  [<ffffffff811bb399>] fsnotify_notify_queue_is_empty+0x9/0x10
>  RSP <ffffc90000253e68>
> ---[ end trace 7dc4a27003f0b575 ]---
> 
> I didn't bisect, but I would guess the culprit is one of the new patches in the
> affected file.
> 
> 22e9cf146d3b fanotify: fix possible false warning when freeing events
> ced89591817c fsnotify: convert notification_mutex to a spinlock
> f82fa3d0e7f5 fsnotify: drop notification_mutex before destroying event
> 782fbc7e8685 fanotify: fix list corruption in fanotify_get_response()
> 56cf1c8a1b35 fsnotify: add a way to stop queueing events on group shutdown

Yes, very likely my patches cause this. But it must be some config issue
because my test machine does not hit this and the code is "obviously
correct" - famous last words ;). I've seen zero-day robot complain as well
so I'll try the config it had sent me and see what's going on (likely
spin_is_locked() does not do what I thought under some configs...).

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

[toc] | [prev] | [next] | [standalone]

#1484886

FromGuenter Roeck <linux@roeck-us.net>
Date2016-09-16 13:20 +0200
Message-ID<shZlv-3Xl-21@gated-at.bofh.it>
In reply to#1484829
On 09/15/2016 11:45 PM, Jan Kara wrote:
> Hi,
>
> On Thu 15-09-16 19:20:10, Guenter Roeck wrote:
>> I see various architectures crashing in -next with the following error.
>>
>> ------------[ cut here ]------------
>> kernel BUG at fs/notify/notification.c:66!
>
> Thanks for report!
>
> ...
>
>> Call Trace:
>>  [<ffffffff811bd532>] inotify_poll+0x42/0x70
>>  [<ffffffff811bfeba>] SyS_epoll_ctl+0x84a/0xf60
>>  [<ffffffff811be4a0>] ? ep_send_events_proc+0x180/0x180
>>  [<ffffffff8176be98>] entry_SYSCALL_64_fastpath+0x13/0x8f
>> Code: 90 90 0f 1f 44 00 00 55 b8 01 00 00 00 48 89 e5 0f c1 05 bb e4 d4 00 83 c0 01 5d c3 66 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 e5 <0f> 0b 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 e5 48 83 ec 10 48
>> RIP  [<ffffffff811bb399>] fsnotify_notify_queue_is_empty+0x9/0x10
>>  RSP <ffffc90000253e68>
>> ---[ end trace 7dc4a27003f0b575 ]---
>>
>> I didn't bisect, but I would guess the culprit is one of the new patches in the
>> affected file.
>>
>> 22e9cf146d3b fanotify: fix possible false warning when freeing events
>> ced89591817c fsnotify: convert notification_mutex to a spinlock
>> f82fa3d0e7f5 fsnotify: drop notification_mutex before destroying event
>> 782fbc7e8685 fanotify: fix list corruption in fanotify_get_response()
>> 56cf1c8a1b35 fsnotify: add a way to stop queueing events on group shutdown
>
> Yes, very likely my patches cause this. But it must be some config issue
> because my test machine does not hit this and the code is "obviously
> correct" - famous last words ;). I've seen zero-day robot complain as well
> so I'll try the config it had sent me and see what's going on (likely
> spin_is_locked() does not do what I thought under some configs...).
>

Your builds are probably all SMP builds. Looks like this happens if SMP is disabled.

Guenter

[toc] | [prev] | [next] | [standalone]

#1484970

FromJan Kara <jack@suse.cz>
Date2016-09-16 15:00 +0200
Message-ID<si0Ui-4PI-1@gated-at.bofh.it>
In reply to#1484886
On Fri 16-09-16 04:10:38, Guenter Roeck wrote:
> On 09/15/2016 11:45 PM, Jan Kara wrote:
> >Hi,
> >
> >On Thu 15-09-16 19:20:10, Guenter Roeck wrote:
> >>I see various architectures crashing in -next with the following error.
> >>
> >>------------[ cut here ]------------
> >>kernel BUG at fs/notify/notification.c:66!
> >
> >Thanks for report!
> >
> >...
> >
> >>Call Trace:
> >> [<ffffffff811bd532>] inotify_poll+0x42/0x70
> >> [<ffffffff811bfeba>] SyS_epoll_ctl+0x84a/0xf60
> >> [<ffffffff811be4a0>] ? ep_send_events_proc+0x180/0x180
> >> [<ffffffff8176be98>] entry_SYSCALL_64_fastpath+0x13/0x8f
> >>Code: 90 90 0f 1f 44 00 00 55 b8 01 00 00 00 48 89 e5 0f c1 05 bb e4 d4 00 83 c0 01 5d c3 66 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 e5 <0f> 0b 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 e5 48 83 ec 10 48
> >>RIP  [<ffffffff811bb399>] fsnotify_notify_queue_is_empty+0x9/0x10
> >> RSP <ffffc90000253e68>
> >>---[ end trace 7dc4a27003f0b575 ]---
> >>
> >>I didn't bisect, but I would guess the culprit is one of the new patches in the
> >>affected file.
> >>
> >>22e9cf146d3b fanotify: fix possible false warning when freeing events
> >>ced89591817c fsnotify: convert notification_mutex to a spinlock
> >>f82fa3d0e7f5 fsnotify: drop notification_mutex before destroying event
> >>782fbc7e8685 fanotify: fix list corruption in fanotify_get_response()
> >>56cf1c8a1b35 fsnotify: add a way to stop queueing events on group shutdown
> >
> >Yes, very likely my patches cause this. But it must be some config issue
> >because my test machine does not hit this and the code is "obviously
> >correct" - famous last words ;). I've seen zero-day robot complain as well
> >so I'll try the config it had sent me and see what's going on (likely
> >spin_is_locked() does not do what I thought under some configs...).
> >
> 
> Your builds are probably all SMP builds. Looks like this happens if SMP
> is disabled.

Right, that was the problem. spin_is_locked() without CONFIG_SPINLOCK_DEBUG
returns always 0.

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

[toc] | [prev] | [next] | [standalone]

#1486990

FromAndrew Morton <akpm@linux-foundation.org>
Date2016-09-20 03:20 +0200
Message-ID<sjhT3-4IM-7@gated-at.bofh.it>
In reply to#1484970
On Tue, 20 Sep 2016 11:10:35 +1000 Michael Ellerman <mpe@ellerman.id.au> wrote:

> Jan Kara <jack@suse.cz> writes:
> > On Fri 16-09-16 04:10:38, Guenter Roeck wrote:
> >> On 09/15/2016 11:45 PM, Jan Kara wrote:
> >> >On Thu 15-09-16 19:20:10, Guenter Roeck wrote:
> >> >>I see various architectures crashing in -next with the following error.
> >> >>------------[ cut here ]------------
> >> >>kernel BUG at fs/notify/notification.c:66!
> >>
> >> >correct" - famous last words ;). I've seen zero-day robot complain as well
> >> >so I'll try the config it had sent me and see what's going on (likely
> >> >spin_is_locked() does not do what I thought under some configs...).
> >> 
> >> Your builds are probably all SMP builds. Looks like this happens if SMP
> >> is disabled.
> >
> > Right, that was the problem. spin_is_locked() without CONFIG_SPINLOCK_DEBUG
> > returns always 0.
> 
> Can we get this fixed soon please? It's breaking all my CI runs.
> 

It should be fixed in next -next.

From: Jan Kara <jack@suse.cz>
Subject: fsnotify-convert-notification_mutex-to-a-spinlock-fix

This is a fixed version of the patch that fixes the BUG_ON hitting on UP
kernels.

Link: http://lkml.kernel.org/r/1474031567-1831-1-git-send-email-jack@suse.cz
Signed-off-by: Jan Kara <jack@suse.cz>
Reviewed-by: Lino Sanfilippo <LinoSanfilippo@gmx.de>
Tested-by: Guenter Roeck <linux@roeck-us.net>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---

 fs/notify/fanotify/fanotify_user.c |    3 ++-
 fs/notify/notification.c           |    9 ++++++---
 2 files changed, 8 insertions(+), 4 deletions(-)

diff -puN fs/notify/fanotify/fanotify_user.c~fsnotify-convert-notification_mutex-to-a-spinlock-fix fs/notify/fanotify/fanotify_user.c
--- a/fs/notify/fanotify/fanotify_user.c~fsnotify-convert-notification_mutex-to-a-spinlock-fix
+++ a/fs/notify/fanotify/fanotify_user.c
@@ -54,7 +54,8 @@ struct kmem_cache *fanotify_perm_event_c
 static struct fsnotify_event *get_one_event(struct fsnotify_group *group,
 					    size_t count)
 {
-	BUG_ON(!spin_is_locked(&group->notification_lock));
+	BUG_ON(IS_ENABLED(CONFIG_SMP) &&
+	       !spin_is_locked(&group->notification_lock));
 
 	pr_debug("%s: group=%p count=%zd\n", __func__, group, count);
 
diff -puN fs/notify/notification.c~fsnotify-convert-notification_mutex-to-a-spinlock-fix fs/notify/notification.c
--- a/fs/notify/notification.c~fsnotify-convert-notification_mutex-to-a-spinlock-fix
+++ a/fs/notify/notification.c
@@ -63,7 +63,8 @@ EXPORT_SYMBOL_GPL(fsnotify_get_cookie);
 /* return true if the notify queue is empty, false otherwise */
 bool fsnotify_notify_queue_is_empty(struct fsnotify_group *group)
 {
-	BUG_ON(!spin_is_locked(&group->notification_lock));
+	BUG_ON(IS_ENABLED(CONFIG_SMP) &&
+	       !spin_is_locked(&group->notification_lock));
 	return list_empty(&group->notification_list) ? true : false;
 }
 
@@ -139,7 +140,8 @@ struct fsnotify_event *fsnotify_remove_f
 {
 	struct fsnotify_event *event;
 
-	BUG_ON(!spin_is_locked(&group->notification_lock));
+	BUG_ON(IS_ENABLED(CONFIG_SMP) &&
+	       !spin_is_locked(&group->notification_lock));
 
 	pr_debug("%s: group=%p\n", __func__, group);
 
@@ -161,7 +163,8 @@ struct fsnotify_event *fsnotify_remove_f
  */
 struct fsnotify_event *fsnotify_peek_first_event(struct fsnotify_group *group)
 {
-	BUG_ON(!spin_is_locked(&group->notification_lock));
+	BUG_ON(IS_ENABLED(CONFIG_SMP) &&
+	       !spin_is_locked(&group->notification_lock));
 
 	return list_first_entry(&group->notification_list,
 				struct fsnotify_event, list);
_

[toc] | [prev] | [next] | [standalone]

#1487277

FromMichael Ellerman <mpe@ellerman.id.au>
Date2016-09-20 14:10 +0200
Message-ID<sjs26-2Lm-29@gated-at.bofh.it>
In reply to#1486990
Andrew Morton <akpm@linux-foundation.org> writes:
> On Tue, 20 Sep 2016 11:10:35 +1000 Michael Ellerman <mpe@ellerman.id.au> wrote:
>> Jan Kara <jack@suse.cz> writes:
>> >
>> > Right, that was the problem. spin_is_locked() without CONFIG_SPINLOCK_DEBUG
>> > returns always 0.
>> 
>> Can we get this fixed soon please? It's breaking all my CI runs.
>
> It should be fixed in next -next.

Great thanks.

I did search LKML to see if Jan had sent a fix but I guess I missed it.

> diff -puN fs/notify/fanotify/fanotify_user.c~fsnotify-convert-notification_mutex-to-a-spinlock-fix fs/notify/fanotify/fanotify_user.c
> --- a/fs/notify/fanotify/fanotify_user.c~fsnotify-convert-notification_mutex-to-a-spinlock-fix
> +++ a/fs/notify/fanotify/fanotify_user.c
> @@ -54,7 +54,8 @@ struct kmem_cache *fanotify_perm_event_c
>  static struct fsnotify_event *get_one_event(struct fsnotify_group *group,
>  					    size_t count)
>  {
> -	BUG_ON(!spin_is_locked(&group->notification_lock));
> +	BUG_ON(IS_ENABLED(CONFIG_SMP) &&
> +	       !spin_is_locked(&group->notification_lock));
  
I thought lockdep_assert_held() was preferred for checks like this that
are purely sanity checking, ie. not part of the algorithm.

cheers

[toc] | [prev] | [next] | [standalone]

#1487941

FromJan Kara <jack@suse.cz>
Date2016-09-21 11:20 +0200
Message-ID<sjLR8-78t-25@gated-at.bofh.it>
In reply to#1487277
On Tue 20-09-16 22:05:00, Michael Ellerman wrote:
> Andrew Morton <akpm@linux-foundation.org> writes:
> > On Tue, 20 Sep 2016 11:10:35 +1000 Michael Ellerman <mpe@ellerman.id.au> wrote:
> >> Jan Kara <jack@suse.cz> writes:
> >> >
> >> > Right, that was the problem. spin_is_locked() without CONFIG_SPINLOCK_DEBUG
> >> > returns always 0.
> >> 
> >> Can we get this fixed soon please? It's breaking all my CI runs.
> >
> > It should be fixed in next -next.
> 
> Great thanks.
> 
> I did search LKML to see if Jan had sent a fix but I guess I missed it.

I didn't send the fix to LKML but only to linux-fsdevel...

> > diff -puN fs/notify/fanotify/fanotify_user.c~fsnotify-convert-notification_mutex-to-a-spinlock-fix fs/notify/fanotify/fanotify_user.c
> > --- a/fs/notify/fanotify/fanotify_user.c~fsnotify-convert-notification_mutex-to-a-spinlock-fix
> > +++ a/fs/notify/fanotify/fanotify_user.c
> > @@ -54,7 +54,8 @@ struct kmem_cache *fanotify_perm_event_c
> >  static struct fsnotify_event *get_one_event(struct fsnotify_group *group,
> >  					    size_t count)
> >  {
> > -	BUG_ON(!spin_is_locked(&group->notification_lock));
> > +	BUG_ON(IS_ENABLED(CONFIG_SMP) &&
> > +	       !spin_is_locked(&group->notification_lock));
>   
> I thought lockdep_assert_held() was preferred for checks like this that
> are purely sanity checking, ie. not part of the algorithm.

As pointed out by other guy, assert_spin_locked() is probably the right way
to do it (it's a wrapper that does essentially what I did here). Somehow I
missed that when looking for the right fix.

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

[toc] | [prev] | [next] | [standalone]

#1486994

FromMichael Ellerman <mpe@ellerman.id.au>
Date2016-09-20 03:20 +0200
Message-ID<sjhT3-4IM-9@gated-at.bofh.it>
In reply to#1484970
Jan Kara <jack@suse.cz> writes:
> On Fri 16-09-16 04:10:38, Guenter Roeck wrote:
>> On 09/15/2016 11:45 PM, Jan Kara wrote:
>> >On Thu 15-09-16 19:20:10, Guenter Roeck wrote:
>> >>I see various architectures crashing in -next with the following error.
>> >>------------[ cut here ]------------
>> >>kernel BUG at fs/notify/notification.c:66!
>>
>> >correct" - famous last words ;). I've seen zero-day robot complain as well
>> >so I'll try the config it had sent me and see what's going on (likely
>> >spin_is_locked() does not do what I thought under some configs...).
>> 
>> Your builds are probably all SMP builds. Looks like this happens if SMP
>> is disabled.
>
> Right, that was the problem. spin_is_locked() without CONFIG_SPINLOCK_DEBUG
> returns always 0.

Can we get this fixed soon please? It's breaking all my CI runs.

cheers

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web