Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1607327 > unrolled thread

mm: BUG in resv_map_release

Started byDmitry Vyukov <dvyukov@google.com>
First post2017-03-23 11:30 +0100
Last post2017-03-24 10:00 +0100
Articles 4 — 2 participants

Back to article view | Back to linux.kernel

Contents

  mm: BUG in resv_map_release Dmitry Vyukov <dvyukov@google.com> - 2017-03-23 11:30 +0100
    Re: mm: BUG in resv_map_release Mike Kravetz <mike.kravetz@oracle.com> - 2017-03-23 18:30 +0100
      Re: mm: BUG in resv_map_release Mike Kravetz <mike.kravetz@oracle.com> - 2017-03-23 19:10 +0100
        Re: mm: BUG in resv_map_release Dmitry Vyukov <dvyukov@google.com> - 2017-03-24 10:00 +0100

#1607327 — mm: BUG in resv_map_release

FromDmitry Vyukov <dvyukov@google.com>
Date2017-03-23 11:30 +0100
Subjectmm: BUG in resv_map_release
Message-ID<to8ad-TC-3@gated-at.bofh.it>
Hello,

I've got the following BUG while running syzkaller fuzzer.
Note the injected kmalloc failure, most likely it's the root cause.


FAULT_INJECTION: forcing a failure.
name failslab, interval 1, probability 0, space 0, times 0
CPU: 2 PID: 12823 Comm: syz-executor1 Not tainted 4.11.0-rc3+ #364
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Call Trace:
 __dump_stack lib/dump_stack.c:16 [inline]
 dump_stack+0x1b8/0x28d lib/dump_stack.c:52
 fail_dump lib/fault-inject.c:45 [inline]
 should_fail+0x78a/0x870 lib/fault-inject.c:154
 should_failslab+0xec/0x120 mm/failslab.c:31
 slab_pre_alloc_hook mm/slab.h:434 [inline]
 slab_alloc mm/slab.c:3394 [inline]
 kmem_cache_alloc_trace+0x206/0x720 mm/slab.c:3636
 kmalloc include/linux/slab.h:490 [inline]
 region_chg+0x429/0xa80 mm/hugetlb.c:402
 hugetlb_reserve_pages+0x16d/0x540 mm/hugetlb.c:4334
 hugetlb_file_setup+0x40c/0x9f0 fs/hugetlbfs/inode.c:1289
 newseg+0x422/0xd30 ipc/shm.c:575
 ipcget_new ipc/util.c:285 [inline]
 ipcget+0x21e/0x580 ipc/util.c:639
 SYSC_shmget ipc/shm.c:673 [inline]
 SyS_shmget+0x158/0x230 ipc/shm.c:657
 entry_SYSCALL_64_fastpath+0x1f/0xc2
RIP: 0033:0x445b79
RSP: 002b:00007f273aba1858 EFLAGS: 00000282 ORIG_RAX: 000000000000001d
RAX: ffffffffffffffda RBX: 0000000000708000 RCX: 0000000000445b79
RDX: 0000000000000900 RSI: 0000000000003000 RDI: 0000000000000000
RBP: 0000000000000086 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000020000000 R11: 0000000000000282 R12: 00000000004a7e31
R13: 0000000000000000 R14: 00007f273aba1618 R15: 00007f273aba1788
------------[ cut here ]------------
kernel BUG at mm/hugetlb.c:742!
invalid opcode: 0000 [#1] SMP KASAN
Dumping ftrace buffer:
   (ftrace buffer empty)
Modules linked in:
CPU: 2 PID: 12823 Comm: syz-executor1 Not tainted 4.11.0-rc3+ #364
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
task: ffff88006be3c0c0 task.stack: ffff88003daa0000
RIP: 0010:resv_map_release+0x265/0x330 mm/hugetlb.c:742
RSP: 0018:ffff88003daa7830 EFLAGS: 00010246
RAX: 0000000000010000 RBX: dffffc0000000000 RCX: ffffc90002253000
RDX: 0000000000010000 RSI: ffffffff81976485 RDI: ffff88006b5bf950
RBP: ffff88003daa78e0 R08: 1ffff1000d7c7a5b R09: 0000000000000000
R10: 0000000000000006 R11: 0000000000000000 R12: ffff88006b5bf958
R13: ffffed0007b54f0f R14: ffff88006b5bf958 R15: ffff88006b5bf958
FS:  00007f273aba2700(0000) GS:ffff88006e200000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f68fabe8000 CR3: 0000000014a09000 CR4: 00000000000026e0
DR0: 0000000020000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000600
Call Trace:
 hugetlbfs_evict_inode+0x7b/0xa0 fs/hugetlbfs/inode.c:493
 evict+0x481/0x920 fs/inode.c:553
 iput_final fs/inode.c:1515 [inline]
 iput+0x62b/0xa20 fs/inode.c:1542
 hugetlb_file_setup+0x593/0x9f0 fs/hugetlbfs/inode.c:1306
 newseg+0x422/0xd30 ipc/shm.c:575
 ipcget_new ipc/util.c:285 [inline]
 ipcget+0x21e/0x580 ipc/util.c:639
 SYSC_shmget ipc/shm.c:673 [inline]
 SyS_shmget+0x158/0x230 ipc/shm.c:657
 entry_SYSCALL_64_fastpath+0x1f/0xc2
RIP: 0033:0x445b79
RSP: 002b:00007f273aba1858 EFLAGS: 00000282 ORIG_RAX: 000000000000001d
RAX: ffffffffffffffda RBX: 0000000000708000 RCX: 0000000000445b79
RDX: 0000000000000900 RSI: 0000000000003000 RDI: 0000000000000000
RBP: 0000000000000086 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000020000000 R11: 0000000000000282 R12: 00000000004a7e31
R13: 0000000000000000 R14: 00007f273aba1618 R15: 00007f273aba1788
Code: 00 fc ff df 48 c7 04 01 00 00 00 00 c7 44 08 08 00 00 00 00 48
81 c4 88 00 00 00 5b 41 5c 41 5d 41 5e 41 5f 5d c3 e8 ab a5 d5 ff <0f>
0b 48 8b bd 70 ff ff ff e8 ad c0 03 00 e9 2c fe ff ff e8 a3
RIP: resv_map_release+0x265/0x330 mm/hugetlb.c:742 RSP: ffff88003daa7830
---[ end trace 575ce95655b30bd0 ]---

On commit 093b995e3b55a0ae0670226ddfcb05bfbf0099ae

[toc] | [next] | [standalone]

#1607733

FromMike Kravetz <mike.kravetz@oracle.com>
Date2017-03-23 18:30 +0100
Message-ID<toeIG-5py-11@gated-at.bofh.it>
In reply to#1607327
On 03/23/2017 03:19 AM, Dmitry Vyukov wrote:
> Hello,
> 
> I've got the following BUG while running syzkaller fuzzer.
> Note the injected kmalloc failure, most likely it's the root cause.

Thanks  Dmitry,

The BUG indicates someone called region_chg() in the process of adding
a hugetlbfs page reservation, but did not complete this 'two step'
process with a call to region_add() or region_abort().  Most likely a
missed call in an error path somewhere.  :(

I'll try to track this down.  The hint of 'injected kmalloc failure'
should help.

-- 
Mike Kravetz

> 
> 
> FAULT_INJECTION: forcing a failure.
> name failslab, interval 1, probability 0, space 0, times 0
> CPU: 2 PID: 12823 Comm: syz-executor1 Not tainted 4.11.0-rc3+ #364
> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
> Call Trace:
>  __dump_stack lib/dump_stack.c:16 [inline]
>  dump_stack+0x1b8/0x28d lib/dump_stack.c:52
>  fail_dump lib/fault-inject.c:45 [inline]
>  should_fail+0x78a/0x870 lib/fault-inject.c:154
>  should_failslab+0xec/0x120 mm/failslab.c:31
>  slab_pre_alloc_hook mm/slab.h:434 [inline]
>  slab_alloc mm/slab.c:3394 [inline]
>  kmem_cache_alloc_trace+0x206/0x720 mm/slab.c:3636
>  kmalloc include/linux/slab.h:490 [inline]
>  region_chg+0x429/0xa80 mm/hugetlb.c:402
>  hugetlb_reserve_pages+0x16d/0x540 mm/hugetlb.c:4334
>  hugetlb_file_setup+0x40c/0x9f0 fs/hugetlbfs/inode.c:1289
>  newseg+0x422/0xd30 ipc/shm.c:575
>  ipcget_new ipc/util.c:285 [inline]
>  ipcget+0x21e/0x580 ipc/util.c:639
>  SYSC_shmget ipc/shm.c:673 [inline]
>  SyS_shmget+0x158/0x230 ipc/shm.c:657
>  entry_SYSCALL_64_fastpath+0x1f/0xc2
> RIP: 0033:0x445b79
> RSP: 002b:00007f273aba1858 EFLAGS: 00000282 ORIG_RAX: 000000000000001d
> RAX: ffffffffffffffda RBX: 0000000000708000 RCX: 0000000000445b79
> RDX: 0000000000000900 RSI: 0000000000003000 RDI: 0000000000000000
> RBP: 0000000000000086 R08: 0000000000000000 R09: 0000000000000000
> R10: 0000000020000000 R11: 0000000000000282 R12: 00000000004a7e31
> R13: 0000000000000000 R14: 00007f273aba1618 R15: 00007f273aba1788
> ------------[ cut here ]------------
> kernel BUG at mm/hugetlb.c:742!
> invalid opcode: 0000 [#1] SMP KASAN
> Dumping ftrace buffer:
>    (ftrace buffer empty)
> Modules linked in:
> CPU: 2 PID: 12823 Comm: syz-executor1 Not tainted 4.11.0-rc3+ #364
> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
> task: ffff88006be3c0c0 task.stack: ffff88003daa0000
> RIP: 0010:resv_map_release+0x265/0x330 mm/hugetlb.c:742
> RSP: 0018:ffff88003daa7830 EFLAGS: 00010246
> RAX: 0000000000010000 RBX: dffffc0000000000 RCX: ffffc90002253000
> RDX: 0000000000010000 RSI: ffffffff81976485 RDI: ffff88006b5bf950
> RBP: ffff88003daa78e0 R08: 1ffff1000d7c7a5b R09: 0000000000000000
> R10: 0000000000000006 R11: 0000000000000000 R12: ffff88006b5bf958
> R13: ffffed0007b54f0f R14: ffff88006b5bf958 R15: ffff88006b5bf958
> FS:  00007f273aba2700(0000) GS:ffff88006e200000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00007f68fabe8000 CR3: 0000000014a09000 CR4: 00000000000026e0
> DR0: 0000000020000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000600
> Call Trace:
>  hugetlbfs_evict_inode+0x7b/0xa0 fs/hugetlbfs/inode.c:493
>  evict+0x481/0x920 fs/inode.c:553
>  iput_final fs/inode.c:1515 [inline]
>  iput+0x62b/0xa20 fs/inode.c:1542
>  hugetlb_file_setup+0x593/0x9f0 fs/hugetlbfs/inode.c:1306
>  newseg+0x422/0xd30 ipc/shm.c:575
>  ipcget_new ipc/util.c:285 [inline]
>  ipcget+0x21e/0x580 ipc/util.c:639
>  SYSC_shmget ipc/shm.c:673 [inline]
>  SyS_shmget+0x158/0x230 ipc/shm.c:657
>  entry_SYSCALL_64_fastpath+0x1f/0xc2
> RIP: 0033:0x445b79
> RSP: 002b:00007f273aba1858 EFLAGS: 00000282 ORIG_RAX: 000000000000001d
> RAX: ffffffffffffffda RBX: 0000000000708000 RCX: 0000000000445b79
> RDX: 0000000000000900 RSI: 0000000000003000 RDI: 0000000000000000
> RBP: 0000000000000086 R08: 0000000000000000 R09: 0000000000000000
> R10: 0000000020000000 R11: 0000000000000282 R12: 00000000004a7e31
> R13: 0000000000000000 R14: 00007f273aba1618 R15: 00007f273aba1788
> Code: 00 fc ff df 48 c7 04 01 00 00 00 00 c7 44 08 08 00 00 00 00 48
> 81 c4 88 00 00 00 5b 41 5c 41 5d 41 5e 41 5f 5d c3 e8 ab a5 d5 ff <0f>
> 0b 48 8b bd 70 ff ff ff e8 ad c0 03 00 e9 2c fe ff ff e8 a3
> RIP: resv_map_release+0x265/0x330 mm/hugetlb.c:742 RSP: ffff88003daa7830
> ---[ end trace 575ce95655b30bd0 ]---
> 
> On commit 093b995e3b55a0ae0670226ddfcb05bfbf0099ae
>

[toc] | [prev] | [next] | [standalone]

#1607761

FromMike Kravetz <mike.kravetz@oracle.com>
Date2017-03-23 19:10 +0100
Message-ID<tofln-5WV-7@gated-at.bofh.it>
In reply to#1607733
On 03/23/2017 10:25 AM, Mike Kravetz wrote:
> On 03/23/2017 03:19 AM, Dmitry Vyukov wrote:
>> Hello,
>>
>> I've got the following BUG while running syzkaller fuzzer.
>> Note the injected kmalloc failure, most likely it's the root cause.
> 
> Thanks  Dmitry,
> 
> The BUG indicates someone called region_chg() in the process of adding
> a hugetlbfs page reservation, but did not complete this 'two step'
> process with a call to region_add() or region_abort().  Most likely a
> missed call in an error path somewhere.  :(
> 
> I'll try to track this down.  The hint of 'injected kmalloc failure'
> should help.

Actually, in this case I believe the bug is in hugetlb_reserve_pages.
It calls region_chg(), but gets an error due to the injected kmalloc
failure.  At this point, the resv_map->adds_in_progress is 0 as it
should be.  However, the error path for hugetlb_reserve_pages calls
region_abort() which will unconditionally decrement adds_in_progress.
So, adds_in_progress goes negative and we eventually BUG.  :(

I'll look for other misuses of region_chg()/region_add()/region_abort()
and put together a patch.

Dmitry, is there some way to run the fuzzer with kmalloc failure injection
and target the hugetlbfs code?  I'm suspect we could flush out other bugs.
I noticed one other you discovered, and will look at that next.

-- 
Mike Kravetz

[toc] | [prev] | [next] | [standalone]

#1608232

FromDmitry Vyukov <dvyukov@google.com>
Date2017-03-24 10:00 +0100
Message-ID<toteF-7pT-1@gated-at.bofh.it>
In reply to#1607761
On Thu, Mar 23, 2017 at 7:02 PM, Mike Kravetz <mike.kravetz@oracle.com> wrote:
> On 03/23/2017 10:25 AM, Mike Kravetz wrote:
>> On 03/23/2017 03:19 AM, Dmitry Vyukov wrote:
>>> Hello,
>>>
>>> I've got the following BUG while running syzkaller fuzzer.
>>> Note the injected kmalloc failure, most likely it's the root cause.
>>
>> Thanks  Dmitry,
>>
>> The BUG indicates someone called region_chg() in the process of adding
>> a hugetlbfs page reservation, but did not complete this 'two step'
>> process with a call to region_add() or region_abort().  Most likely a
>> missed call in an error path somewhere.  :(
>>
>> I'll try to track this down.  The hint of 'injected kmalloc failure'
>> should help.
>
> Actually, in this case I believe the bug is in hugetlb_reserve_pages.
> It calls region_chg(), but gets an error due to the injected kmalloc
> failure.  At this point, the resv_map->adds_in_progress is 0 as it
> should be.  However, the error path for hugetlb_reserve_pages calls
> region_abort() which will unconditionally decrement adds_in_progress.
> So, adds_in_progress goes negative and we eventually BUG.  :(
>
> I'll look for other misuses of region_chg()/region_add()/region_abort()
> and put together a patch.
>
> Dmitry, is there some way to run the fuzzer with kmalloc failure injection
> and target the hugetlbfs code?  I'm suspect we could flush out other bugs.
> I noticed one other you discovered, and will look at that next.

syzkaller systematically targets all of the kernel code. So far I've
seen only these 2 involving hugetlbfs code. I don't think we need to
do anything special for hugetlbfs.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web