Groups > linux.kernel > #1631436 > unrolled thread

net/ipv6: use-after-free in __call_rcu/in6_dev_finish_destroy_rcu

Started by	Andrey Konovalov <andreyknvl@google.com>
First post	2017-04-26 14:40 +0200
Last post	2017-05-02 19:30 +0200
Articles	7 — 3 participants

Back to article view | Back to linux.kernel

  net/ipv6: use-after-free in __call_rcu/in6_dev_finish_destroy_rcu Andrey Konovalov <andreyknvl@google.com> - 2017-04-26 14:40 +0200
    Re: net/ipv6: use-after-free in __call_rcu/in6_dev_finish_destroy_rcu "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-26 16:10 +0200
      Re: net/ipv6: use-after-free in __call_rcu/in6_dev_finish_destroy_rcu Andrey Konovalov <andreyknvl@google.com> - 2017-04-26 16:50 +0200
        Re: net/ipv6: use-after-free in __call_rcu/in6_dev_finish_destroy_rcu "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-26 17:30 +0200
      Re: net/ipv6: use-after-free in __call_rcu/in6_dev_finish_destroy_rcu David Ahern <dsa@cumulusnetworks.com> - 2017-05-02 04:50 +0200
        Re: net/ipv6: use-after-free in __call_rcu/in6_dev_finish_destroy_rcu Andrey Konovalov <andreyknvl@google.com> - 2017-05-02 19:00 +0200
          Re: net/ipv6: use-after-free in __call_rcu/in6_dev_finish_destroy_rcu David Ahern <dsa@cumulusnetworks.com> - 2017-05-02 19:30 +0200

#1631436 — net/ipv6: use-after-free in __call_rcu/in6_dev_finish_destroy_rcu

From	Andrey Konovalov <andreyknvl@google.com>
Date	2017-04-26 14:40 +0200
Subject	net/ipv6: use-after-free in __call_rcu/in6_dev_finish_destroy_rcu
Message-ID	<tAuoF-2hj-5@gated-at.bofh.it>

Hi,

I've got the following error report while fuzzing the kernel with syzkaller.

On commit 5a7ad1146caa895ad718a534399e38bd2ba721b7 (4.11-rc8).

Unfortunately it's not reproducible.

I'm not sure whether is is an issue with rcu or ipv6.

==================================================================
BUG: KASAN: use-after-free in __call_rcu.constprop.77+0x13be/0x1640
kernel/rcu/tree.c:3269 at addr ffff88003b842280
Write of size 8 by task kworker/u10:1/180
CPU: 2 PID: 180 Comm: kworker/u10:1 Not tainted 4.11.0-rc8+ #270
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Workqueue: events_unbound call_usermodehelper_exec_work
Call Trace:
 __dump_stack lib/dump_stack.c:16 [inline]
 dump_stack+0x192/0x22d lib/dump_stack.c:52
 kasan_object_err+0x1c/0x70 mm/kasan/report.c:164
 print_address_description mm/kasan/report.c:202 [inline]
 kasan_report_error mm/kasan/report.c:291 [inline]
 kasan_report+0x252/0x510 mm/kasan/report.c:347
 __asan_report_store8_noabort+0x17/0x20 mm/kasan/report.c:373
 __call_rcu.constprop.77+0x13be/0x1640 kernel/rcu/tree.c:3269
 call_rcu_sched+0x12/0x20 kernel/rcu/tree.c:3288
 free_pid+0x446/0x5d0 kernel/pid.c:293
 __change_pid+0x2a1/0x3d0 kernel/pid.c:411
 detach_pid+0x1f/0x30 kernel/pid.c:416
 __unhash_process kernel/exit.c:74 [inline]
 __exit_signal kernel/exit.c:155 [inline]
 release_task+0xbb0/0x1d90 kernel/exit.c:199
 wait_task_zombie kernel/exit.c:1230 [inline]
 wait_consider_task+0x11fe/0x3410 kernel/exit.c:1458
 do_wait_thread kernel/exit.c:1521 [inline]
 do_wait+0x3ea/0x8e0 kernel/exit.c:1592
 SYSC_wait4 kernel/exit.c:1720 [inline]
 SyS_wait4+0x208/0x340 kernel/exit.c:1689
 call_usermodehelper_exec_sync kernel/kmod.c:292 [inline]
 call_usermodehelper_exec_work+0x1a7/0x2b0 kernel/kmod.c:329
 process_one_work+0x9f7/0x1580 kernel/workqueue.c:2097
 worker_thread+0x1df/0x14b0 kernel/workqueue.c:2231
 kthread+0x31f/0x3f0 kernel/kthread.c:231
 ret_from_fork+0x2c/0x40 arch/x86/entry/entry_64.S:430
Object at ffff88003b842018, in cache kmalloc-1024 size: 1024
Allocated:
PID = 1
 save_stack_trace+0x16/0x20 arch/x86/kernel/stacktrace.c:59
 save_stack+0x43/0xd0 mm/kasan/kasan.c:513
 set_track mm/kasan/kasan.c:525 [inline]
 kasan_kmalloc+0xad/0xe0 mm/kasan/kasan.c:616
 kmem_cache_alloc_trace+0x61/0x170 mm/slub.c:2745
 kmalloc include/linux/slab.h:490 [inline]
 kzalloc include/linux/slab.h:663 [inline]
 ipv6_add_dev+0x199/0x1380 net/ipv6/addrconf.c:380
 addrconf_init+0xd0/0x29a net/ipv6/addrconf.c:6405
 inet6_init+0x2f6/0x584 net/ipv6/af_inet6.c:962
 do_one_initcall+0xf3/0x380 init/main.c:792
 do_initcall_level init/main.c:858 [inline]
 do_initcalls init/main.c:866 [inline]
 do_basic_setup init/main.c:884 [inline]
 kernel_init_freeable+0x54d/0x622 init/main.c:1035
 kernel_init+0x13/0x180 init/main.c:959
 ret_from_fork+0x2c/0x40 arch/x86/entry/entry_64.S:430
Freed:
PID = 6479
 save_stack_trace+0x16/0x20 arch/x86/kernel/stacktrace.c:59
 save_stack+0x43/0xd0 mm/kasan/kasan.c:513
 set_track mm/kasan/kasan.c:525 [inline]
 kasan_slab_free+0x73/0xc0 mm/kasan/kasan.c:589
 slab_free_hook mm/slub.c:1357 [inline]
 slab_free_freelist_hook mm/slub.c:1379 [inline]
 slab_free mm/slub.c:2961 [inline]
 kfree+0x91/0x190 mm/slub.c:3882
 in6_dev_finish_destroy_rcu+0x97/0xc0 net/ipv6/addrconf_core.c:150
 __rcu_reclaim kernel/rcu/rcu.h:118 [inline]
 rcu_do_batch.isra.65+0x6de/0xbd0 kernel/rcu/tree.c:2879
 invoke_rcu_callbacks kernel/rcu/tree.c:3142 [inline]
 __rcu_process_callbacks kernel/rcu/tree.c:3109 [inline]
 rcu_process_callbacks+0x23f/0x810 kernel/rcu/tree.c:3126
 __do_softirq+0x253/0x78b kernel/softirq.c:284
Memory state around the buggy address:
 ffff88003b842180: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
 ffff88003b842200: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>ffff88003b842280: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
                   ^
 ffff88003b842300: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
 ffff88003b842380: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
==================================================================

[toc] | [next] | [standalone]

#1631471

From	"Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Date	2017-04-26 16:10 +0200
Message-ID	<tAvNL-3kF-1@gated-at.bofh.it>
In reply to	#1631436

On Wed, Apr 26, 2017 at 02:34:15PM +0200, Andrey Konovalov wrote:
> Hi,
> 
> I've got the following error report while fuzzing the kernel with syzkaller.
> 
> On commit 5a7ad1146caa895ad718a534399e38bd2ba721b7 (4.11-rc8).
> 
> Unfortunately it's not reproducible.
> 
> I'm not sure whether is is an issue with rcu or ipv6.
> 
> ==================================================================
> BUG: KASAN: use-after-free in __call_rcu.constprop.77+0x13be/0x1640

Does building with CONFIG_DEBUG_OBJECTS_RCU_HEAD=y show any splats?
(Yeah, kind of a stupid question if it is not reproducible, but had
to ask!)

							Thanx, Paul

> kernel/rcu/tree.c:3269 at addr ffff88003b842280
> Write of size 8 by task kworker/u10:1/180
> CPU: 2 PID: 180 Comm: kworker/u10:1 Not tainted 4.11.0-rc8+ #270
> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
> Workqueue: events_unbound call_usermodehelper_exec_work
> Call Trace:
>  __dump_stack lib/dump_stack.c:16 [inline]
>  dump_stack+0x192/0x22d lib/dump_stack.c:52
>  kasan_object_err+0x1c/0x70 mm/kasan/report.c:164
>  print_address_description mm/kasan/report.c:202 [inline]
>  kasan_report_error mm/kasan/report.c:291 [inline]
>  kasan_report+0x252/0x510 mm/kasan/report.c:347
>  __asan_report_store8_noabort+0x17/0x20 mm/kasan/report.c:373
>  __call_rcu.constprop.77+0x13be/0x1640 kernel/rcu/tree.c:3269
>  call_rcu_sched+0x12/0x20 kernel/rcu/tree.c:3288
>  free_pid+0x446/0x5d0 kernel/pid.c:293
>  __change_pid+0x2a1/0x3d0 kernel/pid.c:411
>  detach_pid+0x1f/0x30 kernel/pid.c:416
>  __unhash_process kernel/exit.c:74 [inline]
>  __exit_signal kernel/exit.c:155 [inline]
>  release_task+0xbb0/0x1d90 kernel/exit.c:199
>  wait_task_zombie kernel/exit.c:1230 [inline]
>  wait_consider_task+0x11fe/0x3410 kernel/exit.c:1458
>  do_wait_thread kernel/exit.c:1521 [inline]
>  do_wait+0x3ea/0x8e0 kernel/exit.c:1592
>  SYSC_wait4 kernel/exit.c:1720 [inline]
>  SyS_wait4+0x208/0x340 kernel/exit.c:1689
>  call_usermodehelper_exec_sync kernel/kmod.c:292 [inline]
>  call_usermodehelper_exec_work+0x1a7/0x2b0 kernel/kmod.c:329
>  process_one_work+0x9f7/0x1580 kernel/workqueue.c:2097
>  worker_thread+0x1df/0x14b0 kernel/workqueue.c:2231
>  kthread+0x31f/0x3f0 kernel/kthread.c:231
>  ret_from_fork+0x2c/0x40 arch/x86/entry/entry_64.S:430
> Object at ffff88003b842018, in cache kmalloc-1024 size: 1024
> Allocated:
> PID = 1
>  save_stack_trace+0x16/0x20 arch/x86/kernel/stacktrace.c:59
>  save_stack+0x43/0xd0 mm/kasan/kasan.c:513
>  set_track mm/kasan/kasan.c:525 [inline]
>  kasan_kmalloc+0xad/0xe0 mm/kasan/kasan.c:616
>  kmem_cache_alloc_trace+0x61/0x170 mm/slub.c:2745
>  kmalloc include/linux/slab.h:490 [inline]
>  kzalloc include/linux/slab.h:663 [inline]
>  ipv6_add_dev+0x199/0x1380 net/ipv6/addrconf.c:380
>  addrconf_init+0xd0/0x29a net/ipv6/addrconf.c:6405
>  inet6_init+0x2f6/0x584 net/ipv6/af_inet6.c:962
>  do_one_initcall+0xf3/0x380 init/main.c:792
>  do_initcall_level init/main.c:858 [inline]
>  do_initcalls init/main.c:866 [inline]
>  do_basic_setup init/main.c:884 [inline]
>  kernel_init_freeable+0x54d/0x622 init/main.c:1035
>  kernel_init+0x13/0x180 init/main.c:959
>  ret_from_fork+0x2c/0x40 arch/x86/entry/entry_64.S:430
> Freed:
> PID = 6479
>  save_stack_trace+0x16/0x20 arch/x86/kernel/stacktrace.c:59
>  save_stack+0x43/0xd0 mm/kasan/kasan.c:513
>  set_track mm/kasan/kasan.c:525 [inline]
>  kasan_slab_free+0x73/0xc0 mm/kasan/kasan.c:589
>  slab_free_hook mm/slub.c:1357 [inline]
>  slab_free_freelist_hook mm/slub.c:1379 [inline]
>  slab_free mm/slub.c:2961 [inline]
>  kfree+0x91/0x190 mm/slub.c:3882
>  in6_dev_finish_destroy_rcu+0x97/0xc0 net/ipv6/addrconf_core.c:150
>  __rcu_reclaim kernel/rcu/rcu.h:118 [inline]
>  rcu_do_batch.isra.65+0x6de/0xbd0 kernel/rcu/tree.c:2879
>  invoke_rcu_callbacks kernel/rcu/tree.c:3142 [inline]
>  __rcu_process_callbacks kernel/rcu/tree.c:3109 [inline]
>  rcu_process_callbacks+0x23f/0x810 kernel/rcu/tree.c:3126
>  __do_softirq+0x253/0x78b kernel/softirq.c:284
> Memory state around the buggy address:
>  ffff88003b842180: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>  ffff88003b842200: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> >ffff88003b842280: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>                    ^
>  ffff88003b842300: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>  ffff88003b842380: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> ==================================================================
>

[toc] | [prev] | [next] | [standalone]

#1631494

From	Andrey Konovalov <andreyknvl@google.com>
Date	2017-04-26 16:50 +0200
Message-ID	<tAwqt-3Ay-3@gated-at.bofh.it>
In reply to	#1631471

On Wed, Apr 26, 2017 at 3:59 PM, Paul E. McKenney
<paulmck@linux.vnet.ibm.com> wrote:
> On Wed, Apr 26, 2017 at 02:34:15PM +0200, Andrey Konovalov wrote:
>> Hi,
>>
>> I've got the following error report while fuzzing the kernel with syzkaller.
>>
>> On commit 5a7ad1146caa895ad718a534399e38bd2ba721b7 (4.11-rc8).
>>
>> Unfortunately it's not reproducible.
>>
>> I'm not sure whether is is an issue with rcu or ipv6.
>>
>> ==================================================================
>> BUG: KASAN: use-after-free in __call_rcu.constprop.77+0x13be/0x1640
>
> Does building with CONFIG_DEBUG_OBJECTS_RCU_HEAD=y show any splats?
> (Yeah, kind of a stupid question if it is not reproducible, but had
> to ask!)

Hi Paul,

I'll try enabling this config.

In the meantime, while I was trying to reproduce this issue, I got this warning:

------------[ cut here ]------------
WARNING: CPU: 0 PID: 4590 at kernel/rcu/tree.c:2919
rcu_do_batch.isra.65+0x845/0xbd0
Modules linked in:
CPU: 0 PID: 4590 Comm: syz-executor Not tainted 4.11.0-rc8+ #270
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Call Trace:
 <IRQ>
 __dump_stack lib/dump_stack.c:16
 dump_stack+0x192/0x22d lib/dump_stack.c:52
 __warn+0x19f/0x1e0 kernel/panic.c:549
 warn_slowpath_null+0x2c/0x40 kernel/panic.c:584
 rcu_do_batch.isra.65+0x845/0xbd0 kernel/rcu/tree.c:2919
 invoke_rcu_callbacks kernel/rcu/tree.c:3142
 __rcu_process_callbacks kernel/rcu/tree.c:3109
 rcu_process_callbacks+0x23f/0x810 kernel/rcu/tree.c:3126
 __do_softirq+0x253/0x78b kernel/softirq.c:284
 invoke_softirq kernel/softirq.c:364
 irq_exit+0x149/0x180 kernel/softirq.c:405
 exiting_irq ./arch/x86/include/asm/apic.h:657
 smp_apic_timer_interrupt+0x76/0xa0 arch/x86/kernel/apic/apic.c:962
 apic_timer_interrupt+0x89/0x90
RIP: 0010:do_anonymous_page mm/memory.c:2962
RIP: 0010:handle_pte_fault mm/memory.c:3721
RIP: 0010:__handle_mm_fault+0xdab/0x1c00 mm/memory.c:3841
RSP: 0018:ffff88006a02f7d0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff10
RAX: 000000000005dab0 RBX: 1ffff1000d405f00 RCX: 1ffff1000d28cd3f
RDX: dffffc0000000000 RSI: 8000000000000025 RDI: ffff8800694669f8
RBP: ffff88006a02f9a8 R08: 00000000014280ca R09: 1ffff1000d405eb0
R10: ffff88006a02f660 R11: dffffc0000000000 R12: ffff88006a02f8e0
R13: ffff88006a02f980 R14: ffff8800694669b0 R15: ffff880069466a00
 </IRQ>
 handle_mm_fault+0x1aa/0x450 mm/memory.c:3878
 faultin_page mm/gup.c:408
 __get_user_pages+0x606/0x14a0 mm/gup.c:607
 populate_vma_page_range+0xd9/0x100 mm/gup.c:1062
 __mm_populate+0x278/0x540 mm/gup.c:1112
 mm_populate ./include/linux/mm.h:2132
 vm_mmap_pgoff+0x258/0x280 mm/util.c:314
 SYSC_mmap_pgoff mm/mmap.c:1503
 SyS_mmap_pgoff+0x22c/0x5e0 mm/mmap.c:1461
 SYSC_mmap arch/x86/kernel/sys_x86_64.c:96
 SyS_mmap+0x16/0x20 arch/x86/kernel/sys_x86_64.c:87
 entry_SYSCALL_64_fastpath+0x1a/0xa9 arch/x86/entry/entry_64.S:204
RIP: 0033:0x4458e9
RSP: 002b:00007f3a15ce4b58 EFLAGS: 00000282 ORIG_RAX: 0000000000000009
RAX: ffffffffffffffda RBX: 00007f3a15ce5700 RCX: 00000000004458e9
RDX: 0000000000000002 RSI: 00000000009b8000 RDI: 0000000020000000
RBP: 0000000000000000 R08: ffffffffffffffff R09: 8000000000000000
R10: 0000000000008032 R11: 0000000000000282 R12: 0000000000000000
R13: 0000000000000000 R14: 00007f3a15ce59c0 R15: 00007f3a15ce5700
---[ end trace e36457085170396e ]---

>
>                                                         Thanx, Paul
>
>> kernel/rcu/tree.c:3269 at addr ffff88003b842280
>> Write of size 8 by task kworker/u10:1/180
>> CPU: 2 PID: 180 Comm: kworker/u10:1 Not tainted 4.11.0-rc8+ #270
>> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
>> Workqueue: events_unbound call_usermodehelper_exec_work
>> Call Trace:
>>  __dump_stack lib/dump_stack.c:16 [inline]
>>  dump_stack+0x192/0x22d lib/dump_stack.c:52
>>  kasan_object_err+0x1c/0x70 mm/kasan/report.c:164
>>  print_address_description mm/kasan/report.c:202 [inline]
>>  kasan_report_error mm/kasan/report.c:291 [inline]
>>  kasan_report+0x252/0x510 mm/kasan/report.c:347
>>  __asan_report_store8_noabort+0x17/0x20 mm/kasan/report.c:373
>>  __call_rcu.constprop.77+0x13be/0x1640 kernel/rcu/tree.c:3269
>>  call_rcu_sched+0x12/0x20 kernel/rcu/tree.c:3288
>>  free_pid+0x446/0x5d0 kernel/pid.c:293
>>  __change_pid+0x2a1/0x3d0 kernel/pid.c:411
>>  detach_pid+0x1f/0x30 kernel/pid.c:416
>>  __unhash_process kernel/exit.c:74 [inline]
>>  __exit_signal kernel/exit.c:155 [inline]
>>  release_task+0xbb0/0x1d90 kernel/exit.c:199
>>  wait_task_zombie kernel/exit.c:1230 [inline]
>>  wait_consider_task+0x11fe/0x3410 kernel/exit.c:1458
>>  do_wait_thread kernel/exit.c:1521 [inline]
>>  do_wait+0x3ea/0x8e0 kernel/exit.c:1592
>>  SYSC_wait4 kernel/exit.c:1720 [inline]
>>  SyS_wait4+0x208/0x340 kernel/exit.c:1689
>>  call_usermodehelper_exec_sync kernel/kmod.c:292 [inline]
>>  call_usermodehelper_exec_work+0x1a7/0x2b0 kernel/kmod.c:329
>>  process_one_work+0x9f7/0x1580 kernel/workqueue.c:2097
>>  worker_thread+0x1df/0x14b0 kernel/workqueue.c:2231
>>  kthread+0x31f/0x3f0 kernel/kthread.c:231
>>  ret_from_fork+0x2c/0x40 arch/x86/entry/entry_64.S:430
>> Object at ffff88003b842018, in cache kmalloc-1024 size: 1024
>> Allocated:
>> PID = 1
>>  save_stack_trace+0x16/0x20 arch/x86/kernel/stacktrace.c:59
>>  save_stack+0x43/0xd0 mm/kasan/kasan.c:513
>>  set_track mm/kasan/kasan.c:525 [inline]
>>  kasan_kmalloc+0xad/0xe0 mm/kasan/kasan.c:616
>>  kmem_cache_alloc_trace+0x61/0x170 mm/slub.c:2745
>>  kmalloc include/linux/slab.h:490 [inline]
>>  kzalloc include/linux/slab.h:663 [inline]
>>  ipv6_add_dev+0x199/0x1380 net/ipv6/addrconf.c:380
>>  addrconf_init+0xd0/0x29a net/ipv6/addrconf.c:6405
>>  inet6_init+0x2f6/0x584 net/ipv6/af_inet6.c:962
>>  do_one_initcall+0xf3/0x380 init/main.c:792
>>  do_initcall_level init/main.c:858 [inline]
>>  do_initcalls init/main.c:866 [inline]
>>  do_basic_setup init/main.c:884 [inline]
>>  kernel_init_freeable+0x54d/0x622 init/main.c:1035
>>  kernel_init+0x13/0x180 init/main.c:959
>>  ret_from_fork+0x2c/0x40 arch/x86/entry/entry_64.S:430
>> Freed:
>> PID = 6479
>>  save_stack_trace+0x16/0x20 arch/x86/kernel/stacktrace.c:59
>>  save_stack+0x43/0xd0 mm/kasan/kasan.c:513
>>  set_track mm/kasan/kasan.c:525 [inline]
>>  kasan_slab_free+0x73/0xc0 mm/kasan/kasan.c:589
>>  slab_free_hook mm/slub.c:1357 [inline]
>>  slab_free_freelist_hook mm/slub.c:1379 [inline]
>>  slab_free mm/slub.c:2961 [inline]
>>  kfree+0x91/0x190 mm/slub.c:3882
>>  in6_dev_finish_destroy_rcu+0x97/0xc0 net/ipv6/addrconf_core.c:150
>>  __rcu_reclaim kernel/rcu/rcu.h:118 [inline]
>>  rcu_do_batch.isra.65+0x6de/0xbd0 kernel/rcu/tree.c:2879
>>  invoke_rcu_callbacks kernel/rcu/tree.c:3142 [inline]
>>  __rcu_process_callbacks kernel/rcu/tree.c:3109 [inline]
>>  rcu_process_callbacks+0x23f/0x810 kernel/rcu/tree.c:3126
>>  __do_softirq+0x253/0x78b kernel/softirq.c:284
>> Memory state around the buggy address:
>>  ffff88003b842180: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>>  ffff88003b842200: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>> >ffff88003b842280: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>>                    ^
>>  ffff88003b842300: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>>  ffff88003b842380: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>> ==================================================================
>>
>

[toc] | [prev] | [next] | [standalone]

#1631531

From	"Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Date	2017-04-26 17:30 +0200
Message-ID	<tAx3e-46S-57@gated-at.bofh.it>
In reply to	#1631494

On Wed, Apr 26, 2017 at 04:45:51PM +0200, Andrey Konovalov wrote:
> On Wed, Apr 26, 2017 at 3:59 PM, Paul E. McKenney
> <paulmck@linux.vnet.ibm.com> wrote:
> > On Wed, Apr 26, 2017 at 02:34:15PM +0200, Andrey Konovalov wrote:
> >> Hi,
> >>
> >> I've got the following error report while fuzzing the kernel with syzkaller.
> >>
> >> On commit 5a7ad1146caa895ad718a534399e38bd2ba721b7 (4.11-rc8).
> >>
> >> Unfortunately it's not reproducible.
> >>
> >> I'm not sure whether is is an issue with rcu or ipv6.
> >>
> >> ==================================================================
> >> BUG: KASAN: use-after-free in __call_rcu.constprop.77+0x13be/0x1640
> >
> > Does building with CONFIG_DEBUG_OBJECTS_RCU_HEAD=y show any splats?
> > (Yeah, kind of a stupid question if it is not reproducible, but had
> > to ask!)
> 
> Hi Paul,
> 
> I'll try enabling this config.
> 
> In the meantime, while I was trying to reproduce this issue, I got this warning:
> 
> ------------[ cut here ]------------
> WARNING: CPU: 0 PID: 4590 at kernel/rcu/tree.c:2919
> rcu_do_batch.isra.65+0x845/0xbd0
> Modules linked in:
> CPU: 0 PID: 4590 Comm: syz-executor Not tainted 4.11.0-rc8+ #270
> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
> Call Trace:
>  <IRQ>
>  __dump_stack lib/dump_stack.c:16
>  dump_stack+0x192/0x22d lib/dump_stack.c:52
>  __warn+0x19f/0x1e0 kernel/panic.c:549
>  warn_slowpath_null+0x2c/0x40 kernel/panic.c:584
>  rcu_do_batch.isra.65+0x845/0xbd0 kernel/rcu/tree.c:2919
>  invoke_rcu_callbacks kernel/rcu/tree.c:3142
>  __rcu_process_callbacks kernel/rcu/tree.c:3109
>  rcu_process_callbacks+0x23f/0x810 kernel/rcu/tree.c:3126
>  __do_softirq+0x253/0x78b kernel/softirq.c:284
>  invoke_softirq kernel/softirq.c:364
>  irq_exit+0x149/0x180 kernel/softirq.c:405
>  exiting_irq ./arch/x86/include/asm/apic.h:657
>  smp_apic_timer_interrupt+0x76/0xa0 arch/x86/kernel/apic/apic.c:962
>  apic_timer_interrupt+0x89/0x90
> RIP: 0010:do_anonymous_page mm/memory.c:2962
> RIP: 0010:handle_pte_fault mm/memory.c:3721
> RIP: 0010:__handle_mm_fault+0xdab/0x1c00 mm/memory.c:3841
> RSP: 0018:ffff88006a02f7d0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff10
> RAX: 000000000005dab0 RBX: 1ffff1000d405f00 RCX: 1ffff1000d28cd3f
> RDX: dffffc0000000000 RSI: 8000000000000025 RDI: ffff8800694669f8
> RBP: ffff88006a02f9a8 R08: 00000000014280ca R09: 1ffff1000d405eb0
> R10: ffff88006a02f660 R11: dffffc0000000000 R12: ffff88006a02f8e0
> R13: ffff88006a02f980 R14: ffff8800694669b0 R15: ffff880069466a00
>  </IRQ>
>  handle_mm_fault+0x1aa/0x450 mm/memory.c:3878
>  faultin_page mm/gup.c:408
>  __get_user_pages+0x606/0x14a0 mm/gup.c:607
>  populate_vma_page_range+0xd9/0x100 mm/gup.c:1062
>  __mm_populate+0x278/0x540 mm/gup.c:1112
>  mm_populate ./include/linux/mm.h:2132
>  vm_mmap_pgoff+0x258/0x280 mm/util.c:314
>  SYSC_mmap_pgoff mm/mmap.c:1503
>  SyS_mmap_pgoff+0x22c/0x5e0 mm/mmap.c:1461
>  SYSC_mmap arch/x86/kernel/sys_x86_64.c:96
>  SyS_mmap+0x16/0x20 arch/x86/kernel/sys_x86_64.c:87
>  entry_SYSCALL_64_fastpath+0x1a/0xa9 arch/x86/entry/entry_64.S:204
> RIP: 0033:0x4458e9
> RSP: 002b:00007f3a15ce4b58 EFLAGS: 00000282 ORIG_RAX: 0000000000000009
> RAX: ffffffffffffffda RBX: 00007f3a15ce5700 RCX: 00000000004458e9
> RDX: 0000000000000002 RSI: 00000000009b8000 RDI: 0000000020000000
> RBP: 0000000000000000 R08: ffffffffffffffff R09: 8000000000000000
> R10: 0000000000008032 R11: 0000000000000282 R12: 0000000000000000
> R13: 0000000000000000 R14: 00007f3a15ce59c0 R15: 00007f3a15ce5700
> ---[ end trace e36457085170396e ]---

This is current mainline, somewhere after v4.11-rc1 and linus/master?

If so, this warning indicates that RCU's callback list has been corrupted.
This could be a bug in RCU, and if this was 4.12-rc1 and later, my
first guess would be upcoming changes in RCU callback handling.  But I
am pretty sure that you are not running 4.12-rc1.

So I will instead guess that you are double-call_rcu()-ing (sort of like
double-free, except with call_rcu() and friends instead of kfree()).
Or, alternatively, the call_rcu() counterpart to use-after-free.

							Thanx, Paul

> >> kernel/rcu/tree.c:3269 at addr ffff88003b842280
> >> Write of size 8 by task kworker/u10:1/180
> >> CPU: 2 PID: 180 Comm: kworker/u10:1 Not tainted 4.11.0-rc8+ #270
> >> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
> >> Workqueue: events_unbound call_usermodehelper_exec_work
> >> Call Trace:
> >>  __dump_stack lib/dump_stack.c:16 [inline]
> >>  dump_stack+0x192/0x22d lib/dump_stack.c:52
> >>  kasan_object_err+0x1c/0x70 mm/kasan/report.c:164
> >>  print_address_description mm/kasan/report.c:202 [inline]
> >>  kasan_report_error mm/kasan/report.c:291 [inline]
> >>  kasan_report+0x252/0x510 mm/kasan/report.c:347
> >>  __asan_report_store8_noabort+0x17/0x20 mm/kasan/report.c:373
> >>  __call_rcu.constprop.77+0x13be/0x1640 kernel/rcu/tree.c:3269
> >>  call_rcu_sched+0x12/0x20 kernel/rcu/tree.c:3288
> >>  free_pid+0x446/0x5d0 kernel/pid.c:293
> >>  __change_pid+0x2a1/0x3d0 kernel/pid.c:411
> >>  detach_pid+0x1f/0x30 kernel/pid.c:416
> >>  __unhash_process kernel/exit.c:74 [inline]
> >>  __exit_signal kernel/exit.c:155 [inline]
> >>  release_task+0xbb0/0x1d90 kernel/exit.c:199
> >>  wait_task_zombie kernel/exit.c:1230 [inline]
> >>  wait_consider_task+0x11fe/0x3410 kernel/exit.c:1458
> >>  do_wait_thread kernel/exit.c:1521 [inline]
> >>  do_wait+0x3ea/0x8e0 kernel/exit.c:1592
> >>  SYSC_wait4 kernel/exit.c:1720 [inline]
> >>  SyS_wait4+0x208/0x340 kernel/exit.c:1689
> >>  call_usermodehelper_exec_sync kernel/kmod.c:292 [inline]
> >>  call_usermodehelper_exec_work+0x1a7/0x2b0 kernel/kmod.c:329
> >>  process_one_work+0x9f7/0x1580 kernel/workqueue.c:2097
> >>  worker_thread+0x1df/0x14b0 kernel/workqueue.c:2231
> >>  kthread+0x31f/0x3f0 kernel/kthread.c:231
> >>  ret_from_fork+0x2c/0x40 arch/x86/entry/entry_64.S:430
> >> Object at ffff88003b842018, in cache kmalloc-1024 size: 1024
> >> Allocated:
> >> PID = 1
> >>  save_stack_trace+0x16/0x20 arch/x86/kernel/stacktrace.c:59
> >>  save_stack+0x43/0xd0 mm/kasan/kasan.c:513
> >>  set_track mm/kasan/kasan.c:525 [inline]
> >>  kasan_kmalloc+0xad/0xe0 mm/kasan/kasan.c:616
> >>  kmem_cache_alloc_trace+0x61/0x170 mm/slub.c:2745
> >>  kmalloc include/linux/slab.h:490 [inline]
> >>  kzalloc include/linux/slab.h:663 [inline]
> >>  ipv6_add_dev+0x199/0x1380 net/ipv6/addrconf.c:380
> >>  addrconf_init+0xd0/0x29a net/ipv6/addrconf.c:6405
> >>  inet6_init+0x2f6/0x584 net/ipv6/af_inet6.c:962
> >>  do_one_initcall+0xf3/0x380 init/main.c:792
> >>  do_initcall_level init/main.c:858 [inline]
> >>  do_initcalls init/main.c:866 [inline]
> >>  do_basic_setup init/main.c:884 [inline]
> >>  kernel_init_freeable+0x54d/0x622 init/main.c:1035
> >>  kernel_init+0x13/0x180 init/main.c:959
> >>  ret_from_fork+0x2c/0x40 arch/x86/entry/entry_64.S:430
> >> Freed:
> >> PID = 6479
> >>  save_stack_trace+0x16/0x20 arch/x86/kernel/stacktrace.c:59
> >>  save_stack+0x43/0xd0 mm/kasan/kasan.c:513
> >>  set_track mm/kasan/kasan.c:525 [inline]
> >>  kasan_slab_free+0x73/0xc0 mm/kasan/kasan.c:589
> >>  slab_free_hook mm/slub.c:1357 [inline]
> >>  slab_free_freelist_hook mm/slub.c:1379 [inline]
> >>  slab_free mm/slub.c:2961 [inline]
> >>  kfree+0x91/0x190 mm/slub.c:3882
> >>  in6_dev_finish_destroy_rcu+0x97/0xc0 net/ipv6/addrconf_core.c:150
> >>  __rcu_reclaim kernel/rcu/rcu.h:118 [inline]
> >>  rcu_do_batch.isra.65+0x6de/0xbd0 kernel/rcu/tree.c:2879
> >>  invoke_rcu_callbacks kernel/rcu/tree.c:3142 [inline]
> >>  __rcu_process_callbacks kernel/rcu/tree.c:3109 [inline]
> >>  rcu_process_callbacks+0x23f/0x810 kernel/rcu/tree.c:3126
> >>  __do_softirq+0x253/0x78b kernel/softirq.c:284
> >> Memory state around the buggy address:
> >>  ffff88003b842180: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> >>  ffff88003b842200: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> >> >ffff88003b842280: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> >>                    ^
> >>  ffff88003b842300: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> >>  ffff88003b842380: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> >> ==================================================================
> >>
> >
>

[toc] | [prev] | [next] | [standalone]

#1634090

From	David Ahern <dsa@cumulusnetworks.com>
Date	2017-05-02 04:50 +0200
Message-ID	<tCw30-rv-11@gated-at.bofh.it>
In reply to	#1631471

On 4/26/17 9:15 AM, Andrey Konovalov wrote:
> +David
> 
> I've enabled CONFIG_DEBUG_OBJECTS_RCU_HEAD and this is what I get.
> 
> Apparently the rcu warning is related to the fib6_del_route bug I've
> been trying to reproduce:
> https://groups.google.com/forum/#!msg/syzkaller/3SS80JbVPKA/2tfIAcW7DwAJ
> 
> Adding David, who provided the fix:
> https://patchwork.ozlabs.org/patch/754913/
> 
> I've managed to extract a reproducer, attached together with the
> .config that I used.
> 
> On commit 5a7ad1146caa895ad718a534399e38bd2ba721b7 (4.11-rc8) with
> David's patch applied.
> 
> ------------[ cut here ]------------
> WARNING: CPU: 1 PID: 5911 at lib/debugobjects.c:289
> debug_print_object+0x175/0x210
> ODEBUG: activate active (active state 1) object type: rcu_head hint:
>         (null)
> Modules linked in:
> CPU: 1 PID: 5911 Comm: a.out Not tainted 4.11.0-rc8+ #271
> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
> Call Trace:
>  __dump_stack lib/dump_stack.c:16
>  dump_stack+0x192/0x22d lib/dump_stack.c:52
>  __warn+0x19f/0x1e0 kernel/panic.c:549
>  warn_slowpath_fmt+0xe0/0x120 kernel/panic.c:564
>  debug_print_object+0x175/0x210 lib/debugobjects.c:286
>  debug_object_activate+0x574/0x7e0 lib/debugobjects.c:442
>  debug_rcu_head_queue kernel/rcu/rcu.h:75
>  __call_rcu.constprop.76+0xff/0x9c0 kernel/rcu/tree.c:3229
>  call_rcu_sched+0x12/0x20 kernel/rcu/tree.c:3288
>  rt6_rcu_free net/ipv6/ip6_fib.c:158
>  rt6_release+0x1ea/0x290 net/ipv6/ip6_fib.c:188
>  fib6_del_route net/ipv6/ip6_fib.c:1461

I think I got to the bottom of this one.

With your config, ip6_tunnel is compiled in.

The program runs in a very tight loop, calling 'unshare -n' and then
spawns 2 sets of 14 threads running random ioctl calls. The networking
sequence:

1. New network namespace created via unshare -n
- ip6tnl0 device is created in down state

2. address added to ip6tnl0 (equivalent to ip -6 addr add dev ip6tnl0
fd00::bb/1)
- the host route is created and inserted into FIB

3. ip6tnl0 is brought up - starts DAD on the address

4. exit namespace
- teardown / cleanup sequence starts
- lo teardown appears to happen BEFORE teardown of ip6tunl0
  + removes host route from FIB
  + host route added to rcu callback list: call_rcu(&rt->dst.rcu_head,
dst_rcu_free);
  + rcu callback has not run yet, so rt is NOT on the gc list so it has
    NOT been marked obsolete

5. worker_thread runs addrconf_dad_completed
- calls ipv6_ifa_notify which inserts the host route

All of that happens very quickly. The result is that a route that has
been deleted and added to the RCU list is re-inserted into the FIB. What
happens next depends on order -- in this case the exit namespace
eventually gets to cleaning up ip6tnl0 which removes the host route from
the FIB, calls the rcu function for cleanup -- and triggers the double
rcu trace.

I have a hack that flags this sequence and prevents the re-insertion
following DAD. That allows the command to run until it consumes all 2G
of memory the VM has -- about 600+ iterations without triggering any
stack traces.

[toc] | [prev] | [next] | [standalone]

#1634528

From	Andrey Konovalov <andreyknvl@google.com>
Date	2017-05-02 19:00 +0200
Message-ID	<tCJjA-P5-3@gated-at.bofh.it>
In reply to	#1634090

On Tue, May 2, 2017 at 4:44 AM, David Ahern <dsa@cumulusnetworks.com> wrote:
> On 4/26/17 9:15 AM, Andrey Konovalov wrote:
>> +David
>>
>> I've enabled CONFIG_DEBUG_OBJECTS_RCU_HEAD and this is what I get.
>>
>> Apparently the rcu warning is related to the fib6_del_route bug I've
>> been trying to reproduce:
>> https://groups.google.com/forum/#!msg/syzkaller/3SS80JbVPKA/2tfIAcW7DwAJ
>>
>> Adding David, who provided the fix:
>> https://patchwork.ozlabs.org/patch/754913/
>>
>> I've managed to extract a reproducer, attached together with the
>> .config that I used.
>>
>> On commit 5a7ad1146caa895ad718a534399e38bd2ba721b7 (4.11-rc8) with
>> David's patch applied.
>>
>> ------------[ cut here ]------------
>> WARNING: CPU: 1 PID: 5911 at lib/debugobjects.c:289
>> debug_print_object+0x175/0x210
>> ODEBUG: activate active (active state 1) object type: rcu_head hint:
>>         (null)
>> Modules linked in:
>> CPU: 1 PID: 5911 Comm: a.out Not tainted 4.11.0-rc8+ #271
>> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
>> Call Trace:
>>  __dump_stack lib/dump_stack.c:16
>>  dump_stack+0x192/0x22d lib/dump_stack.c:52
>>  __warn+0x19f/0x1e0 kernel/panic.c:549
>>  warn_slowpath_fmt+0xe0/0x120 kernel/panic.c:564
>>  debug_print_object+0x175/0x210 lib/debugobjects.c:286
>>  debug_object_activate+0x574/0x7e0 lib/debugobjects.c:442
>>  debug_rcu_head_queue kernel/rcu/rcu.h:75
>>  __call_rcu.constprop.76+0xff/0x9c0 kernel/rcu/tree.c:3229
>>  call_rcu_sched+0x12/0x20 kernel/rcu/tree.c:3288
>>  rt6_rcu_free net/ipv6/ip6_fib.c:158
>>  rt6_release+0x1ea/0x290 net/ipv6/ip6_fib.c:188
>>  fib6_del_route net/ipv6/ip6_fib.c:1461
>
> I think I got to the bottom of this one.
>
> With your config, ip6_tunnel is compiled in.
>
> The program runs in a very tight loop, calling 'unshare -n' and then
> spawns 2 sets of 14 threads running random ioctl calls. The networking
> sequence:
>
> 1. New network namespace created via unshare -n
> - ip6tnl0 device is created in down state
>
> 2. address added to ip6tnl0 (equivalent to ip -6 addr add dev ip6tnl0
> fd00::bb/1)
> - the host route is created and inserted into FIB
>
> 3. ip6tnl0 is brought up - starts DAD on the address
>
> 4. exit namespace
> - teardown / cleanup sequence starts
> - lo teardown appears to happen BEFORE teardown of ip6tunl0
>   + removes host route from FIB
>   + host route added to rcu callback list: call_rcu(&rt->dst.rcu_head,
> dst_rcu_free);
>   + rcu callback has not run yet, so rt is NOT on the gc list so it has
>     NOT been marked obsolete
>
> 5. worker_thread runs addrconf_dad_completed
> - calls ipv6_ifa_notify which inserts the host route
>
> All of that happens very quickly. The result is that a route that has
> been deleted and added to the RCU list is re-inserted into the FIB. What
> happens next depends on order -- in this case the exit namespace
> eventually gets to cleaning up ip6tnl0 which removes the host route from
> the FIB, calls the rcu function for cleanup -- and triggers the double
> rcu trace.
>
> I have a hack that flags this sequence and prevents the re-insertion
> following DAD. That allows the command to run until it consumes all 2G
> of memory the VM has -- about 600+ iterations without triggering any
> stack traces.

Hi David,

Thanks for looking into this!

Do you have a patch that I could test?

I also reported another issue recently, that might also be related to this one:
https://groups.google.com/forum/#!topic/syzkaller/Rt0pgY4wfiw

Thanks!

[toc] | [prev] | [next] | [standalone]

#1634542

From	David Ahern <dsa@cumulusnetworks.com>
Date	2017-05-02 19:30 +0200
Message-ID	<tCJMB-1dV-13@gated-at.bofh.it>
In reply to	#1634528

On 5/2/17 10:58 AM, Andrey Konovalov wrote:
> Do you have a patch that I could test?

not yet.

> 
> I also reported another issue recently, that might also be related to this one:
> https://groups.google.com/forum/#!topic/syzkaller/Rt0pgY4wfiw

different problem. I can still trigger this one with the reproducer you sent

[toc] | [prev] | [standalone]

csiph-web

net/ipv6: use-after-free in __call_rcu/in6_dev_finish_destroy_rcu

Contents

#1631436 — net/ipv6: use-after-free in __call_rcu/in6_dev_finish_destroy_rcu

#1631471

#1631494

#1631531

#1634090

#1634528

#1634542