Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1284191 > unrolled thread
| Started by | Dmitry Vyukov <dvyukov@google.com> |
|---|---|
| First post | 2015-12-04 21:10 +0100 |
| Last post | 2015-12-09 10:10 +0100 |
| Articles | 8 on this page of 28 — 4 participants |
Back to article view | Back to linux.kernel
use-after-free in __perf_install_in_context Dmitry Vyukov <dvyukov@google.com> - 2015-12-04 21:10 +0100
Re: use-after-free in __perf_install_in_context Alexei Starovoitov <alexei.starovoitov@gmail.com> - 2015-12-04 21:40 +0100
Re: use-after-free in __perf_install_in_context Dmitry Vyukov <dvyukov@google.com> - 2015-12-04 22:10 +0100
Re: use-after-free in __perf_install_in_context Dmitry Vyukov <dvyukov@google.com> - 2015-12-07 12:10 +0100
Re: use-after-free in __perf_install_in_context Dmitry Vyukov <dvyukov@google.com> - 2015-12-07 12:10 +0100
Re: use-after-free in __perf_install_in_context Dmitry Vyukov <dvyukov@google.com> - 2015-12-07 12:30 +0100
Re: use-after-free in __perf_install_in_context Peter Zijlstra <peterz@infradead.org> - 2015-12-07 16:40 +0100
Re: use-after-free in __perf_install_in_context Dmitry Vyukov <dvyukov@google.com> - 2015-12-07 17:10 +0100
Re: use-after-free in __perf_install_in_context Alexei Starovoitov <alexei.starovoitov@gmail.com> - 2015-12-08 04:30 +0100
Re: use-after-free in __perf_install_in_context Dmitry Vyukov <dvyukov@google.com> - 2015-12-08 17:20 +0100
Re: use-after-free in __perf_install_in_context Alexei Starovoitov <alexei.starovoitov@gmail.com> - 2015-12-08 19:00 +0100
Re: use-after-free in __perf_install_in_context Dmitry Vyukov <dvyukov@google.com> - 2015-12-08 19:00 +0100
Re: use-after-free in __perf_install_in_context Alexei Starovoitov <alexei.starovoitov@gmail.com> - 2015-12-08 19:10 +0100
Re: use-after-free in __perf_install_in_context Dmitry Vyukov <dvyukov@google.com> - 2015-12-08 19:40 +0100
Re: use-after-free in __perf_install_in_context Alexei Starovoitov <alexei.starovoitov@gmail.com> - 2015-12-08 21:00 +0100
Re: use-after-free in __perf_install_in_context Dmitry Vyukov <dvyukov@google.com> - 2015-12-09 10:20 +0100
Re: use-after-free in __perf_install_in_context Alexei Starovoitov <alexei.starovoitov@gmail.com> - 2015-12-10 05:00 +0100
Re: use-after-free in __perf_install_in_context Peter Zijlstra <peterz@infradead.org> - 2015-12-10 10:10 +0100
Re: use-after-free in __perf_install_in_context Alexei Starovoitov <alexei.starovoitov@gmail.com> - 2015-12-10 18:10 +0100
Re: use-after-free in __perf_install_in_context Ingo Molnar <mingo@kernel.org> - 2015-12-11 09:20 +0100
Re: use-after-free in __perf_install_in_context Peter Zijlstra <peterz@infradead.org> - 2015-12-08 17:50 +0100
Re: use-after-free in __perf_install_in_context Dmitry Vyukov <dvyukov@google.com> - 2015-12-08 20:20 +0100
Re: use-after-free in __perf_install_in_context Peter Zijlstra <peterz@infradead.org> - 2015-12-10 21:00 +0100
Re: use-after-free in __perf_install_in_context Peter Zijlstra <peterz@infradead.org> - 2015-12-08 17:30 +0100
Re: use-after-free in __perf_install_in_context Dmitry Vyukov <dvyukov@google.com> - 2015-12-08 18:00 +0100
Re: use-after-free in __perf_install_in_context Peter Zijlstra <peterz@infradead.org> - 2015-12-08 17:30 +0100
Re: use-after-free in __perf_install_in_context Ingo Molnar <mingo@kernel.org> - 2015-12-08 20:00 +0100
Re: use-after-free in __perf_install_in_context Peter Zijlstra <peterz@infradead.org> - 2015-12-09 10:10 +0100
Page 2 of 2 — ← Prev page 1 [2]
| From | Peter Zijlstra <peterz@infradead.org> |
|---|---|
| Date | 2015-12-08 17:50 +0100 |
| Message-ID | <qDtCF-4NA-1@gated-at.bofh.it> |
| In reply to | #1285706 |
On Mon, Dec 07, 2015 at 05:09:21PM +0100, Dmitry Vyukov wrote:
> If your audit does not give any results, can you give me a patch that
> prints rcu callback submission stacks in KASAN reports?
Just because my brain is fried for today, I figured I'd give it a go.
Completely untested..
---
include/linux/slub_def.h | 2 ++
kernel/rcu/tree_plugin.h | 1 +
mm/slub.c | 34 ++++++++++++++++++++++++++++++++--
3 files changed, 35 insertions(+), 2 deletions(-)
diff --git a/include/linux/slub_def.h b/include/linux/slub_def.h
index 33885118523c..445b586c0bfa 100644
--- a/include/linux/slub_def.h
+++ b/include/linux/slub_def.h
@@ -129,4 +129,6 @@ static inline void *virt_to_obj(struct kmem_cache *s,
void object_err(struct kmem_cache *s, struct page *page,
u8 *object, char *reason);
+void object_set_indirect(const void *addr);
+
#endif /* _LINUX_SLUB_DEF_H */
diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index 630c19772630..4e1e79e01e34 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -659,6 +659,7 @@ static void rcu_preempt_do_callbacks(void)
*/
void call_rcu(struct rcu_head *head, rcu_callback_t func)
{
+ object_set_indirect(head);
__call_rcu(head, func, rcu_state_p, -1, 0);
}
EXPORT_SYMBOL_GPL(call_rcu);
diff --git a/mm/slub.c b/mm/slub.c
index 46997517406e..6977dc7cffcd 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -196,7 +196,12 @@ struct track {
unsigned long when; /* When did the operation occur */
};
-enum track_item { TRACK_ALLOC, TRACK_FREE };
+enum track_item {
+ TRACK_ALLOC = 0,
+ TRACK_FREE,
+ TRACK_INDIRECT,
+ TRACK_NR,
+};
#ifdef CONFIG_SYSFS
static int sysfs_slab_add(struct kmem_cache *);
@@ -551,6 +556,7 @@ static void init_tracking(struct kmem_cache *s, void *object)
set_track(s, object, TRACK_FREE, 0UL);
set_track(s, object, TRACK_ALLOC, 0UL);
+ set_track(s, object, TRACK_INDIRECT, 0UL);
}
static void print_track(const char *s, struct track *t)
@@ -579,6 +585,7 @@ static void print_tracking(struct kmem_cache *s, void *object)
print_track("Allocated", get_track(s, object, TRACK_ALLOC));
print_track("Freed", get_track(s, object, TRACK_FREE));
+ print_track("Indirect", get_track(s, object, TRACK_INDIRECT));
}
static void print_page_info(struct page *page)
@@ -652,6 +659,29 @@ static void print_trailer(struct kmem_cache *s, struct page *page, u8 *p)
dump_stack();
}
+void object_set_indirect(const void *addr)
+{
+ if ((addr >= (void *)PAGE_OFFSET) &&
+ (addr < high_memory)) {
+ struct page *page = virt_to_head_page(addr);
+
+ if (PageSlab(page)) {
+ void *object;
+ struct kmem_cache *cache = page->slab_cache;
+ void *last_object;
+
+ object = virt_to_obj(cache, page_address(page), addr);
+ last_object = page_address(page) +
+ page->objects * cache->size;
+
+ if (unlikely(object > last_object))
+ object = last_object; /* we hit into padding */
+
+ set_track(cache, object, TRACK_INDIRECT, (unsigned long)addr);
+ }
+ }
+}
+
void object_err(struct kmem_cache *s, struct page *page,
u8 *object, char *reason)
{
@@ -767,7 +797,7 @@ static int check_pad_bytes(struct kmem_cache *s, struct page *page, u8 *p)
if (s->flags & SLAB_STORE_USER)
/* We also have user information there */
- off += 2 * sizeof(struct track);
+ off += TRACK_NR * sizeof(struct track);
if (s->size == off)
return 1;
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Dmitry Vyukov <dvyukov@google.com> |
|---|---|
| Date | 2015-12-08 20:20 +0100 |
| Message-ID | <qDvXP-6mP-13@gated-at.bofh.it> |
| In reply to | #1286675 |
On Tue, Dec 8, 2015 at 5:44 PM, Peter Zijlstra <peterz@infradead.org> wrote:
> On Mon, Dec 07, 2015 at 05:09:21PM +0100, Dmitry Vyukov wrote:
>> If your audit does not give any results, can you give me a patch that
>> prints rcu callback submission stacks in KASAN reports?
>
> Just because my brain is fried for today, I figured I'd give it a go.
>
> Completely untested..
>
> ---
> include/linux/slub_def.h | 2 ++
> kernel/rcu/tree_plugin.h | 1 +
> mm/slub.c | 34 ++++++++++++++++++++++++++++++++--
> 3 files changed, 35 insertions(+), 2 deletions(-)
>
> diff --git a/include/linux/slub_def.h b/include/linux/slub_def.h
> index 33885118523c..445b586c0bfa 100644
> --- a/include/linux/slub_def.h
> +++ b/include/linux/slub_def.h
> @@ -129,4 +129,6 @@ static inline void *virt_to_obj(struct kmem_cache *s,
> void object_err(struct kmem_cache *s, struct page *page,
> u8 *object, char *reason);
>
> +void object_set_indirect(const void *addr);
> +
> #endif /* _LINUX_SLUB_DEF_H */
> diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
> index 630c19772630..4e1e79e01e34 100644
> --- a/kernel/rcu/tree_plugin.h
> +++ b/kernel/rcu/tree_plugin.h
> @@ -659,6 +659,7 @@ static void rcu_preempt_do_callbacks(void)
> */
> void call_rcu(struct rcu_head *head, rcu_callback_t func)
> {
> + object_set_indirect(head);
> __call_rcu(head, func, rcu_state_p, -1, 0);
> }
> EXPORT_SYMBOL_GPL(call_rcu);
> diff --git a/mm/slub.c b/mm/slub.c
> index 46997517406e..6977dc7cffcd 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
> @@ -196,7 +196,12 @@ struct track {
> unsigned long when; /* When did the operation occur */
> };
>
> -enum track_item { TRACK_ALLOC, TRACK_FREE };
> +enum track_item {
> + TRACK_ALLOC = 0,
> + TRACK_FREE,
> + TRACK_INDIRECT,
> + TRACK_NR,
> +};
>
> #ifdef CONFIG_SYSFS
> static int sysfs_slab_add(struct kmem_cache *);
> @@ -551,6 +556,7 @@ static void init_tracking(struct kmem_cache *s, void *object)
>
> set_track(s, object, TRACK_FREE, 0UL);
> set_track(s, object, TRACK_ALLOC, 0UL);
> + set_track(s, object, TRACK_INDIRECT, 0UL);
> }
>
> static void print_track(const char *s, struct track *t)
> @@ -579,6 +585,7 @@ static void print_tracking(struct kmem_cache *s, void *object)
>
> print_track("Allocated", get_track(s, object, TRACK_ALLOC));
> print_track("Freed", get_track(s, object, TRACK_FREE));
> + print_track("Indirect", get_track(s, object, TRACK_INDIRECT));
> }
>
> static void print_page_info(struct page *page)
> @@ -652,6 +659,29 @@ static void print_trailer(struct kmem_cache *s, struct page *page, u8 *p)
> dump_stack();
> }
>
> +void object_set_indirect(const void *addr)
> +{
> + if ((addr >= (void *)PAGE_OFFSET) &&
> + (addr < high_memory)) {
> + struct page *page = virt_to_head_page(addr);
> +
> + if (PageSlab(page)) {
> + void *object;
> + struct kmem_cache *cache = page->slab_cache;
> + void *last_object;
> +
> + object = virt_to_obj(cache, page_address(page), addr);
> + last_object = page_address(page) +
> + page->objects * cache->size;
> +
> + if (unlikely(object > last_object))
> + object = last_object; /* we hit into padding */
> +
> + set_track(cache, object, TRACK_INDIRECT, (unsigned long)addr);
> + }
> + }
> +}
> +
> void object_err(struct kmem_cache *s, struct page *page,
> u8 *object, char *reason)
> {
> @@ -767,7 +797,7 @@ static int check_pad_bytes(struct kmem_cache *s, struct page *page, u8 *p)
>
> if (s->flags & SLAB_STORE_USER)
> /* We also have user information there */
> - off += 2 * sizeof(struct track);
> + off += TRACK_NR * sizeof(struct track);
>
> if (s->size == off)
> return 1;
Peter,
Tested with your patches.
The additional WARNING does not fire.
For the rcu stacks, I had to change two more 2's to TRACK_NR and also
moved memorization from call_rcu to __call_rcu, but now it is working.
Two reports with indirect stack:
BUG: KASAN: use-after-free in __lock_acquire+0x4e99/0x5100 at addr
ffff88003096c640
BUG kmalloc-1024 (Not tainted): kasan: bad access detected
-----------------------------------------------------------------------------
irq event stamp: 136
hardirqs last enabled at (135): [<ffffffff814c12e4>]
__raw_spin_lock_init+0x24/0x120 kernel/locking/spinlock_debug.c:23
hardirqs last disabled at (136): [<ffffffff8183c967>]
__slab_alloc+0x37/0x90 mm/slub.c:2487
softirqs last enabled at (38): [<ffffffff8138417d>]
__do_softirq+0x5ed/0xb40 kernel/softirq.c:299
softirqs last disabled at (31): [< inline >] invoke_softirq
kernel/softirq.c:350
softirqs last disabled at (31): [<ffffffff813849d5>]
irq_exit+0x165/0x1e0 kernel/softirq.c:391
INFO: Allocated in alloc_perf_context+0x4c/0x100 age=220 cpu=1 pid=15649
[< none >] ___slab_alloc+0x648/0x8c0 mm/slub.c:2468
[< none >] __slab_alloc+0x4c/0x90 mm/slub.c:2497
[< inline >] slab_alloc_node mm/slub.c:2560
[< inline >] slab_alloc mm/slub.c:2602
[< none >] kmem_cache_alloc_trace+0x23c/0x3f0 mm/slub.c:2619
[< inline >] kmalloc include/linux/slab.h:458
[< inline >] kzalloc include/linux/slab.h:602
[< none >] alloc_perf_context+0x4c/0x100 kernel/events/core.c:3405
[< none >] find_get_context+0x187/0x830 kernel/events/core.c:3512
[< none >] SYSC_perf_event_open+0xe50/0x21a0 kernel/events/core.c:8381
[< none >] SyS_perf_event_open+0x39/0x50 kernel/events/core.c:8242
[< none >] tracesys_phase2+0x88/0x8d arch/x86/entry/entry_64.S:269
INFO: Freed in free_ctx+0x4b/0x70 age=89 cpu=3 pid=21
[< none >] __slab_free+0x21e/0x3e0 mm/slub.c:2678
[< inline >] slab_free mm/slub.c:2833
[< none >] kfree+0x26f/0x3e0 mm/slub.c:3662
[< none >] free_ctx+0x4b/0x70 kernel/events/core.c:872
[< inline >] __rcu_reclaim kernel/rcu/rcu.h:118
[< inline >] rcu_do_batch kernel/rcu/tree.c:2693
[< inline >] invoke_rcu_callbacks kernel/rcu/tree.c:2961
[< inline >] __rcu_process_callbacks kernel/rcu/tree.c:2928
[< none >] rcu_process_callbacks+0x631/0x19e0 kernel/rcu/tree.c:2945
[< none >] __do_softirq+0x2e5/0xb40 kernel/softirq.c:273
[< none >] run_ksoftirqd+0x20/0x50 kernel/softirq.c:662
[< none >] smpboot_thread_fn+0x41d/0x850 kernel/smpboot.c:163
[< none >] kthread+0x21d/0x2e0 drivers/block/aoe/aoecmd.c:1314
[< none >] ret_from_fork+0x3f/0x70 arch/x86/entry/entry_64.S:468
INFO: Indirect in __call_rcu.constprop.63+0x7e/0xe30 age=126 cpu=3 pid=15645
[< none >] __call_rcu.constprop.63+0x7e/0xe30 kernel/rcu/tree.c:3048
[< none >] call_rcu_sched+0x26/0x30 kernel/rcu/tree.c:3117
[< none >] put_ctx+0x150/0x250 kernel/events/core.c:891
[< inline >] __free_event kernel/events/core.c:3707
[< none >] _free_event+0x2c7/0xaa0 kernel/events/core.c:3738
[< none >] put_event+0x294/0x4d0 kernel/events/core.c:3834
[< none >] perf_release+0x3c/0x60 kernel/events/core.c:3849
[< none >] __fput+0x244/0x860 fs/file_table.c:208
[< none >] ____fput+0x15/0x20 fs/file_table.c:244
[< none >] task_work_run+0x130/0x240 kernel/task_work.c:115
[< inline >] exit_task_work include/linux/task_work.h:21
[< none >] do_exit+0x885/0x3050 kernel/exit.c:750
[< none >] do_group_exit+0xec/0x390 kernel/exit.c:880
[< none >] get_signal+0x677/0x1bf0 kernel/signal.c:2307
[< none >] do_signal+0x7e/0x2170 arch/x86/kernel/signal.c:709
[< none >] exit_to_usermode_loop+0xfe/0x1e0
arch/x86/entry/common.c:247
[< inline >] prepare_exit_to_usermode arch/x86/entry/common.c:282
[< none >] syscall_return_slowpath+0x16b/0x240
arch/x86/entry/common.c:344
[< none >] int_ret_from_sys_call+0x25/0x9f
arch/x86/entry/entry_64.S:281
INFO: Slab 0xffffea0000c25a00 objects=21 used=1 fp=0xffff88003096d7a8
flags=0x1fffc0000004080
INFO: Object 0xffff88003096c620 @offset=17952 fp=0x (null)
CPU: 1 PID: 15687 Comm: syzkaller_execu Tainted: G B
4.4.0-rc3+ #155
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
0000000000000001 ffff88003794f680 ffffffff82e0f8b8 0000000041b58ab3
ffffffff87a9a27d ffffffff82e0f806 ffff880033c62e80 ffffffff87abb3d1
ffff88003e806d00 0000000000000008 ffff88003096c620 ffff88003794f680
Call Trace:
[< inline >] __dump_stack lib/dump_stack.c:15
[<ffffffff82e0f8b8>] dump_stack+0xb2/0xfa lib/dump_stack.c:50
[<ffffffff818369dc>] print_trailer+0x12c/0x210 mm/slub.c:659
[<ffffffff81841a3f>] object_err+0x2f/0x40 mm/slub.c:689
[< inline >] print_address_description mm/kasan/report.c:138
[<ffffffff818457a9>] kasan_report_error+0x5d9/0x860 mm/kasan/report.c:251
[< inline >] kasan_report mm/kasan/report.c:274
[<ffffffff81845bd4>] __asan_report_load8_noabort+0x54/0x70
mm/kasan/report.c:295
[<ffffffff814b25f9>] __lock_acquire+0x4e99/0x5100 kernel/locking/lockdep.c:3092
[<ffffffff814b517d>] lock_acquire+0x19d/0x3f0 kernel/locking/lockdep.c:3585
[< inline >] __raw_spin_lock include/linux/spinlock_api_smp.h:144
[<ffffffff86a89591>] _raw_spin_lock+0x31/0x40 kernel/locking/spinlock.c:151
[< inline >] perf_ctx_lock kernel/events/core.c:351
[<ffffffff816e15d9>] __perf_install_in_context+0x109/0xa00
kernel/events/core.c:2083
[<ffffffff816cb7ba>] remote_function+0x14a/0x200 kernel/events/core.c:74
[<ffffffff81572787>] generic_exec_single+0x2a7/0x490 kernel/smp.c:156
[<ffffffff81573350>] smp_call_function_single+0x200/0x310 kernel/smp.c:300
[<ffffffff816c9bd3>] task_function_call+0x123/0x160 kernel/events/core.c:101
[<ffffffff816d1bf1>] perf_install_in_context+0x201/0x340
kernel/events/core.c:2164
[<ffffffff816f62e5>] SYSC_perf_event_open+0x1465/0x21a0
kernel/events/core.c:8546
[<ffffffff816ff449>] SyS_perf_event_open+0x39/0x50 kernel/events/core.c:8242
[<ffffffff86a8a3b6>] entry_SYSCALL_64_fastpath+0x16/0x7a
arch/x86/entry/entry_64.S:185
BUG: KASAN: use-after-free in __lock_acquire+0x4e99/0x5100 at addr
ffff88005f77aee0
Read of size 8 by task syzkaller_execu/30337
=============================================================================
BUG kmalloc-1024 (Not tainted): kasan: bad access detected
-----------------------------------------------------------------------------
Disabling lock debugging due to kernel taint
INFO: Allocated in alloc_perf_context+0x4c/0x100 age=124 cpu=3 pid=30312
[< none >] ___slab_alloc+0x648/0x8c0 mm/slub.c:2468
[< none >] __slab_alloc+0x4c/0x90 mm/slub.c:2497
[< inline >] slab_alloc_node mm/slub.c:2560
[< inline >] slab_alloc mm/slub.c:2602
[< none >] kmem_cache_alloc_trace+0x23c/0x3f0 mm/slub.c:2619
[< inline >] kmalloc include/linux/slab.h:458
[< inline >] kzalloc include/linux/slab.h:602
[< none >] alloc_perf_context+0x4c/0x100 kernel/events/core.c:3405
[< none >] find_get_context+0x187/0x830 kernel/events/core.c:3512
[< none >] SYSC_perf_event_open+0xe50/0x21a0 kernel/events/core.c:8381
[< none >] SyS_perf_event_open+0x39/0x50 kernel/events/core.c:8242
[< none >] tracesys_phase2+0x88/0x8d arch/x86/entry/entry_64.S:269
INFO: Freed in free_ctx+0x4b/0x70 age=74 cpu=2 pid=16
[< none >] __slab_free+0x21e/0x3e0 mm/slub.c:2678
[< inline >] slab_free mm/slub.c:2833
[< none >] kfree+0x26f/0x3e0 mm/slub.c:3662
[< none >] free_ctx+0x4b/0x70 kernel/events/core.c:872
[< inline >] __rcu_reclaim kernel/rcu/rcu.h:118
[< inline >] rcu_do_batch kernel/rcu/tree.c:2693
[< inline >] invoke_rcu_callbacks kernel/rcu/tree.c:2961
[< inline >] __rcu_process_callbacks kernel/rcu/tree.c:2928
[< none >] rcu_process_callbacks+0x631/0x19e0 kernel/rcu/tree.c:2945
[< none >] __do_softirq+0x2e5/0xb40 kernel/softirq.c:273
[< none >] run_ksoftirqd+0x20/0x50 kernel/softirq.c:662
[< none >] smpboot_thread_fn+0x41d/0x850 kernel/smpboot.c:163
[< none >] kthread+0x21d/0x2e0 drivers/block/aoe/aoecmd.c:1314
[< none >] ret_from_fork+0x3f/0x70 arch/x86/entry/entry_64.S:468
INFO: Indirect in __call_rcu.constprop.63+0x7e/0xe30 age=88 cpu=2 pid=30312
[< none >] __call_rcu.constprop.63+0x7e/0xe30 kernel/rcu/tree.c:3048
[< none >] call_rcu_sched+0x26/0x30 kernel/rcu/tree.c:3117
[< none >] put_ctx+0x150/0x250 kernel/events/core.c:891
[< inline >] perf_event_exit_task_context kernel/events/core.c:8860
[< none >] perf_event_exit_task+0x706/0xae0 kernel/events/core.c:8887
[< none >] do_exit+0x892/0x3050 kernel/exit.c:759
[< none >] do_group_exit+0xec/0x390 kernel/exit.c:880
[< none >] get_signal+0x677/0x1bf0 kernel/signal.c:2307
[< none >] do_signal+0x7e/0x2170 arch/x86/kernel/signal.c:709
[< none >] exit_to_usermode_loop+0xfe/0x1e0
arch/x86/entry/common.c:247
[< inline >] prepare_exit_to_usermode arch/x86/entry/common.c:282
[< none >] syscall_return_slowpath+0x16b/0x240
arch/x86/entry/common.c:344
[< none >] int_ret_from_sys_call+0x25/0x9f
arch/x86/entry/entry_64.S:281
INFO: Slab 0xffffea00017dde00 objects=21 used=8 fp=0xffff88005f77aec0
flags=0x5fffc0000004080
INFO: Object 0xffff88005f77aec0 @offset=11968 fp=0xffff88005f77b498
CPU: 3 PID: 30337 Comm: syzkaller_execu Tainted: G B
4.4.0-rc3+ #155
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
0000000000000003 ffff880065547680 ffffffff82e0f8b8 0000000041b58ab3
ffffffff87a9a27d ffffffff82e0f806 ffff880064972e80 ffffffff87abb3d1
ffff88003e806d00 0000000000000008 ffff88005f77aec0 ffff880065547680
Call Trace:
[< inline >] __dump_stack lib/dump_stack.c:15
[<ffffffff82e0f8b8>] dump_stack+0xb2/0xfa lib/dump_stack.c:50
[<ffffffff818369dc>] print_trailer+0x12c/0x210 mm/slub.c:659
[<ffffffff81841a3f>] object_err+0x2f/0x40 mm/slub.c:689
[< inline >] print_address_description mm/kasan/report.c:138
[<ffffffff818457a9>] kasan_report_error+0x5d9/0x860 mm/kasan/report.c:251
[< inline >] kasan_report mm/kasan/report.c:274
[<ffffffff81845bd4>] __asan_report_load8_noabort+0x54/0x70
mm/kasan/report.c:295
[<ffffffff814b25f9>] __lock_acquire+0x4e99/0x5100 kernel/locking/lockdep.c:3092
[<ffffffff814b517d>] lock_acquire+0x19d/0x3f0 kernel/locking/lockdep.c:3585
[< inline >] __raw_spin_lock include/linux/spinlock_api_smp.h:144
[<ffffffff86a89591>] _raw_spin_lock+0x31/0x40 kernel/locking/spinlock.c:151
[< inline >] perf_ctx_lock kernel/events/core.c:351
[<ffffffff816e15d9>] __perf_install_in_context+0x109/0xa00
kernel/events/core.c:2083
[<ffffffff816cb7ba>] remote_function+0x14a/0x200 kernel/events/core.c:74
[<ffffffff81572787>] generic_exec_single+0x2a7/0x490 kernel/smp.c:156
[<ffffffff81573350>] smp_call_function_single+0x200/0x310 kernel/smp.c:300
[<ffffffff816c9bd3>] task_function_call+0x123/0x160 kernel/events/core.c:101
[<ffffffff816d1bf1>] perf_install_in_context+0x201/0x340
kernel/events/core.c:2164
[<ffffffff816f62e5>] SYSC_perf_event_open+0x1465/0x21a0
kernel/events/core.c:8546
[<ffffffff816ff449>] SyS_perf_event_open+0x39/0x50 kernel/events/core.c:8242
[<ffffffff86a8a3b6>] entry_SYSCALL_64_fastpath+0x16/0x7a
arch/x86/entry/entry_64.S:185
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Peter Zijlstra <peterz@infradead.org> |
|---|---|
| Date | 2015-12-10 21:00 +0100 |
| Message-ID | <qEfxE-28N-3@gated-at.bofh.it> |
| In reply to | #1286783 |
On Tue, Dec 08, 2015 at 08:14:58PM +0100, Dmitry Vyukov wrote:
> Tested with your patches.
> The additional WARNING does not fire.
> For the rcu stacks, I had to change two more 2's to TRACK_NR and also
> moved memorization from call_rcu to __call_rcu, but now it is working.
> Two reports with indirect stack:
Ah nice, and sorry for the oversights. Obviously one can add this
indirect marker to __queue_work() as well to also cover the workqueue
offloading some other sites do.
> [<ffffffff81845bd4>] __asan_report_load8_noabort+0x54/0x70 mm/kasan/report.c:295
> [<ffffffff814b25f9>] __lock_acquire+0x4e99/0x5100 kernel/locking/lockdep.c:3092
> [<ffffffff814b517d>] lock_acquire+0x19d/0x3f0 kernel/locking/lockdep.c:3585
> [< inline >] __raw_spin_lock include/linux/spinlock_api_smp.h:144
> [<ffffffff86a89591>] _raw_spin_lock+0x31/0x40 kernel/locking/spinlock.c:151
> [< inline >] perf_ctx_lock kernel/events/core.c:351
So this here takes locks on cpuctx->ctx.lock and cpuctx->task_ctx->lock
and it is the latter that is pointing out into space.
> [<ffffffff816e15d9>] __perf_install_in_context+0x109/0xa00 kernel/events/core.c:2083
> [<ffffffff816cb7ba>] remote_function+0x14a/0x200 kernel/events/core.c:74
> [<ffffffff81572787>] generic_exec_single+0x2a7/0x490 kernel/smp.c:156
> [<ffffffff81573350>] smp_call_function_single+0x200/0x310 kernel/smp.c:300
> [<ffffffff816c9bd3>] task_function_call+0x123/0x160 kernel/events/core.c:101
> [<ffffffff816d1bf1>] perf_install_in_context+0x201/0x340 kernel/events/core.c:2164
> [<ffffffff816f62e5>] SYSC_perf_event_open+0x1465/0x21a0 kernel/events/core.c:8546
> [<ffffffff816ff449>] SyS_perf_event_open+0x39/0x50 kernel/events/core.c:8242
> [<ffffffff86a8a3b6>] entry_SYSCALL_64_fastpath+0x16/0x7a arch/x86/entry/entry_64.S:185
Now I managed to produce a related WARN:
[ 2338.884942] ------------[ cut here ]------------
[ 2338.890112] WARNING: CPU: 13 PID: 35162 at ../kernel/events/core.c:2702 task_ctx_sched_out+0x6b/0x80()
[ 2338.900504] Modules linked in:
[ 2338.903933] CPU: 13 PID: 35162 Comm: bash Not tainted 4.4.0-rc4-dirty #244
[ 2338.911610] Hardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.02.0002.122320131210 12/23/2013
[ 2338.923071] ffffffff81f1468e ffff8807c6457cb8 ffffffff815c680c 0000000000000000
[ 2338.931382] ffff8807c6457cf0 ffffffff810c8a56 ffffe8ffff8c1bd0 ffff8808132ed400
[ 2338.939678] 0000000000000286 ffff880813170380 ffff8808132ed400 ffff8807c6457d00
[ 2338.947987] Call Trace:
[ 2338.950726] [<ffffffff815c680c>] dump_stack+0x4e/0x82
[ 2338.956474] [<ffffffff810c8a56>] warn_slowpath_common+0x86/0xc0
[ 2338.963195] [<ffffffff810c8b4a>] warn_slowpath_null+0x1a/0x20
[ 2338.969720] [<ffffffff811a49cb>] task_ctx_sched_out+0x6b/0x80
[ 2338.976244] [<ffffffff811a62d2>] perf_event_exec+0xe2/0x180
[ 2338.982575] [<ffffffff8121fb6f>] setup_new_exec+0x6f/0x1b0
[ 2338.988810] [<ffffffff8126de83>] load_elf_binary+0x393/0x1660
[ 2338.995339] [<ffffffff811dc772>] ? get_user_pages+0x52/0x60
[ 2339.001669] [<ffffffff8121e297>] search_binary_handler+0x97/0x200
[ 2339.008581] [<ffffffff8121f8b3>] do_execveat_common.isra.33+0x543/0x6e0
[ 2339.016072] [<ffffffff8121fcea>] SyS_execve+0x3a/0x50
[ 2339.021819] [<ffffffff819fc165>] stub_execve+0x5/0x5
[ 2339.027469] [<ffffffff819fbeb2>] ? entry_SYSCALL_64_fastpath+0x12/0x71
[ 2339.034860] ---[ end trace ee1337c59a0ddeac ]---
Which is an existing WARN_ON_ONCE() indicating that cpuctx->task_ctx is
not what we expected it to be.
The below patch appears (its so far managed to run longer than previous
attempts, but I'm sure it'll explode the moment I've send this email) to
have cured it.
I'm not sure I can explain your problem with this, but I figure its
worth a try.
---
kernel/events/core.c | 15 +++++----------
1 file changed, 5 insertions(+), 10 deletions(-)
diff --git a/kernel/events/core.c b/kernel/events/core.c
index c3d61b92d805..d5293325d8c5 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -3136,15 +3136,16 @@ static int event_enable_on_exec(struct perf_event *event,
* Enable all of a task's events that have been marked enable-on-exec.
* This expects task == current.
*/
-static void perf_event_enable_on_exec(struct perf_event_context *ctx)
+static void perf_event_enable_on_exec(int ctxn)
{
- struct perf_event_context *clone_ctx = NULL;
+ struct perf_event_context *ctx, *clone_ctx = NULL;
struct perf_event *event;
unsigned long flags;
int enabled = 0;
int ret;
local_irq_save(flags);
+ ctx = current->perf_event_ctxp[ctxn];
if (!ctx || !ctx->nr_events)
goto out;
@@ -3187,17 +3188,11 @@ static void perf_event_enable_on_exec(struct perf_event_context *ctx)
void perf_event_exec(void)
{
- struct perf_event_context *ctx;
int ctxn;
rcu_read_lock();
- for_each_task_context_nr(ctxn) {
- ctx = current->perf_event_ctxp[ctxn];
- if (!ctx)
- continue;
-
- perf_event_enable_on_exec(ctx);
- }
+ for_each_task_context_nr(ctxn)
+ perf_event_enable_on_exec(ctxn);
rcu_read_unlock();
}
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Peter Zijlstra <peterz@infradead.org> |
|---|---|
| Date | 2015-12-08 17:30 +0100 |
| Message-ID | <qDtjk-4FX-17@gated-at.bofh.it> |
| In reply to | #1284191 |
On Fri, Dec 04, 2015 at 09:04:35PM +0100, Dmitry Vyukov wrote: > I was able to reproduce it by > restricting syscalls only to perf_event_open, perf ioctls and bpf > syscall. Does that include cpu hotplug? Sasha just reminded me there is a swevent use-after-free issue on hotplug. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Dmitry Vyukov <dvyukov@google.com> |
|---|---|
| Date | 2015-12-08 18:00 +0100 |
| Message-ID | <qDtMm-4Ra-13@gated-at.bofh.it> |
| In reply to | #1286654 |
On Tue, Dec 8, 2015 at 5:27 PM, Peter Zijlstra <peterz@infradead.org> wrote: > On Fri, Dec 04, 2015 at 09:04:35PM +0100, Dmitry Vyukov wrote: > >> I was able to reproduce it by >> restricting syscalls only to perf_event_open, perf ioctls and bpf >> syscall. > > Does that include cpu hotplug? I have CONFIG_HOTPLUG_CPU enabled. But I don't do actual plug/unplug. > Sasha just reminded me there is a swevent use-after-free issue on > hotplug. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Peter Zijlstra <peterz@infradead.org> |
|---|---|
| Date | 2015-12-08 17:30 +0100 |
| Message-ID | <qDtjk-4FX-27@gated-at.bofh.it> |
| In reply to | #1284191 |
On Fri, Dec 04, 2015 at 09:04:35PM +0100, Dmitry Vyukov wrote:
> Hello,
>
> While running syzkaller fuzzer I am seeing lots of the following
> use-after-free reports. Unfortunately all my numerous attempts to
> reproduce them in a controlled environment failed. They pop up during
> fuzzing periodically (once in several hours in a single VM), but
> whenever I try to stress-replay what happened in the VM before the
> report, the use-after-free does not reproduce.
Typical that :/
> Can somebody knowledgeable in perf subsystem look at the report? Maybe
> it is possible to figure out what happened based purely on the report.
> I can pretty reliably test any proposed fixes.
So I'm still going over the code, but meanwhile I tried reproducing this
using the perf_fuzzer and some debug code, but no luck with that.
Since you seem to be able to reproduce, could you do a run with the
below patch in to see if it tickles something?
---
diff --git a/kernel/events/core.c b/kernel/events/core.c
index 36babfd..58e504c 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -872,13 +872,22 @@ static void free_ctx(struct rcu_head *head)
kfree(ctx);
}
+#define for_each_task_context_nr(ctxn) \
+ for ((ctxn) = 0; (ctxn) < perf_nr_task_contexts; (ctxn)++)
+
static void put_ctx(struct perf_event_context *ctx)
{
if (atomic_dec_and_test(&ctx->refcount)) {
if (ctx->parent_ctx)
put_ctx(ctx->parent_ctx);
- if (ctx->task)
+ if (ctx->task) {
+ int ctxn;
+
+ for_each_task_context_nr(ctxn)
+ WARN_ON(ctx->task->perf_event_ctxp[ctxn] == ctx);
+
put_task_struct(ctx->task);
+ }
call_rcu(&ctx->rcu_head, free_ctx);
}
}
@@ -2649,9 +2658,6 @@ static void perf_pmu_sched_task(struct task_struct *prev,
static void perf_event_switch(struct task_struct *task,
struct task_struct *next_prev, bool sched_in);
-#define for_each_task_context_nr(ctxn) \
- for ((ctxn) = 0; (ctxn) < perf_nr_task_contexts; (ctxn)++)
-
/*
* Called from scheduler to remove the events of the current task,
* with interrupts disabled.
@@ -8931,6 +8937,8 @@ again:
mutex_unlock(&ctx->mutex);
+ task->perf_event_ctxp[ctxn] = NULL;
+
put_ctx(ctx);
}
}
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Ingo Molnar <mingo@kernel.org> |
|---|---|
| Date | 2015-12-08 20:00 +0100 |
| Message-ID | <qDvEt-60F-7@gated-at.bofh.it> |
| In reply to | #1286657 |
* Peter Zijlstra <peterz@infradead.org> wrote: > On Fri, Dec 04, 2015 at 09:04:35PM +0100, Dmitry Vyukov wrote: > > Hello, > > > > While running syzkaller fuzzer I am seeing lots of the following > > use-after-free reports. Unfortunately all my numerous attempts to > > reproduce them in a controlled environment failed. They pop up during > > fuzzing periodically (once in several hours in a single VM), but > > whenever I try to stress-replay what happened in the VM before the > > report, the use-after-free does not reproduce. > > Typical that :/ > > > Can somebody knowledgeable in perf subsystem look at the report? Maybe > > it is possible to figure out what happened based purely on the report. > > I can pretty reliably test any proposed fixes. > > So I'm still going over the code, but meanwhile I tried reproducing this > using the perf_fuzzer and some debug code, but no luck with that. > > Since you seem to be able to reproduce, could you do a run with the > below patch in to see if it tickles something? Btw., could we add more redundancy / debug code to the refcounting code? It seems to be a frequent source of very hard to find/fix races/bugs - so it should be ripe for some extra debug infrastructure ... Thanks, Ingo -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Peter Zijlstra <peterz@infradead.org> |
|---|---|
| Date | 2015-12-09 10:10 +0100 |
| Message-ID | <qDIV4-6nQ-25@gated-at.bofh.it> |
| In reply to | #1286774 |
On Tue, Dec 08, 2015 at 07:57:38PM +0100, Ingo Molnar wrote: > Btw., could we add more redundancy / debug code to the refcounting code? It seems > to be a frequent source of very hard to find/fix races/bugs - so it should be ripe > for some extra debug infrastructure ... I'll try, but its not easy. The biggest problem so far is the ctx::parent relation. We cannot easily track the reverse of that because of locking (we'd need to hold two ctx->lock, and we cannot because perf_event_context_sched_out()). The other two relations we have reverse maps for: task->perf_event_ctxp[] <-> ctx->task event->ctx <-> ctx->event_list Also, all 3 relations are (more or less) protected under ctx->lock: - unclone_ctx(): removes the ctx->parent link, with ctx->lock held - perf_remove_from_context(): removes the event from ctx::event_list, with ctx->lock held; leaves event->ctx set, because nasty games with ctx migration, it will get set to the new context, but can never be NULL. - perf_event_exit_task_context(): clears task->perf_event_ctxp[], with ctx->lock held; leaves ctx->task set, still looking into this. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [standalone]
Page 2 of 2 — ← Prev page 1 [2]
Back to top | Article view | linux.kernel
csiph-web