Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start

From	Josh Poimboeuf <jpoimboe@redhat.com>
Newsgroups	linux.kernel
Subject	Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start
Date	2016-11-28 23:00 +0100
Message-ID	<sIC7T-7Cs-11@gated-at.bofh.it> (permalink)
References	<sH6a6-5Oe-27@gated-at.bofh.it>
Organization	linux.* mail to news gateway

Show all headers | View raw

On Thu, Nov 24, 2016 at 12:33:48PM -0500, Vince Weaver wrote:
> 
> This is on a skylake machine, linus git as of yesterday after the various 
> kasan-related fixes went in.  Not sure if there were any that hadn't hit 
> upstream yet.
> 
> Anyway I can't tell from this one what the actual trigger is.  After this 
> mess the fuzzer process was locked, udev started complaining, and it 
> eventually died completely after a few hours of repeated messages like 
> this.
> 
> [38898.373183] INFO: rcu_sched self-detected stall on CPU
> [38898.378452]  7-...: (5249 ticks this GP) idle=727/140000000000001/0 softirq=3141908/3141908 fqs=2625 
> [38898.381211] INFO: rcu_sched detected stalls on CPUs/tasks:
> [38898.381214]  0-...: (1 GPs behind) idle=05f/140000000000001/2 softirq=3285458/3285459 fqs=2625 
> [38898.381217]  7-...: (5249 ticks this GP) idle=727/140000000000001/0 softirq=3141908/3141908 fqs=2625 
> [38898.381218]  (detected by 1, t=5252 jiffies, g=3685053, c=3685052, q=32)
> [38898.381244] ==================================================================
> [38898.381247] BUG: KASAN: stack-out-of-bounds in __unwind_start+0x1a2/0x1c0 at addr ffff8801e9727c28
> [38898.381248] Read of size 8 by task swapper/1/0
> [38898.381250] page:ffffea0007a5c9c0 count:0 mapcount:0 mapping:          (null) index:0x0
> [38898.381251] flags: 0x2ffff8000000000()
> [38898.381251] page dumped because: kasan: bad access detected
> [38898.381328] Memory state around the buggy address:
> [38898.381330]  ffff8801e9727b00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> [38898.381331]  ffff8801e9727b80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> [38898.381332] >ffff8801e9727c00: 00 00 00 00 f1 f1 f1 f1 00 00 00 00 f3 f3 f3 f3
> [38898.381333]                                   ^
> [38898.381334]  ffff8801e9727c80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f1
> [38898.381335]  ffff8801e9727d00: f1 f1 f1 00 00 00 f4 f2 f2 f2 f2 00 00 00 00 f3
> [38898.381335] ==================================================================
> [38898.510702]   (t=5284 jiffies g=3685053 c=3685052 q=32)
> 
> (That's all, the report above repeats but no useful things like a 
> backtrace are ever printed)

After looking at the RCU stall detection code, I think the KASAN error
and missing stack dump aren't very surprising.  RCU calls the scheduler
dump_cpu_task() function, which seems inherently problematic: it tries
to dump the stack of a task while it's running on another CPU.

There are some issues with that:

1) There's no way to find the starting frame of a currently running task
   from another CPU.

   In fact, I'm wondering how dump_cpu_task() ever worked at all?  It
   seems like you'd have to get lucky that the sp/bp registers stored by
   the last call to schedule() happen to point to a currently valid
   stack frame.

2) Even if there were a way to find the starting frame, it's racy
   because the target task could be overwriting the stack while we're
   reading it.

3) IRQ/exception stack dumps would be missing anyway because the stack
   dump code only looks at the current CPU's interrupt stacks.

Maybe dump_cpu_task() should instead run the stack dump directly from
the target CPU, e.g. with trigger_single_cpu_backtrace() or
smp_call_function_single()?

Paul, Peter, Ingo, any thoughts?

-- 
Josh

Back to linux.kernel | Previous | Next — Next in thread | Find similar | Unroll thread

Thread

Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start Josh Poimboeuf <jpoimboe@redhat.com> - 2016-11-28 23:00 +0100
  Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-11-29 01:50 +0100
    Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start Josh Poimboeuf <jpoimboe@redhat.com> - 2016-11-29 07:00 +0100
      Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start Peter Zijlstra <peterz@infradead.org> - 2016-11-29 10:20 +0100
        Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-11-29 15:10 +0100
          Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start Josh Poimboeuf <jpoimboe@redhat.com> - 2016-11-29 16:10 +0100
            Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start Petr Mladek <pmladek@suse.com> - 2016-11-29 17:20 +0100
              Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-11-29 19:10 +0100
            Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-11-29 18:00 +0100
              Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start Josh Poimboeuf <jpoimboe@redhat.com> - 2016-11-29 18:20 +0100
                Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-11-29 18:40 +0100
                Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start Petr Mladek <pmladek@suse.com> - 2016-11-30 11:00 +0100
      Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-11-29 11:30 +0100
      Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start Peter Zijlstra <peterz@infradead.org> - 2016-11-29 13:50 +0100
        Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-11-29 16:20 +0100
          Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start Petr Mladek <pmladek@suse.com> - 2016-11-29 17:30 +0100
            Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start Peter Zijlstra <peterz@infradead.org> - 2016-11-29 18:20 +0100
              Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-11-29 20:50 +0100
                Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start Peter Zijlstra <peterz@infradead.org> - 2016-11-29 21:00 +0100
                Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-11-29 21:10 +0100
                Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-11-29 21:40 +0100
                Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start Josh Poimboeuf <jpoimboe@redhat.com> - 2016-11-30 20:20 +0100
                Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-11-30 21:00 +0100
                Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start Peter Zijlstra <peterz@infradead.org> - 2016-12-01 07:00 +0100
                Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-12-01 13:40 +0100
                Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start Peter Zijlstra <peterz@infradead.org> - 2016-12-01 17:50 +0100
                Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-12-01 18:10 +0100
              Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start Petr Mladek <pmladek@suse.com> - 2016-11-30 11:10 +0100
                Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start Peter Zijlstra <peterz@infradead.org> - 2016-11-30 12:10 +0100

csiph-web