Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1522953 > unrolled thread
| Started by | Vince Weaver <vincent.weaver@maine.edu> |
|---|---|
| First post | 2016-11-15 18:50 +0100 |
| Last post | 2016-11-15 22:00 +0100 |
| Articles | 20 on this page of 30 — 7 participants |
Back to article view | Back to linux.kernel
perf: fuzzer KASAN unwind_get_return_address Vince Weaver <vincent.weaver@maine.edu> - 2016-11-15 18:50 +0100
Re: perf: fuzzer KASAN unwind_get_return_address Peter Zijlstra <peterz@infradead.org> - 2016-11-15 20:00 +0100
Re: perf: fuzzer KASAN unwind_get_return_address Vince Weaver <vincent.weaver@maine.edu> - 2016-11-15 20:10 +0100
Re: perf: fuzzer KASAN unwind_get_return_address Josh Poimboeuf <jpoimboe@redhat.com> - 2016-11-15 22:00 +0100
Re: perf: fuzzer KASAN unwind_get_return_address Peter Zijlstra <peterz@infradead.org> - 2016-11-16 14:10 +0100
Re: perf: fuzzer KASAN unwind_get_return_address Dmitry Vyukov <dvyukov@google.com> - 2016-11-16 14:20 +0100
Re: perf: fuzzer KASAN unwind_get_return_address Josh Poimboeuf <jpoimboe@redhat.com> - 2016-11-16 15:40 +0100
Re: perf: fuzzer KASAN unwind_get_return_address Josh Poimboeuf <jpoimboe@redhat.com> - 2016-11-16 16:00 +0100
Re: perf: fuzzer KASAN unwind_get_return_address Peter Zijlstra <peterz@infradead.org> - 2016-11-16 16:00 +0100
Re: perf: fuzzer KASAN unwind_get_return_address Josh Poimboeuf <jpoimboe@redhat.com> - 2016-11-17 05:50 +0100
Re: perf: fuzzer KASAN unwind_get_return_address Peter Zijlstra <peterz@infradead.org> - 2016-11-17 10:20 +0100
Re: perf: fuzzer KASAN unwind_get_return_address Josh Poimboeuf <jpoimboe@redhat.com> - 2016-11-17 18:10 +0100
Re: perf: fuzzer KASAN unwind_get_return_address Vince Weaver <vincent.weaver@maine.edu> - 2016-11-17 18:20 +0100
Re: perf: fuzzer KASAN unwind_get_return_address Dmitry Vyukov <dvyukov@google.com> - 2016-11-17 18:50 +0100
Re: perf: fuzzer KASAN unwind_get_return_address Josh Poimboeuf <jpoimboe@redhat.com> - 2016-11-17 18:40 +0100
Re: perf: fuzzer KASAN unwind_get_return_address Vince Weaver <vincent.weaver@maine.edu> - 2016-11-17 19:20 +0100
Re: perf: fuzzer KASAN unwind_get_return_address Peter Zijlstra <peterz@infradead.org> - 2016-11-17 18:20 +0100
Re: perf: fuzzer KASAN unwind_get_return_address Peter Zijlstra <peterz@infradead.org> - 2016-11-17 18:20 +0100
[tip:perf/urgent] perf/x86/intel: Cure bogus unwind from PEBS entries tip-bot for Peter Zijlstra <tipbot@zytor.com> - 2016-11-22 13:40 +0100
Re: perf: fuzzer KASAN unwind_get_return_address Josh Poimboeuf <jpoimboe@redhat.com> - 2016-11-17 18:50 +0100
Re: perf: fuzzer KASAN unwind_get_return_address Peter Zijlstra <peterz@infradead.org> - 2016-11-16 16:00 +0100
Re: perf: fuzzer KASAN unwind_get_return_address Vince Weaver <vincent.weaver@maine.edu> - 2016-11-16 16:10 +0100
[PATCH 1/2] unwind: prevent KASAN false positive warnings in guess unwinder Josh Poimboeuf <jpoimboe@redhat.com> - 2016-11-17 18:20 +0100
Re: [PATCH 1/2] unwind: prevent KASAN false positive warnings in guess unwinder Josh Poimboeuf <jpoimboe@redhat.com> - 2016-11-17 21:30 +0100
Re: [PATCH 1/2] unwind: prevent KASAN false positive warnings in guess unwinder Ingo Molnar <mingo@kernel.org> - 2016-11-18 09:40 +0100
[tip:x86/urgent] x86/unwind: Prevent KASAN false positive warnings in guess unwinder tip-bot for Josh Poimboeuf <tipbot@zytor.com> - 2016-11-18 10:10 +0100
[PATCH 2/2] dumpstack: prevent KASAN false positive warnings Josh Poimboeuf <jpoimboe@redhat.com> - 2016-11-17 18:20 +0100
[tip:x86/urgent] x86/dumpstack: Prevent KASAN false positive warnings tip-bot for Josh Poimboeuf <tipbot@zytor.com> - 2016-11-18 10:10 +0100
Re: perf: fuzzer KASAN unwind_get_return_address Dmitry Vyukov <dvyukov@google.com> - 2016-11-15 20:10 +0100
Re: perf: fuzzer KASAN unwind_get_return_address Josh Poimboeuf <jpoimboe@redhat.com> - 2016-11-15 22:00 +0100
Page 1 of 2 [1] 2 Next page →
| From | Vince Weaver <vincent.weaver@maine.edu> |
|---|---|
| Date | 2016-11-15 18:50 +0100 |
| Subject | perf: fuzzer KASAN unwind_get_return_address |
| Message-ID | <sDQ1P-7ZM-27@gated-at.bofh.it> |
Running on my haswell machine with the imc/uncore patch applied, the perf_fuzzer next tripped over this issue. [ 202.034495] BAD LUCK: lost 371 message(s) from NMI context! [ 202.034496] ================================================================== [ 202.048327] BUG: KASAN: stack-out-of-bounds in unwind_get_return_address+0x35/0x80 at addr ffff8800cff0bd90 [ 202.058826] Read of size 8 by task perf_fuzzer/16254 [ 202.064186] page:ffffea00033fc2c0 count:1 mapcount:0 mapping: (null) index:0x0^Ac [ 202.073068] flags: 0x1ffff8000000400(reserved) [ 202.077885] page dumped because: kasan: bad access detected [ 202.083880] CPU: 4 PID: 16254 Comm: perf_fuzzer Not tainted 4.9.0-rc5+ #5 [ 202.091204] Hardware name: LENOVO 10AM000AUS/SHARKBAY, BIOS FBKT72AUS 01/26/2014 [ 202.099181] ffff8800cff0b1d8^Ac ffffffff816bb796^Ac ffff8800cff0b270^Ac ffff8800cff0bd90^Ac [ 202.107896] ffff8800cff0b260^Ac ffffffff812fbe95^Ac 00007ffc9d1ab480^Ac 0000000000000000^Ac [ 202.116638] ffffffff8125117d^Ac 0000000000000092^Ac 0000000000000000^Ac ffff8800cff0b7c0^Ac [ 202.125339] Call Trace: [ 202.127994] <NMI> [<ffffffff816bb796>] dump_stack+0x63/0x8d [ 202.134184] [<ffffffff812fbe95>] kasan_report_error+0x495/0x4c0 [ 202.140680] [<ffffffff8125117d>] ? perf_output_begin+0x28d/0x4c0 [ 202.147228] [<ffffffff812fc319>] kasan_report+0x39/0x40 [ 202.152987] [<ffffffff81095ce5>] ? unwind_get_return_address+0x35/0x80 [ 202.160094] [<ffffffff812fa8fe>] __asan_load8+0x5e/0x70 [ 202.165859] [<ffffffff81095ce5>] unwind_get_return_address+0x35/0x80 [ 202.172817] [<ffffffff8100b08d>] perf_callchain_kernel+0x22d/0x270 [ 202.179590] [<ffffffff812fa7c4>] ? __asan_load4+0x24/0x80 [ 202.185548] [<ffffffff8100ae60>] ? arch_perf_update_userpage+0x130/0x130 [ 202.192849] [<ffffffff81252aaa>] get_perf_callchain+0x24a/0x3e0 [ 202.199339] [<ffffffff81252860>] ? put_callchain_buffers+0x50/0x50 [ 202.206092] [<ffffffff81095b17>] ? perf_get_regs_user+0x327/0x380 [ 202.212751] [<ffffffff81135fd0>] ? lock_release+0x30/0x540 [ 202.218803] [<ffffffff81252d05>] perf_callchain+0xc5/0xe0 [ 202.224767] [<ffffffff812fa7c4>] ? __asan_load4+0x24/0x80 [ 202.230696] [<ffffffff8124dbf9>] perf_prepare_sample+0x489/0x630 [ 202.237275] [<ffffffff81135fd0>] ? lock_release+0x30/0x540 [ 202.243266] [<ffffffff8124de9c>] ? perf_event_output_forward+0xfc/0x130 [ 202.250472] [<ffffffff8124dda0>] ? perf_prepare_sample+0x630/0x630 [ 202.257251] [<ffffffff8124e0ae>] perf_event_output+0xae/0x130 [ 202.263564] [<ffffffff8124e000>] ? perf_event_output_backward+0x130/0x130 [ 202.270964] [<ffffffff8124e000>] ? perf_event_output_backward+0x130/0x130 [ 202.278373] [<ffffffff81247cc2>] ? perf_event_update_userpage+0x212/0x2b0 [ 202.285772] [<ffffffff81247ab0>] ? perf_event_task_disable+0xc0/0xc0 [ 202.292744] [<ffffffff812fac4f>] ? __asan_loadN+0xf/0x20 [ 202.298581] [<ffffffff8101757d>] ? setup_pebs_sample_data+0x68d/0x830 [ 202.305622] [<ffffffff81017a91>] __intel_pmu_pebs_event+0x221/0x3a0 [ 202.312469] [<ffffffff81135e4d>] ? lock_acquire+0x3d/0x190 [ 202.318523] [<ffffffff81017870>] ? pebs_update_state+0x150/0x150 [ 202.325060] [<ffffffff8104c6ec>] ? get_stack_info+0x3c/0x150 [ 202.331259] [<ffffffff810106b7>] ? __intel_pmu_enable_all+0x77/0xf0 [ 202.338128] [<ffffffff812fa7c4>] ? __asan_load4+0x24/0x80 [ 202.344059] [<ffffffff81018b50>] ? intel_pmu_disable_bts+0x60/0x60 [ 202.350823] [<ffffffff812fa7c4>] ? __asan_load4+0x24/0x80 [ 202.356740] [<ffffffff81252d05>] ? perf_callchain+0xc5/0xe0 [ 202.362855] [<ffffffff81135fd0>] ? lock_release+0x30/0x540 [ 202.368855] [<ffffffff8124dc31>] ? perf_prepare_sample+0x4c1/0x630 [ 202.375619] [<ffffffff8124de84>] ? perf_event_output_forward+0xe4/0x130 [ 202.382849] [<ffffffff81017ffc>] intel_pmu_drain_pebs_nhm+0x3ec/0x530 [ 202.389899] [<ffffffff81017c10>] ? __intel_pmu_pebs_event+0x3a0/0x3a0 [ 202.396959] [<ffffffff81247caa>] ? perf_event_update_userpage+0x1fa/0x2b0 [ 202.406800] [<ffffffff81247cc2>] ? perf_event_update_userpage+0x212/0x2b0 [ 202.416486] [<ffffffff81247ab0>] ? perf_event_task_disable+0xc0/0xc0 [ 202.425720] [<ffffffff8101a832>] ? intel_pmu_lbr_read+0x32/0x790 [ 202.434566] [<ffffffff8123ba26>] ? __perf_event_overflow+0x116/0x280 [ 202.443735] [<ffffffff810144d8>] ? intel_bts_interrupt+0x88/0x1b0 [ 202.452538] [<ffffffff81012c7e>] intel_pmu_handle_irq+0x3ae/0x690 [ 202.461407] [<ffffffff810128d0>] ? intel_pmu_save_and_restart+0x80/0x80 [ 202.470877] [<ffffffff81135fd0>] ? lock_release+0x30/0x540 [ 202.479131] [<ffffffff81088eeb>] ? native_apic_msr_write+0x2b/0x30 [ 202.488181] [<ffffffff8108899c>] ? x2apic_send_IPI_self+0x3c/0x50 [ 202.497066] [<ffffffff81055d72>] ? native_sched_clock+0x62/0x140 [ 202.505919] [<ffffffff810081fd>] perf_event_nmi_handler+0x2d/0x50 [ 202.514832] [<ffffffff8104da91>] nmi_handle+0xb1/0x1d0 [ 202.522697] [<ffffffff8104d9e5>] ? nmi_handle+0x5/0x1d0 [ 202.530610] [<ffffffff8104e185>] default_do_nmi+0xe5/0x140 [ 202.538765] [<ffffffff8104e332>] do_nmi+0x152/0x1b0 [ 202.546254] [<ffffffff81b8f171>] end_repeat_nmi+0x1a/0x1e [ 202.554257] [<ffffffff810106b7>] ? __intel_pmu_enable_all+0x77/0xf0 [ 202.563167] [<ffffffff812475eb>] ? perf_event_task_tick+0x48b/0x5f0 [ 202.572060] [<ffffffff812475eb>] ? perf_event_task_tick+0x48b/0x5f0 [ 202.580864] [<ffffffff812475eb>] ? perf_event_task_tick+0x48b/0x5f0 [ 202.589703] <EOE> <IRQ> [<ffffffff81101571>] scheduler_tick+0xb1/0x150 [ 202.598985] [<ffffffff8116e7e7>] update_process_times+0x47/0x60 [ 202.607433] [<ffffffff81185e53>] tick_sched_handle.isra.14+0x33/0x80 [ 202.616314] [<ffffffff811869cb>] tick_sched_timer+0x4b/0x90 [ 202.624322] [<ffffffff8116fbfe>] __hrtimer_run_queues+0x21e/0x540 [ 202.632864] [<ffffffff81186980>] ? tick_sched_do_timer+0x50/0x50 [ 202.641337] [<ffffffff8116f9e0>] ? retrigger_next_event+0xa0/0xa0 [ 202.649947] [<ffffffff8117b8f6>] ? ktime_get_update_offsets_now+0xe6/0x190 [ 202.659411] [<ffffffff811707f0>] ? hrtimer_interrupt+0xb0/0x220 [ 202.667864] [<ffffffff8117082f>] hrtimer_interrupt+0xef/0x220 [ 202.676069] [<ffffffff8123b020>] ? perf_cgroup_attach+0xb0/0xb0 [ 202.684444] [<ffffffff8107ec2f>] local_apic_timer_interrupt+0x4f/0x80 [ 202.693422] [<ffffffff81b903d7>] smp_apic_timer_interrupt+0x57/0x70 [ 202.702203] [<ffffffff81b8f6a2>] apic_timer_interrupt+0x82/0x90 [ 202.710591] <EOI> [<ffffffff8123b020>] ? perf_cgroup_attach+0xb0/0xb0 [ 202.719609] [<ffffffff8118dc3a>] ? smp_call_function_single+0x14a/0x1b0 [ 202.728811] [<ffffffff8118dc30>] ? smp_call_function_single+0x140/0x1b0 [ 202.738039] [<ffffffff8118daf0>] ? generic_exec_single+0x170/0x170 [ 202.746727] [<ffffffff8123b020>] ? perf_cgroup_attach+0xb0/0xb0 [ 202.755181] [<ffffffff81238e48>] event_function_call+0x268/0x270 [ 202.763687] [<ffffffff812426d0>] ? task_ctx_sched_out+0x60/0x60 [ 202.772057] [<ffffffff81238be0>] ? task_function_call+0xc0/0xc0 [ 202.780404] [<ffffffff812426d0>] ? task_ctx_sched_out+0x60/0x60 [ 202.788768] [<ffffffff81238e79>] ? _perf_event_disable+0x29/0x70 [ 202.797258] [<ffffffff812383d0>] ? update_group_times+0x50/0x50 [ 202.805667] [<ffffffff81238e97>] ? _perf_event_disable+0x47/0x70 [ 202.814188] [<ffffffff8113a4d7>] ? do_raw_spin_unlock+0x97/0x130 [ 202.822733] [<ffffffff81238e50>] ? event_function_call+0x270/0x270 [ 202.831462] [<ffffffff81238ea8>] _perf_event_disable+0x58/0x70 [ 202.839778] [<ffffffff812386a3>] perf_event_for_each_child+0x53/0xd0 [ 202.848576] [<ffffffff81247a51>] perf_event_task_disable+0x61/0xc0 [ 202.857303] [<ffffffff810daee2>] SyS_prctl+0x3f2/0x690 [ 202.864853] [<ffffffff810daaf0>] ? SyS_umask+0x40/0x40 [ 202.872375] [<ffffffff81136c6a>] ? lockdep_sys_exit+0x1a/0xa0 [ 202.880517] [<ffffffff81004016>] ? lockdep_sys_exit_thunk+0x16/0x30 [ 202.889310] [<ffffffff81b8dabb>] entry_SYSCALL_64_fastpath+0x1e/0xb2 [ 202.898177] Memory state around the buggy address: [ 202.905288] ffff8800cff0bc80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 202.915044] ffff8800cff0bd00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 202.924697] >ffff8800cff0bd80: f3 f3 f3 f3 f3 f3 f3 f3 00 00 00 00 00 00 00 00 [ 202.934420] ^ [ 202.940352] ffff8800cff0be00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 202.950141] ffff8800cff0be80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 202.959835] ==================================================================
[toc] | [next] | [standalone]
| From | Peter Zijlstra <peterz@infradead.org> |
|---|---|
| Date | 2016-11-15 20:00 +0100 |
| Message-ID | <sDR7A-fI-19@gated-at.bofh.it> |
| In reply to | #1522953 |
On Tue, Nov 15, 2016 at 12:43:56PM -0500, Vince Weaver wrote: > > Running on my haswell machine with the imc/uncore patch applied, the > perf_fuzzer next tripped over this issue. > > [ 202.034495] BAD LUCK: lost 371 message(s) from NMI context! > [ 202.034496] ================================================================== > [ 202.048327] BUG: KASAN: stack-out-of-bounds in unwind_get_return_address+0x35/0x80 at addr ffff8800cff0bd90 > [ 202.058826] Read of size 8 by task perf_fuzzer/16254 > [ 202.064186] page:ffffea00033fc2c0 count:1 mapcount:0 mapping: (null) index:0x0^Ac > [ 202.073068] flags: 0x1ffff8000000400(reserved) > [ 202.077885] page dumped because: kasan: bad access detected > [ 202.083880] CPU: 4 PID: 16254 Comm: perf_fuzzer Not tainted 4.9.0-rc5+ #5 > [ 202.091204] Hardware name: LENOVO 10AM000AUS/SHARKBAY, BIOS FBKT72AUS 01/26/2014 > [ 202.099181] ffff8800cff0b1d8^Ac ffffffff816bb796^Ac ffff8800cff0b270^Ac ffff8800cff0bd90^Ac > [ 202.107896] ffff8800cff0b260^Ac ffffffff812fbe95^Ac 00007ffc9d1ab480^Ac 0000000000000000^Ac > [ 202.116638] ffffffff8125117d^Ac 0000000000000092^Ac 0000000000000000^Ac ffff8800cff0b7c0^Ac > [ 202.125339] Call Trace: > [ 202.127994] <NMI> [<ffffffff816bb796>] dump_stack+0x63/0x8d > [ 202.134184] [<ffffffff812fbe95>] kasan_report_error+0x495/0x4c0 > [ 202.140680] [<ffffffff8125117d>] ? perf_output_begin+0x28d/0x4c0 > [ 202.147228] [<ffffffff812fc319>] kasan_report+0x39/0x40 > [ 202.152987] [<ffffffff81095ce5>] ? unwind_get_return_address+0x35/0x80 > [ 202.160094] [<ffffffff812fa8fe>] __asan_load8+0x5e/0x70 > [ 202.165859] [<ffffffff81095ce5>] unwind_get_return_address+0x35/0x80 Josh, any ideas? > [ 202.172817] [<ffffffff8100b08d>] perf_callchain_kernel+0x22d/0x270 > [ 202.179590] [<ffffffff812fa7c4>] ? __asan_load4+0x24/0x80 > [ 202.185548] [<ffffffff8100ae60>] ? arch_perf_update_userpage+0x130/0x130 > [ 202.192849] [<ffffffff81252aaa>] get_perf_callchain+0x24a/0x3e0 > [ 202.199339] [<ffffffff81252860>] ? put_callchain_buffers+0x50/0x50 > [ 202.206092] [<ffffffff81095b17>] ? perf_get_regs_user+0x327/0x380 > [ 202.212751] [<ffffffff81135fd0>] ? lock_release+0x30/0x540 > [ 202.218803] [<ffffffff81252d05>] perf_callchain+0xc5/0xe0 > [ 202.224767] [<ffffffff812fa7c4>] ? __asan_load4+0x24/0x80 > [ 202.230696] [<ffffffff8124dbf9>] perf_prepare_sample+0x489/0x630 > [ 202.237275] [<ffffffff81135fd0>] ? lock_release+0x30/0x540 > [ 202.243266] [<ffffffff8124de9c>] ? perf_event_output_forward+0xfc/0x130 > [ 202.250472] [<ffffffff8124dda0>] ? perf_prepare_sample+0x630/0x630 > [ 202.257251] [<ffffffff8124e0ae>] perf_event_output+0xae/0x130 > [ 202.263564] [<ffffffff8124e000>] ? perf_event_output_backward+0x130/0x130 > [ 202.270964] [<ffffffff8124e000>] ? perf_event_output_backward+0x130/0x130 > [ 202.278373] [<ffffffff81247cc2>] ? perf_event_update_userpage+0x212/0x2b0 > [ 202.285772] [<ffffffff81247ab0>] ? perf_event_task_disable+0xc0/0xc0 > [ 202.292744] [<ffffffff812fac4f>] ? __asan_loadN+0xf/0x20 > [ 202.298581] [<ffffffff8101757d>] ? setup_pebs_sample_data+0x68d/0x830 > [ 202.305622] [<ffffffff81017a91>] __intel_pmu_pebs_event+0x221/0x3a0 > [ 202.312469] [<ffffffff81135e4d>] ? lock_acquire+0x3d/0x190 > [ 202.318523] [<ffffffff81017870>] ? pebs_update_state+0x150/0x150 > [ 202.325060] [<ffffffff8104c6ec>] ? get_stack_info+0x3c/0x150 > [ 202.331259] [<ffffffff810106b7>] ? __intel_pmu_enable_all+0x77/0xf0 > [ 202.338128] [<ffffffff812fa7c4>] ? __asan_load4+0x24/0x80 > [ 202.344059] [<ffffffff81018b50>] ? intel_pmu_disable_bts+0x60/0x60 > [ 202.350823] [<ffffffff812fa7c4>] ? __asan_load4+0x24/0x80 > [ 202.356740] [<ffffffff81252d05>] ? perf_callchain+0xc5/0xe0 > [ 202.362855] [<ffffffff81135fd0>] ? lock_release+0x30/0x540 > [ 202.368855] [<ffffffff8124dc31>] ? perf_prepare_sample+0x4c1/0x630 > [ 202.375619] [<ffffffff8124de84>] ? perf_event_output_forward+0xe4/0x130 > [ 202.382849] [<ffffffff81017ffc>] intel_pmu_drain_pebs_nhm+0x3ec/0x530 > [ 202.389899] [<ffffffff81017c10>] ? __intel_pmu_pebs_event+0x3a0/0x3a0 > [ 202.396959] [<ffffffff81247caa>] ? perf_event_update_userpage+0x1fa/0x2b0 > [ 202.406800] [<ffffffff81247cc2>] ? perf_event_update_userpage+0x212/0x2b0 > [ 202.416486] [<ffffffff81247ab0>] ? perf_event_task_disable+0xc0/0xc0 > [ 202.425720] [<ffffffff8101a832>] ? intel_pmu_lbr_read+0x32/0x790 > [ 202.434566] [<ffffffff8123ba26>] ? __perf_event_overflow+0x116/0x280 > [ 202.443735] [<ffffffff810144d8>] ? intel_bts_interrupt+0x88/0x1b0 > [ 202.452538] [<ffffffff81012c7e>] intel_pmu_handle_irq+0x3ae/0x690 > [ 202.461407] [<ffffffff810128d0>] ? intel_pmu_save_and_restart+0x80/0x80 > [ 202.470877] [<ffffffff81135fd0>] ? lock_release+0x30/0x540 > [ 202.479131] [<ffffffff81088eeb>] ? native_apic_msr_write+0x2b/0x30 > [ 202.488181] [<ffffffff8108899c>] ? x2apic_send_IPI_self+0x3c/0x50 > [ 202.497066] [<ffffffff81055d72>] ? native_sched_clock+0x62/0x140 > [ 202.505919] [<ffffffff810081fd>] perf_event_nmi_handler+0x2d/0x50 > [ 202.514832] [<ffffffff8104da91>] nmi_handle+0xb1/0x1d0 > [ 202.522697] [<ffffffff8104d9e5>] ? nmi_handle+0x5/0x1d0 > [ 202.530610] [<ffffffff8104e185>] default_do_nmi+0xe5/0x140 > [ 202.538765] [<ffffffff8104e332>] do_nmi+0x152/0x1b0 > [ 202.546254] [<ffffffff81b8f171>] end_repeat_nmi+0x1a/0x1e > [ 202.554257] [<ffffffff810106b7>] ? __intel_pmu_enable_all+0x77/0xf0 > [ 202.563167] [<ffffffff812475eb>] ? perf_event_task_tick+0x48b/0x5f0 > [ 202.572060] [<ffffffff812475eb>] ? perf_event_task_tick+0x48b/0x5f0 > [ 202.580864] [<ffffffff812475eb>] ? perf_event_task_tick+0x48b/0x5f0 > [ 202.589703] <EOE> <IRQ> [<ffffffff81101571>] scheduler_tick+0xb1/0x150 > [ 202.598985] [<ffffffff8116e7e7>] update_process_times+0x47/0x60 > [ 202.607433] [<ffffffff81185e53>] tick_sched_handle.isra.14+0x33/0x80 > [ 202.616314] [<ffffffff811869cb>] tick_sched_timer+0x4b/0x90 > [ 202.624322] [<ffffffff8116fbfe>] __hrtimer_run_queues+0x21e/0x540 > [ 202.632864] [<ffffffff81186980>] ? tick_sched_do_timer+0x50/0x50 > [ 202.641337] [<ffffffff8116f9e0>] ? retrigger_next_event+0xa0/0xa0 > [ 202.649947] [<ffffffff8117b8f6>] ? ktime_get_update_offsets_now+0xe6/0x190 > [ 202.659411] [<ffffffff811707f0>] ? hrtimer_interrupt+0xb0/0x220 > [ 202.667864] [<ffffffff8117082f>] hrtimer_interrupt+0xef/0x220 > [ 202.676069] [<ffffffff8123b020>] ? perf_cgroup_attach+0xb0/0xb0 > [ 202.684444] [<ffffffff8107ec2f>] local_apic_timer_interrupt+0x4f/0x80 > [ 202.693422] [<ffffffff81b903d7>] smp_apic_timer_interrupt+0x57/0x70 > [ 202.702203] [<ffffffff81b8f6a2>] apic_timer_interrupt+0x82/0x90 > [ 202.710591] <EOI> [<ffffffff8123b020>] ? perf_cgroup_attach+0xb0/0xb0 > [ 202.719609] [<ffffffff8118dc3a>] ? smp_call_function_single+0x14a/0x1b0 > [ 202.728811] [<ffffffff8118dc30>] ? smp_call_function_single+0x140/0x1b0 > [ 202.738039] [<ffffffff8118daf0>] ? generic_exec_single+0x170/0x170 > [ 202.746727] [<ffffffff8123b020>] ? perf_cgroup_attach+0xb0/0xb0 > [ 202.755181] [<ffffffff81238e48>] event_function_call+0x268/0x270 > [ 202.763687] [<ffffffff812426d0>] ? task_ctx_sched_out+0x60/0x60 > [ 202.772057] [<ffffffff81238be0>] ? task_function_call+0xc0/0xc0 > [ 202.780404] [<ffffffff812426d0>] ? task_ctx_sched_out+0x60/0x60 > [ 202.788768] [<ffffffff81238e79>] ? _perf_event_disable+0x29/0x70 > [ 202.797258] [<ffffffff812383d0>] ? update_group_times+0x50/0x50 > [ 202.805667] [<ffffffff81238e97>] ? _perf_event_disable+0x47/0x70 > [ 202.814188] [<ffffffff8113a4d7>] ? do_raw_spin_unlock+0x97/0x130 > [ 202.822733] [<ffffffff81238e50>] ? event_function_call+0x270/0x270 > [ 202.831462] [<ffffffff81238ea8>] _perf_event_disable+0x58/0x70 > [ 202.839778] [<ffffffff812386a3>] perf_event_for_each_child+0x53/0xd0 > [ 202.848576] [<ffffffff81247a51>] perf_event_task_disable+0x61/0xc0 > [ 202.857303] [<ffffffff810daee2>] SyS_prctl+0x3f2/0x690 > [ 202.864853] [<ffffffff810daaf0>] ? SyS_umask+0x40/0x40 > [ 202.872375] [<ffffffff81136c6a>] ? lockdep_sys_exit+0x1a/0xa0 > [ 202.880517] [<ffffffff81004016>] ? lockdep_sys_exit_thunk+0x16/0x30 > [ 202.889310] [<ffffffff81b8dabb>] entry_SYSCALL_64_fastpath+0x1e/0xb2 > [ 202.898177] Memory state around the buggy address: > [ 202.905288] ffff8800cff0bc80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > [ 202.915044] ffff8800cff0bd00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > [ 202.924697] >ffff8800cff0bd80: f3 f3 f3 f3 f3 f3 f3 f3 00 00 00 00 00 00 00 00 > [ 202.934420] ^ > [ 202.940352] ffff8800cff0be00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > [ 202.950141] ffff8800cff0be80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > [ 202.959835] ================================================================== >
[toc] | [prev] | [next] | [standalone]
| From | Vince Weaver <vincent.weaver@maine.edu> |
|---|---|
| Date | 2016-11-15 20:10 +0100 |
| Message-ID | <sDRhf-ya-3@gated-at.bofh.it> |
| In reply to | #1523006 |
On Tue, 15 Nov 2016, Peter Zijlstra wrote:
> On Tue, Nov 15, 2016 at 12:43:56PM -0500, Vince Weaver wrote:
> >
> > Running on my haswell machine with the imc/uncore patch applied, the
> > perf_fuzzer next tripped over this issue.
> >
> > [ 202.034495] BAD LUCK: lost 371 message(s) from NMI context!
> > [ 202.034496] ==================================================================
> > [ 202.048327] BUG: KASAN: stack-out-of-bounds in unwind_get_return_address+0x35/0x80 at addr ffff8800cff0bd90
> > [ 202.058826] Read of size 8 by task perf_fuzzer/16254
> > [ 202.064186] page:ffffea00033fc2c0 count:1 mapcount:0 mapping: (null) index:0x0^Ac
> > [ 202.073068] flags: 0x1ffff8000000400(reserved)
> > [ 202.077885] page dumped because: kasan: bad access detected
> > [ 202.083880] CPU: 4 PID: 16254 Comm: perf_fuzzer Not tainted 4.9.0-rc5+ #5
> > [ 202.091204] Hardware name: LENOVO 10AM000AUS/SHARKBAY, BIOS FBKT72AUS 01/26/2014
> > [ 202.099181] ffff8800cff0b1d8^Ac ffffffff816bb796^Ac ffff8800cff0b270^Ac ffff8800cff0bd90^Ac
> > [ 202.107896] ffff8800cff0b260^Ac ffffffff812fbe95^Ac 00007ffc9d1ab480^Ac 0000000000000000^Ac
> > [ 202.116638] ffffffff8125117d^Ac 0000000000000092^Ac 0000000000000000^Ac ffff8800cff0b7c0^Ac
> > [ 202.125339] Call Trace:
> > [ 202.127994] <NMI> [<ffffffff816bb796>] dump_stack+0x63/0x8d
> > [ 202.134184] [<ffffffff812fbe95>] kasan_report_error+0x495/0x4c0
> > [ 202.140680] [<ffffffff8125117d>] ? perf_output_begin+0x28d/0x4c0
> > [ 202.147228] [<ffffffff812fc319>] kasan_report+0x39/0x40
> > [ 202.152987] [<ffffffff81095ce5>] ? unwind_get_return_address+0x35/0x80
> > [ 202.160094] [<ffffffff812fa8fe>] __asan_load8+0x5e/0x70
> > [ 202.165859] [<ffffffff81095ce5>] unwind_get_return_address+0x35/0x80
>
> Josh, any ideas?
From what I can tell this maps to:
unsigned long unwind_get_return_address(struct unwind_state *state)
{
unsigned long addr;
unsigned long *addr_p = unwind_get_return_address_ptr(state);
if (unwind_done(state))
return 0;
>> addr = ftrace_graph_ret_addr(state->task, &state->graph_idx, *addr_p,
addr_p);
return __kernel_text_address(addr) ? addr : 0;
}
[toc] | [prev] | [next] | [standalone]
| From | Josh Poimboeuf <jpoimboe@redhat.com> |
|---|---|
| Date | 2016-11-15 22:00 +0100 |
| Message-ID | <sDSZH-1sK-3@gated-at.bofh.it> |
| In reply to | #1523020 |
On Tue, Nov 15, 2016 at 02:05:50PM -0500, Vince Weaver wrote:
> On Tue, 15 Nov 2016, Peter Zijlstra wrote:
>
> > On Tue, Nov 15, 2016 at 12:43:56PM -0500, Vince Weaver wrote:
> > >
> > > Running on my haswell machine with the imc/uncore patch applied, the
> > > perf_fuzzer next tripped over this issue.
> > >
> > > [ 202.034495] BAD LUCK: lost 371 message(s) from NMI context!
> > > [ 202.034496] ==================================================================
> > > [ 202.048327] BUG: KASAN: stack-out-of-bounds in unwind_get_return_address+0x35/0x80 at addr ffff8800cff0bd90
> > > [ 202.058826] Read of size 8 by task perf_fuzzer/16254
> > > [ 202.064186] page:ffffea00033fc2c0 count:1 mapcount:0 mapping: (null) index:0x0^Ac
> > > [ 202.073068] flags: 0x1ffff8000000400(reserved)
> > > [ 202.077885] page dumped because: kasan: bad access detected
> > > [ 202.083880] CPU: 4 PID: 16254 Comm: perf_fuzzer Not tainted 4.9.0-rc5+ #5
> > > [ 202.091204] Hardware name: LENOVO 10AM000AUS/SHARKBAY, BIOS FBKT72AUS 01/26/2014
> > > [ 202.099181] ffff8800cff0b1d8^Ac ffffffff816bb796^Ac ffff8800cff0b270^Ac ffff8800cff0bd90^Ac
> > > [ 202.107896] ffff8800cff0b260^Ac ffffffff812fbe95^Ac 00007ffc9d1ab480^Ac 0000000000000000^Ac
> > > [ 202.116638] ffffffff8125117d^Ac 0000000000000092^Ac 0000000000000000^Ac ffff8800cff0b7c0^Ac
> > > [ 202.125339] Call Trace:
> > > [ 202.127994] <NMI> [<ffffffff816bb796>] dump_stack+0x63/0x8d
> > > [ 202.134184] [<ffffffff812fbe95>] kasan_report_error+0x495/0x4c0
> > > [ 202.140680] [<ffffffff8125117d>] ? perf_output_begin+0x28d/0x4c0
> > > [ 202.147228] [<ffffffff812fc319>] kasan_report+0x39/0x40
> > > [ 202.152987] [<ffffffff81095ce5>] ? unwind_get_return_address+0x35/0x80
> > > [ 202.160094] [<ffffffff812fa8fe>] __asan_load8+0x5e/0x70
> > > [ 202.165859] [<ffffffff81095ce5>] unwind_get_return_address+0x35/0x80
> >
> > Josh, any ideas?
>
> From what I can tell this maps to:
>
> unsigned long unwind_get_return_address(struct unwind_state *state)
> {
> unsigned long addr;
> unsigned long *addr_p = unwind_get_return_address_ptr(state);
>
> if (unwind_done(state))
> return 0;
>
> >> addr = ftrace_graph_ret_addr(state->task, &state->graph_idx, *addr_p,
> addr_p);
>
> return __kernel_text_address(addr) ? addr : 0;
> }
Hi Vince,
Would you mind posting a disassembly of unwind_get_return_address()?
Any idea how recreatable it is? (In particular I'd be interested in
seeing this dump with the latest unwinder improvements in the -tip tree,
which dump the pt_regs associated with an interrupt.)
--
Josh
[toc] | [prev] | [next] | [standalone]
| From | Peter Zijlstra <peterz@infradead.org> |
|---|---|
| Date | 2016-11-16 14:10 +0100 |
| Message-ID | <sE88q-3gV-19@gated-at.bofh.it> |
| In reply to | #1523058 |
On Tue, Nov 15, 2016 at 02:57:48PM -0600, Josh Poimboeuf wrote:
> Would you mind posting a disassembly of unwind_get_return_address()?
$ objdump -D ivb-dbg/vmlinux | awk '/<[^>]*>:/ { p=0; } /<unwind_get_return_address>:/ { p=1; } { if (p) print $0; }'
ffffffff811afd10 <unwind_get_return_address>:
ffffffff811afd10: e8 eb cc f4 01 callq ffffffff830fca00 <__fentry__>
ffffffff811afd15: 48 b8 00 00 00 00 00 movabs $0xdffffc0000000000,%rax
ffffffff811afd1c: fc ff df
ffffffff811afd1f: 48 89 fa mov %rdi,%rdx
ffffffff811afd22: 53 push %rbx
ffffffff811afd23: 48 89 fb mov %rdi,%rbx
ffffffff811afd26: 48 c1 ea 03 shr $0x3,%rdx
ffffffff811afd2a: 48 83 ec 18 sub $0x18,%rsp
ffffffff811afd2e: 0f b6 14 02 movzbl (%rdx,%rax,1),%edx
ffffffff811afd32: 48 89 f8 mov %rdi,%rax
ffffffff811afd35: 83 e0 07 and $0x7,%eax
ffffffff811afd38: 83 c0 03 add $0x3,%eax
ffffffff811afd3b: 38 d0 cmp %dl,%al
ffffffff811afd3d: 7c 04 jl ffffffff811afd43 <unwind_get_return_address+0x33>
ffffffff811afd3f: 84 d2 test %dl,%dl
ffffffff811afd41: 75 75 jne ffffffff811afdb8 <unwind_get_return_address+0xa8>
ffffffff811afd43: 8b 03 mov (%rbx),%eax
ffffffff811afd45: 85 c0 test %eax,%eax
ffffffff811afd47: 75 08 jne ffffffff811afd51 <unwind_get_return_address+0x41>
ffffffff811afd49: 48 83 c4 18 add $0x18,%rsp
ffffffff811afd4d: 31 c0 xor %eax,%eax
ffffffff811afd4f: 5b pop %rbx
ffffffff811afd50: c3 retq
ffffffff811afd51: 48 8d 7b 38 lea 0x38(%rbx),%rdi
ffffffff811afd55: 48 b8 00 00 00 00 00 movabs $0xdffffc0000000000,%rax
ffffffff811afd5c: fc ff df
ffffffff811afd5f: 48 89 fa mov %rdi,%rdx
ffffffff811afd62: 48 c1 ea 03 shr $0x3,%rdx
ffffffff811afd66: 80 3c 02 00 cmpb $0x0,(%rdx,%rax,1)
ffffffff811afd6a: 75 53 jne ffffffff811afdbf <unwind_get_return_address+0xaf>
ffffffff811afd6c: 48 b8 00 00 00 00 00 movabs $0xdffffc0000000000,%rax
ffffffff811afd73: fc ff df
ffffffff811afd76: 48 8b 4b 38 mov 0x38(%rbx),%rcx
ffffffff811afd7a: 48 89 ca mov %rcx,%rdx
ffffffff811afd7d: 48 c1 ea 03 shr $0x3,%rdx
ffffffff811afd81: 80 3c 02 00 cmpb $0x0,(%rdx,%rax,1)
ffffffff811afd85: 75 3f jne ffffffff811afdc6 <unwind_get_return_address+0xb6>
ffffffff811afd87: 48 8d 7b 28 lea 0x28(%rbx),%rdi
ffffffff811afd8b: 48 8b 11 mov (%rcx),%rdx
ffffffff811afd8e: 48 b8 00 00 00 00 00 movabs $0xdffffc0000000000,%rax
ffffffff811afd95: fc ff df
ffffffff811afd98: 48 8d 73 30 lea 0x30(%rbx),%rsi
ffffffff811afd9c: 49 89 f8 mov %rdi,%r8
ffffffff811afd9f: 49 c1 e8 03 shr $0x3,%r8
ffffffff811afda3: 41 80 3c 00 00 cmpb $0x0,(%r8,%rax,1)
ffffffff811afda8: 75 2e jne ffffffff811afdd8 <unwind_get_return_address+0xc8>
ffffffff811afdaa: 48 8b 7b 28 mov 0x28(%rbx),%rdi
ffffffff811afdae: 48 83 c4 18 add $0x18,%rsp
ffffffff811afdb2: 5b pop %rbx
ffffffff811afdb3: e9 08 98 2a 00 jmpq ffffffff814595c0 <ftrace_graph_ret_addr>
ffffffff811afdb8: e8 53 7d 42 00 callq ffffffff815d7b10 <__asan_report_load4_noabort>
ffffffff811afdbd: eb 84 jmp ffffffff811afd43 <unwind_get_return_address+0x33>
ffffffff811afdbf: e8 9c 7d 42 00 callq ffffffff815d7b60 <__asan_report_load8_noabort>
ffffffff811afdc4: eb a6 jmp ffffffff811afd6c <unwind_get_return_address+0x5c>
ffffffff811afdc6: 48 89 cf mov %rcx,%rdi
ffffffff811afdc9: 48 89 0c 24 mov %rcx,(%rsp)
ffffffff811afdcd: e8 8e 7d 42 00 callq ffffffff815d7b60 <__asan_report_load8_noabort>
ffffffff811afdd2: 48 8b 0c 24 mov (%rsp),%rcx
ffffffff811afdd6: eb af jmp ffffffff811afd87 <unwind_get_return_address+0x77>
ffffffff811afdd8: 48 89 74 24 10 mov %rsi,0x10(%rsp)
ffffffff811afddd: 48 89 54 24 08 mov %rdx,0x8(%rsp)
ffffffff811afde2: 48 89 0c 24 mov %rcx,(%rsp)
ffffffff811afde6: e8 75 7d 42 00 callq ffffffff815d7b60 <__asan_report_load8_noabort>
ffffffff811afdeb: 48 8b 74 24 10 mov 0x10(%rsp),%rsi
ffffffff811afdf0: 48 8b 54 24 08 mov 0x8(%rsp),%rdx
ffffffff811afdf5: 48 8b 0c 24 mov (%rsp),%rcx
ffffffff811afdf9: eb af jmp ffffffff811afdaa <unwind_get_return_address+0x9a>
ffffffff811afdfb: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
> Any idea how recreatable it is? (In particular I'd be interested in
> seeing this dump with the latest unwinder improvements in the -tip tree,
> which dump the pt_regs associated with an interrupt.)
Fairly reproducable it seems, doesn't seem to include pt_regs dumps
though :/
tip/master as of this morning.
3==================================================================
3BUG: KASAN: stack-out-of-bounds in unwind_next_frame+0x1ba/0x1f0 at addr ffff88042fc87be0
3Read of size 8 by task swapper/28/0
0page:ffffea0010bf21c0 count:1 mapcount:0 mapping: (null) index:0x0c
0flags: 0x2ffff8000000400(reserved)
1page dumped because: kasan: bad access detected
dCPU: 28 PID: 0 Comm: swapper/28 Not tainted 4.9.0-rc5-00530-gd8866fc-dirty #2
dHardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.02.0002.122320131210 12/23/2013
dCall Trace:
d <NMI>
d ? dump_stack+0x5e/0x89
d ? kasan_report_error+0x4a5/0x4d0
d ? __asan_report_load8_noabort+0x45/0x50
d ? __kernel_text_address+0x20/0xa0
d ? unwind_next_frame+0x1ba/0x1f0
d ? unwind_next_frame+0x1ba/0x1f0
d ? perf_callchain_kernel+0x33c/0x540
d ? arch_perf_update_userpage+0x340/0x340
d ? get_perf_callchain+0x24d/0x610
d ? put_callchain_buffers+0x50/0x50
d ? number+0x653/0x830
d ? perf_callchain+0x126/0x190
d ? perf_prepare_sample+0x720/0x1010
d ? perf_event_output_forward+0x81/0xf0
d ? perf_prepare_sample+0x1010/0x1010
d ? pointer+0x880/0x880
d ? perf_event_update_userpage+0x16/0x730
d ? __perf_event_overflow+0x1a0/0x510
d ? intel_pmu_handle_irq+0x34b/0xa90
d ? intel_pmu_save_and_restart+0xd0/0xd0
d ? acpi_os_read_memory+0x205/0x23c
d ? format_decode+0xc5/0x7a0
d ? vunmap_page_range+0x26a/0x400
d ? ghes_copy_tofrom_phys+0x141/0x270
d ? ghes_read_estatus+0x112/0x5a0
d ? ghes_copy_tofrom_phys+0x270/0x270
d ? early_printk+0xa4/0xd0
d ? devkmsg_sysctl_set_loglvl+0x160/0x160
d ? perf_event_nmi_handler+0x28/0x40
d ? nmi_handle+0xa1/0x250
d ? default_do_nmi+0x61/0x170
d ? do_nmi+0x191/0x200
d ? end_repeat_nmi+0x1a/0x1e
d ? format_decode+0xc5/0x7a0
d ? format_decode+0xc5/0x7a0
d ? format_decode+0xc5/0x7a0
d <EOE>
d <IRQ>
d ? vsnprintf+0xfc/0x15e0
d ? pointer+0x880/0x880
d ? x86_pmu_enable_all+0x1c0/0x1c0
d ? vscnprintf+0x9/0x30
d ? early_vprintk+0xb0/0x130
d ? trace_raw_output_console+0x160/0x160
d ? memcpy+0x34/0x50
d ? x86_pmu_commit_txn+0x180/0x260
d ? events_sysfs_show+0xb0/0xb0
d ? save_stack+0x33/0xb0
d ? hrtimer_init+0x120/0x120
d ? timerqueue_del+0x62/0x140
d ? perf_event_update_userpage+0x16/0x730
d ? perf_event_update_userpage+0x16/0x730
d ? x86_perf_event_set_period+0x239/0x450
d ? perf_event_update_userpage+0x16/0x730
d ? x86_pmu_enable+0x5f7/0xaa0
d ? printk+0xb6/0xef
d ? printk_emit+0xa0/0xa0
d ? _raw_spin_unlock_irqrestore+0x42/0x70
d ? ___ratelimit+0x1e4/0x3f0
d ? irq_work_run_list+0xa1/0xf0
d ? irq_work_run+0x14/0x40
d ? smp_call_function_single_interrupt+0x60/0x80
d ? call_function_single_interrupt+0x89/0x90
d <EOI>
d ? cpuidle_enter_state+0x113/0x780
d ? cpuidle_enter_state+0x10e/0x780
d ? cpu_load_update_nohz_stop+0x155/0x1b0
d ? cpu_startup_entry+0x19a/0x2c0
d ? start_cpu+0x5/0x14
3Memory state around the buggy address:
3 ffff88042fc87a80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3 ffff88042fc87b00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3>ffff88042fc87b80: 00 00 00 00 00 00 00 00 00 00 00 00 f1 f1 f1 f1
3 ^
3 ffff88042fc87c00: 00 f4 f4 f4 f3 f3 f3 f3 00 00 00 00 00 00 00 00
3 ffff88042fc87c80: 00 f1 f1 f1 f1 00 00 00 00 00 00 00 00 00 00 00
3==================================================================
3==================================================================
3BUG: KASAN: stack-out-of-bounds in unwind_next_frame+0x1ba/0x1f0 at addr ffff880424a97878
3Read of size 8 by task perf_fuzzer/3451
0page:ffffea001092a5c0 count:0 mapcount:0 mapping: (null) index:0x0c
0flags: 0x2ffff8000000000()
1page dumped because: kasan: bad access detected
dCPU: 28 PID: 3451 Comm: perf_fuzzer Not tainted 4.9.0-rc5-00530-gd8866fc-dirty #2
dHardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.02.0002.122320131210 12/23/2013
dCall Trace:
d ? dump_stack+0x5e/0x89
d ? kasan_report_error+0x4a5/0x4d0
d ? __asan_report_load8_noabort+0x45/0x50
d ? __kernel_text_address+0x20/0xa0
d ? unwind_next_frame+0x1ba/0x1f0
d ? unwind_next_frame+0x1ba/0x1f0
d ? perf_callchain_kernel+0x33c/0x540
d ? arch_perf_update_userpage+0x340/0x340
d ? get_perf_callchain+0x24d/0x610
d ? put_callchain_buffers+0x50/0x50
d ? ipv6_flowlabel_opt+0x1111/0x17d0
d ? perf_log_itrace_start+0x3a0/0x3a0
d ? cpumask_next_and+0x5a/0xa0
d ? ktime_get_raw_fast_ns+0xd3/0x1e0
d ? perf_callchain+0x126/0x190
d ? perf_prepare_sample+0x720/0x1010
d ? perf_event_output_forward+0x81/0xf0
d ? perf_prepare_sample+0x1010/0x1010
d ? perf_event_update_userpage+0x16/0x730
d ? kasan_unpoison_shadow+0x31/0x40
d ? get_page_from_freelist+0x52e/0x2310
d ? perf_output_begin+0x3a1/0x9b0
d ? cpu_clock_event_add+0x17/0x20
d ? __perf_event_overflow+0x1a0/0x510
d ? perf_swevent_overflow+0x156/0x1f0
d ? perf_tp_event+0x3e8/0x5c0
d ? perf_output_begin_backward+0x960/0x960
d ? perf_tp_event_match.isra.85.part.86+0x140/0x140
d ? __mark_inode_dirty+0x459/0xa50
d ? legitimize_path.isra.28+0x6b/0x150
d ? unlazy_walk+0x456/0x790
d ? memset+0x1f/0x40
d ? perf_trace_writeback_dirty_inode_template+0x3af/0x610
d ? save_stack+0x33/0xb0
d ? inode_congested+0x450/0x450
d ? dput+0x1de/0x530
d ? walk_component+0x2cc/0xdc0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? pick_link+0xbe0/0xbe0
d ? inode_congested+0x450/0x450
d ? __mark_inode_dirty+0x459/0xa50
d ? proc_sys_setattr+0x84/0xb0
d ? notify_change+0x4d6/0xc40
d ? security_inode_need_killpriv+0x58/0x80
d ? do_truncate+0xd7/0x160
d ? file_open_root+0x1a0/0x1a0
d ? path_openat+0x97f/0x3b30
d ? vfs_rename+0x14a0/0x14a0
d ? getname_flags+0xba/0x500
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? do_filp_open+0x175/0x230
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? may_open_dev+0xc0/0xc0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? do_sys_open+0x16d/0x310
d ? SyS_write+0xab/0x160
d ? filp_open+0x50/0x50
d ? task_stopped_code+0xf0/0xf0
d ? trace_hardirqs_on_thunk+0x1a/0x1c
d ? entry_SYSCALL_64_fastpath+0x18/0xa8
3Memory state around the buggy address:
3 ffff880424a97700: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3 ffff880424a97780: 00 00 00 00 00 00 00 00 f3 f3 f3 f3 f3 f3 f3 f3
3>ffff880424a97800: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f1
3 ^
3 ffff880424a97880: f1 f1 f1 04 f4 f4 f4 f2 f2 f2 f2 00 f4 f4 f4 f3
3 ffff880424a97900: f3 f3 f3 00 00 00 00 00 00 00 00 00 00 00 00 00
3==================================================================
[toc] | [prev] | [next] | [standalone]
| From | Dmitry Vyukov <dvyukov@google.com> |
|---|---|
| Date | 2016-11-16 14:20 +0100 |
| Message-ID | <sE8i6-3kM-7@gated-at.bofh.it> |
| In reply to | #1523455 |
On Wed, Nov 16, 2016 at 2:03 PM, Peter Zijlstra <peterz@infradead.org> wrote:
> On Tue, Nov 15, 2016 at 02:57:48PM -0600, Josh Poimboeuf wrote:
>> Would you mind posting a disassembly of unwind_get_return_address()?
>
> $ objdump -D ivb-dbg/vmlinux | awk '/<[^>]*>:/ { p=0; } /<unwind_get_return_address>:/ { p=1; } { if (p) print $0; }'
>
> ffffffff811afd10 <unwind_get_return_address>:
> ffffffff811afd10: e8 eb cc f4 01 callq ffffffff830fca00 <__fentry__>
> ffffffff811afd15: 48 b8 00 00 00 00 00 movabs $0xdffffc0000000000,%rax
> ffffffff811afd1c: fc ff df
> ffffffff811afd1f: 48 89 fa mov %rdi,%rdx
> ffffffff811afd22: 53 push %rbx
> ffffffff811afd23: 48 89 fb mov %rdi,%rbx
> ffffffff811afd26: 48 c1 ea 03 shr $0x3,%rdx
> ffffffff811afd2a: 48 83 ec 18 sub $0x18,%rsp
> ffffffff811afd2e: 0f b6 14 02 movzbl (%rdx,%rax,1),%edx
> ffffffff811afd32: 48 89 f8 mov %rdi,%rax
> ffffffff811afd35: 83 e0 07 and $0x7,%eax
> ffffffff811afd38: 83 c0 03 add $0x3,%eax
> ffffffff811afd3b: 38 d0 cmp %dl,%al
> ffffffff811afd3d: 7c 04 jl ffffffff811afd43 <unwind_get_return_address+0x33>
> ffffffff811afd3f: 84 d2 test %dl,%dl
> ffffffff811afd41: 75 75 jne ffffffff811afdb8 <unwind_get_return_address+0xa8>
> ffffffff811afd43: 8b 03 mov (%rbx),%eax
> ffffffff811afd45: 85 c0 test %eax,%eax
> ffffffff811afd47: 75 08 jne ffffffff811afd51 <unwind_get_return_address+0x41>
> ffffffff811afd49: 48 83 c4 18 add $0x18,%rsp
> ffffffff811afd4d: 31 c0 xor %eax,%eax
> ffffffff811afd4f: 5b pop %rbx
> ffffffff811afd50: c3 retq
> ffffffff811afd51: 48 8d 7b 38 lea 0x38(%rbx),%rdi
> ffffffff811afd55: 48 b8 00 00 00 00 00 movabs $0xdffffc0000000000,%rax
> ffffffff811afd5c: fc ff df
> ffffffff811afd5f: 48 89 fa mov %rdi,%rdx
> ffffffff811afd62: 48 c1 ea 03 shr $0x3,%rdx
> ffffffff811afd66: 80 3c 02 00 cmpb $0x0,(%rdx,%rax,1)
> ffffffff811afd6a: 75 53 jne ffffffff811afdbf <unwind_get_return_address+0xaf>
> ffffffff811afd6c: 48 b8 00 00 00 00 00 movabs $0xdffffc0000000000,%rax
> ffffffff811afd73: fc ff df
> ffffffff811afd76: 48 8b 4b 38 mov 0x38(%rbx),%rcx
> ffffffff811afd7a: 48 89 ca mov %rcx,%rdx
> ffffffff811afd7d: 48 c1 ea 03 shr $0x3,%rdx
> ffffffff811afd81: 80 3c 02 00 cmpb $0x0,(%rdx,%rax,1)
> ffffffff811afd85: 75 3f jne ffffffff811afdc6 <unwind_get_return_address+0xb6>
> ffffffff811afd87: 48 8d 7b 28 lea 0x28(%rbx),%rdi
> ffffffff811afd8b: 48 8b 11 mov (%rcx),%rdx
> ffffffff811afd8e: 48 b8 00 00 00 00 00 movabs $0xdffffc0000000000,%rax
> ffffffff811afd95: fc ff df
> ffffffff811afd98: 48 8d 73 30 lea 0x30(%rbx),%rsi
> ffffffff811afd9c: 49 89 f8 mov %rdi,%r8
> ffffffff811afd9f: 49 c1 e8 03 shr $0x3,%r8
> ffffffff811afda3: 41 80 3c 00 00 cmpb $0x0,(%r8,%rax,1)
> ffffffff811afda8: 75 2e jne ffffffff811afdd8 <unwind_get_return_address+0xc8>
> ffffffff811afdaa: 48 8b 7b 28 mov 0x28(%rbx),%rdi
> ffffffff811afdae: 48 83 c4 18 add $0x18,%rsp
> ffffffff811afdb2: 5b pop %rbx
> ffffffff811afdb3: e9 08 98 2a 00 jmpq ffffffff814595c0 <ftrace_graph_ret_addr>
> ffffffff811afdb8: e8 53 7d 42 00 callq ffffffff815d7b10 <__asan_report_load4_noabort>
> ffffffff811afdbd: eb 84 jmp ffffffff811afd43 <unwind_get_return_address+0x33>
> ffffffff811afdbf: e8 9c 7d 42 00 callq ffffffff815d7b60 <__asan_report_load8_noabort>
> ffffffff811afdc4: eb a6 jmp ffffffff811afd6c <unwind_get_return_address+0x5c>
> ffffffff811afdc6: 48 89 cf mov %rcx,%rdi
> ffffffff811afdc9: 48 89 0c 24 mov %rcx,(%rsp)
> ffffffff811afdcd: e8 8e 7d 42 00 callq ffffffff815d7b60 <__asan_report_load8_noabort>
> ffffffff811afdd2: 48 8b 0c 24 mov (%rsp),%rcx
> ffffffff811afdd6: eb af jmp ffffffff811afd87 <unwind_get_return_address+0x77>
> ffffffff811afdd8: 48 89 74 24 10 mov %rsi,0x10(%rsp)
> ffffffff811afddd: 48 89 54 24 08 mov %rdx,0x8(%rsp)
> ffffffff811afde2: 48 89 0c 24 mov %rcx,(%rsp)
> ffffffff811afde6: e8 75 7d 42 00 callq ffffffff815d7b60 <__asan_report_load8_noabort>
> ffffffff811afdeb: 48 8b 74 24 10 mov 0x10(%rsp),%rsi
> ffffffff811afdf0: 48 8b 54 24 08 mov 0x8(%rsp),%rdx
> ffffffff811afdf5: 48 8b 0c 24 mov (%rsp),%rcx
> ffffffff811afdf9: eb af jmp ffffffff811afdaa <unwind_get_return_address+0x9a>
> ffffffff811afdfb: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
>
>> Any idea how recreatable it is? (In particular I'd be interested in
>> seeing this dump with the latest unwinder improvements in the -tip tree,
>> which dump the pt_regs associated with an interrupt.)
>
> Fairly reproducable it seems, doesn't seem to include pt_regs dumps
> though :/
>
> tip/master as of this morning.
Can you print the stack that it gets after unwinding? If we will see
some garbage there, then it will confirm that it reads from redzones.
You can check taint before/after unwind and dump the stack iff kernel
become tainted during unwind.
> 3==================================================================
> 3BUG: KASAN: stack-out-of-bounds in unwind_next_frame+0x1ba/0x1f0 at addr ffff88042fc87be0
> 3Read of size 8 by task swapper/28/0
> 0page:ffffea0010bf21c0 count:1 mapcount:0 mapping: (null) index:0x0c
> 0flags: 0x2ffff8000000400(reserved)
> 1page dumped because: kasan: bad access detected
> dCPU: 28 PID: 0 Comm: swapper/28 Not tainted 4.9.0-rc5-00530-gd8866fc-dirty #2
> dHardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.02.0002.122320131210 12/23/2013
> dCall Trace:
> d <NMI>
> d ? dump_stack+0x5e/0x89
> d ? kasan_report_error+0x4a5/0x4d0
> d ? __asan_report_load8_noabort+0x45/0x50
> d ? __kernel_text_address+0x20/0xa0
> d ? unwind_next_frame+0x1ba/0x1f0
> d ? unwind_next_frame+0x1ba/0x1f0
> d ? perf_callchain_kernel+0x33c/0x540
> d ? arch_perf_update_userpage+0x340/0x340
> d ? get_perf_callchain+0x24d/0x610
> d ? put_callchain_buffers+0x50/0x50
> d ? number+0x653/0x830
> d ? perf_callchain+0x126/0x190
> d ? perf_prepare_sample+0x720/0x1010
> d ? perf_event_output_forward+0x81/0xf0
> d ? perf_prepare_sample+0x1010/0x1010
> d ? pointer+0x880/0x880
> d ? perf_event_update_userpage+0x16/0x730
> d ? __perf_event_overflow+0x1a0/0x510
> d ? intel_pmu_handle_irq+0x34b/0xa90
> d ? intel_pmu_save_and_restart+0xd0/0xd0
> d ? acpi_os_read_memory+0x205/0x23c
> d ? format_decode+0xc5/0x7a0
> d ? vunmap_page_range+0x26a/0x400
> d ? ghes_copy_tofrom_phys+0x141/0x270
> d ? ghes_read_estatus+0x112/0x5a0
> d ? ghes_copy_tofrom_phys+0x270/0x270
> d ? early_printk+0xa4/0xd0
> d ? devkmsg_sysctl_set_loglvl+0x160/0x160
> d ? perf_event_nmi_handler+0x28/0x40
> d ? nmi_handle+0xa1/0x250
> d ? default_do_nmi+0x61/0x170
> d ? do_nmi+0x191/0x200
> d ? end_repeat_nmi+0x1a/0x1e
> d ? format_decode+0xc5/0x7a0
> d ? format_decode+0xc5/0x7a0
> d ? format_decode+0xc5/0x7a0
> d <EOE>
> d <IRQ>
> d ? vsnprintf+0xfc/0x15e0
> d ? pointer+0x880/0x880
> d ? x86_pmu_enable_all+0x1c0/0x1c0
> d ? vscnprintf+0x9/0x30
> d ? early_vprintk+0xb0/0x130
> d ? trace_raw_output_console+0x160/0x160
> d ? memcpy+0x34/0x50
> d ? x86_pmu_commit_txn+0x180/0x260
> d ? events_sysfs_show+0xb0/0xb0
> d ? save_stack+0x33/0xb0
> d ? hrtimer_init+0x120/0x120
> d ? timerqueue_del+0x62/0x140
> d ? perf_event_update_userpage+0x16/0x730
> d ? perf_event_update_userpage+0x16/0x730
> d ? x86_perf_event_set_period+0x239/0x450
> d ? perf_event_update_userpage+0x16/0x730
> d ? x86_pmu_enable+0x5f7/0xaa0
> d ? printk+0xb6/0xef
> d ? printk_emit+0xa0/0xa0
> d ? _raw_spin_unlock_irqrestore+0x42/0x70
> d ? ___ratelimit+0x1e4/0x3f0
> d ? irq_work_run_list+0xa1/0xf0
> d ? irq_work_run+0x14/0x40
> d ? smp_call_function_single_interrupt+0x60/0x80
> d ? call_function_single_interrupt+0x89/0x90
> d <EOI>
> d ? cpuidle_enter_state+0x113/0x780
> d ? cpuidle_enter_state+0x10e/0x780
> d ? cpu_load_update_nohz_stop+0x155/0x1b0
> d ? cpu_startup_entry+0x19a/0x2c0
> d ? start_cpu+0x5/0x14
> 3Memory state around the buggy address:
> 3 ffff88042fc87a80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 3 ffff88042fc87b00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 3>ffff88042fc87b80: 00 00 00 00 00 00 00 00 00 00 00 00 f1 f1 f1 f1
> 3 ^
> 3 ffff88042fc87c00: 00 f4 f4 f4 f3 f3 f3 f3 00 00 00 00 00 00 00 00
> 3 ffff88042fc87c80: 00 f1 f1 f1 f1 00 00 00 00 00 00 00 00 00 00 00
> 3==================================================================
>
>
> 3==================================================================
> 3BUG: KASAN: stack-out-of-bounds in unwind_next_frame+0x1ba/0x1f0 at addr ffff880424a97878
> 3Read of size 8 by task perf_fuzzer/3451
> 0page:ffffea001092a5c0 count:0 mapcount:0 mapping: (null) index:0x0c
> 0flags: 0x2ffff8000000000()
> 1page dumped because: kasan: bad access detected
> dCPU: 28 PID: 3451 Comm: perf_fuzzer Not tainted 4.9.0-rc5-00530-gd8866fc-dirty #2
> dHardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.02.0002.122320131210 12/23/2013
> dCall Trace:
> d ? dump_stack+0x5e/0x89
> d ? kasan_report_error+0x4a5/0x4d0
> d ? __asan_report_load8_noabort+0x45/0x50
> d ? __kernel_text_address+0x20/0xa0
> d ? unwind_next_frame+0x1ba/0x1f0
> d ? unwind_next_frame+0x1ba/0x1f0
> d ? perf_callchain_kernel+0x33c/0x540
> d ? arch_perf_update_userpage+0x340/0x340
> d ? get_perf_callchain+0x24d/0x610
> d ? put_callchain_buffers+0x50/0x50
> d ? ipv6_flowlabel_opt+0x1111/0x17d0
> d ? perf_log_itrace_start+0x3a0/0x3a0
> d ? cpumask_next_and+0x5a/0xa0
> d ? ktime_get_raw_fast_ns+0xd3/0x1e0
> d ? perf_callchain+0x126/0x190
> d ? perf_prepare_sample+0x720/0x1010
> d ? perf_event_output_forward+0x81/0xf0
> d ? perf_prepare_sample+0x1010/0x1010
> d ? perf_event_update_userpage+0x16/0x730
> d ? kasan_unpoison_shadow+0x31/0x40
> d ? get_page_from_freelist+0x52e/0x2310
> d ? perf_output_begin+0x3a1/0x9b0
> d ? cpu_clock_event_add+0x17/0x20
> d ? __perf_event_overflow+0x1a0/0x510
> d ? perf_swevent_overflow+0x156/0x1f0
> d ? perf_tp_event+0x3e8/0x5c0
> d ? perf_output_begin_backward+0x960/0x960
> d ? perf_tp_event_match.isra.85.part.86+0x140/0x140
> d ? __mark_inode_dirty+0x459/0xa50
> d ? legitimize_path.isra.28+0x6b/0x150
> d ? unlazy_walk+0x456/0x790
> d ? memset+0x1f/0x40
> d ? perf_trace_writeback_dirty_inode_template+0x3af/0x610
> d ? save_stack+0x33/0xb0
> d ? inode_congested+0x450/0x450
> d ? dput+0x1de/0x530
> d ? walk_component+0x2cc/0xdc0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? pick_link+0xbe0/0xbe0
> d ? inode_congested+0x450/0x450
> d ? __mark_inode_dirty+0x459/0xa50
> d ? proc_sys_setattr+0x84/0xb0
> d ? notify_change+0x4d6/0xc40
> d ? security_inode_need_killpriv+0x58/0x80
> d ? do_truncate+0xd7/0x160
> d ? file_open_root+0x1a0/0x1a0
> d ? path_openat+0x97f/0x3b30
> d ? vfs_rename+0x14a0/0x14a0
> d ? getname_flags+0xba/0x500
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? do_filp_open+0x175/0x230
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? may_open_dev+0xc0/0xc0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? do_sys_open+0x16d/0x310
> d ? SyS_write+0xab/0x160
> d ? filp_open+0x50/0x50
> d ? task_stopped_code+0xf0/0xf0
> d ? trace_hardirqs_on_thunk+0x1a/0x1c
> d ? entry_SYSCALL_64_fastpath+0x18/0xa8
> 3Memory state around the buggy address:
> 3 ffff880424a97700: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 3 ffff880424a97780: 00 00 00 00 00 00 00 00 f3 f3 f3 f3 f3 f3 f3 f3
> 3>ffff880424a97800: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f1
> 3 ^
> 3 ffff880424a97880: f1 f1 f1 04 f4 f4 f4 f2 f2 f2 f2 00 f4 f4 f4 f3
> 3 ffff880424a97900: f3 f3 f3 00 00 00 00 00 00 00 00 00 00 00 00 00
> 3==================================================================
>
>
>
[toc] | [prev] | [next] | [standalone]
| From | Josh Poimboeuf <jpoimboe@redhat.com> |
|---|---|
| Date | 2016-11-16 15:40 +0100 |
| Message-ID | <sE9xw-48I-35@gated-at.bofh.it> |
| In reply to | #1523455 |
On Wed, Nov 16, 2016 at 02:03:37PM +0100, Peter Zijlstra wrote: > On Tue, Nov 15, 2016 at 02:57:48PM -0600, Josh Poimboeuf wrote: > > Would you mind posting a disassembly of unwind_get_return_address()? > > Any idea how recreatable it is? (In particular I'd be interested in > > seeing this dump with the latest unwinder improvements in the -tip tree, > > which dump the pt_regs associated with an interrupt.) > > Fairly reproducable it seems, doesn't seem to include pt_regs dumps > though :/ > > tip/master as of this morning. Thanks. This is actually a different issue than the one reported by Vince. In this case FRAME_POINTER is disabled, so it uses the "guess" unwinder which scans every address on the stack, looking for text addresses. So the kasan errors are expected. (The missing pt_regs are also expected: the guess unwinder doesn't show them.) I'll work up a patch to fix this. I still have no idea what's causing Vince's bug in the frame pointer unwinder. > 3================================================================== > 3BUG: KASAN: stack-out-of-bounds in unwind_next_frame+0x1ba/0x1f0 at addr ffff88042fc87be0 > 3Read of size 8 by task swapper/28/0 > 0page:ffffea0010bf21c0 count:1 mapcount:0 mapping: (null) index:0x0c > 0flags: 0x2ffff8000000400(reserved) > 1page dumped because: kasan: bad access detected > dCPU: 28 PID: 0 Comm: swapper/28 Not tainted 4.9.0-rc5-00530-gd8866fc-dirty #2 > dHardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.02.0002.122320131210 12/23/2013 > dCall Trace: > d <NMI> > d ? dump_stack+0x5e/0x89 > d ? kasan_report_error+0x4a5/0x4d0 > d ? __asan_report_load8_noabort+0x45/0x50 > d ? __kernel_text_address+0x20/0xa0 > d ? unwind_next_frame+0x1ba/0x1f0 > d ? unwind_next_frame+0x1ba/0x1f0 > d ? perf_callchain_kernel+0x33c/0x540 > d ? arch_perf_update_userpage+0x340/0x340 > d ? get_perf_callchain+0x24d/0x610 > d ? put_callchain_buffers+0x50/0x50 > d ? number+0x653/0x830 > d ? perf_callchain+0x126/0x190 > d ? perf_prepare_sample+0x720/0x1010 > d ? perf_event_output_forward+0x81/0xf0 > d ? perf_prepare_sample+0x1010/0x1010 > d ? pointer+0x880/0x880 > d ? perf_event_update_userpage+0x16/0x730 > d ? __perf_event_overflow+0x1a0/0x510 > d ? intel_pmu_handle_irq+0x34b/0xa90 > d ? intel_pmu_save_and_restart+0xd0/0xd0 > d ? acpi_os_read_memory+0x205/0x23c > d ? format_decode+0xc5/0x7a0 > d ? vunmap_page_range+0x26a/0x400 > d ? ghes_copy_tofrom_phys+0x141/0x270 > d ? ghes_read_estatus+0x112/0x5a0 > d ? ghes_copy_tofrom_phys+0x270/0x270 > d ? early_printk+0xa4/0xd0 > d ? devkmsg_sysctl_set_loglvl+0x160/0x160 > d ? perf_event_nmi_handler+0x28/0x40 > d ? nmi_handle+0xa1/0x250 > d ? default_do_nmi+0x61/0x170 > d ? do_nmi+0x191/0x200 > d ? end_repeat_nmi+0x1a/0x1e > d ? format_decode+0xc5/0x7a0 > d ? format_decode+0xc5/0x7a0 > d ? format_decode+0xc5/0x7a0 > d <EOE> > d <IRQ> > d ? vsnprintf+0xfc/0x15e0 > d ? pointer+0x880/0x880 > d ? x86_pmu_enable_all+0x1c0/0x1c0 > d ? vscnprintf+0x9/0x30 > d ? early_vprintk+0xb0/0x130 > d ? trace_raw_output_console+0x160/0x160 > d ? memcpy+0x34/0x50 > d ? x86_pmu_commit_txn+0x180/0x260 > d ? events_sysfs_show+0xb0/0xb0 > d ? save_stack+0x33/0xb0 > d ? hrtimer_init+0x120/0x120 > d ? timerqueue_del+0x62/0x140 > d ? perf_event_update_userpage+0x16/0x730 > d ? perf_event_update_userpage+0x16/0x730 > d ? x86_perf_event_set_period+0x239/0x450 > d ? perf_event_update_userpage+0x16/0x730 > d ? x86_pmu_enable+0x5f7/0xaa0 > d ? printk+0xb6/0xef > d ? printk_emit+0xa0/0xa0 > d ? _raw_spin_unlock_irqrestore+0x42/0x70 > d ? ___ratelimit+0x1e4/0x3f0 > d ? irq_work_run_list+0xa1/0xf0 > d ? irq_work_run+0x14/0x40 > d ? smp_call_function_single_interrupt+0x60/0x80 > d ? call_function_single_interrupt+0x89/0x90 > d <EOI> > d ? cpuidle_enter_state+0x113/0x780 > d ? cpuidle_enter_state+0x10e/0x780 > d ? cpu_load_update_nohz_stop+0x155/0x1b0 > d ? cpu_startup_entry+0x19a/0x2c0 > d ? start_cpu+0x5/0x14 > 3Memory state around the buggy address: > 3 ffff88042fc87a80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > 3 ffff88042fc87b00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > 3>ffff88042fc87b80: 00 00 00 00 00 00 00 00 00 00 00 00 f1 f1 f1 f1 > 3 ^ > 3 ffff88042fc87c00: 00 f4 f4 f4 f3 f3 f3 f3 00 00 00 00 00 00 00 00 > 3 ffff88042fc87c80: 00 f1 f1 f1 f1 00 00 00 00 00 00 00 00 00 00 00 > 3================================================================== > > > 3================================================================== > 3BUG: KASAN: stack-out-of-bounds in unwind_next_frame+0x1ba/0x1f0 at addr ffff880424a97878 > 3Read of size 8 by task perf_fuzzer/3451 > 0page:ffffea001092a5c0 count:0 mapcount:0 mapping: (null) index:0x0c > 0flags: 0x2ffff8000000000() > 1page dumped because: kasan: bad access detected > dCPU: 28 PID: 3451 Comm: perf_fuzzer Not tainted 4.9.0-rc5-00530-gd8866fc-dirty #2 > dHardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.02.0002.122320131210 12/23/2013 > dCall Trace: > d ? dump_stack+0x5e/0x89 > d ? kasan_report_error+0x4a5/0x4d0 > d ? __asan_report_load8_noabort+0x45/0x50 > d ? __kernel_text_address+0x20/0xa0 > d ? unwind_next_frame+0x1ba/0x1f0 > d ? unwind_next_frame+0x1ba/0x1f0 > d ? perf_callchain_kernel+0x33c/0x540 > d ? arch_perf_update_userpage+0x340/0x340 > d ? get_perf_callchain+0x24d/0x610 > d ? put_callchain_buffers+0x50/0x50 > d ? ipv6_flowlabel_opt+0x1111/0x17d0 > d ? perf_log_itrace_start+0x3a0/0x3a0 > d ? cpumask_next_and+0x5a/0xa0 > d ? ktime_get_raw_fast_ns+0xd3/0x1e0 > d ? perf_callchain+0x126/0x190 > d ? perf_prepare_sample+0x720/0x1010 > d ? perf_event_output_forward+0x81/0xf0 > d ? perf_prepare_sample+0x1010/0x1010 > d ? perf_event_update_userpage+0x16/0x730 > d ? kasan_unpoison_shadow+0x31/0x40 > d ? get_page_from_freelist+0x52e/0x2310 > d ? perf_output_begin+0x3a1/0x9b0 > d ? cpu_clock_event_add+0x17/0x20 > d ? __perf_event_overflow+0x1a0/0x510 > d ? perf_swevent_overflow+0x156/0x1f0 > d ? perf_tp_event+0x3e8/0x5c0 > d ? perf_output_begin_backward+0x960/0x960 > d ? perf_tp_event_match.isra.85.part.86+0x140/0x140 > d ? __mark_inode_dirty+0x459/0xa50 > d ? legitimize_path.isra.28+0x6b/0x150 > d ? unlazy_walk+0x456/0x790 > d ? memset+0x1f/0x40 > d ? perf_trace_writeback_dirty_inode_template+0x3af/0x610 > d ? save_stack+0x33/0xb0 > d ? inode_congested+0x450/0x450 > d ? dput+0x1de/0x530 > d ? walk_component+0x2cc/0xdc0 > d ? save_stack+0x33/0xb0 > d ? save_stack+0x33/0xb0 > d ? save_stack+0x33/0xb0 > d ? save_stack+0x33/0xb0 > d ? pick_link+0xbe0/0xbe0 > d ? inode_congested+0x450/0x450 > d ? __mark_inode_dirty+0x459/0xa50 > d ? proc_sys_setattr+0x84/0xb0 > d ? notify_change+0x4d6/0xc40 > d ? security_inode_need_killpriv+0x58/0x80 > d ? do_truncate+0xd7/0x160 > d ? file_open_root+0x1a0/0x1a0 > d ? path_openat+0x97f/0x3b30 > d ? vfs_rename+0x14a0/0x14a0 > d ? getname_flags+0xba/0x500 > d ? save_stack+0x33/0xb0 > d ? save_stack+0x33/0xb0 > d ? save_stack+0x33/0xb0 > d ? save_stack+0x33/0xb0 > d ? save_stack+0x33/0xb0 > d ? save_stack+0x33/0xb0 > d ? save_stack+0x33/0xb0 > d ? save_stack+0x33/0xb0 > d ? save_stack+0x33/0xb0 > d ? save_stack+0x33/0xb0 > d ? save_stack+0x33/0xb0 > d ? save_stack+0x33/0xb0 > d ? save_stack+0x33/0xb0 > d ? save_stack+0x33/0xb0 > d ? save_stack+0x33/0xb0 > d ? save_stack+0x33/0xb0 > d ? save_stack+0x33/0xb0 > d ? save_stack+0x33/0xb0 > d ? save_stack+0x33/0xb0 > d ? save_stack+0x33/0xb0 > d ? save_stack+0x33/0xb0 > d ? save_stack+0x33/0xb0 > d ? save_stack+0x33/0xb0 > d ? save_stack+0x33/0xb0 > d ? save_stack+0x33/0xb0 > d ? save_stack+0x33/0xb0 > d ? save_stack+0x33/0xb0 > d ? save_stack+0x33/0xb0 > d ? save_stack+0x33/0xb0 > d ? do_filp_open+0x175/0x230 > d ? save_stack+0x33/0xb0 > d ? save_stack+0x33/0xb0 > d ? may_open_dev+0xc0/0xc0 > d ? save_stack+0x33/0xb0 > d ? save_stack+0x33/0xb0 > d ? save_stack+0x33/0xb0 > d ? save_stack+0x33/0xb0 > d ? save_stack+0x33/0xb0 > d ? do_sys_open+0x16d/0x310 > d ? SyS_write+0xab/0x160 > d ? filp_open+0x50/0x50 > d ? task_stopped_code+0xf0/0xf0 > d ? trace_hardirqs_on_thunk+0x1a/0x1c > d ? entry_SYSCALL_64_fastpath+0x18/0xa8 > 3Memory state around the buggy address: > 3 ffff880424a97700: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > 3 ffff880424a97780: 00 00 00 00 00 00 00 00 f3 f3 f3 f3 f3 f3 f3 f3 > 3>ffff880424a97800: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f1 > 3 ^ > 3 ffff880424a97880: f1 f1 f1 04 f4 f4 f4 f2 f2 f2 f2 00 f4 f4 f4 f3 > 3 ffff880424a97900: f3 f3 f3 00 00 00 00 00 00 00 00 00 00 00 00 00 > 3================================================================== -- Josh
[toc] | [prev] | [next] | [standalone]
| From | Josh Poimboeuf <jpoimboe@redhat.com> |
|---|---|
| Date | 2016-11-16 16:00 +0100 |
| Message-ID | <sE9QR-4fj-15@gated-at.bofh.it> |
| In reply to | #1523536 |
On Wed, Nov 16, 2016 at 03:49:43PM +0100, Peter Zijlstra wrote: > On Wed, Nov 16, 2016 at 08:37:46AM -0600, Josh Poimboeuf wrote: > > On Wed, Nov 16, 2016 at 02:03:37PM +0100, Peter Zijlstra wrote: > > > On Tue, Nov 15, 2016 at 02:57:48PM -0600, Josh Poimboeuf wrote: > > > > Would you mind posting a disassembly of unwind_get_return_address()? > > > > Any idea how recreatable it is? (In particular I'd be interested in > > > > seeing this dump with the latest unwinder improvements in the -tip tree, > > > > which dump the pt_regs associated with an interrupt.) > > > > > > Fairly reproducable it seems, doesn't seem to include pt_regs dumps > > > though :/ > > > > > > tip/master as of this morning. > > > > Thanks. This is actually a different issue than the one reported by > > Vince. In this case FRAME_POINTER is disabled, so it uses the "guess" > > unwinder which scans every address on the stack, looking for text > > addresses. So the kasan errors are expected. > > > > (The missing pt_regs are also expected: the guess unwinder doesn't show > > them.) > > > > I'll work up a patch to fix this. I still have no idea what's causing > > Vince's bug in the frame pointer unwinder. > > Hurm,.. by the number of '?' entries in Vince's backtrace I was assuming > it was without frame pointers. When frame pointers are disabled, *all* the addresses are prefixed with '?'. When frame pointers are enabled, and there are a lot of '?' addresses, it usually means the containing functions reserved a lot of stack space and the printed addresses are mostly leftovers from previous runs. > Let me enable those and run again, it didn't insta-trigger like it does > without. Thanks! -- Josh
[toc] | [prev] | [next] | [standalone]
| From | Peter Zijlstra <peterz@infradead.org> |
|---|---|
| Date | 2016-11-16 16:00 +0100 |
| Message-ID | <sE9QR-4fj-5@gated-at.bofh.it> |
| In reply to | #1523536 |
On Wed, Nov 16, 2016 at 03:49:43PM +0100, Peter Zijlstra wrote:
> Let me enable those and run again, it didn't insta-trigger like it does
> without.
Tada!
$ objdump -D ivb-dbg/vmlinux | awk '/<[^>]*>:/ { p = 0; } /<unwind_get_return_address>:/ { p = 1; } { if (p) print $0; }'
ffffffff811c70d0 <unwind_get_return_address>:
ffffffff811c70d0: e8 8b 61 0e 02 callq ffffffff832ad260 <__fentry__>
ffffffff811c70d5: 48 b8 00 00 00 00 00 movabs $0xdffffc0000000000,%rax
ffffffff811c70dc: fc ff df
ffffffff811c70df: 55 push %rbp
ffffffff811c70e0: 48 89 fa mov %rdi,%rdx
ffffffff811c70e3: 48 89 e5 mov %rsp,%rbp
ffffffff811c70e6: 48 c1 ea 03 shr $0x3,%rdx
ffffffff811c70ea: 41 56 push %r14
ffffffff811c70ec: 41 55 push %r13
ffffffff811c70ee: 41 54 push %r12
ffffffff811c70f0: 53 push %rbx
ffffffff811c70f1: 48 89 fb mov %rdi,%rbx
ffffffff811c70f4: 48 83 ec 10 sub $0x10,%rsp
ffffffff811c70f8: 0f b6 14 02 movzbl (%rdx,%rax,1),%edx
ffffffff811c70fc: 48 89 f8 mov %rdi,%rax
ffffffff811c70ff: 83 e0 07 and $0x7,%eax
ffffffff811c7102: 83 c0 03 add $0x3,%eax
ffffffff811c7105: 38 d0 cmp %dl,%al
ffffffff811c7107: 7c 08 jl ffffffff811c7111 <unwind_get_return_address+0x41>
ffffffff811c7109: 84 d2 test %dl,%dl
ffffffff811c710b: 0f 85 0e 01 00 00 jne ffffffff811c721f <unwind_get_return_address+0x14f>
ffffffff811c7111: 8b 03 mov (%rbx),%eax
ffffffff811c7113: 85 c0 test %eax,%eax
ffffffff811c7115: 0f 84 c9 00 00 00 je ffffffff811c71e4 <unwind_get_return_address+0x114>
ffffffff811c711b: 48 8d 7b 40 lea 0x40(%rbx),%rdi
ffffffff811c711f: 48 b8 00 00 00 00 00 movabs $0xdffffc0000000000,%rax
ffffffff811c7126: fc ff df
ffffffff811c7129: 48 89 fa mov %rdi,%rdx
ffffffff811c712c: 48 c1 ea 03 shr $0x3,%rdx
ffffffff811c7130: 80 3c 02 00 cmpb $0x0,(%rdx,%rax,1)
ffffffff811c7134: 0f 85 ef 00 00 00 jne ffffffff811c7229 <unwind_get_return_address+0x159>
ffffffff811c713a: 4c 8b 63 40 mov 0x40(%rbx),%r12
ffffffff811c713e: 4d 85 e4 test %r12,%r12
ffffffff811c7141: 0f 84 ac 00 00 00 je ffffffff811c71f3 <unwind_get_return_address+0x123>
ffffffff811c7147: 49 8d bc 24 88 00 00 lea 0x88(%r12),%rdi
ffffffff811c714e: 00
ffffffff811c714f: 48 b8 00 00 00 00 00 movabs $0xdffffc0000000000,%rax
ffffffff811c7156: fc ff df
ffffffff811c7159: 48 89 f9 mov %rdi,%rcx
ffffffff811c715c: 48 c1 e9 03 shr $0x3,%rcx
ffffffff811c7160: 80 3c 01 00 cmpb $0x0,(%rcx,%rax,1)
ffffffff811c7164: 0f 85 4f 01 00 00 jne ffffffff811c72b9 <unwind_get_return_address+0x1e9>
ffffffff811c716a: 41 f6 84 24 88 00 00 testb $0x3,0x88(%r12)
ffffffff811c7171: 00 03
ffffffff811c7173: 75 6f jne ffffffff811c71e4 <unwind_get_return_address+0x114>
ffffffff811c7175: 49 83 ec 80 sub $0xffffffffffffff80,%r12
ffffffff811c7179: 48 b8 00 00 00 00 00 movabs $0xdffffc0000000000,%rax
ffffffff811c7180: fc ff df
ffffffff811c7183: 4c 89 e2 mov %r12,%rdx
ffffffff811c7186: 48 c1 ea 03 shr $0x3,%rdx
ffffffff811c718a: 80 3c 02 00 cmpb $0x0,(%rdx,%rax,1)
ffffffff811c718e: 0f 85 2f 01 00 00 jne ffffffff811c72c3 <unwind_get_return_address+0x1f3>
ffffffff811c7194: 4c 8d 73 28 lea 0x28(%rbx),%r14
ffffffff811c7198: 49 8b 14 24 mov (%r12),%rdx
ffffffff811c719c: 48 b8 00 00 00 00 00 movabs $0xdffffc0000000000,%rax
ffffffff811c71a3: fc ff df
ffffffff811c71a6: 48 8d 73 30 lea 0x30(%rbx),%rsi
ffffffff811c71aa: 4c 89 f1 mov %r14,%rcx
ffffffff811c71ad: 48 c1 e9 03 shr $0x3,%rcx
ffffffff811c71b1: 80 3c 01 00 cmpb $0x0,(%rcx,%rax,1)
ffffffff811c71b5: 0f 85 15 01 00 00 jne ffffffff811c72d0 <unwind_get_return_address+0x200>
ffffffff811c71bb: 48 8b 7b 28 mov 0x28(%rbx),%rdi
ffffffff811c71bf: 4c 89 e1 mov %r12,%rcx
ffffffff811c71c2: e8 59 7a 2c 00 callq ffffffff8148ec20 <ftrace_graph_ret_addr>
ffffffff811c71c7: 48 89 c7 mov %rax,%rdi
ffffffff811c71ca: 49 89 c5 mov %rax,%r13
ffffffff811c71cd: e8 9e 30 0c 00 callq ffffffff8128a270 <__kernel_text_address>
ffffffff811c71d2: 89 c2 mov %eax,%edx
ffffffff811c71d4: 4c 89 e8 mov %r13,%rax
ffffffff811c71d7: 85 d2 test %edx,%edx
ffffffff811c71d9: 75 0b jne ffffffff811c71e6 <unwind_get_return_address+0x116>
ffffffff811c71db: 80 3d 18 29 f9 02 00 cmpb $0x0,0x2f92918(%rip) # ffffffff84159afa <__print_once.27085>
ffffffff811c71e2: 74 4f je ffffffff811c7233 <unwind_get_return_address+0x163>
ffffffff811c71e4: 31 c0 xor %eax,%eax
ffffffff811c71e6: 48 83 c4 10 add $0x10,%rsp
ffffffff811c71ea: 5b pop %rbx
ffffffff811c71eb: 41 5c pop %r12
ffffffff811c71ed: 41 5d pop %r13
ffffffff811c71ef: 41 5e pop %r14
ffffffff811c71f1: 5d pop %rbp
ffffffff811c71f2: c3 retq
ffffffff811c71f3: 48 8d 7b 38 lea 0x38(%rbx),%rdi
ffffffff811c71f7: 48 b8 00 00 00 00 00 movabs $0xdffffc0000000000,%rax
ffffffff811c71fe: fc ff df
ffffffff811c7201: 48 89 fa mov %rdi,%rdx
ffffffff811c7204: 48 c1 ea 03 shr $0x3,%rdx
ffffffff811c7208: 80 3c 02 00 cmpb $0x0,(%rdx,%rax,1)
ffffffff811c720c: 0f 85 9d 00 00 00 jne ffffffff811c72af <unwind_get_return_address+0x1df>
ffffffff811c7212: 48 8b 43 38 mov 0x38(%rbx),%rax
ffffffff811c7216: 4c 8d 60 08 lea 0x8(%rax),%r12
ffffffff811c721a: e9 5a ff ff ff jmpq ffffffff811c7179 <unwind_get_return_address+0xa9>
ffffffff811c721f: e8 6c b0 45 00 callq ffffffff81622290 <__asan_report_load4_noabort>
ffffffff811c7224: e9 e8 fe ff ff jmpq ffffffff811c7111 <unwind_get_return_address+0x41>
ffffffff811c7229: e8 b2 b0 45 00 callq ffffffff816222e0 <__asan_report_load8_noabort>
ffffffff811c722e: e9 07 ff ff ff jmpq ffffffff811c713a <unwind_get_return_address+0x6a>
ffffffff811c7233: 4c 89 f2 mov %r14,%rdx
ffffffff811c7236: c6 05 bd 28 f9 02 01 movb $0x1,0x2f928bd(%rip) # ffffffff84159afa <__print_once.27085>
ffffffff811c723d: 48 b8 00 00 00 00 00 movabs $0xdffffc0000000000,%rax
ffffffff811c7244: fc ff df
ffffffff811c7247: 48 c1 ea 03 shr $0x3,%rdx
ffffffff811c724b: 80 3c 02 00 cmpb $0x0,(%rdx,%rax,1)
ffffffff811c724f: 75 4d jne ffffffff811c729e <unwind_get_return_address+0x1ce>
ffffffff811c7251: 48 b8 00 00 00 00 00 movabs $0xdffffc0000000000,%rax
ffffffff811c7258: fc ff df
ffffffff811c725b: 48 8b 5b 28 mov 0x28(%rbx),%rbx
ffffffff811c725f: 48 8d bb c0 04 00 00 lea 0x4c0(%rbx),%rdi
ffffffff811c7266: 48 89 fa mov %rdi,%rdx
ffffffff811c7269: 48 c1 ea 03 shr $0x3,%rdx
ffffffff811c726d: 0f b6 04 02 movzbl (%rdx,%rax,1),%eax
ffffffff811c7271: 84 c0 test %al,%al
ffffffff811c7273: 74 04 je ffffffff811c7279 <unwind_get_return_address+0x1a9>
ffffffff811c7275: 3c 03 cmp $0x3,%al
ffffffff811c7277: 7e 2f jle ffffffff811c72a8 <unwind_get_return_address+0x1d8>
ffffffff811c7279: 44 8b 83 c0 04 00 00 mov 0x4c0(%rbx),%r8d
ffffffff811c7280: 48 8d 8b 58 06 00 00 lea 0x658(%rbx),%rcx
ffffffff811c7287: 4c 89 e2 mov %r12,%rdx
ffffffff811c728a: 4c 89 ee mov %r13,%rsi
ffffffff811c728d: 48 c7 c7 e0 1d 45 83 mov $0xffffffff83451de0,%rdi
ffffffff811c7294: e8 49 8c 35 00 callq ffffffff8151fee2 <printk_deferred>
ffffffff811c7299: e9 46 ff ff ff jmpq ffffffff811c71e4 <unwind_get_return_address+0x114>
ffffffff811c729e: 4c 89 f7 mov %r14,%rdi
ffffffff811c72a1: e8 3a b0 45 00 callq ffffffff816222e0 <__asan_report_load8_noabort>
ffffffff811c72a6: eb a9 jmp ffffffff811c7251 <unwind_get_return_address+0x181>
ffffffff811c72a8: e8 e3 af 45 00 callq ffffffff81622290 <__asan_report_load4_noabort>
ffffffff811c72ad: eb ca jmp ffffffff811c7279 <unwind_get_return_address+0x1a9>
ffffffff811c72af: e8 2c b0 45 00 callq ffffffff816222e0 <__asan_report_load8_noabort>
ffffffff811c72b4: e9 59 ff ff ff jmpq ffffffff811c7212 <unwind_get_return_address+0x142>
ffffffff811c72b9: e8 22 b0 45 00 callq ffffffff816222e0 <__asan_report_load8_noabort>
ffffffff811c72be: e9 a7 fe ff ff jmpq ffffffff811c716a <unwind_get_return_address+0x9a>
ffffffff811c72c3: 4c 89 e7 mov %r12,%rdi
ffffffff811c72c6: e8 15 b0 45 00 callq ffffffff816222e0 <__asan_report_load8_noabort>
ffffffff811c72cb: e9 c4 fe ff ff jmpq ffffffff811c7194 <unwind_get_return_address+0xc4>
ffffffff811c72d0: 4c 89 f7 mov %r14,%rdi
ffffffff811c72d3: 48 89 75 d0 mov %rsi,-0x30(%rbp)
ffffffff811c72d7: 48 89 55 d8 mov %rdx,-0x28(%rbp)
ffffffff811c72db: e8 00 b0 45 00 callq ffffffff816222e0 <__asan_report_load8_noabort>
ffffffff811c72e0: 48 8b 75 d0 mov -0x30(%rbp),%rsi
ffffffff811c72e4: 48 8b 55 d8 mov -0x28(%rbp),%rdx
ffffffff811c72e8: e9 ce fe ff ff jmpq ffffffff811c71bb <unwind_get_return_address+0xeb>
ffffffff811c72ed: 0f 1f 00 nopl (%rax)
---
3==================================================================
3BUG: KASAN: stack-out-of-bounds in unwind_get_return_address+0x1fb/0x220 at addr ffff88042f88bba0
3Read of size 8 by task swapper/2/0
0page:ffffea0010be22c0 count:1 mapcount:0 mapping: (null) index:0x0c
0flags: 0x2ffff8000000400(reserved)
1page dumped because: kasan: bad access detected
dCPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.9.0-rc5-00530-gd8866fc-dirty #3
dHardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.02.0002.122320131210 12/23/2013
dCall Trace:
d <NMI>
d dump_stack+0x67/0x94
d kasan_report_error+0x4a1/0x4d0
d ? printk+0xef/0xef
d __asan_report_load8_noabort+0x43/0x50
d ? unwind_get_return_address+0x1fb/0x220
d unwind_get_return_address+0x1fb/0x220
d perf_callchain_kernel+0x356/0x550
d ? arch_perf_update_userpage+0x350/0x350
d ? __perf_event_header__init_id+0x500/0x500
d get_perf_callchain+0x276/0x670
d ? put_callchain_buffers+0x50/0x50
d ? sched_clock_cpu+0x11c/0x1a0
d perf_callchain+0x128/0x1a0
d perf_prepare_sample+0x70e/0xfb0
d perf_event_output_forward+0x93/0x110
d ? perf_prepare_sample+0xfb0/0xfb0
d ? arch_perf_update_userpage+0x26c/0x350
d ? sched_clock_cpu+0x11c/0x1a0
d __perf_event_overflow+0x1a3/0x570
d perf_event_overflow+0x14/0x20
d __intel_pmu_pebs_event+0x3ca/0x610
d ? pebs_update_state+0x310/0x310
d ? acpi_map_lookup+0x40/0xad
d ? intel_pmu_disable_bts+0xc0/0xc0
d ? acpi_map_lookup+0x40/0xad
d ? put_dec+0x1c/0xb0
d ? number+0x71c/0xa70
d ? put_dec+0xb0/0xb0
d intel_pmu_drain_pebs_nhm+0x5f6/0xbf0
d ? __intel_pmu_pebs_event+0x610/0x610
d ? early_serial_putc+0x41/0x70
d ? early_serial_write+0x7c/0xf0
d ? trace_raw_output_console+0x160/0x160
d intel_pmu_handle_irq+0x4b2/0xa90
d ? intel_pmu_save_and_restart+0xe0/0xe0
d ? acpi_os_read_memory+0x228/0x262
d ? acpi_os_get_timer+0x1a/0x1a
d ? vunmap_page_range+0x269/0x400
d ? ghes_copy_tofrom_phys+0x149/0x270
d ? ghes_read_estatus+0x11e/0x6b0
d ? ghes_copy_tofrom_phys+0x270/0x270
d perf_event_nmi_handler+0x2d/0x50
d nmi_handle+0x9e/0x250
d default_do_nmi+0x111/0x180
d do_nmi+0x1a2/0x210
d end_repeat_nmi+0x1a/0x1e
dRIP: 0010:irq_exit+0x10/0x1d0
dRSP: 0000:ffff88042f887fc8 EFLAGS: 00000046c
dRAX: 0000000000000000 RBX: ffffffff83a77980 RCX: 1ffff10080965faf
dRDX: 1ffff10085f13747 RSI: 0000000000000000 RDI: ffff88042f89ba38
dRBP: ffff88042f887fd0 R08: ffff8804060b1a08 R09: 1ffff10085f1276e
dR10: ffffed0080c16369 R11: ffff88042f89dd04 R12: 00000023af3410aa
dR13: 0000000000000004 R14: 0000000000000004 R15: 0000000000000180
d ? irq_exit+0x10/0x1d0
d ? irq_exit+0x10/0x1d0
d <EOE>
d <IRQ>
d smp_call_function_single_interrupt+0x70/0x90
d call_function_single_interrupt+0x90/0xa0
dRIP: 0010:cpuidle_enter_state+0x121/0x7a0
dRSP: 0000:ffff88042caffe28 EFLAGS: 00000246c ORIG_RAX: ffffffffffffff04
dRAX: 0000000000000000 RBX: ffff88042f8ab720 RCX: 000000000000001f
dRDX: 1ffff10085f142f9 RSI: 000000002dd33691 RDI: ffff88042f8a17c8
dRBP: ffff88042caffe88 R08: 0000000000000018 R09: ffffffff83f3f320
dR10: 071c71c71c71c71c R11: ffff88042f89dd04 R12: 00000023af3410aa
dR13: 0000000000000004 R14: 0000000000000004 R15: 0000000000000180
d <EOI>
d ? cpuidle_enter_state+0x11c/0x7a0
d cpuidle_enter+0x17/0x20
d call_cpuidle+0x47/0xc0
d ? cpuidle_select+0x59/0x80
d cpu_startup_entry+0x1a6/0x2d0
d start_secondary+0x245/0x2d0
d start_cpu+0x5/0x14
3Memory state around the buggy address:
3 ffff88042f88ba80: f2 00 00 f4 f4 f2 f2 f2 f2 00 00 f4 f4 f3 f3 f3
3 ffff88042f88bb00: f3 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3>ffff88042f88bb80: f1 f1 f1 f1 f1 f1 f1 f1 00 f4 f4 f4 f2 f2 f2 f2
3 ^
3 ffff88042f88bc00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3 ffff88042f88bc80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3==================================================================
4Disabling lock debugging due to kernel taint
3==================================================================
3BUG: KASAN: stack-out-of-bounds in unwind_next_frame+0x5fc/0x780 at addr ffff88042f88bb98
3Read of size 8 by task swapper/2/0
0page:ffffea0010be22c0 count:1 mapcount:0 mapping: (null) index:0x0c
0flags: 0x2ffff8000000400(reserved)
1page dumped because: kasan: bad access detected
dCPU: 2 PID: 0 Comm: swapper/2 Tainted: G B 4.9.0-rc5-00530-gd8866fc-dirty #3
dHardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.02.0002.122320131210 12/23/2013
dCall Trace:
d <NMI>
d dump_stack+0x67/0x94
d kasan_report_error+0x4a1/0x4d0
d ? kasan_report_error+0x420/0x4d0
d __asan_report_load8_noabort+0x43/0x50
d ? unwind_next_frame+0x5fc/0x780
d unwind_next_frame+0x5fc/0x780
d perf_callchain_kernel+0x341/0x550
d ? arch_perf_update_userpage+0x350/0x350
d ? __perf_event_header__init_id+0x500/0x500
d get_perf_callchain+0x276/0x670
d ? put_callchain_buffers+0x50/0x50
d ? sched_clock_cpu+0x11c/0x1a0
d perf_callchain+0x128/0x1a0
d perf_prepare_sample+0x70e/0xfb0
d perf_event_output_forward+0x93/0x110
d ? perf_prepare_sample+0xfb0/0xfb0
d ? arch_perf_update_userpage+0x26c/0x350
d ? sched_clock_cpu+0x11c/0x1a0
d __perf_event_overflow+0x1a3/0x570
d perf_event_overflow+0x14/0x20
d __intel_pmu_pebs_event+0x3ca/0x610
d ? pebs_update_state+0x310/0x310
d ? acpi_map_lookup+0x40/0xad
d ? intel_pmu_disable_bts+0xc0/0xc0
d ? acpi_map_lookup+0x40/0xad
d ? put_dec+0x1c/0xb0
d ? number+0x71c/0xa70
d ? put_dec+0xb0/0xb0
d intel_pmu_drain_pebs_nhm+0x5f6/0xbf0
d ? __intel_pmu_pebs_event+0x610/0x610
d ? early_serial_putc+0x41/0x70
d ? early_serial_write+0x7c/0xf0
d ? trace_raw_output_console+0x160/0x160
d intel_pmu_handle_irq+0x4b2/0xa90
d ? intel_pmu_save_and_restart+0xe0/0xe0
d ? acpi_os_read_memory+0x228/0x262
d ? acpi_os_get_timer+0x1a/0x1a
d ? vunmap_page_range+0x269/0x400
d ? ghes_copy_tofrom_phys+0x149/0x270
d ? ghes_read_estatus+0x11e/0x6b0
d ? ghes_copy_tofrom_phys+0x270/0x270
d perf_event_nmi_handler+0x2d/0x50
d nmi_handle+0x9e/0x250
d default_do_nmi+0x111/0x180
d do_nmi+0x1a2/0x210
d end_repeat_nmi+0x1a/0x1e
dRIP: 0010:irq_exit+0x10/0x1d0
dRSP: 0000:ffff88042f887fc8 EFLAGS: 00000046c
dRAX: 0000000000000000 RBX: ffffffff83a77980 RCX: 1ffff10080965faf
dRDX: 1ffff10085f13747 RSI: 0000000000000000 RDI: ffff88042f89ba38
dRBP: ffff88042f887fd0 R08: ffff8804060b1a08 R09: 1ffff10085f1276e
dR10: ffffed0080c16369 R11: ffff88042f89dd04 R12: 00000023af3410aa
dR13: 0000000000000004 R14: 0000000000000004 R15: 0000000000000180
d ? irq_exit+0x10/0x1d0
d ? irq_exit+0x10/0x1d0
d <EOE>
d <IRQ>
d smp_call_function_single_interrupt+0x70/0x90
d call_function_single_interrupt+0x90/0xa0
dRIP: 0010:cpuidle_enter_state+0x121/0x7a0
dRSP: 0000:ffff88042caffe28 EFLAGS: 00000246c ORIG_RAX: ffffffffffffff04
dRAX: 0000000000000000 RBX: ffff88042f8ab720 RCX: 000000000000001f
dRDX: 1ffff10085f142f9 RSI: 000000002dd33691 RDI: ffff88042f8a17c8
dRBP: ffff88042caffe88 R08: 0000000000000018 R09: ffffffff83f3f320
dR10: 071c71c71c71c71c R11: ffff88042f89dd04 R12: 00000023af3410aa
dR13: 0000000000000004 R14: 0000000000000004 R15: 0000000000000180
d <EOI>
d ? cpuidle_enter_state+0x11c/0x7a0
d cpuidle_enter+0x17/0x20
d call_cpuidle+0x47/0xc0
d ? cpuidle_select+0x59/0x80
d cpu_startup_entry+0x1a6/0x2d0
d start_secondary+0x245/0x2d0
d start_cpu+0x5/0x14
3Memory state around the buggy address:
3 ffff88042f88ba80: f2 00 00 f4 f4 f2 f2 f2 f2 00 00 f4 f4 f3 f3 f3
3 ffff88042f88bb00: f3 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3>ffff88042f88bb80: f1 f1 f1 f1 f1 f1 f1 f1 00 f4 f4 f4 f2 f2 f2 f2
3 ^
3 ffff88042f88bc00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3 ffff88042f88bc80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3==================================================================
[toc] | [prev] | [next] | [standalone]
| From | Josh Poimboeuf <jpoimboe@redhat.com> |
|---|---|
| Date | 2016-11-17 05:50 +0100 |
| Message-ID | <sEmO5-4fO-5@gated-at.bofh.it> |
| In reply to | #1523554 |
On Wed, Nov 16, 2016 at 03:58:49PM +0100, Peter Zijlstra wrote:
> 3BUG: KASAN: stack-out-of-bounds in unwind_get_return_address+0x1fb/0x220 at addr ffff88042f88bba0
So I dug through the disassembly (thanks for the vmlinux), and I'm
pretty sure the stack-out-of-bounds address is on the NMI stack, in the
kasan redzone in the stack frame of intel_pmu_handle_irq().
What's weird though is that perf_callchain_kernel() passes the pt_regs
from the IRQ, not from the NMI. The unwinder should have started from
the IRQ stack. But somehow it ended up unwinding to the middle of the
NMI stack.
So it seems like stack corruption in the IRQ or task stack, with a frame
pointer that points back to the middle of the NMI stack for some reason.
But then again, the kasan error report dumped the stack fine. So that
would seem to rule out stack corruption... So I have no idea what's
going on.
I got perf_fuzzer running and tried to recreate, but no luck.
Peter or Vince, can you try to recreate with this patch? It dumps the
raw stack contents during a stack dump. Hopefully that would give a
clue about what's going wrong.
diff --git a/arch/x86/kernel/dumpstack.c b/arch/x86/kernel/dumpstack.c
index 499aa6f..67ff3ac 100644
--- a/arch/x86/kernel/dumpstack.c
+++ b/arch/x86/kernel/dumpstack.c
@@ -48,6 +48,30 @@ static void printk_stack_address(unsigned long address, int reliable,
printk("%s %s%pB\n", log_lvl, reliable ? "" : "? ", (void *)address);
}
+static void raw_stack_dump(struct stack_info *info)
+{
+ unsigned long *s, word[4];
+ int skip = 0;
+
+ for (s = info->begin; s < info->end; s += 4) {
+ word[0] = READ_ONCE_NOCHECK(s[0]);
+ word[1] = READ_ONCE_NOCHECK(s[1]);
+ word[2] = READ_ONCE_NOCHECK(s[2]);
+ word[3] = READ_ONCE_NOCHECK(s[3]);
+
+ if (!word[0] && !word[1] && !word[2] && !word[3]) {
+ if (!skip)
+ printk("%p: %016x ...\n", s, 0);
+ skip = 1;
+ continue;
+ }
+
+ skip = 0;
+ printk("%p: %016lx %016lx %016lx %016lx\n",
+ s, word[0], word[1], word[2], word[3]);
+ }
+}
+
void show_trace_log_lvl(struct task_struct *task, struct pt_regs *regs,
unsigned long *stack, char *log_lvl)
{
@@ -156,6 +180,8 @@ void show_trace_log_lvl(struct task_struct *task, struct pt_regs *regs,
if (str_end)
printk("%s <%s>\n", log_lvl, str_end);
+
+ raw_stack_dump(&stack_info);
}
}
[toc] | [prev] | [next] | [standalone]
| From | Peter Zijlstra <peterz@infradead.org> |
|---|---|
| Date | 2016-11-17 10:20 +0100 |
| Message-ID | <sEr1n-78g-17@gated-at.bofh.it> |
| In reply to | #1524084 |
On Thu, Nov 17, 2016 at 10:04:46AM +0100, Peter Zijlstra wrote: > On Wed, Nov 16, 2016 at 10:48:28PM -0600, Josh Poimboeuf wrote: > > Peter or Vince, can you try to recreate with this patch? It dumps the > > raw stack contents during a stack dump. Hopefully that would give a > > clue about what's going wrong. > > > Here goes... I'll do another run and get you the results of that as > well. This one is funny, I've not seen that WARNING before. Let me do a third run. 4WARNING: unrecognized kernel stack return address ffff88072f20bef8 at ffff88072f20bdd0 in swapper/10:0 3================================================================== 3BUG: KASAN: stack-out-of-bounds in unwind_get_return_address+0x1fb/0x220 at addr ffff88072f20bda8 3Read of size 8 by task swapper/10/0 0page:ffffea001cbc82c0 count:1 mapcount:0 mapping: (null) index:0x0c 0flags: 0x6ffff8000000400(reserved) 1page dumped because: kasan: bad access detected dCPU: 10 PID: 0 Comm: swapper/10 Not tainted 4.9.0-rc5-00530-gd8866fc-dirty #4 dHardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.02.0002.122320131210 12/23/2013 dCall Trace: d <NMI> d dump_stack+0x67/0x94 d kasan_report_error+0x4a1/0x4d0 d ? unwind_get_return_address+0x1fb/0x220 d ? kasan_report_error+0x5/0x4d0 d __asan_report_load8_noabort+0x43/0x50 d ? unwind_get_return_address+0x130/0x220 d ? unwind_get_return_address+0x1fb/0x220 d unwind_get_return_address+0x1fb/0x220 d perf_callchain_kernel+0x356/0x550 d ? arch_perf_update_userpage+0x350/0x350 d ? ftrace_ops_list_func+0x252/0x370 d ? perf_callchain_kernel+0x5/0x550 d ? perf_callchain+0x128/0x1a0 d get_perf_callchain+0x276/0x670 d ? put_callchain_buffers+0x50/0x50 d ? get_perf_callchain+0x5/0x670 d ? rcu_is_watching+0x30/0x70 d perf_callchain+0x128/0x1a0 d ? setup_pebs_sample_data+0xd48/0x18e0 d perf_prepare_sample+0x70e/0xfb0 d ? __rcu_read_lock+0x5/0x50 d perf_event_output+0x93/0x110 d ? perf_event_output_backward+0x110/0x110 d ? setup_pebs_sample_data+0xd48/0x18e0 d __intel_pmu_pebs_event+0x2b1/0x610 d ? pebs_update_state+0x310/0x310 d ? perf_cgroup_attach+0xb0/0xb0 d ? ctx_resched+0x1a0/0x1a0 d ? rcu_is_watching+0x30/0x70 d ? intel_pmu_disable_bts+0xc0/0xc0 d ? rcu_is_watching+0x30/0x70 d ? ftrace_ops_list_func+0x252/0x370 d ? intel_pmu_drain_pebs_nhm+0x5f6/0xbf0 d ? ftrace_call+0x5/0x34 d ? __intel_pmu_pebs_event+0x5/0x610 d intel_pmu_drain_pebs_nhm+0x5f6/0xbf0 d ? __intel_pmu_pebs_event+0x610/0x610 d ? ftrace_call+0x5/0x34 d ? intel_pmu_drain_pebs_nhm+0x5/0xbf0 d intel_pmu_handle_irq+0x4b2/0xa90 d ? intel_pmu_save_and_restart+0xe0/0xe0 d ? ftrace_ops_list_func+0x252/0x370 d ? perf_event_nmi_handler+0x2d/0x50 d ? ftrace_call+0x5/0x34 d ? ftrace_call+0x5/0x34 d ? ftrace_ops_list_func+0x252/0x370 d ? intel_pmu_handle_irq+0x5/0xa90 d ? perf_event_nmi_handler+0x5/0x50 d perf_event_nmi_handler+0x2d/0x50 d nmi_handle+0x9e/0x250 d ? nmi_handle+0x5/0x250 d default_do_nmi+0x67/0x180 d do_nmi+0x1a2/0x210 d ? ctx_resched+0x1a0/0x1a0 d ? perf_cgroup_attach+0xb0/0xb0 d end_repeat_nmi+0x1a/0x1e dRIP: 0010:remote_function+0x113/0x180 dRSP: 0018:ffff88072f207f60 EFLAGS: 00000806c dRAX: 0000000000000000 RBX: ffff880421affd60 RCX: 1ffff1008435ffaf dRDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88072f224f10 dRBP: ffff88072f207f80 R08: 00000000000004c1 R09: ffffed00e5e426ca dR10: 00000000ffffffff R11: 1ffff100e5e426ec R12: ffffffff814f8270 dR13: ffff880421affd78 R14: ffff880421affcc0 R15: ffffffff814e6a70 d ? perf_cgroup_attach+0xb0/0xb0 d ? ctx_resched+0x1a0/0x1a0 d ? remote_function+0x113/0x180 d ? remote_function+0x113/0x180 d <EOE> ffff88072f20a000: 0000000000000000 ... ffff88072f20a5a0: 0000000000000000 0000000000000000 0000000000000000 ffff8807af20a9a6 ffff88072f20a5c0: ffff88072f20a9ae 0000000000000060 1ffff100e5e414c3 00000000ffffffff ffff88072f20a5e0: 0020000000000000 0000000000000000 00000000fffffffd 0000000000000001 ffff88072f20a600: 0000000000000180 0000000000000010 ffff1060ffffff09 0000000041b58ab3 ffff88072f20a620: ffff8807af20aa0e ffff88072f20aa16 0000000000000060 1ffff100e5e414d0 ffff88072f20a640: 00000000ffffffff 0020000000000000 0000000000000000 00000000fffffffd ffff88072f20a660: 0000000000000001 0000000000000180 ffffffff8348be10 ffff1060ffffff09 ffff88072f20a680: 0000000041b58ab3 ffffffff83a40503 ffffffff823ab960 ffff88072f20a758 ffff88072f20a6a0: ffff88072f313830 ffffffff823b7c54 ffff88072f20a7b0 ffff88072f20a7b8 ffff88072f20a6c0: ffff88072f20a7b0 ffff88072f20a9a7 0000000000000001 ffffffff8348bec9 ffff88072f20a6e0: ffff88072f20aa16 0000000000000018 dffffc0000000000 ffff88072f20a870 ffff88072f20a700: ffff88072f20a7c0 ffff88072f20a7e8 ffffffff823b7c54 ffff88072f20a818 ffff88072f20a720: ffff88072f20a820 ffff88072f20a818 ffff88072f20aa0f 1ffff100e5e414ec ffff88072f20a740: ffffed00e5e41504 000000007fffffff ffff8807af20aa0e ffffffff8348beca ffff88072f20a760: 0000000041b58ab3 ffffffff83a405aa ffffffff823b6ef0 ffff880402cb94a8 ffff88072f20a780: ffff1060ffffff09 ffff88072f20aa00 1ffff100e5e41515 ffff88072f20a890 ffff88072f20a7a0: ffffffff814e6b83 ffffffffffffffff ffff88072f20a868 fffffbfff07121c8 ffff88072f20a7c0: ffffed00e5e414ff 1ffff100e5e41515 ffff88072f20a988 ffffffff814e6b83 ffff88072f20a7e0: ffffffffffffffff ffff88072f20a890 ffffffff823b8d95 0000000041b58ab3 ffff88072f20a800: ffffffff839ff69c ffffffff823b8d00 ffffed00e5e41515 ffff880700000020 ffff88072f20a820: ffff88072f20a8a0 ffff88072f20a858 ffff88072f20a8c8 ffff88072f20aa00 ffff88072f20a840: dffffc0000000000 ffff88072f20a890 ffffffff813c861f ffff88072f20aec8 ffff88072f20a860: 0000000000000000 0000000000000113 0000000000000180 ffffffff838ad5e6 ffff88072f20a880: ffffffff838ad5e6 ffff88072f20aa00 ffff88072f20a9b0 ffffffff813c883f ffff88072f20a8a0: 000000000000000f 0000000041b58ab3 ffffffff83a0c758 ffffffff813c86d0 ffff88072f20a8c0: ffff88072f20af30 0000000000000000 ffff88072f20a988 ffffffff832ad2e7 ffff88072f20a8e0: ffff88072f20a998 ffffffff832ad2e7 ffff88072f20a9b4 0000000000000000 ffff88072f20a900: ffff88072f20a9c0 0000000000000113 ffff88072f20ab48 ffff880700000020 ffff88072f20a920: 0000000000000000 ffffffff839fbe88 0000000000000042 0000000000000001 ffff88072f20a940: ffff0a00ffffff05 ffff88072f20aa1b ffff88072f20ad50 ffff88072f20aa1c ffff88072f20a960: 0000000000000000 ffff88072f20a9c0 ffffffff823adbcd 0000000041b58ab3 ffff88072f20a980: ffffffff83a40559 1ffff100e5e4153c ffff0a00ffffff05 1ffff100e5e4153c ffff88072f20a9a0: ffff88072f20aa00 ffff88072f20ad35 ffff88072f20af30 ffff0a00ffffff05 ffff88072f20a9c0: ffff88072f20ab48 ffffffff823af620 ffff88072f20aa88 ffffffff832ad2e7 ffff88072f20a9e0: 0000000041b58ab3 ffffffff83a40559 ffffffff823af530 0000000000000000 ffff88072f20aa00: 665f65746f6d6572 2b6e6f6974636e75 78302f3331317830 3035780000303831 ffff88072f20aa20: 0000300030657800 ffffffff83a13036 0000000000000053 0000000000000001 ffff88072f20aa40: ffff0a00ffffff05 ffff88072f20ab25 ffff88072f20ae7a ffff88072f20ab26 ffff88072f20aa60: ffff88072f20af30 ffff88072f20ad30 ffffffff823adb30 1ffff100e5e41558 ffff88072f20aa80: 00000000ffffffff 0020f100e5e4155c ffff0a00ffffff00 1ffff10000000010 ffff88072f20aaa0: ffff88072f20ab00 ffff88072f20aa80 ffff88072f20b010 ffff103000001005 ffff88072f20aac0: 0000000041b58ab3 ffffffff83a40503 ffffffff823ab960 ffffffffffffffff ffff88072f20aae0: 3266323061616330 6666666638383037 ffffffff00000010 0000000000000000 ffff88072f20ab00: ffffffff814e6a70 ffff88072f20af10 ffff103000001009 0000000041b58ab3 ffff88072f20ab20: ffff88072f20af30 ffff88072f20ad75 ffff88072f20ad30 1ffff100e5e41570 ffff88072f20ab40: ffffffffffffffff 002088072f20ac08 ffffffff823b6600 ffffffff00000010 ffff88072f20ab60: ffff88072f20ad00 ffffffff00000010 ffff88072f20ad10 ffff103000001009 ffff88072f20ab80: 0000000041b58ab3 ffffffff83a40503 ffffffff823ab960 ffffffff839f498b ffff88072f20aba0: 3833396634393862 6666666666666666 ffff88072f20ac87 ffff88072f20ac08 ffff88072f20abc0: ffffffff823ad785 ffff88072f20acc8 0000000000000001 ffff88072f20ad85 ffff88072f20abe0: ffff88072f20ac08 ffffffff816213e5 ffff88072f20ad85 ffffffff839f498f ffff88072f20ac00: dffffc0000000000 ffff88072f20ace8 ffffffff823b72eb ffff88072f20b028 ffff88072f20ac20: ffff88072f20b030 ffffffff839f4990 ffff88072f20ad30 1ffff100e5e4158c ffff88072f20ac40: ffffed00e5e41606 0000000000000200 ffff88072f20af30 0000000000000001 ffff88072f20ac60: 0000000041b58ab3 ffffffff83a405aa ffffffff823b6ef0 ffff88072f20acc8 ffff88072f20ac80: ffff103000001000 1ffff100e5e41595 ffffffff83c88b40 ffff88072f20af10 ffff88072f20aca0: ffffffff81338af4 0000000041b58ab3 ffff88072f20acd0 ffffffff811c0a51 ffff88072f20acc0: ffff88072f20ad86 dffffc0000000000 ffff88072f20ad00 ffffffff811c0afc ffff88072f20ace0: ffff88072f20ad30 00000000ffffffff 1ffff100e5e415a2 ffffffff83c88b40 ffff88072f20ad00: ffff88072f20af78 ffffffff81338af4 0000000041b58ab3 ffffffff83a03c41 ffff88072f20ad20: ffffffff81338a00 ffff0a00ffffff00 3730383866666666 3030646130326632 ffff88072f20ad40: 666666666666203a 6138333331386666 3066666666203030 6666666666303061 ffff88072f20ad60: 3636303320363636 3032363636363636 3636362030333033 3336363636363636 ffff88072f20ad80: 1f000a3633363336 ffffffff83c88b40 ffff88072f20b008 ffffffff81338af4 ffff88072f20ada0: 0000000041b58ab3 ffffffff83a03c41 ffffffff81338a00 0000000041b58ab3 ffff88072f20adc0: 6177647261486401 3a656d616e206572 43206c65746e4920 ffff880402cb9498 ffff88072f20ade0: ffff88072f20ae30 ffffffff81448fa2 534f4942202c5a47 ffff880402cb94a8 ffff88072f20ae00: ffffffff8133c807 00000000ffffffff 0000000000000000 ffff88072f20bef8 ffff88072f20ae20: ffffffff8342d740 ffff88072f20afc0 ffff88072f20aee8 ffffffff832ad2e7 ffff88072f20ae40: ffff880402cb9498 ffff88072f20ae98 ffffffff81448fa2 ffff880402cb9498 ffff88072f20ae60: ffff880402cb94a8 ffffffff8133c807 00000000ffffffff 0000000000000018 ffff88072f20ae80: ffff88072f20bef8 ffffffff839f4970 ffff88072f20b028 ffff88072f20af50 ffff88072f20aea0: ffffffff832ad2e7 ffffffff81448fa2 ffffffff8342d740 ffff880402cb94a8 ffff88072f20aec0: ffff880402cb9498 ffff88072f20aff8 ffffffff81448fa2 ffff88072c4c8000 ffff88072f20aee0: ffff880402cb94a8 ffff88072f20b028 ffffffff839f4970 ffff88072f20b068 ffff88072f20af00: 0000000000000000 0000000000000000 ffff88072f20b028 ffffffff839f4970 ffff88072f20af20: ffffffff832ad2e7 ffffffff81338a05 ffffffff81448fa2 ffff880402cb94a8 ffff88072f20af40: ffff880402cb94a8 ffff88072f20b0a8 00000000ffffffff 0000000000000018 ffff88072f20af60: ffff88072f20bef8 ffffffff839f4970 ffff88072f20b028 ffff88072f20aff8 ffff88072f20af80: ffffffff8133c807 0000000000000000 00000000ffffffff 0000000000000000 ffff88072f20afa0: ffff88072f20af80 ffffffff8133c7c5 ffff88072f20bef8 0000000000000000 ffff88072f20afc0: 0000000000000000 000000002f20b080 1ffff100e5e41601 0000000000000018 ffff88072f20afe0: ffff88072f20bef8 ffff88072c4c8000 ffff88072f20c000 ffff88072f20b0a8 ffff88072f20b000: ffffffff8151fea9 0000000041b58ab3 ffffffff839ff69c ffffffff8151fdf3 ffff88072f20b020: ffffffff839ff69c ffffffff00000030 ffff88072f20b0b8 ffff88072f20b068 ffff88072f20b040: ffff88072f20b020 ffffffff839f4970 ffff88072f20bff8 ffffffff8151fdf8 ffff88072f20b060: ffffffff813b9d58 ffff88072f20b090 ffff88072f20b040 ffff88072f20b020 ffff88072f20b080: ffff88072f20b090 ffff88072f20b040 ffff88072f20b020 ffff88072f20b060 ffff88072f20b0a0: 0000000000000009 ffff88072f20b178 ffffffff811504e6 ffffffff814e6b83 ffff88072f20b0c0: ffffffff839fbd23 000000002f20b188 0000000000000020 ffffffff839fbd5b ffff88072f20b0e0: ffffffff839fbd1f 0000000000000005 ffff88072f20a000 ffff88072f20c000 ffff88072f20b100: ffff88072f207f60 0000000000000002 ffff88072f200000 ffff88072f208000 ffff88072f20b120: ffff88072c4d7d78 0000000000000024 ffff88072c4c8000 0000000000000000 ffff88072f20b140: ffff88072f207f80 0000000000000000 0000000000000086 0000000000000000 ffff88072f20b160: ffff88072f20b420 ffffed00e5e41685 dffffc0000000000 ffff88072f20b188 ffff88072f20b180: ffffffff811505c4 ffff88072f20b1a8 ffffffff8238dfb3 ffff88072f20b238 ffff88072f20b1a0: ffff88072f20bda8 ffff88072f20b228 ffffffff816221c1 ffffffff811c72cb ffff88072f20b1c0: 1ffff100e5e41669 ffff88072f20b238 0000000000000086 ffffffff81621d25 ffff88072f20b1e0: dffffc0000000000 1ffff100e5e4166a 1ffff100e5e417b5 1ffff100e5e41669 ffff88072f20b200: ffff88072f20b310 ffff88072f20bda8 ffff88072f20b420 ffffed00e5e41685 ffff88072f20b220: dffffc0000000000 ffff88072f20b260 ffffffff81622323 ffff88072f20bda8 ffff88072f20b240: ffff88072f20bda8 0000000000000008 ffffffff811c7200 ffffffff811c72cb ffff88072f20b260: ffff88072f20b2a0 ffffffff811c72cb ffff88072f20b338 ffff88072f20b288 ffff88072f20b280: ffff88072f20b310 ffff88072f20b390 ffff88072f20b420 ffffed00e5e41685 ffff88072f20b2a0: ffff88072f20b3b8 ffffffff8100f356 000000000001bdc0 ffff88072f20b460 ffff88072f20b2c0: ffff88070b9fcec0 ffff88072f20b42c 1ffff100e5e4165e ffff88072f20b432 ffff88072f20b2e0: 0000000000000004 ffff88072f20b428 0000000041b58ab3 ffffffff839f613c ffff88072f20b300: ffffffff8100f000 ffff88072f20b488 0000000000000005 ffff88072f20a000 ffff88072f20b320: ffff88072f20c000 ffff88072f207f60 0000000000000020 ffff88072c4c8000 ffff88072f20b340: 0000000000000000 ffff88072f20bda0 0000000000000000 ffff88072f20b420 ffff88072f20b360: ffffffff81448fa2 ffffffff8100f005 ffff880402cb94a8 ffffffff81514438 ffff88072f20b380: ffff880405b11980 0000000000000040 000000000001bdc0 ffff88072f20b460 ffff88072f20b3a0: ffff88070b9fcec0 1ffff100e5e41680 ffff88070b9fcec0 ffff88072f20b488 ffff88072f20b3c0: ffffffff81513f16 ffff88072f20b574 ffff880405b11980 ffff880700000000 ffff88072f20b3e0: ffff88072f20b6e0 ffff880700000000 ffff880700000001 000000000000000c ffff88072f20b400: 0000000041b58ab3 ffffffff83a0a369 ffffffff81513ca0 0000000000000001 ffff88072f20b420: ffff88070b9fcec0 000000020000007f ffff88072f000001 ffffffff81513ca5 ffff88072f20b440: 1ffff100e5e416ef ffff88072f20b588 ffffffff8135d050 00000000000000ff ffff88072f20b460: ffff880405b11980 0000000000000040 ffff88072f20b6e0 ffff88072f224f08 ffff88072f20b480: 0000000000000000 ffff88072f20b4d0 ffffffff81514438 ffffffff00000001 ffff88072f20b4a0: ffffffff81027da8 ffff88072f20b574 ffff880405b11980 ffff88072f20b570 ffff88072f20b4c0: ffff880405b11980 ffff88072f20b7c0 ffff88072f20b540 ffffffff8150ac7e ffff88072f20b4e0: ffff880405b11980 0000000100000000 ffffffff81359595 ffff880402cb9498 ffff88072f20b500: 00000000000000ff ffff88072f20b576 ffff88072f20b6e0 1ffff100e5e416aa ffff88072f20b520: ffff880405b11980 ffff88072f20b7c0 ffff88072f20b6e0 dffffc0000000000 ffff88072f20b540: ffff88072f20b630 ffffffff8150b7d3 0000000041b58ab3 ffffffff83a0d7b0 ffff88072f20b560: ffffffff8150b740 ffff880405b11980 0048000100000009 00000000000000a8 ffff88072f20b580: ffff88072f20b6e0 ffff88072f20b6e0 ffff88072c610210 ffff88072f20b7c0 ffff88072f20b5a0: ffff880405b11980 00000000000000ff ffff88072f20b630 ffffffff81027da8 ffff88072f20b5c0: ffff88072c610210 ffff88072f20bef8 ffff880405b11980 ffff88072f20b770 ffff88072f20b5e0: ffffffff84156db0 ffff88072f20b7d0 ffff88072f213440 ffff880405b11a68 ffff88072f20b600: ffff880700000000 ffff88072c610210 ffff88072c610210 ffff88072f20b7c0 ffff88072f20b620: 0000000000000001 ffff880405b11980 ffff88072f20b9b0 ffffffff81028f01 ffff88072f20b640: 0000000000000000 ... ffff88072f20b660: 0000000000000000 1ffff100e5e416d8 1ffffffff082adb6 0000000000000007 ffff88072f20b680: fffffbfff082adb7 0000000000000000 ffff88072f20b7c0 ffff88072f20bef8 ffff88072f20b6a0: ffff880405b11980 0000000000000005 ffff88072c610580 ffff88072f20b6e0 ffff88072f20b6c0: 0000000041b58ab3 ffffffff839f20b0 ffffffff81028c50 0000000000000000 ffff88072f20b6e0: ffffffff814e6a70 ffff880421affcc0 ffff880421affd78 ffffffff814f8270 ffff88072f20b700: ffff88072f20bd50 ffff880421affd60 1ffff100e5e426ec 00000000ffffffff ffff88072f20b720: ffffed00e5e426ca 00000000000004c1 0000000000000000 1ffff1008435ffaf ffff88072f20b740: 0000000000000000 0000000000000000 ffff88072f224f10 ffffffffffffffff ffff88072f20b760: ffffffff8135d050 0000000000000010 0000000000000803 ffff88072f20bd48 ffff88072f20b780: 0000000000000018 ffffffff839f2108 ffffffff8102b200 0000000000000000 ffff88072f20b7a0: 0000000000000000 ... ffff88072f20b7c0: 0000000000000000 0000000000000000 0000000000000000 0000000000000100 ffff88072f20b7e0: 0000000000000000 0000000000000000 0000000005080021 00000000000000ff ffff88072f20b800: ffffffff8135d050 0000000000000000 0000004c8c598365 0000000000013955 ffff88072f20b820: 0000000000000000 000000000000000a ffff88070b9fcec0 0000000000000000 ffff88072f20b840: 0000000000000000 ... ffff88072f20b860: 0000000000000000 0000000000000000 0000000000000000 ffff880402cb9498 ffff88072f20b880: ffff88072f20b8d0 ffffffff81448fa2 0000000000000000 ffff880402cb94a8 ffff88072f20b8a0: ffffffff81029856 0000000000000000 dffffc0000000000 ffff88072f20bb08 ffff88072f20b8c0: ffff880405b11980 ffff88072f213440 ffff88072f20b988 ffffffff832ad2e7 ffff88072f20b8e0: 0000000000000000 ... ffff88072f20b900: ffff88072f20bb30 0000000000000000 0000000000000000 0000000000000000 ffff88072f20b920: 0000000000000008 0000000000000000 fffffbfff082adbc ffff88072c610580 ffff88072f20b940: ffff88072c610000 ffff88072f20bef8 ffff880405b11980 0000000000000000 ffff88072f20b960: ffffffff81028c55 0000000000000000 0000000000000000 0000000000000000 ffff88072f20b980: 0000000000000000 0000000000000000 dffffc0000000000 ffff88072f20bb08 ffff88072f20b9a0: ffff880405b11980 ffff88072f213440 ffff88072f20bb30 ffffffff81029856 ffff88072f20b9c0: 0000000000000000 0000000000000000 ffff88072f213d78 ffff88072f20bef8 ffff88072f20b9e0: 1ffff100e5e41745 ffff88072c610000 ffff88072f20ba88 ffff880402cb9407 ffff88072f20ba00: ffffed00e5e427af ffff88072c610580 fffffbfff082adbc ffff88072f20bac8 ffff88072f20ba20: ffff88072f20ba88 0000000041b58ab3 ffffffff839f20d0 ffffffff81029260 ffff88072f20ba40: 0000004c8bbedff6 0000000000000001 ffff88072f20bb08 ffffffff832ad2e7 ffff88072f20ba60: 0000000000000000 ffff88072f20bb30 0000000000000000 0000000000000000 ffff88072f20ba80: ffff88072f20bdf8 0000000000000008 0000000000000000 0000000000000000 ffff88072f20baa0: dffffc0000000000 ffff88072f213440 1ffffffff082adb9 0000000000000390 ffff88072f20bac0: 0000000040000000 0000000000000000 0000000000000000 ffff88072f213d90 ffff88072f20bae0: ffffffff81029265 0000000000029fff ffff88072f20bb00 fffffffffffffff8 ffff88072f20bb00: 000000000000000f 0000000000000001 0000004c8bbee2c1 ffffffff83c16120 ffff88072f20bb20: 0000004c8bbedff6 ffff88072f213440 ffff88072f20bdf8 ffffffff8101e752 ffff88072f20bb40: ffff88072f213d78 1ffff100e5e41770 ffffed00e5e427af 0000006400000000 ffff88072f20bb60: 0000000000000000 ffff88072f20bef8 0000000000000000 0000000000000000 ffff88072f20bb80: 0000000041b58ab3 ffffffff839f1dd0 ffffffff8101e2a0 0000000000000000 ffff88072f20bba0: 0000000000000000 ... ffff88072f20bcc0: ffff880402cb9498 ffff88072f20bd18 ffffffff81448fa2 0000000000000000 ffff88072f20bce0: ffff880402cb94a8 ffffffff8100a25d ffff88072f20bef8 0000004c8bbee2c1 ffff88072f20bd00: ffffffff83c16120 0000004c8bbedff6 ffffffff83c16130 ffff88072f20bdd0 ffff88072f20bd20: ffffffff832ad2e7 0000004c8bbedff6 ffffffff83c16130 ffff88072f20bdf0 ffff88072f20bd40: ffffffff832ad2e7 ffff88072f20be18 ffff88072f20bda0 ffffffff81448fa2 ffff88072f20bd60: 0000000000000000 ffffed00e5e426ca 00000000000004c1 0000004c8bbee2c1 ffff88072f20bd80: 000000000000001f 0000000000000000 000000002dd2f949 ffff88072f20bef8 ffff88072f20bda0: 000000000000001f ffffffff8101e2a5 ffff88072f20bef8 0000000000000000 ffff88072f20bdc0: ffff88072f20bf90 ffffffff8100a235 ffff88072f20bef8 0000004c8bbee2c1 ffff88072f20bde0: ffffffff83c16120 0000004c8bbedff6 ffffffff83c16130 ffff88072f20be18 ffff88072f20be00: ffffffff8100a25d dffffc0000000000 0000000000000000 ffff88072f20be80 ffff88072f20be20: ffffffff81150f5e 0000000000000000 ffffffff81150ec5 ffff88072f20bef8 ffff88072f20be40: 0000000000000000 ffffffff83c64ce8 0000000000000000 ffff88072f20bef8 ffff88072f20be60: 0000000000000000 0000000000000007 ffff88072c4c877c ffff88072f20bf90 ffff88072f20be80: ffff88072f20bea8 ffffffff81151a97 ffff88072f20bee8 ffff88072f20bef8 ffff88072f20bea0: ffff88072c4c8000 ffff88072f20bee8 ffffffff81151d52 0000000000000000 ffff88072f20bec0: 0000000000000001 ffffffff814f8270 ffff880421affd78 ffff880421affcc0 ffff88072f20bee0: ffffffff814e6a70 ffff88072f20bef9 ffffffff832ad08e ffffffff814e6a70 ffff88072f20bf00: ffff880421affcc0 ffff880421affd78 ffffffff814f8270 ffff88072f207f80 ffff88072f20bf20: ffff880421affd60 1ffff100e5e426ec 00000000ffffffff ffffed00e5e426ca ffff88072f20bf40: 00000000000004c1 0000000000000000 1ffff1008435ffaf 0000000000000000 ffff88072f20bf60: 0000000000000000 ffff88072f224f10 ffffffffffffffff ffffffff814e6b83 ffff88072f20bf80: 0000000000000010 0000000000000806 ffff88072f207f60 0000000000000018 ffff88072f20bfa0: ffffffff814e6b83 0000000000000010 0000000000000806 ffff88072f207f60 ffff88072f20bfc0: 0000000000000018 0000000000000001 0000000000000000 ffffffff814e6b83 ffff88072f20bfe0: 0000000000000010 0000000000000806 ffff88072f207f60 0000000000000018 d <IRQ> d flush_smp_call_function_queue+0x111/0x310 d generic_smp_call_function_single_interrupt+0x13/0x30 d smp_call_function_single_interrupt+0x64/0x90 d call_function_single_interrupt+0x90/0xa0 dRIP: 0010:cpuidle_enter_state+0x121/0x7a0 dRSP: 0018:ffff88072c4d7e28 EFLAGS: 00000246c ORIG_RAX: ffffffffffffff04 dRAX: 0000000000000000 RBX: ffff88072f22b720 RCX: 000000000000001f dRDX: 1ffff100e5e442f9 RSI: 000000002dd2f949 RDI: ffff88072f2217c8 dRBP: ffff88072c4d7e88 R08: 0000000000000007 R09: ffffffff83f3f320 dR10: 071c71c71c71c71c R11: ffff88072f21dd04 R12: 0000004c8bbe92c9 dR13: 0000000000000001 R14: 0000000000000001 R15: 0000000000000060 d <EOI> ffff88072f200000: 0000000000000000 ... ffff88072f207560: 0000000000000000 ffff880413a83468 ffff880413a83300 0000000000000000 ffff88072f207580: ffff88072f2075e0 ffffffff814ffc80 0000000000000000 ffff88072f2075e0 ffff88072f2075a0: ffffffff8101b2ee ffff88072f213440 0000000000000000 ffff88072f224ff4 ffff88072f2075c0: ffff88072f2075e0 ffffffff8101d12c ffffffff83c167c0 ffff88072f213440 ffff88072f2075e0: ffff88072f2075f0 ffffffff8101d2a0 ffff88072f207688 ffffffff8100de4e ffff88072f207600: ffffed00e5e40eda ffff88072f213654 ffff880413a83300 ffff880413a83300 ffff88072f207620: ffffffff83c167c0 000000490bb75162 ffff88072f224f08 ffff88072f213658 ffff88072f207640: ffff88072f2076a0 ffff88072f213650 ffffffff00000000 ffff88072f213440 ffff88072f207660: ffffffff83c167c0 ffffffff83c167c0 ffff88072f224ff4 ffff880413a83310 ffff88072f207680: dffffc0000000000 ffff88072f2076a0 ffffffff814eaff9 ffff88072f224f00 ffff88072f2076a0: ffff88072f2076f8 ffffffff814f55aa 000000012f224f08 ffff88072f224f08 ffff88072f2076c0: ffff880413a83300 ffff88072f224ff0 ffff880413a83300 ffff88072f224f08 ffff88072f2076e0: 0000000000000000 ffff880413a83388 ffff88072f224f08 ffff88072f207740 ffff88072f207700: ffffffff814f5fc2 00000000e0a8761e 0000000000000000 ffff88072f224f08 ffff88072f207720: ffff88072f224f08 0000000000000000 ffff880413a83300 ffff880402cb9498 ffff88072f207740: ffff88072f207790 ffffffff81448fa2 ffff880421affd20 ffff880402cb94a8 ffff88072f207760: ffffffff812f57f4 ffff88072a547d10 000000000000000a ffff88072a547d00 ffff88072f207780: 0000000000000000 ffff88072f207d78 ffff88072f207848 ffff880402cb9498 ffff88072f2077a0: ffff88072f2077f0 ffffffff81448fa2 ffffffff814d88ec ffff880402cb94a8 ffff88072f2077c0: ffffffff812f6675 ffff88072f2079b8 ffff88072f207cd8 dffffc0000000000 ffff88072f2077e0: 0000000000000000 ffffed00e5e40f3c ffff88072f2078a8 ffffffff832ad2e7 ffff88072f207800: 1ffff100e54a8fac 000000000000000a 0000000000000004 0000000000000000 ffff88072f207820: ffff88072f207a60 ffff88072f207839 ffffffff832ac633 ffffed00e5e40f3c ffff88072f207840: 0000000000000007 000000000000001e 0000000000000010 0000000000000007 ffff88072f207860: fffffbfff082b618 ffff880729fcbf98 ffff88072f215ca0 000000000000001e ffff88072f207880: ffff88072f207890 ffffffff823d7aa5 ffff88072f2078d0 ffffffff8238dbde ffff88072f2078a0: 0000000000000050 ffff88072f2079b8 ffff88072f207cd8 dffffc0000000000 ffff88072f2078c0: 0000000000000000 ffffed00e5e40f3c ffff88072f207a60 ffffffff812f6390 ffff88072f2078e0: ffffffff812da912 ffff88072c494200 ffff880402cb9498 ffff88072f207948 ffff88072f207900: ffffffff81448fa2 1ffff100e5e40f33 0000000002cb94a8 ffff88072f207d04 ffff88072f207920: ffff88072f2079e0 ffff88072f207ae8 ffff88072f207ae0 ffff88072f207cec ffff88072f207940: ffffed00e5e40f9b ffff880729fcbf90 ffff880402cb9498 ffff88072f2079a8 ffff88072f207960: ffffffff81448fa2 ffff88072c471a00 ffff880402cb94a8 ffffffff8101a831 ffff88072f207980: 0000000000000000 ffff880405b11b10 0000000000000002 ffff88072c489f00 ffff88072f2079a0: dffffc0000000000 ffff88072f207a60 ffffffff832ad2e7 ffff88072f207a08 ffff88072f2079c0: ffffffff81448fa2 0000000000000040 ffff880402cb94a8 ffff880402cb9498 ffff88072f2079e0: ffff88072f207a30 ffffffff81448fa2 ffff88072f213968 ffff880402cb94a8 ffff88072f207a00: ffffffff8101c939 ffff880402cb9498 ffff88072f207a60 ffffffff81448fa2 ffff88072f207a20: ffff88072f207c80 ffff880402cb94a8 ffffffff810191fa 0000000000000000 ffff88072f207a40: ffff88072c49f980 ffff88072f213970 ffff88072f213440 0000000000000002 ffff88072f207a60: ffff88072f207b18 ffffffff832ad2e7 ffff88072f213440 0000000000000002 ffff88072f207a80: ffff88072f207b38 ffffffff832ad2e7 ffff88072f207b60 0000000200000005 ffff88072f207aa0: 0000000000000004 ffff88072f213b60 ffff88072f207a98 ffffffff81019250 ffff88072f207ac0: 0000000000000000 0000000000000000 0000000000000004 0000000000000001 ffff88072f207ae0: ffff88072c49f980 0000000000000000 ffffffff832ab465 0000000000000001 ffff88072f207b00: ffff88072f213440 0000000000000000 ffffffff810190d5 ffff88072f207b28 ffff88072f207b20: ffffffff832ab465 ffff88072f207b60 ffffffff810191fa 0000000000000000 ffff88072f207b40: ffff88072f207b60 ffffffff810191fa dffffc0000000000 ffff88072f207c88 ffff88072f207b60: ffff88072f207c48 ffffffff8100cac2 ffffffff832ad2e7 ffff88072f224f08 ffff88072f207b80: 1ffffffff082adc1 0000000000029fff 1ffff100e5e40f78 ffff88072f213b60 ffff88072f207ba0: ffff880400000002 ffff880700000000 ffff88072f207c80 ffff88072f213440 ffff88072f207bc0: 0000000041b58ab3 ffffffff839f6109 ffffffff8100c720 ffff88072f207c80 ffff88072f207be0: 0000000200000000 ffff88072f213440 0000000181448fa2 ffffffff8100c725 ffff88072f207c00: ffffffff839f60e8 ffffffff81009c50 0000000000000008 ffff880402cb9498 ffff88072f207c20: ffff88072f207c70 ffffffff81448fa2 ffff88072f213440 ffff880402cb94a8 ffff88072f207c40: ffffffff814ffc80 ffff880405b11980 0000000000000000 ffff880402cb9498 ffff88072f207c60: ffff88072f207cb0 ffffffff81448fa2 ffff88072f207d28 ffff880402cb94a8 ffff88072f207c80: ffffffff814ffc80 ffff880405b11980 0000000000000000 ffff880405b11b40 ffff88072f207ca0: ffff880405b11980 0000000000000001 ffff88072f207d68 ffffffff832ad2e7 ffff88072f207cc0: ffff880402cb9498 ffff88072f207d18 ffffffff81448fa2 ffffffff832ad2e7 ffff88072f207ce0: ffff880402cb94a8 ffffffff8101d12c ffff88072f213440 0000000000000000 ffff88072f207d00: dffffc0000000000 0000000000000000 0000000000000002 ffff88072f207dd0 ffff88072f207d20: ffffffff832ad2e7 0000000000000000 0000000000000002 ffff88072f207df0 ffff88072f207d40: ffffffff832ad2e7 ffff88072f207e18 ffffffff832ad2e7 ffffffff814ffc80 ffff88072f207d60: ffffffff8102bd25 ffffed00e5e426ca 00000000000004c1 0000000000000001 ffff88072f207d80: 00000000000003f1 0000000000000000 0000000000000000 0000000000000000 ffff88072f207da0: 0000000000000186 ffffffff8102e295 0000000000000000 0000000000000002 ffff88072f207dc0: 0000000000000400 ffffffff8101d105 ffff880402cb9498 ffff88072f207e28 ffff88072f207de0: ffffffff81448fa2 0000000000000000 ffff880402cb94a8 ffffffff814f840a ffff88072f207e00: 0000000000000000 ffff880405b11980 ffff88072f224f10 0000000000000000 ffff88072f207e20: ffff88072f224f08 ffff88072f207ee0 ffffffff832ad2e7 0000000000000000 ffff88072f207e40: dffffc0000000000 0000000000000000 1ffffffff0782d03 ffff88072f207f50 ffff88072f207e60: ffffffff83c167c0 ffff88072f213654 ffff88072f213658 ffffed00e5e426ca ffff88072f207e80: 00000000000004c1 0000000000000000 000000000000038f 1ffff100e5e426c8 ffff88072f207ea0: 0000000000000000 ffff88072f224f10 ffffffff83c167c0 ffffffff832ab465 ffff88072f207ec0: ffff88072f207ed8 ffffffff814eaff9 ffff88072f224f00 ffff88072f207f08 ffff88072f207ee0: ffff88072f207ef0 ffffffff832ab465 ffff88072f207f50 ffffffff814f840a ffff88072f207f00: 0000000000000000 ffff88072f207f50 ffffffff814f840a ffff88072f224fe8 ffff88072f207f20: ffff88072f224f98 ffff880421affd60 ffffffff814f8270 ffff880421affd78 ffff88072f207f40: ffff880421affcc0 ffffffff814e6a70 ffff88072f207f80 ffffffff814e6b6e ffff88072f207f60: 0000000000000001 ffff880421affca8 dffffc0000000000 0000000000000000 ffff88072f207f80: ffff88072f207fc0 ffffffff813b0dc1 ffff88072f207fa8 ffffffff83a77980 ffff88072f207fa0: 0000004c8bbe92c9 0000000000000001 0000000000000001 0000000000000060 ffff88072f207fc0: ffff88072f207fd0 ffffffff813b2693 ffff88072f207fe8 ffffffff8119e8f4 ffff88072f207fe0: ffff88072f22b720 ffff88072c4d7d79 ffffffff832adc10 ffff88072c4d7d78 d ? cpuidle_enter_state+0x11c/0x7a0 d ? cpuidle_enter_state+0x5/0x7a0 d ? cpuidle_enter+0x5/0x20 d cpuidle_enter+0x17/0x20 d call_cpuidle+0x47/0xc0 d ? call_cpuidle+0x5/0xc0 d cpu_startup_entry+0x1a6/0x2d0 d start_secondary+0x245/0x2d0 d start_cpu+0x5/0x14 ffff88072c4d0000: 0000000057ac6e9d 0000000000000000 0000000000000000 0000000000000000 ffff88072c4d0020: 0000000000000000 ... ffff88072c4d7620: 0000000000000010 0000000000000000 ffff88072c4d7930 ffffffff81510c31 ffff88072c4d7640: 0000000000000000 ... ffff88072c4d7660: 0000000000000010 0000000000000000 ffff88072c4d7970 ffffffff81510c31 ffff88072c4d7680: 0000000000000000 ffff88041fb1d8a8 0000000000000000 ffff88041fb1d800 ffff88072c4d76a0: 1ffff100e589aed8 ffffffe400000000 ffff88041e47cc80 ffff88072c4d79a0 ffff88072c4d76c0: 0000000041b58ab3 ffffffff83a0d8d8 ffffffff81510890 ffff88041fb1d800 ffff88072c4d76e0: 1ffff100e589aee0 ffffffe400000000 ffff88041e47cc80 ffff88072c4d79e0 ffff88072c4d7700: 0000000041b58ab3 ffffffff83a0d8d8 ffffffff81510890 0000000000000000 ffff88072c4d7720: 0000000000000000 ... ffff88072c4d7800: 0000000041b58ab3 ffffffff83a0d6b0 ffffffff814e3a20 0000000000000000 ffff88072c4d7820: 0000000000000000 ... ffff88072c4d7860: ffffffff8150cec6 0000000000000003 ffff88072f21bb20 1ffff100e589af38 ffff88072c4d7880: 0000000000000001 ffff880403093830 ffff88072c4d7948 ffffffff832ad2e7 ffff88072c4d78a0: ffff88072c4d78c8 ffffffff816213e5 ffff88072c4d79a0 ffff88072c4d7ce8 ffff88072c4d78c0: ffff88072c4d7bf8 ffff88072c4d7930 ffffffff8150f4b3 ffff880727eb3000 ffff88072c4d78e0: 0000000000000040 0000000000000000 dffffc0000000000 ffff880403093300 ffff88072c4d7900: 1ffff10080612706 0000000000000001 ffff88072c4c8380 ffff88072c4d79a0 ffff88072c4d7920: ffffffff81359c15 ffff88041e47ce88 ffff88072c4d7c08 ffffffff814e3bf8 ffff88072c4d7940: ffff88072c4d7970 ffff88072c4d7958 ffffffff81359c15 ffff88072c4d7bf8 ffff88072c4d7960: 0000000000000003 ffff88072f21bb20 ffffffff814f2df9 ffff88072c4c8000 ffff88072c4d7980: ffffffff814e3a20 ffff88072c4d7b40 dffffc0000000000 ffff880402cbcc90 ffff88072c4d79a0: ffff88072c4d7a58 ffffffff832ad2e7 ffffffff814f2df9 ffff88072c4c8000 ffff88072c4d79c0: ffffffff814e3a20 ffff88072c4d7b90 ffff88072c4d7ad0 ffff880402cbce88 ffff88072c4d79e0: ffff88072c4d7a98 ffffffff832ad2e7 ffff88072c4c8898 dffffc0000000000 ffff88072c4d7a00: dffffc0000000000 000000000000000a 1ffff100e5899112 ffff88072c4d7b40 ffff88072c4d7a20: 0000000000000000 0000000000000000 ffffffff81359c15 0000000000000000 ffff88072c4d7a40: dffffc0000000000 ffff880403093300 1ffff100e5899112 ffff88072c4d7a68 ffff88072c4d7a60: ffffffff81359c15 ffff88072c4d7ad0 ffff88072c4c8000 ffffffff814e3a20 ffff88072c4d7a80: ffff88072c4d7ad0 ffffffff814f2df9 ffff88072f220e5c ffffffff812a5c3b ffff88072c4d7aa0: ffff88072c4c8000 ffff88072f220d80 ffff880421f61e80 0000000000000000 ffff88072c4d7ac0: ffff88072c4c847c ffff88072c4d7b80 ffffffff832ad2e7 ffffffff814fe10a ffff88072c4d7ae0: ffff88072f21bb08 ffffffff8329e69c ffff88072f220d80 ffff88072c4d7bf8 ffff88072c4d7b00: ffff880421f61e80 ffff8804192d2400 ffff88072c4c8418 ffff88072c4c8898 ffff88072c4d7b20: dffffc0000000000 0000000000000003 ffff880402cb9498 ffff88072c4d7b88 ffff88072c4d7b40: ffffffff81448fa2 ffff88072f220d80 ffff880402cb94a8 ffffffff812faffe ffff88072c4d7b60: 0000000100006f11 ffff88072a547d00 ffff88072f220d80 dffffc0000000000 ffff88072c4d7b80: 0000000000000004 ffff88072c4d7c40 ffffffff832ad2e7 ffffffff832ab608 ffff88072c4d7ba0: ffff88072c4c8000 ffff88072c4d7bf8 ffff880402cb9498 ffff88072c4d7c08 ffff88072c4d7bc0: ffffffff81448fa2 ffff880402cb9498 ffff88072c4d7c20 ffffffff81448fa2 ffff88072c4d7be0: ffff88083fffa140 ffff880402cb94a8 ffffffff815834a0 00000000000249af ffff88072c4d7c00: ffff88072c4d7e50 ffff88072c4d7e50 0000000000000000 ffff88083fffc998 ffff88072c4d7c20: ffff88072c4d7cd8 ffffffff832ad2e7 ffff880402cb9498 ffff88072c4d7c88 ffff88072c4d7c40: ffffffff81448fa2 ffff880402cb9498 ffff880402cb94a8 ffffffff82d315e9 ffff88072c4d7c60: 0000000000188347 ffff88072f21dcc8 ffff88072f21dcc4 0000000000000000 ffff88072c4d7c80: 0000000000000001 ffff88072c4d7d40 ffffffff832ad2e7 ffff880402cb9498 ffff88072c4d7ca0: ffff88072c4d7cf0 ffff880402cb9498 ffff88072c4d7d00 ffff88072c4d7ea0 ffff88072c4d7cc0: ffffffff813a55db ffff880402cb94a8 ffffffff832ab076 ffffffff83f3f320 ffff88072c4d7ce0: ffff880402cb9498 ffff88072c4d7d38 ffffffff81448fa2 0000000000000000 ffff88072c4d7d00: ffff880402cb94a8 ffffffff82d2d9ae ffff88072f22b720 ffffffff83f3f320 ffff88072c4d7d20: 0000000000000001 0000000000000001 0000000000000060 ffff88072c4d7df0 ffff88072c4d7d40: ffffffff832ad2e7 ffffffff812b9035 ffff880402cb9498 ffff88072c4d7da8 ffff88072c4d7d60: ffffffff81448fa2 ffff88072c4d7e88 ffff880402cb94a8 0000000000000060 ffff88072c4d7d80: 0000000000000001 0000000000000001 0000004c8bbe92c9 ffff88072c4d7e88 ffff88072c4d7da0: ffff88072f22b720 ffff88072f21dd04 071c71c71c71c71c ffffffff83f3f320 ffff88072c4d7dc0: 0000000000000007 0000000000000000 000000000000001f 1ffff100e5e442f9 ffff88072c4d7de0: 000000002dd2f949 ffff88072f2217c8 ffffffffffffff04 ffffffff82d2d9d1 ffff88072c4d7e00: 0000000000000010 0000000000000246 ffff88072c4d7e28 0000000000000018 ffff88072c4d7e20: ffffffff82d2d9cc ffff88072f22b720 ffff88072f22b720 ffffffff82d2d8b5 ffff88072c4d7e40: 0000000000000007 ffffffff82d2e0d5 0000004c8ba3686a 0000004c8bbe92c9 ffff88072c4d7e60: ffff88072c4c8000 ffffffff83f3f320 000000000000000a ffff88072f22b720 ffff88072c4d7e80: ffffffff83f3f320 ffff88072c4d7e98 ffffffff82d2e0e7 ffff88072c4d7ec8 ffff88072c4d7ea0: ffffffff81312417 ffffffff813123d5 ffff88072c4d7f18 ffffed00e5899000 ffff88072c4d7ec0: ffff88072c4c8000 ffff88072c4d7f18 ffffffff81312756 ffffed00e5899000 ffff88072c4d7ee0: ffff88072c4c8000 ffff88072c4c8000 000000000000000a ffff88072f213100 ffff88072c4d7f00: 00000000002ab0ae 0000000000000000 0000000000000000 ffff88072c4d7f48 ffff88072c4d7f20: ffffffff811a0e95 0000000000000000 0000000000000000 0000000000000000 ffff88072c4d7f40: 0000000000000000 0000000000000000 ffffffff810001a5 0000000000000000 ffff88072c4d7f60: 0000000000000000 ... 3Memory state around the buggy address: 3 ffff88072f20bc80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 3 ffff88072f20bd00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 3>ffff88072f20bd80: f3 f3 f3 f3 f3 f3 f3 f3 00 00 00 00 00 00 00 00 3 ^ 3 ffff88072f20be00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 3 ffff88072f20be80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 3==================================================================
[toc] | [prev] | [next] | [standalone]
| From | Josh Poimboeuf <jpoimboe@redhat.com> |
|---|---|
| Date | 2016-11-17 18:10 +0100 |
| Message-ID | <sEymf-3v8-107@gated-at.bofh.it> |
| In reply to | #1524178 |
On Thu, Nov 17, 2016 at 09:25:58AM -0500, Vince Weaver wrote: > On Thu, 17 Nov 2016, Josh Poimboeuf wrote: > > > On Thu, Nov 17, 2016 at 10:48:27AM +0100, Dmitry Vyukov wrote: > > > Just in case, there is currently a known KASAN false positive related > > > to longjmp's on GPFs. When a syscall hits GPF stack is unwound to > > > kernel entry point, this leaves a bunch of stray poisoned redzones on > > > the thread stack. They later cause false stack-out-of-bounds reports. > > > > > > But this does not seem to be the case here. Kernel is not tainted. And > > > shadow at the bottom of the reports looks sane. > > > > > > But if that's the case somehow, we will need to add > > > kasan_unpoison_remaining_stack() call before a longjmp like we did for > > > jprobe_return(): > > > https://groups.google.com/d/msg/kasan-dev/Hzox58yZ4MU/TOdFoWMuBQAJ > > > > I'm pretty sure this isn't a KASAN false positive. The unwinder does > > actually seem to be accessing a bad area of the stack, in the middle of > > a function's stack frame. > > I'm having trouble reproducing it on a few other machines I have fuzzing. > So there might be some kernel option contributing, I need to compare > .configs. > > Also the machine that easily triggers the problem I'm compiling with > gcc-5.4 where the machines I can't are using gcc-4.9. I believe KASAN only works with gcc 5 and later, so that would explain why you aren't seeing it with gcc 4.9. -- Josh
[toc] | [prev] | [next] | [standalone]
| From | Vince Weaver <vincent.weaver@maine.edu> |
|---|---|
| Date | 2016-11-17 18:20 +0100 |
| Message-ID | <sEyvU-3yz-17@gated-at.bofh.it> |
| In reply to | #1524502 |
On Thu, 17 Nov 2016, Dmitry Vyukov wrote: > On Thu, Nov 17, 2016 at 3:36 PM, Josh Poimboeuf <jpoimboe@redhat.com> wrote: > > On Thu, Nov 17, 2016 at 09:25:58AM -0500, Vince Weaver wrote: > >> On Thu, 17 Nov 2016, Josh Poimboeuf wrote: > >> > >> > On Thu, Nov 17, 2016 at 10:48:27AM +0100, Dmitry Vyukov wrote: > >> > > Just in case, there is currently a known KASAN false positive related > >> > > to longjmp's on GPFs. When a syscall hits GPF stack is unwound to > >> > > kernel entry point, this leaves a bunch of stray poisoned redzones on > >> > > the thread stack. They later cause false stack-out-of-bounds reports. > >> > > > >> > > But this does not seem to be the case here. Kernel is not tainted. And > >> > > shadow at the bottom of the reports looks sane. > >> > > > >> > > But if that's the case somehow, we will need to add > >> > > kasan_unpoison_remaining_stack() call before a longjmp like we did for > >> > > jprobe_return(): > >> > > https://groups.google.com/d/msg/kasan-dev/Hzox58yZ4MU/TOdFoWMuBQAJ > >> > > >> > I'm pretty sure this isn't a KASAN false positive. The unwinder does > >> > actually seem to be accessing a bad area of the stack, in the middle of > >> > a function's stack frame. > >> > >> I'm having trouble reproducing it on a few other machines I have fuzzing. > >> So there might be some kernel option contributing, I need to compare > >> .configs. > >> > >> Also the machine that easily triggers the problem I'm compiling with > >> gcc-5.4 where the machines I can't are using gcc-4.9. > > > > I believe KASAN only works with gcc 5 and later, so that would explain > > why you aren't seeing it with gcc 4.9. > > Right. 4.9 has limited support for KASAN. It supports general > instrumentation, but only with CONFIG_KASAN_OUTLINE, and it does not > support stack poisoning. Which is required to detect stack OOBs. I guess it's time to update the other machines to debian-unstable then. I didn't really need to be able to run dmesg as non-root anyway. I would actually be compiling the kernels with gcc-6.2 rather than gcc-5 but that seems to not work currently. Haven't had time to see if that's a known issue or not. Vince
[toc] | [prev] | [next] | [standalone]
| From | Dmitry Vyukov <dvyukov@google.com> |
|---|---|
| Date | 2016-11-17 18:50 +0100 |
| Message-ID | <sEyvU-3yz-19@gated-at.bofh.it> |
| In reply to | #1524502 |
On Thu, Nov 17, 2016 at 3:36 PM, Josh Poimboeuf <jpoimboe@redhat.com> wrote: > On Thu, Nov 17, 2016 at 09:25:58AM -0500, Vince Weaver wrote: >> On Thu, 17 Nov 2016, Josh Poimboeuf wrote: >> >> > On Thu, Nov 17, 2016 at 10:48:27AM +0100, Dmitry Vyukov wrote: >> > > Just in case, there is currently a known KASAN false positive related >> > > to longjmp's on GPFs. When a syscall hits GPF stack is unwound to >> > > kernel entry point, this leaves a bunch of stray poisoned redzones on >> > > the thread stack. They later cause false stack-out-of-bounds reports. >> > > >> > > But this does not seem to be the case here. Kernel is not tainted. And >> > > shadow at the bottom of the reports looks sane. >> > > >> > > But if that's the case somehow, we will need to add >> > > kasan_unpoison_remaining_stack() call before a longjmp like we did for >> > > jprobe_return(): >> > > https://groups.google.com/d/msg/kasan-dev/Hzox58yZ4MU/TOdFoWMuBQAJ >> > >> > I'm pretty sure this isn't a KASAN false positive. The unwinder does >> > actually seem to be accessing a bad area of the stack, in the middle of >> > a function's stack frame. >> >> I'm having trouble reproducing it on a few other machines I have fuzzing. >> So there might be some kernel option contributing, I need to compare >> .configs. >> >> Also the machine that easily triggers the problem I'm compiling with >> gcc-5.4 where the machines I can't are using gcc-4.9. > > I believe KASAN only works with gcc 5 and later, so that would explain > why you aren't seeing it with gcc 4.9. Right. 4.9 has limited support for KASAN. It supports general instrumentation, but only with CONFIG_KASAN_OUTLINE, and it does not support stack poisoning. Which is required to detect stack OOBs.
[toc] | [prev] | [next] | [standalone]
| From | Josh Poimboeuf <jpoimboe@redhat.com> |
|---|---|
| Date | 2016-11-17 18:40 +0100 |
| Message-ID | <sEymf-3v8-109@gated-at.bofh.it> |
| In reply to | #1524178 |
On Thu, Nov 17, 2016 at 10:48:27AM +0100, Dmitry Vyukov wrote: > Just in case, there is currently a known KASAN false positive related > to longjmp's on GPFs. When a syscall hits GPF stack is unwound to > kernel entry point, this leaves a bunch of stray poisoned redzones on > the thread stack. They later cause false stack-out-of-bounds reports. > > But this does not seem to be the case here. Kernel is not tainted. And > shadow at the bottom of the reports looks sane. > > But if that's the case somehow, we will need to add > kasan_unpoison_remaining_stack() call before a longjmp like we did for > jprobe_return(): > https://groups.google.com/d/msg/kasan-dev/Hzox58yZ4MU/TOdFoWMuBQAJ I'm pretty sure this isn't a KASAN false positive. The unwinder does actually seem to be accessing a bad area of the stack, in the middle of a function's stack frame. -- Josh
[toc] | [prev] | [next] | [standalone]
| From | Vince Weaver <vincent.weaver@maine.edu> |
|---|---|
| Date | 2016-11-17 19:20 +0100 |
| Message-ID | <sEymf-3v8-111@gated-at.bofh.it> |
| In reply to | #1524596 |
On Thu, 17 Nov 2016, Josh Poimboeuf wrote: > On Thu, Nov 17, 2016 at 10:48:27AM +0100, Dmitry Vyukov wrote: > > Just in case, there is currently a known KASAN false positive related > > to longjmp's on GPFs. When a syscall hits GPF stack is unwound to > > kernel entry point, this leaves a bunch of stray poisoned redzones on > > the thread stack. They later cause false stack-out-of-bounds reports. > > > > But this does not seem to be the case here. Kernel is not tainted. And > > shadow at the bottom of the reports looks sane. > > > > But if that's the case somehow, we will need to add > > kasan_unpoison_remaining_stack() call before a longjmp like we did for > > jprobe_return(): > > https://groups.google.com/d/msg/kasan-dev/Hzox58yZ4MU/TOdFoWMuBQAJ > > I'm pretty sure this isn't a KASAN false positive. The unwinder does > actually seem to be accessing a bad area of the stack, in the middle of > a function's stack frame. I'm having trouble reproducing it on a few other machines I have fuzzing. So there might be some kernel option contributing, I need to compare .configs. Also the machine that easily triggers the problem I'm compiling with gcc-5.4 where the machines I can't are using gcc-4.9. Vince
[toc] | [prev] | [next] | [standalone]
| From | Peter Zijlstra <peterz@infradead.org> |
|---|---|
| Date | 2016-11-17 18:20 +0100 |
| Message-ID | <sEyvU-3yz-21@gated-at.bofh.it> |
| In reply to | #1524084 |
On Thu, Nov 17, 2016 at 09:18:48AM -0600, Josh Poimboeuf wrote: > On Thu, Nov 17, 2016 at 10:04:46AM +0100, Peter Zijlstra wrote: > > On Wed, Nov 16, 2016 at 10:48:28PM -0600, Josh Poimboeuf wrote: > > > Peter or Vince, can you try to recreate with this patch? It dumps the > > > raw stack contents during a stack dump. Hopefully that would give a > > > clue about what's going wrong. > > > > > > Here goes... I'll do another run and get you the results of that as > > well. > > Thanks, I just waded through this and it turned up some good clues. And > according to 'git blame', you might be able to help :-) > > It's not stack corruption. Instead it looks like > __intel_pmu_pebs_event() is creating a bad or stale pt_regs which gets > passed to the unwinder. Specifically, regs->bp points to a seemingly > random address on the NMI stack. Which seems odd, considering the code > itself is running on the same NMI stack. > > I don't know much about the PEBS code but it seems like it's passing > some stale data. Either that or there's some NMI nesting going on. Ooh, indeed. The PEBS record can be quite stale by the time we get to the interrupt. Using those registers for an unwind is 'interesting' at best. Esp. with the multi-pebs stuff that's landed this can be very very stale, but even single pebs can have a radically different stack at interrupt time than we had at record time -- imagine a (i)ret happening in between. Let me consider that code, and what to do about this; its been a while since I went over all that.
[toc] | [prev] | [next] | [standalone]
| From | Peter Zijlstra <peterz@infradead.org> |
|---|---|
| Date | 2016-11-17 18:20 +0100 |
| Message-ID | <sEyvV-3yz-75@gated-at.bofh.it> |
| In reply to | #1524515 |
On Thu, Nov 17, 2016 at 05:07:00PM +0100, Peter Zijlstra wrote:
> On Thu, Nov 17, 2016 at 09:18:48AM -0600, Josh Poimboeuf wrote:
> > Thanks, I just waded through this and it turned up some good clues. And
> > according to 'git blame', you might be able to help :-)
> >
> > It's not stack corruption. Instead it looks like
> > __intel_pmu_pebs_event() is creating a bad or stale pt_regs which gets
> > passed to the unwinder. Specifically, regs->bp points to a seemingly
> > random address on the NMI stack. Which seems odd, considering the code
> > itself is running on the same NMI stack.
> >
> > I don't know much about the PEBS code but it seems like it's passing
> > some stale data. Either that or there's some NMI nesting going on.
So the puzzle was BP,SP pointing into the NMI stack at random spots. But
I think I can explain this; if the event has a very _very_ short period,
then the tail __intel_pmu_enable_all() call from the PMI handler will
'insta' trigger a record and raise another PMI.
We then get back-to-back NMIs with a record pointing to a now
overwritten stack.
The other scenario, where there is an (i)ret between the record and the
interrupt would be less confusing but still wrong.
Solve this by always using iregs->{bp,sp} for callchains.
The below patch still copies the record BP,SP when !CALLCHAINS &&
SAMPLE_REGS; does this make sense?
The fuzzer is still running with this patch applied.. I'll let it run
for a while.
---
arch/x86/events/intel/ds.c | 35 +++++++++++++++++++++++------------
arch/x86/events/perf_event.h | 2 +-
2 files changed, 24 insertions(+), 13 deletions(-)
diff --git a/arch/x86/events/intel/ds.c b/arch/x86/events/intel/ds.c
index 0319311dbdbb..be202390bbd3 100644
--- a/arch/x86/events/intel/ds.c
+++ b/arch/x86/events/intel/ds.c
@@ -1108,20 +1108,20 @@ static void setup_pebs_sample_data(struct perf_event *event,
}
/*
- * We use the interrupt regs as a base because the PEBS record
- * does not contain a full regs set, specifically it seems to
- * lack segment descriptors, which get used by things like
- * user_mode().
+ * We use the interrupt regs as a base because the PEBS record does not
+ * contain a full regs set, specifically it seems to lack segment
+ * descriptors, which get used by things like user_mode().
*
- * In the simple case fix up only the IP and BP,SP regs, for
- * PERF_SAMPLE_IP and PERF_SAMPLE_CALLCHAIN to function properly.
- * A possible PERF_SAMPLE_REGS will have to transfer all regs.
+ * In the simple case fix up only the IP for PERF_SAMPLE_IP.
+ *
+ * We must however always use BP,SP from iregs for the unwinder to stay
+ * sane; the record BP,SP can point into thin air when the record is
+ * from a previous PMI context or an (I)RET happend between the record
+ * and PMI.
*/
*regs = *iregs;
regs->flags = pebs->flags;
set_linear_ip(regs, pebs->ip);
- regs->bp = pebs->bp;
- regs->sp = pebs->sp;
if (sample_type & PERF_SAMPLE_REGS_INTR) {
regs->ax = pebs->ax;
@@ -1130,10 +1130,21 @@ static void setup_pebs_sample_data(struct perf_event *event,
regs->dx = pebs->dx;
regs->si = pebs->si;
regs->di = pebs->di;
- regs->bp = pebs->bp;
- regs->sp = pebs->sp;
- regs->flags = pebs->flags;
+ /*
+ * Per the above; only set BP,SP if we don't need callchains.
+ *
+ * XXX: does this make sense?
+ */
+ if (!(sample_type & PERF_SAMPLE_CALLCHAIN)) {
+ regs->bp = pebs->bp;
+ regs->sp = pebs->sp;
+ }
+
+ /*
+ * Preserve PERF_EFLAGS_VM from set_linear_ip().
+ */
+ regs->flags = pebs->flags | (regs->flags & PERF_EFLAGS_VM);
#ifndef CONFIG_X86_32
regs->r8 = pebs->r8;
regs->r9 = pebs->r9;
diff --git a/arch/x86/events/perf_event.h b/arch/x86/events/perf_event.h
index 5874d8de1f8d..a77ee026643d 100644
--- a/arch/x86/events/perf_event.h
+++ b/arch/x86/events/perf_event.h
@@ -113,7 +113,7 @@ struct debug_store {
* Per register state.
*/
struct er_account {
- raw_spinlock_t lock; /* per-core: protect structure */
+ raw_spinlock_t lock; /* per-core: protect structure */
u64 config; /* extra MSR config */
u64 reg; /* extra MSR number */
atomic_t ref; /* reference count */
[toc] | [prev] | [next] | [standalone]
| From | tip-bot for Peter Zijlstra <tipbot@zytor.com> |
|---|---|
| Date | 2016-11-22 13:40 +0100 |
| Subject | [tip:perf/urgent] perf/x86/intel: Cure bogus unwind from PEBS entries |
| Message-ID | <sGiwG-79h-21@gated-at.bofh.it> |
| In reply to | #1524540 |
Commit-ID: b8000586c90b4804902058a38d3a59ce5708e695
Gitweb: http://git.kernel.org/tip/b8000586c90b4804902058a38d3a59ce5708e695
Author: Peter Zijlstra <peterz@infradead.org>
AuthorDate: Thu, 17 Nov 2016 18:17:31 +0100
Committer: Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 22 Nov 2016 12:36:58 +0100
perf/x86/intel: Cure bogus unwind from PEBS entries
Vince Weaver reported that perf_fuzzer + KASAN detects that PEBS event
unwinds sometimes do 'weird' things. In particular, we seemed to be
ending up unwinding from random places on the NMI stack.
While it was somewhat expected that the event record BP,SP would not
match the interrupt BP,SP in that the interrupt is strictly later than
the record event, it was overlooked that it could be on an already
overwritten stack.
Therefore, don't copy the recorded BP,SP over the interrupted BP,SP
when we need stack unwinds.
Note that its still possible the unwind doesn't full match the actual
event, as its entirely possible to have done an (I)RET between record
and interrupt, but on average it should still point in the general
direction of where the event came from. Also, it's the best we can do,
considering.
The particular scenario that triggered the bogus NMI stack unwind was
a PEBS event with very short period, upon enabling the event at the
tail of the PMI handler (FREEZE_ON_PMI is not used), it instantly
triggers a record (while still on the NMI stack) which in turn
triggers the next PMI. This then causes back-to-back NMIs and we'll
try and unwind the stack-frame from the last NMI, which obviously is
now overwritten by our own.
Analyzed-by: Josh Poimboeuf <jpoimboe@redhat.com>
Reported-by: Vince Weaver <vincent.weaver@maine.edu>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Arnaldo Carvalho de Melo <acme@kernel.org>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Stephane Eranian <eranian@gmail.com>
Cc: Stephane Eranian <eranian@google.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: davej@codemonkey.org.uk <davej@codemonkey.org.uk>
Cc: dvyukov@google.com <dvyukov@google.com>
Cc: stable@vger.kernel.org
Fixes: ca037701a025 ("perf, x86: Add PEBS infrastructure")
Link: http://lkml.kernel.org/r/20161117171731.GV3157@twins.programming.kicks-ass.net
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
arch/x86/events/intel/ds.c | 35 +++++++++++++++++++++++------------
arch/x86/events/perf_event.h | 2 +-
2 files changed, 24 insertions(+), 13 deletions(-)
diff --git a/arch/x86/events/intel/ds.c b/arch/x86/events/intel/ds.c
index 0319311..be20239 100644
--- a/arch/x86/events/intel/ds.c
+++ b/arch/x86/events/intel/ds.c
@@ -1108,20 +1108,20 @@ static void setup_pebs_sample_data(struct perf_event *event,
}
/*
- * We use the interrupt regs as a base because the PEBS record
- * does not contain a full regs set, specifically it seems to
- * lack segment descriptors, which get used by things like
- * user_mode().
+ * We use the interrupt regs as a base because the PEBS record does not
+ * contain a full regs set, specifically it seems to lack segment
+ * descriptors, which get used by things like user_mode().
*
- * In the simple case fix up only the IP and BP,SP regs, for
- * PERF_SAMPLE_IP and PERF_SAMPLE_CALLCHAIN to function properly.
- * A possible PERF_SAMPLE_REGS will have to transfer all regs.
+ * In the simple case fix up only the IP for PERF_SAMPLE_IP.
+ *
+ * We must however always use BP,SP from iregs for the unwinder to stay
+ * sane; the record BP,SP can point into thin air when the record is
+ * from a previous PMI context or an (I)RET happend between the record
+ * and PMI.
*/
*regs = *iregs;
regs->flags = pebs->flags;
set_linear_ip(regs, pebs->ip);
- regs->bp = pebs->bp;
- regs->sp = pebs->sp;
if (sample_type & PERF_SAMPLE_REGS_INTR) {
regs->ax = pebs->ax;
@@ -1130,10 +1130,21 @@ static void setup_pebs_sample_data(struct perf_event *event,
regs->dx = pebs->dx;
regs->si = pebs->si;
regs->di = pebs->di;
- regs->bp = pebs->bp;
- regs->sp = pebs->sp;
- regs->flags = pebs->flags;
+ /*
+ * Per the above; only set BP,SP if we don't need callchains.
+ *
+ * XXX: does this make sense?
+ */
+ if (!(sample_type & PERF_SAMPLE_CALLCHAIN)) {
+ regs->bp = pebs->bp;
+ regs->sp = pebs->sp;
+ }
+
+ /*
+ * Preserve PERF_EFLAGS_VM from set_linear_ip().
+ */
+ regs->flags = pebs->flags | (regs->flags & PERF_EFLAGS_VM);
#ifndef CONFIG_X86_32
regs->r8 = pebs->r8;
regs->r9 = pebs->r9;
diff --git a/arch/x86/events/perf_event.h b/arch/x86/events/perf_event.h
index 5874d8d..a77ee02 100644
--- a/arch/x86/events/perf_event.h
+++ b/arch/x86/events/perf_event.h
@@ -113,7 +113,7 @@ struct debug_store {
* Per register state.
*/
struct er_account {
- raw_spinlock_t lock; /* per-core: protect structure */
+ raw_spinlock_t lock; /* per-core: protect structure */
u64 config; /* extra MSR config */
u64 reg; /* extra MSR number */
atomic_t ref; /* reference count */
[toc] | [prev] | [next] | [standalone]
| From | Josh Poimboeuf <jpoimboe@redhat.com> |
|---|---|
| Date | 2016-11-17 18:50 +0100 |
| Message-ID | <sEyvU-3yz-23@gated-at.bofh.it> |
| In reply to | #1524084 |
On Thu, Nov 17, 2016 at 10:04:46AM +0100, Peter Zijlstra wrote: > On Wed, Nov 16, 2016 at 10:48:28PM -0600, Josh Poimboeuf wrote: > > Peter or Vince, can you try to recreate with this patch? It dumps the > > raw stack contents during a stack dump. Hopefully that would give a > > clue about what's going wrong. > > > Here goes... I'll do another run and get you the results of that as > well. Thanks, I just waded through this and it turned up some good clues. And according to 'git blame', you might be able to help :-) It's not stack corruption. Instead it looks like __intel_pmu_pebs_event() is creating a bad or stale pt_regs which gets passed to the unwinder. Specifically, regs->bp points to a seemingly random address on the NMI stack. Which seems odd, considering the code itself is running on the same NMI stack. I don't know much about the PEBS code but it seems like it's passing some stale data. Either that or there's some NMI nesting going on. -- Josh
[toc] | [prev] | [next] | [standalone]
Page 1 of 2 [1] 2 Next page →
Back to top | Article view | linux.kernel
csiph-web