Groups > linux.kernel > #1485173 > unrolled thread

perf: perf_fuzzer crashing due to slab poison

Started by	Vince Weaver <vincent.weaver@maine.edu>
First post	2016-09-16 18:50 +0200
Last post	2016-09-19 16:50 +0200
Articles	2 — 1 participant

Back to article view | Back to linux.kernel

  perf: perf_fuzzer crashing due to slab poison Vince Weaver <vincent.weaver@maine.edu> - 2016-09-16 18:50 +0200
    Re: perf: perf_fuzzer crashing due to slab poison Vince Weaver <vincent.weaver@maine.edu> - 2016-09-19 16:50 +0200

#1485173 — perf: perf_fuzzer crashing due to slab poison

From	Vince Weaver <vincent.weaver@maine.edu>
Date	2016-09-16 18:50 +0200
Subject	perf: perf_fuzzer crashing due to slab poison
Message-ID	<si4uR-7hN-5@gated-at.bofh.it>

Trying to replicate the cgroup problem on my haswell machine and tripped 
over this other (probably unrelated) bug that not only crashed the machine 
but took out the whole local network (due to the ethernet card getting 
stuck somehow).

It looks like slab poison in some of those registers :(  Are there any 
options I should be enabling to help debug this kind of thing?  Could we
somehow write useful info (rather than just 6b6b6b) into the freed memory 
to give hints when debugging?

Vince

	Linux version 4.8.0-rc6+ x86_64
	Processor: Intel 6/60/3
	/proc/sys/kernel/perf_event_max_sample_rate currently: 1750/s
	/proc/sys/kernel/perf_event_paranoid currently: 0
	To reproduce, try: ./perf_fuzzer -s 30000 -r 1473974214


[33967.807734] general protection fault: 0000 [#1] SMP
[33967.813503] Modules linked in: binfmt_misc intel_rapl iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm snd_hda_codec_realtek snd_hda_codec_hdmi iTCO_wdt snd_hda_codec_generic irqbypass iTCO_vendor_support snd_hda_intel crct10dif_pclmul snd_hda_codec crc32_pclmul ghash_clmulni_intel snd_hda_core ppdev aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper evdev snd_hwdep snd_pcm cryptd i915 snd_timer psmouse pcspkr serio_raw drm_kms_helper snd mei_me tpm_tis tpm_tis_core video battery soundcore sg lpc_ich mei mfd_core parport_pc wmi drm i2c_i801 i2c_algo_bit i2c_smbus parport tpm button sr_mod cdrom sd_mod xhci_pci xhci_hcd ahci ehci_pci libahci ehci_hcd libata e1000e ptp usbcore crc32c_intel scsi_mod usb_common pps_core fan thermal
[33967.890749] CPU: 0 PID: 8251 Comm: perf_fuzzer Tainted: G        W       4.8.0-rc6+ #194
[33967.899990] Hardware name: LENOVO 10AM000AUS/SHARKBAY, BIOS FBKT72AUS 01/26/2014
[33967.908498] task: ffff8801163ae700 task.stack: ffff880115e90000
[33967.915441] RIP: 0010:[<ffffffff81079886>]  [<ffffffff81079886>] wait_consider_task+0x16/0xc50
[33967.925281] RSP: 0018:ffff880115e93de8  EFLAGS: 00010296
[33967.931544] RAX: 6b6b6b6b6b6b6b6b RBX: ffff880115e93eb0 RCX: ffff8801191acc00
[33967.939740] RDX: 6b6b6b6b6b6b675b RSI: 0000000000000000 RDI: ffff880115e93eb0
[33967.947927] RBP: ffff880115e93e48 R08: 00000000b085cee5 R09: 4522cf5100000000
[33967.956100] R10: 00000000001fa23f R11: 0000000000000000 R12: ffff8801163aeb00
[33967.964270] R13: ffff8801163ae700 R14: ffff8801163ae700 R15: 6b6b6b6b6b6b675b
[33967.972445] FS:  00007f1dc0a19700(0000) GS:ffff88011ea00000(0000) knlGS:0000000000000000
[33967.981633] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[33967.988283] CR2: 00007ffd20fbb058 CR3: 000000011825c000 CR4: 00000000001407f0
[33967.996446] DR0: 0000000000000000 DR1: 0000000000000ff0 DR2: 0000000000000000
[33968.004615] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[33968.012753] Stack:
[33968.015417]  0000000000000282 000000011ea18c18 ffffffff81c05098 ffffffff81c05080
[33968.023943]  ffff8801163aeb28 ffff8801163ae700 ffff8801163ae700 ffff880115e93eb0
[33968.032492]  ffff8801163aeb00 ffff8801163ae700 ffff8801163ae700 6b6b6b6b6b6b675b
[33968.040996] Call Trace:
[33968.044109]  [<ffffffff8107a5cf>] do_wait+0x10f/0x250
[33968.050021]  [<ffffffff8107b886>] SyS_wait4+0x66/0xd0
[33968.055975]  [<ffffffff810791d0>] ? task_stopped_code+0x60/0x60
[33968.062833]  [<ffffffff81003b5e>] do_syscall_64+0x5e/0xc0
[33968.069103]  [<ffffffff817250ea>] entry_SYSCALL64_slow_path+0x25/0x25
[33968.076515] Code: 02 00 e9 62 ff ff ff 4c 89 ef e8 16 5f 07 00 e9 38 ff ff ff 90 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 53 48 83 ec 38 <44> 8b b2 7c 03 00 00 41 83 fe 10 74 29 8b 07 49 89 fd 89 f3 49 
[33968.099081] RIP  [<ffffffff81079886>] wait_consider_task+0x16/0xc50
[33968.106371]  RSP <ffff880115e93de8>
[33968.112742] ---[ end trace dfb54c93a465ccd8 ]---
[33968.112743] general protection fault: 0000 [#2] SMP
[33968.112756] Modules linked in: binfmt_misc intel_rapl iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm snd_hda_codec_realtek snd_hda_codec_hdmi iTCO_wdt snd_hda_codec_generic irqbypass iTCO_vendor_support snd_hda_intel crct10dif_pclmul snd_hda_codec crc32_pclmul ghash_clmulni_intel snd_hda_core ppdev aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper evdev snd_hwdep snd_pcm cryptd i915 snd_timer psmouse pcspkr serio_raw drm_kms_helper snd mei_me tpm_tis tpm_tis_core video battery soundcore sg lpc_ich mei mfd_core parport_pc wmi drm i2c_i801 i2c_algo_bit i2c_smbus parport tpm button sr_mod cdrom sd_mod xhci_pci xhci_hcd ahci ehci_pci libahci ehci_hcd libata e1000e ptp usbcore crc32c_intel scsi_mod usb_common pps_core fan thermal
[33968.112757] CPU: 1 PID: 9323 Comm: perf_fuzzer Tainted: G      D W       4.8.0-rc6+ #194
[33968.112757] Hardware name: LENOVO 10AM000AUS/SHARKBAY, BIOS FBKT72AUS 01/26/2014
[33968.112758] task: ffff880114a9e300 task.stack: ffff880115c4c000
[33968.112761] RIP: 0010:[<ffffffff811320b5>]  [<ffffffff811320b5>] syscall_unregfunc+0x45/0x90
[33968.112761] RSP: 0018:ffff880115c4faf0  EFLAGS: 00010283
[33968.112762] RAX: 6b6b6b6b6b6b6b6b RBX: ffffffff81d1c7c8 RCX: ffff8801170af010
[33968.112762] RDX: ffff8801170aeb70 RSI: ffff8801155d67a0 RDI: ffff8801155d64c0
[33968.112762] RBP: ffff880115c4faf0 R08: 0000000000000000 R09: 0000000000000000
[33968.112763] R10: ffff880114a9e300 R11: 0000000000000000 R12: ffffffff81d1c7c0
[33968.112763] R13: ffffffff81c107a0 R14: ffff880114acacd0 R15: ffffffff81003030
[33968.112764] FS:  00007f1dc0a19700(0000) GS:ffff88011ea40000(0000) knlGS:0000000000000000
[33968.112764] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[33968.112764] CR2: 00007ffe09882bdc CR3: 0000000001c06000 CR4: 00000000001406e0
[33968.112765] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[33968.112765] DR3: 0000000000008788 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[33968.112766] Stack:
[33968.112767]  ffff880115c4fb28 ffffffff81131c68 ffffffff81c107a0 0000000000000003
[33968.112768]  ffff880116dd4c00 ffff880116dd4c40 ffff880114cfaaa0 ffff880115c4fb50
[33968.112769]  ffffffff8114cf13 0000000000000082 ffffffff81c107a0 0000000000000000
[33968.112769] Call Trace:
[33968.112771]  [<ffffffff81131c68>] tracepoint_probe_unregister+0x188/0x1e0
[33968.112772]  [<ffffffff8114cf13>] trace_event_reg+0x43/0xd0
[33968.112773]  [<ffffffff81150713>] perf_trace_event_unreg.isra.2+0x33/0x90
[33968.112774]  [<ffffffff81150a78>] perf_trace_destroy+0x38/0x50
[33968.112776]  [<ffffffff8116a859>] tp_perf_event_destroy+0x9/0x10
[33968.112777]  [<ffffffff81172e45>] _free_event+0xd5/0x330
[33968.112778]  [<ffffffff81173534>] put_event+0x14/0x20
[33968.112779]  [<ffffffff81173770>] perf_event_release_kernel+0x230/0x2d0
[33968.112780]  [<ffffffff81173573>] ? perf_event_release_kernel+0x33/0x2d0
[33968.112781]  [<ffffffff81173820>] perf_release+0x10/0x20
[33968.112784]  [<ffffffff81211a9f>] __fput+0xdf/0x1f0
[33968.112785]  [<ffffffff81211bee>] ____fput+0xe/0x10
[33968.112786]  [<ffffffff81095f2e>] task_work_run+0x7e/0xa0
[33968.112788]  [<ffffffff8107ad36>] do_exit+0x2f6/0xb10
[33968.112789]  [<ffffffff81086a92>] ? get_signal+0xc2/0x6d0
[33968.112790]  [<ffffffff8107b5e0>] do_group_exit+0x50/0xd0
[33968.112791]  [<ffffffff81086c5f>] get_signal+0x28f/0x6d0
[33968.112793]  [<ffffffff8109dec7>] ? finish_task_switch+0xa7/0x220
[33968.112795]  [<ffffffff8102d518>] do_signal+0x28/0x760
[33968.112796]  [<ffffffff8116c2a6>] ? perf_trace_run_bpf_submit+0x76/0xb0
[33968.112797]  [<ffffffff810030e5>] ? perf_trace_sys_exit+0xb5/0xd0
[33968.112798]  [<ffffffff8100329c>] exit_to_usermode_loop+0x8c/0xd0
[33968.112799]  [<ffffffff81003a87>] prepare_exit_to_usermode+0x37/0x50
[33968.112800]  [<ffffffff817259a5>] retint_user+0x8/0x10
[33968.112810] Code: e5 e8 90 29 5f 00 48 c7 c7 40 d5 c0 81 48 8b b7 e0 02 00 00 48 8d be 20 fd ff ff 48 81 ff 40 d5 c0 81 74 44 48 8b 86 30 03 00 00 <48> 8b 48 10 48 83 c0 10 48 39 c8 48 8d 91 60 fb ff ff 74 ce 48 
[33968.112812] RIP  [<ffffffff811320b5>] syscall_unregfunc+0x45/0x90
[33968.112812]  RSP <ffff880115c4faf0>
[33968.112816] ---[ end trace dfb54c93a465ccd9 ]---
[33968.112817] Fixing recursive fault but reboot is needed!
[33989.573587] INFO: rcu_sched detected stalls on CPUs/tasks:
[33989.579895] 	1-...: (1 GPs behind) idle=4b9/140000000000000/0 softirq=1278733/1278734 fqs=2446 
[33989.589632] 	(detected by 0, t=5256 jiffies, g=1198134, c=1198133, q=79)
[33989.597186] Task dump for CPU 1:
[33989.601057] perf_fuzzer     S 0000000000000000     0  8251   3914 0x0000000a
[33989.609084]  0000000000000000 0000000000000000 ffff880115e93eb8 0000000000000086
[33989.617545]  ffffffff8107ad6a ffff880100000000 ffffffff00000000 0000000000000086
[33989.626041]  0000000115c3d000 ffffffff81c05098 ffffffff81c05080 ffffffff81c05080
[33989.634494] Call Trace:
[33989.637532]  [<ffffffff8107ad6a>] ? do_exit+0x32a/0xb10
[33989.643525]  [<ffffffff810c7e58>] ? do_raw_write_lock+0x48/0xc0
[33989.650259]  [<ffffffff81724be0>] ? _raw_write_lock_irq+0x40/0x50
[33989.657201]  [<ffffffff8107ad6a>] ? do_exit+0x32a/0xb10
[33989.663229]  [<ffffffff8107ad6a>] ? do_exit+0x32a/0xb10
[33989.669250]  [<ffffffff81726737>] ? rewind_stack_do_exit+0x17/0x20

[toc] | [next] | [standalone]

#1486563

From	Vince Weaver <vincent.weaver@maine.edu>
Date	2016-09-19 16:50 +0200
Message-ID	<sj83p-6ES-53@gated-at.bofh.it>
In reply to	#1485173

I've updated both the haswell and skylake machine to stock 4.8-rc7
and both are still getting slab corruption in task_structs when running 
the perf_fuzzer.

On skylake:
[35484.208607] Slab corruption (Not tainted): task_struct start=ffffa181f19b6140, len=3648
[35484.216764] 010: 6a 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  jkkkkkkkkkkkkkkk
[35484.224534] Single bit error detected. Probably bad RAM.
[35484.229932] Run memtest86+ or a similar memory test tool.

On haswell it seems random which bug gets hit first, the task_struct slab
corruption one or the cgroup_attach hang.  This morning it hit the 
latter.

[36074.181453] INFO: rcu_sched detected stalls on CPUs/tasks:
[36074.187387]  4-...: (32 GPs behind) idle=5c9/1/2 softirq=206028/206028 fqs=2626 
[36074.195292]  (detected by 3, t=5254 jiffies, g=254720, c=254719, q=131)
[36074.202396] Task dump for CPU 4:
[36074.205831] swapper/4       R  running task        0     0      1 0x00000008
[36074.213447]  0000000000000246 ffff88011978fe70 0000000000000000 00000000000003b8
[36074.221529]  000020ca48fcb186 000020ca49179475 ffff880119790000 ffffe8ffffd0a400
[36074.229523]  ffffffff81cd40e0 ffffffff81d32580 0000000000000004 ffff88011978fec0
[36074.237523] Call Trace:
[36074.240142]  [<ffffffff815b0b97>] ? cpuidle_enter+0x17/0x20
[36074.246093]  [<ffffffff810bba3a>] ? call_cpuidle+0x2a/0x50
[36074.251975]  [<ffffffff810bbe11>] ? cpu_startup_entry+0x291/0x340
[36074.258553]  [<ffffffff8104feb3>] ? start_secondary+0x133/0x140
[36102.071290] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [perf_fuzzer:2966]
[36102.152977] CPU: 3 PID: 2966 Comm: perf_fuzzer Not tainted 4.8.0-rc7 #195
[36102.160250] Hardware name: LENOVO 10AM000AUS/SHARKBAY, BIOS FBKT72AUS 01/26/2014
[36102.168163] task: ffff8800bbfc6380 task.stack: ffff880113708000
[36102.174558] RIP: 0010:[<ffffffff81100291>]  [<ffffffff81100291>] smp_call_function_single+0xb1/0x110
[36102.184364] RSP: 0018:ffff88011370bdb8  EFLAGS: 00000202
[36102.190026] RAX: 0000000000000000 RBX: 0000000000000004 RCX: 0000000000000830
[36102.197657] RDX: 0000000000000001 RSI: 00000000000008fb RDI: 0000000000000830
[36102.205321] RBP: ffff88011370bdf8 R08: 00000000c48d3dfe R09: 1e3cdd0700000000
[36102.212995] R10: 0000000000000001 R11: 0000000000000000 R12: ffffffff8116ab00
[36102.220647] R13: ffff8800be96e800 R14: ffffffff81170220 R15: 0000000000000000
[36102.228254] FS:  00007fb736b6d700(0000) GS:ffff88011eac0000(0000) knlGS:0000000000000000
[36102.236921] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[36102.243106] CR2: 0000000000835620 CR3: 0000000112503000 CR4: 00000000001407e0
[36102.250733] DR0: 0000000000000000 DR1: 000000ec4dbff2cf DR2: 0000000000000000
[36102.258420] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[36102.266020] Stack:
[36102.268202]  ffff8800bbfc6380 0000000000000246 0000000000000000 ffffffff8116ab00
[36102.276774]  ffff88011370be28 0000000000000003 ffff88011eb208e8 ffff8800be96e800
[36102.285405]  ffff88011370be70 ffffffff811697ee 0000000000000000 ffff8800be96e800
[36102.294030] Call Trace:
[36102.297207]  [<ffffffff8116ab00>] ? perf_cgroup_attach+0x50/0x50
[36102.304170]  [<ffffffff811697ee>] event_function_call+0x13e/0x150
[36102.311235]  [<ffffffff81170220>] ? __perf_cgroup_move+0x60/0x60
[36102.318241]  [<ffffffff81169190>] ? update_group_times+0x40/0x40
[36102.325246]  [<ffffffff81169860>] ? _perf_event_disable+0x60/0x60
[36102.332295]  [<ffffffff811698ab>] _perf_event_enable+0x4b/0x70
[36102.339097]  [<ffffffff811693ba>] perf_event_for_each_child+0x3a/0xa0
[36102.346479]  [<ffffffff81172397>] perf_event_task_enable+0x57/0xa0
[36102.353676]  [<ffffffff8108c3cc>] SyS_prctl+0x35c/0x460
[36102.359855]  [<ffffffff81003016>] ? lockdep_sys_exit_thunk+0x16/0x30
[36102.367159]  [<ffffffff81727c36>] entry_SYSCALL_64_fastpath+0x1e/0xad
[36102.374596] Code: df 4c 89 e2 e8 51 fe ff ff 48 83 c4 30 5b 41 5c 5d c3 48 8d 75 d0 48 89 d1 89 df 4c 89 e2 e8 37 fe ff ff 8b 55 e8 83 e2 01 74 0a <f3> 90 8b 55 e8 83 e2 01 75 f6 48 83 c4 30 5b 41 5c 5d c3 8b 05 
[36130.069119] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [perf_fuzzer:2966]
....

[toc] | [prev] | [standalone]

csiph-web

perf: perf_fuzzer crashing due to slab poison

Contents

#1485173 — perf: perf_fuzzer crashing due to slab poison

#1486563