Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1220097 > unrolled thread

Re: perf: fuzzer PEBS record without PEBS event! crash

Started byStephane Eranian <eranian@googlemail.com>
First post2015-09-07 13:50 +0200
Last post2015-09-08 17:40 +0200
Articles 4 — 2 participants

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: perf: fuzzer PEBS record without PEBS event! crash Stephane Eranian <eranian@googlemail.com> - 2015-09-07 13:50 +0200
    Re: perf: fuzzer PEBS record without PEBS event! crash Vince Weaver <vincent.weaver@maine.edu> - 2015-09-08 17:30 +0200
      Re: perf: fuzzer PEBS record without PEBS event! crash Stephane Eranian <eranian@googlemail.com> - 2015-09-08 17:30 +0200
        Re: perf: fuzzer PEBS record without PEBS event! crash Vince Weaver <vincent.weaver@maine.edu> - 2015-09-08 17:40 +0200

#1220097 — Re: perf: fuzzer PEBS record without PEBS event! crash

FromStephane Eranian <eranian@googlemail.com>
Date2015-09-07 13:50 +0200
SubjectRe: perf: fuzzer PEBS record without PEBS event! crash
Message-ID<q635U-4u5-31@gated-at.bofh.it>
On Mon, Sep 7, 2015 at 2:44 AM, Vince Weaver <vincent.weaver@maine.edu> wrote:
> On Sun, 6 Sep 2015, Stephane Eranian wrote:
>
>> > [ 1800.285152] ------------[ cut here ]------------
>> > [ 1800.290757] WARNING: CPU: 0 PID: 0 at arch/x86/kernel/cpu/perf_event_intel_ds.c:1210 intel_pmu_drain_pebs_nhm+0x305/0x320()
>> > [ 1800.303301] PEBS record without PEBS event! status=2 pebs_enabled=1 active_mask=200000000
>> > [ 1800.311916] Modules linked in: binfmt_misc intel_rapl iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel snd_hda_codec_hdmi hmac drbg ansi_cprng i915 aesni_intel aes_x86_64 snd_hda_codec_realtek snd_hda_codec_generic lrw snd_hda_intel snd_hda_codec iTCO_wdt gf128mul drm_kms_helper snd_hda_core ppdev iTCO_vendor_support evdev glue_helper ablk_helper cryptd parport_pc snd_hwdep snd_pcm snd_timer psmouse tpm_tis tpm drm i2c_algo_bit sg parport snd pcspkr video serio_raw processor soundcore mei_me lpc_ich mfd_core mei i2c_i801 wmi button battery sr_mod sd_mod cdrom ehci_pci ehci_hcd xhci_pci xhci_hcd ahci libahci libata e1000e ptp usbcore crc32c_intel scsi_mod pps_core usb_common fan thermal
>> > [ 1800.390813] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W       4.2.0+ #173
>> > [ 1800.399488] Hardware name: LENOVO 10AM000AUS/SHARKBAY, BIOS FBKT72AUS 01/26/2014
>> > [ 1800.408243]  ffffffff81a0d828 ffff88011ea03aa8 ffffffff8169da0c ffff88011ea03af0
>> > [ 1800.417089]  ffff88011ea03ae0 ffffffff8106e812 0000000000000001 ffff880119000000
>> > [ 1800.426024]  ffff88011ea0c120 ffff88011900ffc0 ffff880119000000 ffff88011ea03b40
>> > [ 1800.434946] Call Trace:
>> > [ 1800.438453]  <IRQ>  [<ffffffff8169da0c>] dump_stack+0x44/0x55
>> > [ 1800.445504]  [<ffffffff8106e812>] warn_slowpath_common+0x82/0xc0
>> > [ 1800.452832]  [<ffffffff8106e89c>] warn_slowpath_fmt+0x4c/0x50
>> > [ 1800.459879]  [<ffffffff8102f7d5>] intel_pmu_drain_pebs_nhm+0x305/0x320
>> > [ 1800.467725]  [<ffffffff810b6b69>] ? __lock_acquire.isra.31+0x349/0xfc0
>> > [ 1800.475613]  [<ffffffff8102ec38>] intel_pmu_drain_pebs_buffer+0x18/0x20
>>
>>
>> What would help here is if you could get the source line number. You can do that
>> with gdb:
>>      gdb vmlinux
>>      (gdb) info line *0xffffffff8102ec38
>
> (gdb) info line *0xffffffff8102ec38
> Line 567 of "arch/x86/kernel/cpu/perf_event_intel_ds.c"
>    starts at address 0xffffffff8102ec38 <intel_pmu_drain_pebs_buffer+24>
>    and ends at 0xffffffff8102ec40 <release_pebs_buffer>.
>
> Line 567 isn't that helpful, it's just the last line of
> intel_pmu_drain_pebs_buffer() but that's possibly because I have done a
> git pull since the time I built the kernel so the source tree doesn't
> match up.
>
Do it again with the matching source tree. Addresses change all the time.

> Vince
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1220928

FromVince Weaver <vincent.weaver@maine.edu>
Date2015-09-08 17:30 +0200
Message-ID<q6t0n-8ch-11@gated-at.bofh.it>
In reply to#1220097
On Mon, 7 Sep 2015, Stephane Eranian wrote:

> On Mon, Sep 7, 2015 at 2:44 AM, Vince Weaver <vincent.weaver@maine.edu> wrote:
> > On Sun, 6 Sep 2015, Stephane Eranian wrote:
> >
> >> > [ 1800.285152] ------------[ cut here ]------------
> >> > [ 1800.290757] WARNING: CPU: 0 PID: 0 at arch/x86/kernel/cpu/perf_event_intel_ds.c:1210 intel_pmu_drain_pebs_nhm+0x305/0x320()
> >> > [ 1800.303301] PEBS record without PEBS event! status=2 pebs_enabled=1 active_mask=200000000
> >> > [ 1800.311916] Modules linked in: binfmt_misc intel_rapl iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel snd_hda_codec_hdmi hmac drbg ansi_cprng i915 aesni_intel aes_x86_64 snd_hda_codec_realtek snd_hda_codec_generic lrw snd_hda_intel snd_hda_codec iTCO_wdt gf128mul drm_kms_helper snd_hda_core ppdev iTCO_vendor_support evdev glue_helper ablk_helper cryptd parport_pc snd_hwdep snd_pcm snd_timer psmouse tpm_tis tpm drm i2c_algo_bit sg parport snd pcspkr video serio_raw processor soundcore mei_me lpc_ich mfd_core mei i2c_i801 wmi button battery sr_mod sd_mod cdrom ehci_pci ehci_hcd xhci_pci xhci_hcd ahci libahci libata e1000e ptp usbcore crc32c_intel scsi_mod pps_core usb_common fan thermal
> >> > [ 1800.390813] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W       4.2.0+ #173
> >> > [ 1800.399488] Hardware name: LENOVO 10AM000AUS/SHARKBAY, BIOS FBKT72AUS 01/26/2014
> >> > [ 1800.408243]  ffffffff81a0d828 ffff88011ea03aa8 ffffffff8169da0c ffff88011ea03af0
> >> > [ 1800.417089]  ffff88011ea03ae0 ffffffff8106e812 0000000000000001 ffff880119000000
> >> > [ 1800.426024]  ffff88011ea0c120 ffff88011900ffc0 ffff880119000000 ffff88011ea03b40
> >> > [ 1800.434946] Call Trace:
> >> > [ 1800.438453]  <IRQ>  [<ffffffff8169da0c>] dump_stack+0x44/0x55
> >> > [ 1800.445504]  [<ffffffff8106e812>] warn_slowpath_common+0x82/0xc0
> >> > [ 1800.452832]  [<ffffffff8106e89c>] warn_slowpath_fmt+0x4c/0x50
> >> > [ 1800.459879]  [<ffffffff8102f7d5>] intel_pmu_drain_pebs_nhm+0x305/0x320
> >> > [ 1800.467725]  [<ffffffff810b6b69>] ? __lock_acquire.isra.31+0x349/0xfc0
> >> > [ 1800.475613]  [<ffffffff8102ec38>] intel_pmu_drain_pebs_buffer+0x18/0x20
> >>
> >>
> >> What would help here is if you could get the source line number. You can do that
> >> with gdb:
> >>      gdb vmlinux
> >>      (gdb) info line *0xffffffff8102ec38
> >
> > (gdb) info line *0xffffffff8102ec38
> > Line 567 of "arch/x86/kernel/cpu/perf_event_intel_ds.c"
> >    starts at address 0xffffffff8102ec38 <intel_pmu_drain_pebs_buffer+24>
> >    and ends at 0xffffffff8102ec40 <release_pebs_buffer>.
> >
> > Line 567 isn't that helpful, it's just the last line of
> > intel_pmu_drain_pebs_buffer() but that's possibly because I have done a
> > git pull since the time I built the kernel so the source tree doesn't
> > match up.
> >
> Do it again with the matching source tree. Addresses change all the time.

I re-compiled with current git, and the source matches the kernel exactly 
now.  I managed to trigger the warning again (though it didn't trigger a 
crash yet).  Still got line 567

[ 6890.067085] WARNING: CPU: 7 PID: 7 at arch/x86/kernel/cpu/perf_event_intel_ds.c:1210 intel_pmu_drain_pebs_nhm+0x305/0x320()
[ 6890.078963] PEBS record without PEBS event! status=0 pebs_enabled=1 active_mask=200000001
[ 6890.199394] Call Trace:
[ 6890.202024]  [<ffffffff8169ffe0>] dump_stack+0x44/0x55
[ 6890.207516]  [<ffffffff8106e8b2>] warn_slowpath_common+0x82/0xc0
[ 6890.213990]  [<ffffffff8106e93c>] warn_slowpath_fmt+0x4c/0x50
[ 6890.220128]  [<ffffffff8102f7d5>] intel_pmu_drain_pebs_nhm+0x305/0x320
[ 6890.227092]  [<ffffffff8102ec38>] intel_pmu_drain_pebs_buffer+0x18/0x20

(gdb) info line *0xffffffff8102ec38
Line 567 of "arch/x86/kernel/cpu/perf_event_intel_ds.c"
   starts at address 0xffffffff8102ec38 <intel_pmu_drain_pebs_buffer+24>
   and ends at 0xffffffff8102ec40 <release_pebs_buffer>.


which is

562:static inline void intel_pmu_drain_pebs_buffer(void)
563:{
564:        struct pt_regs regs;
565:
566:        x86_pmu.drain_pebs(&regs);
567: }

which seems odd, though I guess because it is inlined?

Vince
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1220931

FromStephane Eranian <eranian@googlemail.com>
Date2015-09-08 17:30 +0200
Message-ID<q6t0n-8ch-27@gated-at.bofh.it>
In reply to#1220928
On Tue, Sep 8, 2015 at 5:32 PM, Vince Weaver <vincent.weaver@maine.edu> wrote:
> On Mon, 7 Sep 2015, Stephane Eranian wrote:
>
>> On Mon, Sep 7, 2015 at 2:44 AM, Vince Weaver <vincent.weaver@maine.edu> wrote:
>> > On Sun, 6 Sep 2015, Stephane Eranian wrote:
>> >
>> >> > [ 1800.285152] ------------[ cut here ]------------
>> >> > [ 1800.290757] WARNING: CPU: 0 PID: 0 at arch/x86/kernel/cpu/perf_event_intel_ds.c:1210 intel_pmu_drain_pebs_nhm+0x305/0x320()
>> >> > [ 1800.303301] PEBS record without PEBS event! status=2 pebs_enabled=1 active_mask=200000000
>> >> > [ 1800.311916] Modules linked in: binfmt_misc intel_rapl iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel snd_hda_codec_hdmi hmac drbg ansi_cprng i915 aesni_intel aes_x86_64 snd_hda_codec_realtek snd_hda_codec_generic lrw snd_hda_intel snd_hda_codec iTCO_wdt gf128mul drm_kms_helper snd_hda_core ppdev iTCO_vendor_support evdev glue_helper ablk_helper cryptd parport_pc snd_hwdep snd_pcm snd_timer psmouse tpm_tis tpm drm i2c_algo_bit sg parport snd pcspkr video serio_raw processor soundcore mei_me lpc_ich mfd_core mei i2c_i801 wmi button battery sr_mod sd_mod cdrom ehci_pci ehci_hcd xhci_pci xhci_hcd ahci libahci libata e1000e ptp usbcore crc32c_intel scsi_mod pps_core usb_common fan thermal
>> >> > [ 1800.390813] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W       4.2.0+ #173
>> >> > [ 1800.399488] Hardware name: LENOVO 10AM000AUS/SHARKBAY, BIOS FBKT72AUS 01/26/2014
>> >> > [ 1800.408243]  ffffffff81a0d828 ffff88011ea03aa8 ffffffff8169da0c ffff88011ea03af0
>> >> > [ 1800.417089]  ffff88011ea03ae0 ffffffff8106e812 0000000000000001 ffff880119000000
>> >> > [ 1800.426024]  ffff88011ea0c120 ffff88011900ffc0 ffff880119000000 ffff88011ea03b40
>> >> > [ 1800.434946] Call Trace:
>> >> > [ 1800.438453]  <IRQ>  [<ffffffff8169da0c>] dump_stack+0x44/0x55
>> >> > [ 1800.445504]  [<ffffffff8106e812>] warn_slowpath_common+0x82/0xc0
>> >> > [ 1800.452832]  [<ffffffff8106e89c>] warn_slowpath_fmt+0x4c/0x50
>> >> > [ 1800.459879]  [<ffffffff8102f7d5>] intel_pmu_drain_pebs_nhm+0x305/0x320
>> >> > [ 1800.467725]  [<ffffffff810b6b69>] ? __lock_acquire.isra.31+0x349/0xfc0
>> >> > [ 1800.475613]  [<ffffffff8102ec38>] intel_pmu_drain_pebs_buffer+0x18/0x20
>> >>
>> >>
>> >> What would help here is if you could get the source line number. You can do that
>> >> with gdb:
>> >>      gdb vmlinux
>> >>      (gdb) info line *0xffffffff8102ec38
>> >
>> > (gdb) info line *0xffffffff8102ec38
>> > Line 567 of "arch/x86/kernel/cpu/perf_event_intel_ds.c"
>> >    starts at address 0xffffffff8102ec38 <intel_pmu_drain_pebs_buffer+24>
>> >    and ends at 0xffffffff8102ec40 <release_pebs_buffer>.
>> >
>> > Line 567 isn't that helpful, it's just the last line of
>> > intel_pmu_drain_pebs_buffer() but that's possibly because I have done a
>> > git pull since the time I built the kernel so the source tree doesn't
>> > match up.
>> >
>> Do it again with the matching source tree. Addresses change all the time.
>
> I re-compiled with current git, and the source matches the kernel exactly
> now.  I managed to trigger the warning again (though it didn't trigger a
> crash yet).  Still got line 567
>
> [ 6890.067085] WARNING: CPU: 7 PID: 7 at arch/x86/kernel/cpu/perf_event_intel_ds.c:1210 intel_pmu_drain_pebs_nhm+0x305/0x320()
> [ 6890.078963] PEBS record without PEBS event! status=0 pebs_enabled=1 active_mask=200000001
> [ 6890.199394] Call Trace:
> [ 6890.202024]  [<ffffffff8169ffe0>] dump_stack+0x44/0x55
> [ 6890.207516]  [<ffffffff8106e8b2>] warn_slowpath_common+0x82/0xc0
> [ 6890.213990]  [<ffffffff8106e93c>] warn_slowpath_fmt+0x4c/0x50
> [ 6890.220128]  [<ffffffff8102f7d5>] intel_pmu_drain_pebs_nhm+0x305/0x320
> [ 6890.227092]  [<ffffffff8102ec38>] intel_pmu_drain_pebs_buffer+0x18/0x20
>
> (gdb) info line *0xffffffff8102ec38
> Line 567 of "arch/x86/kernel/cpu/perf_event_intel_ds.c"
>    starts at address 0xffffffff8102ec38 <intel_pmu_drain_pebs_buffer+24>
>    and ends at 0xffffffff8102ec40 <release_pebs_buffer>.
>
>
> which is
>
> 562:static inline void intel_pmu_drain_pebs_buffer(void)
> 563:{
> 564:        struct pt_regs regs;
> 565:
> 566:        x86_pmu.drain_pebs(&regs);
> 567: }
>
> which seems odd, though I guess because it is inlined?
>
Has to do with inlining. Make the function not inlined.

static void intel_pmu_drain_pebs_buffer(void) __attribute__((noinline));
static inline void intel_pmu_drain_pebs_buffer(void)
{
        struct pt_regs regs;

        x86_pmu.drain_pebs(&regs);
}
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1220936

FromVince Weaver <vincent.weaver@maine.edu>
Date2015-09-08 17:40 +0200
Message-ID<q6ta2-8oh-19@gated-at.bofh.it>
In reply to#1220931
On Tue, 8 Sep 2015, Stephane Eranian wrote:

> Has to do with inlining. Make the function not inlined.

OK I'll do that.  

This current kernel isn't crashing, but I'm getting this a lot

[ 7877.413225] PEBS record without PEBS event! status=1 pebs_enabled=8 active_mask=0

vince@haswell:~$ dmesg | grep PEBS | wc -l
36

let me recompile without inlining and see what I get.

Vince

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web