Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1477384 > unrolled thread

[PATCH v2 0/5] perf, bts: Fallout from the fuzzer for perf/urgent

Back to article view | Back to linux.kernel

Contents

  [PATCH v2 0/5] perf, bts: Fallout from the fuzzer for perf/urgent Alexander Shishkin <alexander.shishkin@linux.intel.com> - 2016-09-06 15:30 +0200
    [PATCH v2 4/5] perf/x86/intel/bts: Fix BTS PMI detection Alexander Shishkin <alexander.shishkin@linux.intel.com> - 2016-09-06 15:40 +0200
    [PATCH v2 5/5] perf/x86/intel/bts: Kill a silly warning Alexander Shishkin <alexander.shishkin@linux.intel.com> - 2016-09-06 15:50 +0200
    Re: [PATCH v2 0/5] perf, bts: Fallout from the fuzzer for perf/urgent Ingo Molnar <mingo@kernel.org> - 2016-09-06 19:20 +0200
      Re: [PATCH v2 0/5] perf, bts: Fallout from the fuzzer for  perf/urgent Vince Weaver <vince@deater.net> - 2016-09-07 02:20 +0200
      Re: [PATCH v2 0/5] perf, bts: Fallout from the fuzzer for perf/urgent Alexander Shishkin <alexander.shishkin@linux.intel.com> - 2016-09-07 17:30 +0200
        Re: [PATCH v2 0/5] perf, bts: Fallout from the fuzzer for  perf/urgent Vince Weaver <vince@deater.net> - 2016-09-07 17:40 +0200
          Re: [PATCH v2 0/5] perf, bts: Fallout from the fuzzer for perf/urgent Peter Zijlstra <peterz@infradead.org> - 2016-09-07 18:40 +0200
          Re: [PATCH v2 0/5] perf, bts: Fallout from the fuzzer for perf/urgent Alexander Shishkin <alexander.shishkin@linux.intel.com> - 2016-09-07 20:40 +0200
            Re: [PATCH v2 0/5] perf, bts: Fallout from the fuzzer for  perf/urgent Vince Weaver <vincent.weaver@maine.edu> - 2016-09-08 05:40 +0200
              Re: [PATCH v2 0/5] perf, bts: Fallout from the fuzzer for perf/urgent Alexander Shishkin <alexander.shishkin@linux.intel.com> - 2016-09-08 11:00 +0200
                Re: [PATCH v2 0/5] perf, bts: Fallout from the fuzzer for  perf/urgent Vince Weaver <vincent.weaver@maine.edu> - 2016-09-08 15:00 +0200
        Re: [PATCH v2 0/5] perf, bts: Fallout from the fuzzer for perf/urgent Ingo Molnar <mingo@kernel.org> - 2016-09-08 08:30 +0200
        Re: [PATCH v2 0/5] perf, bts: Fallout from the fuzzer for perf/urgent Ingo Molnar <mingo@kernel.org> - 2016-09-08 08:30 +0200
          Re: [PATCH v2 0/5] perf, bts: Fallout from the fuzzer for perf/urgent Alexander Shishkin <alexander.shishkin@linux.intel.com> - 2016-09-08 10:50 +0200

#1477384 — [PATCH v2 0/5] perf, bts: Fallout from the fuzzer for perf/urgent

Hi,

There were more bugs since the previous version, plus the BTS barriers
got fixed. With these patches, my testcase keeps running and no
spurious NMI warnings pop up any more.

Original story:

Recently Vince has reported warnings and panics coming from the
general direction of AUX tracing. I found two bugs which manifest
similarly, one in intel_bts driver and one in AUX unmapping path.

Both are triggered by racing SET_OUTPUT against mmap_close while
running AUX tracing. I have a test case that set fire to the kernel
within a few seconds by doing this, which I can share if anyone
cares.

These are all good candidates for 4.7-stable and the BTS ones can be
theoretically backported further.

Alexander Shishkin (5):
  perf: Fix a race between mmap_close and set_output of AUX events
  perf: Fix aux_mmap_count vs aux_refcount order
  perf/x86/intel/bts: Fix confused ordering of PMU callbacks
  perf/x86/intel/bts: Fix BTS PMI detection
  perf/x86/intel/bts: Kill a silly warning

 arch/x86/events/intel/bts.c | 123 +++++++++++++++++++++++++++++++++-----------
 kernel/events/core.c        |  31 ++++++++---
 kernel/events/ring_buffer.c |  16 +++---
 3 files changed, 129 insertions(+), 41 deletions(-)

-- 
2.9.3

[toc] | [next] | [standalone]

#1477393 — [PATCH v2 4/5] perf/x86/intel/bts: Fix BTS PMI detection

Since BTS doesn't have a dedicated PMI status bit, the driver needs to
take extra care to check for the condition that triggers it to avoid
spurious NMI warnings.

Regardless of the local BTS context state, the only way of knowing that
the NMI is ours is to compare the write pointer against the interrupt
threshold.

Signed-off-by: Alexander Shishkin <alexander.shishkin@linux.intel.com>
---
 arch/x86/events/intel/bts.c | 19 +++++++++++++++----
 1 file changed, 15 insertions(+), 4 deletions(-)

diff --git a/arch/x86/events/intel/bts.c b/arch/x86/events/intel/bts.c
index 61e1d713b1..9233edf993 100644
--- a/arch/x86/events/intel/bts.c
+++ b/arch/x86/events/intel/bts.c
@@ -446,26 +446,37 @@ bts_buffer_reset(struct bts_buffer *buf, struct perf_output_handle *handle)
 
 int intel_bts_interrupt(void)
 {
+	struct debug_store *ds = this_cpu_ptr(&cpu_hw_events)->ds;
 	struct bts_ctx *bts = this_cpu_ptr(&bts_ctx);
 	struct perf_event *event = bts->handle.event;
 	struct bts_buffer *buf;
 	s64 old_head;
-	int err = -ENOSPC;
+	int err = -ENOSPC, handled = 0;
+
+	/*
+	 * The only surefire way of knowing if this NMI is ours is by checking
+	 * the write ptr against the PMI threshold.
+	 */
+	if (ds->bts_index >= ds->bts_interrupt_threshold)
+		handled = 1;
 
 	/*
 	 * this is wrapped in intel_bts_enable_local/intel_bts_disable_local,
 	 * so we can only be INACTIVE or STOPPED
 	 */
 	if (READ_ONCE(bts->state) == BTS_STATE_STOPPED)
-		return 0;
+		return handled;
 
 	buf = perf_get_aux(&bts->handle);
+	if (!buf)
+		return handled;
+
 	/*
 	 * Skip snapshot counters: they don't use the interrupt, but
 	 * there's no other way of telling, because the pointer will
 	 * keep moving
 	 */
-	if (!buf || buf->snapshot)
+	if (buf->snapshot)
 		return 0;
 
 	old_head = local_read(&buf->head);
@@ -473,7 +484,7 @@ int intel_bts_interrupt(void)
 
 	/* no new data */
 	if (old_head == local_read(&buf->head))
-		return 0;
+		return handled;
 
 	perf_aux_output_end(&bts->handle, local_xchg(&buf->data_size, 0),
 			    !!local_xchg(&buf->lost, 0));
-- 
2.9.3

[toc] | [prev] | [next] | [standalone]

#1477410 — [PATCH v2 5/5] perf/x86/intel/bts: Kill a silly warning

At the moment, intel_bts will WARN() out if there is more than one
event writing to the same ring buffer, via SET_OUTPUT, and will only
send data from one event to a buffer.

There is no reason to have this warning in, so kill it.

Signed-off-by: Alexander Shishkin <alexander.shishkin@linux.intel.com>
---
 arch/x86/events/intel/bts.c | 2 --
 1 file changed, 2 deletions(-)

diff --git a/arch/x86/events/intel/bts.c b/arch/x86/events/intel/bts.c
index 9233edf993..bdcd651099 100644
--- a/arch/x86/events/intel/bts.c
+++ b/arch/x86/events/intel/bts.c
@@ -378,8 +378,6 @@ bts_buffer_reset(struct bts_buffer *buf, struct perf_output_handle *handle)
 		return 0;
 
 	head = handle->head & ((buf->nr_pages << PAGE_SHIFT) - 1);
-	if (WARN_ON_ONCE(head != local_read(&buf->head)))
-		return -EINVAL;
 
 	phys = &buf->buf[buf->cur_buf];
 	space = phys->offset + phys->displacement + phys->size - head;
-- 
2.9.3

[toc] | [prev] | [next] | [standalone]

#1477683

* Alexander Shishkin <alexander.shishkin@linux.intel.com> wrote:

> Hi,
> 
> There were more bugs since the previous version, plus the BTS barriers got 
> fixed. With these patches, my testcase keeps running and no spurious NMI 
> warnings pop up any more.

Could you please also run the fuzzer that Vince uses, does it now pass on hardware 
you have access to?

I'd like to make "passes the fuzzer" a standard requirement before new changes are 
accepted to perf core.

Thanks,

	Ingo

[toc] | [prev] | [next] | [standalone]

#1477893 — Re: [PATCH v2 0/5] perf, bts: Fallout from the fuzzer for perf/urgent

On Tue, 6 Sep 2016, Ingo Molnar wrote:

> > There were more bugs since the previous version, plus the BTS barriers got 
> > fixed. With these patches, my testcase keeps running and no spurious NMI 
> > warnings pop up any more.
> 
> Could you please also run the fuzzer that Vince uses, does it now pass 
> on hardware you have access to?

I ran 4.8-rc5 with these patches applied on a Skylake machine that shows 
the various bts problems pretty quickly normally.

It ran a while and didn't hit any of them, but it did fall over with RCU 
errors that are (probably) unrelated.  

I'd have a better report but I'm having a lot of trouble getting kernel 
oopses to appear on the serial console with this new install on the 
skylake machine.  It may not be totally systemd's fault but I am going to 
blame it anyway.

Vince

[toc] | [prev] | [next] | [standalone]

#1478401

Ingo Molnar <mingo@kernel.org> writes:

> * Alexander Shishkin <alexander.shishkin@linux.intel.com> wrote:
>
>> Hi,
>> 
>> There were more bugs since the previous version, plus the BTS barriers got 
>> fixed. With these patches, my testcase keeps running and no spurious NMI 
>> warnings pop up any more.
>
> Could you please also run the fuzzer that Vince uses, does it now pass on hardware 
> you have access to?

Sure. And yes, I did catch a warning, which calls for one more patch
(below). Also one unrelated thing in PEBS that Peter fixed.

> I'd like to make "passes the fuzzer" a standard requirement before new changes are 
> accepted to perf core.

Let's make it so.

For the sake of consistency, this one needs to go before 3/5. I'll
re-send the whole series, though, if need be. I've got 2 perf_fuzzers
running on this meanwhile.

From c170c607bfdc3804578033faa43f342a5d95eb6c Mon Sep 17 00:00:00 2001
From: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Date: Wed, 7 Sep 2016 18:05:08 +0300
Subject: [PATCH] perf/x86/intel: Don't disable "intel_bts" around "intel"
 event batching

At the moment, intel_bts events get disabled from intel PMU's disable
callback, which includes event scheduling transactions of said PMU,
which have nothing to do with intel_bts events.

We do want to keep intel_bts events off inside the PMI handler to
avoid filling up their buffer too soon.

This patch moves intel_bts enabling/disabling directly to the PMI
handler.

Signed-off-by: Alexander Shishkin <alexander.shishkin@linux.intel.com>
---
 arch/x86/events/intel/core.c | 15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)

diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index 88792f846d..e2d71513c9 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -1730,9 +1730,11 @@ static __initconst const u64 knl_hw_cache_extra_regs
  * disabled state if called consecutively.
  *
  * During consecutive calls, the same disable value will be written to related
- * registers, so the PMU state remains unchanged. hw.state in
- * intel_bts_disable_local will remain PERF_HES_STOPPED too in consecutive
- * calls.
+ * registers, so the PMU state remains unchanged.
+ *
+ * intel_bts events don't coexist with intel pmu's BTS events because of
+ * x86_add_exclusive(x86_lbr_exclusive_lbr); there's no need to keep them
+ * disabled around intel pmu's event batching etc, only inside the PMI handler.
  */
 static void __intel_pmu_disable_all(void)
 {
@@ -1742,8 +1744,6 @@ static void __intel_pmu_disable_all(void)
 
 	if (test_bit(INTEL_PMC_IDX_FIXED_BTS, cpuc->active_mask))
 		intel_pmu_disable_bts();
-	else
-		intel_bts_disable_local();
 
 	intel_pmu_pebs_disable_all();
 }
@@ -1771,8 +1771,7 @@ static void __intel_pmu_enable_all(int added, bool pmi)
 			return;
 
 		intel_pmu_enable_bts(event->hw.config);
-	} else
-		intel_bts_enable_local();
+	}
 }
 
 static void intel_pmu_enable_all(int added)
@@ -2076,6 +2075,7 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
 	 */
 	if (!x86_pmu.late_ack)
 		apic_write(APIC_LVTPC, APIC_DM_NMI);
+	intel_bts_disable_local();
 	__intel_pmu_disable_all();
 	handled = intel_pmu_drain_bts_buffer();
 	handled += intel_bts_interrupt();
@@ -2175,6 +2175,7 @@ done:
 	/* Only restore PMU state when it's active. See x86_pmu_disable(). */
 	if (cpuc->enabled)
 		__intel_pmu_enable_all(0, true);
+	intel_bts_enable_local();
 
 	/*
 	 * Only unmask the NMI after the overflow counters
-- 
2.9.3

[toc] | [prev] | [next] | [standalone]

#1478426 — Re: [PATCH v2 0/5] perf, bts: Fallout from the fuzzer for perf/urgent

On Wed, 7 Sep 2016, Alexander Shishkin wrote:

> Sure. And yes, I did catch a warning, which calls for one more patch
> (below). Also one unrelated thing in PEBS that Peter fixed.

Does that fix this which I just got on my skylake machine (4.8-rc5 with 
your other 5 patches applied)

[ 5351.822559] WARNING: CPU: 3 PID: 19191 at arch/x86/events/intel/bts.c:344 event_function+0xa1/0x160
[ 5351.823895] CPU: 3 PID: 19191 Comm: perf_fuzzer Not tainted 4.8.0-rc5+ #3
[ 5351.823896] Hardware name: LENOVO 10FY0017US/SKYBAY, BIOS FWKT53A   06/06/2016
[ 5351.823897]  0000000000000086 000000001a69ecd3 ffffffff8cf1f9f5 0000000000000000
[ 5351.823898]  0000000000000000 ffffffff8cc7b624 ffffcaa8ffcc26f0 ffff9c676e1aba00
[ 5351.823900]  ffff9c6771222800 0000000000000000 ffff9c676e1aba00 ffff9c676f6f7e40
[ 5351.823901] Call Trace:
[ 5351.823905]  [<ffffffff8cf1f9f5>] ? dump_stack+0x5c/0x77
[ 5351.823907]  [<ffffffff8cc7b624>] ? __warn+0xc4/0xe0
[ 5351.823909]  [<ffffffff8cd624d1>] ? event_function+0xa1/0x160
[ 5351.823910]  [<ffffffff8cd69db0>] ? ctx_resched+0x50/0x50
[ 5351.823911]  [<ffffffff8cd63a3a>] ? remote_function+0x3a/0x40
[ 5351.823913]  [<ffffffff8ccfaecd>] ? generic_exec_single+0x9d/0x100
[ 5351.823914]  [<ffffffff8ccfaea1>] ? generic_exec_single+0x71/0x100
[ 5351.823916]  [<ffffffff8cd63a00>] ? perf_cgroup_attach+0x70/0x70
[ 5351.823917]  [<ffffffff8ccfaffd>] ? smp_call_function_single+0xcd/0x130
[ 5351.823918]  [<ffffffff8ccfaffd>] ? smp_call_function_single+0xcd/0x130
[ 5351.823919]  [<ffffffff8cd628b9>] ? task_function_call+0x49/0x70
[ 5351.823920]  [<ffffffff8cd62430>] ? cpu_clock_event_read+0x10/0x10
[ 5351.823921]  [<ffffffff8cd67438>] ? event_function_call+0x98/0x100
[ 5351.823922]  [<ffffffff8cd69db0>] ? ctx_resched+0x50/0x50
[ 5351.823923]  [<ffffffff8cd67530>] ? perf_event_disable+0x30/0x30
[ 5351.823924]  [<ffffffff8cd62642>] ? perf_event_for_each_child+0x32/0x90
[ 5351.823925]  [<ffffffff8cd6aa51>] ? perf_event_task_enable+0x61/0xb0
[ 5351.823927]  [<ffffffff8cc9066e>] ? SyS_prctl+0x2ae/0x470
[ 5351.823929]  [<ffffffff8cc03b6f>] ? do_syscall_64+0x5f/0x160
[ 5351.823930]  [<ffffffff8d1e7b65>] ? entry_SYSCALL64_slow_path+0x25/0x25
[ 5351.823931] ---[ end trace bc7b0b7d0c024d60 ]---

[toc] | [prev] | [next] | [standalone]

#1478485

On Wed, Sep 07, 2016 at 11:36:48AM -0400, Vince Weaver wrote:
> On Wed, 7 Sep 2016, Alexander Shishkin wrote:
> 
> >  Also one unrelated thing in PEBS that Peter fixed.

---
Subject: perf,x86: Fix PEBSv3 record drain
From: Peter Zijlstra <peterz@infradead.org>
Date: Wed Sep 7 14:42:55 CEST 2016

Alexander hit the WARN_ON_ONCE(!event) on his Skylake while running
the fuzzer.

This means the PEBSv3 record included a status bit for an inactive
event, something that _should_ not happen.

Move the code that filters the status bits against our known PEBS
events up a spot to guarantee we only deal with events we know about.

Further add "continue" statements to the WARN_ON_ONCE()s such that
we'll not die nor generate silly events in case we ever do hit them
again.

Cc: Vince Weaver <vince@deater.net>
Cc: Stephane Eranian <eranian@google.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Kan Liang <kan.liang@intel.com>
Cc: stable@vger.kernel.org
Fixes: a3d86542de88 ("perf/x86/intel/pebs: Add PEBSv3 decoding")
Reported-by: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Tested-by: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
 arch/x86/events/intel/ds.c |   19 +++++++++++--------
 1 file changed, 11 insertions(+), 8 deletions(-)

--- a/arch/x86/events/intel/ds.c
+++ b/arch/x86/events/intel/ds.c
@@ -1312,18 +1312,18 @@ static void intel_pmu_drain_pebs_nhm(str
 		struct pebs_record_nhm *p = at;
 		u64 pebs_status;
 
-		/* PEBS v3 has accurate status bits */
+		pebs_status = p->status & cpuc->pebs_enabled;
+		pebs_status &= (1ULL << x86_pmu.max_pebs_events) - 1;
+
+		/* PEBS v3 has more accurate status bits */
 		if (x86_pmu.intel_cap.pebs_format >= 3) {
-			for_each_set_bit(bit, (unsigned long *)&p->status,
-					 MAX_PEBS_EVENTS)
+			for_each_set_bit(bit, (unsigned long *)&pebs_status,
+					 x86_pmu.max_pebs_events)
 				counts[bit]++;
 
 			continue;
 		}
 
-		pebs_status = p->status & cpuc->pebs_enabled;
-		pebs_status &= (1ULL << x86_pmu.max_pebs_events) - 1;
-
 		/*
 		 * On some CPUs the PEBS status can be zero when PEBS is
 		 * racing with clearing of GLOBAL_STATUS.
@@ -1371,8 +1371,11 @@ static void intel_pmu_drain_pebs_nhm(str
 			continue;
 
 		event = cpuc->events[bit];
-		WARN_ON_ONCE(!event);
-		WARN_ON_ONCE(!event->attr.precise_ip);
+		if (WARN_ON_ONCE(!event))
+			continue;
+
+		if (WARN_ON_ONCE(!event->attr.precise_ip))
+			continue;
 
 		/* log dropped samples number */
 		if (error[bit])

[toc] | [prev] | [next] | [standalone]

#1478551

Vince Weaver <vince@deater.net> writes:

> On Wed, 7 Sep 2016, Alexander Shishkin wrote:
>
>> Sure. And yes, I did catch a warning, which calls for one more patch
>> (below). Also one unrelated thing in PEBS that Peter fixed.
>
> Does that fix this which I just got on my skylake machine (4.8-rc5 with 
> your other 5 patches applied)
>
> [ 5351.822559] WARNING: CPU: 3 PID: 19191 at arch/x86/events/intel/bts.c:344 event_function+0xa1/0x160

Yes, it fixes a problem that triggered this warning. Can't tell from the
absence of backtrace what's going on here, though.

Regards,
--
Alex

[toc] | [prev] | [next] | [standalone]

#1478762 — Re: [PATCH v2 0/5] perf, bts: Fallout from the fuzzer for perf/urgent

On Wed, 7 Sep 2016, Alexander Shishkin wrote:

> Vince Weaver <vince@deater.net> writes:
> 
> > On Wed, 7 Sep 2016, Alexander Shishkin wrote:
> >
> >> Sure. And yes, I did catch a warning, which calls for one more patch
> >> (below). Also one unrelated thing in PEBS that Peter fixed.
> >
> > Does that fix this which I just got on my skylake machine (4.8-rc5 with 
> > your other 5 patches applied)
> >
> > [ 5351.822559] WARNING: CPU: 3 PID: 19191 at arch/x86/events/intel/bts.c:344 event_function+0xa1/0x160
> 
> Yes, it fixes a problem that triggered this warning. Can't tell from the
> absence of backtrace what's going on here, though.

On the skylake machine with the original 5 patches I got this after 
continuing to fuzz.  Sorry about the lack of frame pointer, next
compile will have it enabled.

If it matters, prior to this I hit the unrelated
[25510.278199] WARNING: CPU: 1 PID: 25405 at kernel/events/core.c:3554 perf_event_read+0x18f/0x1a0


[28682.174684] WARNING: CPU: 7 PID: 31992 at kernel/events/core.c:4961 perf_mmap_close+0x2e1/0x2f0
[28682.280579] CPU: 7 PID: 31992 Comm: perf_fuzzer Tainted: G        W       4.8.0-rc5+ #3
[28682.288739] Hardware name: LENOVO 10FY0017US/SKYBAY, BIOS FWKT53A   06/06/2016
[28682.296096]  0000000000000286 000000008e07b373 ffffffff8cf1f9f5 0000000000000000
[28682.303738]  0000000000000000 ffffffff8cc7b624 ffff9c676e780e00 ffff9c676e029280
[28682.311362]  ffff9c6772ab3398 ffff9c676e029000 ffff9c676e780bb0 ffff9c676f7c0280
[28682.319036] Call Trace:
[28682.321529]  [<ffffffff8cf1f9f5>] ? dump_stack+0x5c/0x77
[28682.326939]  [<ffffffff8cc7b624>] ? __warn+0xc4/0xe0
[28682.332018]  [<ffffffff8cd6dc01>] ? perf_mmap_close+0x2e1/0x2f0
[28682.338083]  [<ffffffff8cd664b0>] ? perf_iterate_ctx+0x150/0x150
[28682.344218]  [<ffffffff8cdb5e7d>] ? remove_vma+0x2d/0x70
[28682.349662]  [<ffffffff8cdb84d6>] ? do_munmap+0x246/0x400
[28682.355197]  [<ffffffff8cdb8c1b>] ? SyS_munmap+0x4b/0x70
[28682.360612]  [<ffffffff8cc03b6f>] ? do_syscall_64+0x5f/0x160
[28682.366404]  [<ffffffff8d1e7b65>] ? entry_SYSCALL64_slow_path+0x25/0x25
[28682.373149] ---[ end trace bc7b0b7d0c024d62 ]---
[28682.377975] ------------[ cut here ]------------
[28682.382681] WARNING: CPU: 4 PID: 0 at kernel/events/ring_buffer.c:543 __rb_free_aux+0x110/0x120
[28682.489412] CPU: 4 PID: 0 Comm: swapper/4 Tainted: G        W       4.8.0-rc5+ #3
[28682.498106] Hardware name: LENOVO 10FY0017US/SKYBAY, BIOS FWKT53A   06/06/2016
[28682.506518]  0000000000000086 6d2f07ae6326ed5e ffffffff8cf1f9f5 0000000000000000
[28682.515223]  0000000000000000 ffffffff8cc7b624 ffff9c676e780e00 ffff9c677dd0b740
[28682.523996]  0000000000000000 ffff9c676e029000 ffff9c676e780e00 00001a16168dab54
[28682.532867] Call Trace:
[28682.536458]  <IRQ>  [<ffffffff8cf1f9f5>] ? dump_stack+0x5c/0x77
[28682.543615]  [<ffffffff8cc7b624>] ? __warn+0xc4/0xe0
[28682.549741]  [<ffffffff8cd700e0>] ? __rb_free_aux+0x110/0x120
[28682.556829]  [<ffffffff8cd70f3a>] ? perf_aux_output_end+0xba/0x100
[28682.564312]  [<ffffffff8cc0c9e1>] ? bts_event_stop+0xc1/0x130
[28682.571238]  [<ffffffff8cd63a00>] ? perf_cgroup_attach+0x70/0x70
[28682.578402]  [<ffffffff8cd621f0>] ? __perf_event_stop+0x40/0x60
[28682.585484]  [<ffffffff8cd63a3a>] ? remote_function+0x3a/0x40
[28682.592644]  [<ffffffff8ccfad46>] ? flush_smp_call_function_queue+0x76/0x160
[28682.600867]  [<ffffffff8cc4e6e9>] ? smp_trace_call_function_single_interrupt+0x29/0xc0
[28682.609914]  [<ffffffff8d1e9ce2>] ? trace_call_function_single_interrupt+0x82/0x90
[28682.618608]  <EOI>  [<ffffffff8d0a9d13>] ? cpuidle_enter_state+0x113/0x260
[28682.626570]  [<ffffffff8ccbd7dc>] ? cpu_startup_entry+0x2cc/0x370
[28682.633675]  [<ffffffff8cc4f07d>] ? start_secondary+0x14d/0x190
[28682.640749] ---[ end trace bc7b0b7d0c024d63 ]---

[toc] | [prev] | [next] | [standalone]

#1478942

Vince Weaver <vincent.weaver@maine.edu> writes:

> On the skylake machine with the original 5 patches I got this after 
> continuing to fuzz.  Sorry about the lack of frame pointer, next
> compile will have it enabled.
>
> If it matters, prior to this I hit the unrelated
> [25510.278199] WARNING: CPU: 1 PID: 25405 at kernel/events/core.c:3554 perf_event_read+0x18f/0x1a0
>
>
> [28682.174684] WARNING: CPU: 7 PID: 31992 at kernel/events/core.c:4961 perf_mmap_close+0x2e1/0x2f0

It just keeps on giving, doesn't it. I've got the same thing here during
the night, looking at it again. Did you capture the seed by any chance?

Thanks,
--
Alex

[toc] | [prev] | [next] | [standalone]

#1479192 — Re: [PATCH v2 0/5] perf, bts: Fallout from the fuzzer for perf/urgent

On Thu, 8 Sep 2016, Alexander Shishkin wrote:

> Vince Weaver <vincent.weaver@maine.edu> writes:
> 
> > On the skylake machine with the original 5 patches I got this after 
> > continuing to fuzz.  Sorry about the lack of frame pointer, next
x> > compile will have it enabled.
> >
> > If it matters, prior to this I hit the unrelated
> > [25510.278199] WARNING: CPU: 1 PID: 25405 at kernel/events/core.c:3554 perf_event_read+0x18f/0x1a0
> >
> >
> > [28682.174684] WARNING: CPU: 7 PID: 31992 at kernel/events/core.c:4961 perf_mmap_close+0x2e1/0x2f0
> 
> It just keeps on giving, doesn't it. I've got the same thing here during
> the night, looking at it again. Did you capture the seed by any chance?

It was triggered by

	Seeding random number generator with 1473284719
        /proc/sys/kernel/perf_event_max_sample_rate currently: 1000/s
        /proc/sys/kernel/perf_event_paranoid currently: 0
        To reproduce, try: ./perf_fuzzer -s 30000 -r 1473284719

but in my experience these things don't always reproduce deterministically.  

I can try reproducing it but I can't right now because overnight the 
machine locked hard (even sysrq isn't helping) with just the unhelpful 
message of

[70953.494637] ------------[ cut here ]------------

not sure if I have the serial console configured wrong or if it just locks 
up before it can send anything.

Anyway I'll head to the lab after my class and reboot things.

Vince

[toc] | [prev] | [next] | [standalone]

#1478785

* Ingo Molnar <mingo@kernel.org> wrote:

> Yeah, please re-send it - and please also Vince's Reported-by tag to all commits 
                                           ^--- add
> that would explain failures that Vince reported.

Thanks,

	Ingo

[toc] | [prev] | [next] | [standalone]

#1478787

* Alexander Shishkin <alexander.shishkin@linux.intel.com> wrote:

> Ingo Molnar <mingo@kernel.org> writes:
> 
> > * Alexander Shishkin <alexander.shishkin@linux.intel.com> wrote:
> >
> >> Hi,
> >> 
> >> There were more bugs since the previous version, plus the BTS barriers got 
> >> fixed. With these patches, my testcase keeps running and no spurious NMI 
> >> warnings pop up any more.
> >
> > Could you please also run the fuzzer that Vince uses, does it now pass on hardware 
> > you have access to?
> 
> Sure. And yes, I did catch a warning, which calls for one more patch
> (below). Also one unrelated thing in PEBS that Peter fixed.
> 
> > I'd like to make "passes the fuzzer" a standard requirement before new changes are 
> > accepted to perf core.
> 
> Let's make it so.
> 
> For the sake of consistency, this one needs to go before 3/5. I'll
> re-send the whole series, though, if need be. I've got 2 perf_fuzzers
> running on this meanwhile.

Yeah, please re-send it - and please also Vince's Reported-by tag to all commits 
that would explain failures that Vince reported.

Also, please document how much and what type of fuzzer testing the series got: 
fuzzer version, time it ran and (rough) hardware it ran on would be useful. (That 
way we can look back later on whether there was any fuzzer testing on AMD systems 
for example, which you might not be able to perform.)

It would also be very, very nice to also add a Documentation/perf/testing.txt step 
by step ELI5 style document that explains how to set up and run the fuzzer!

> + * registers, so the PMU state remains unchanged.
> + *
> + * intel_bts events don't coexist with intel pmu's BTS events because of
> + * x86_add_exclusive(x86_lbr_exclusive_lbr); there's no need to keep them
> + * disabled around intel pmu's event batching etc, only inside the PMI handler.

Pet peeve nit: please capitalize 'PMU' correctly and consistently (upper case). 
This paragraph has both variants: "PMU" and "pmu" which is the worst variant 
really.

Thanks,

	Ingo

[toc] | [prev] | [next] | [standalone]

#1478930

Ingo Molnar <mingo@kernel.org> writes:

> * Alexander Shishkin <alexander.shishkin@linux.intel.com> wrote:
>
>> Ingo Molnar <mingo@kernel.org> writes:
>> 
>> > * Alexander Shishkin <alexander.shishkin@linux.intel.com> wrote:
>> >
>> >> Hi,
>> >> 
>> >> There were more bugs since the previous version, plus the BTS barriers got 
>> >> fixed. With these patches, my testcase keeps running and no spurious NMI 
>> >> warnings pop up any more.
>> >
>> > Could you please also run the fuzzer that Vince uses, does it now pass on hardware 
>> > you have access to?
>> 
>> Sure. And yes, I did catch a warning, which calls for one more patch
>> (below). Also one unrelated thing in PEBS that Peter fixed.
>> 
>> > I'd like to make "passes the fuzzer" a standard requirement before new changes are 
>> > accepted to perf core.
>> 
>> Let's make it so.
>> 
>> For the sake of consistency, this one needs to go before 3/5. I'll
>> re-send the whole series, though, if need be. I've got 2 perf_fuzzers
>> running on this meanwhile.
>
> Yeah, please re-send it - and please also Vince's Reported-by tag to all commits 
> that would explain failures that Vince reported.
>
> Also, please document how much and what type of fuzzer testing the series got: 
> fuzzer version, time it ran and (rough) hardware it ran on would be useful. (That 
> way we can look back later on whether there was any fuzzer testing on AMD systems 
> for example, which you might not be able to perform.)

Not sure if run time is useful with the fuzzer, but otherwise seems
reasonable. How do we put this stuff into a commit message, though?

Perf-Fuzzer-ID: d18d23aae6

?

> It would also be very, very nice to also add a Documentation/perf/testing.txt step 
> by step ELI5 style document that explains how to set up and run the fuzzer!

Actually, I'd like that too as I'm not sure I'm doing it 100% right (had
to patch it some time ago to stop it from segfaulting).

>> + * intel_bts events don't coexist with intel pmu's BTS events because of
>> + * x86_add_exclusive(x86_lbr_exclusive_lbr); there's no need to keep them
>> + * disabled around intel pmu's event batching etc, only inside the PMI handler.
>
> Pet peeve nit: please capitalize 'PMU' correctly and consistently (upper case). 
> This paragraph has both variants: "PMU" and "pmu" which is the worst variant 
> really.

Sure.

Regards,
--
Alex

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web