Re: [PATCH] perf/core: generate overflow signal when samples are dropped (WAS: Re: [REGRESSION] perf/core: PMU interrupts dropped if we entered the kernel in the "skid" region)

From	Alexey Budankov <alexey.budankov@linux.intel.com>
Newsgroups	linux.kernel
Subject	Re: [PATCH] perf/core: generate overflow signal when samples are dropped (WAS: Re: [REGRESSION] perf/core: PMU interrupts dropped if we entered the kernel in the "skid" region)
Date	2017-06-29 10:20 +0200
Message-ID	<tXCQa-5a5-5@gated-at.bofh.it> (permalink)
References	(3 earlier) <tXdf3-2en-9@gated-at.bofh.it> <tXieJ-5vN-11@gated-at.bofh.it> <tXiRs-5KR-23@gated-at.bofh.it> <tXkzU-6QG-13@gated-at.bofh.it> <tXkTf-7cj-7@gated-at.bofh.it>
Organization	Intel Corp.

Show all headers | View raw

Hi Folks,

On 28.06.2017 16:07, Mark Rutland wrote:
> On Wed, Jun 28, 2017 at 08:40:30AM -0400, Vince Weaver wrote:
>> On Wed, 28 Jun 2017, Mark Rutland wrote:
>>
>>> On Wed, Jun 28, 2017 at 11:12:48AM +0100, Mark Rutland wrote:
>>
>>> Instead of bailing out early in perf_event_overflow, we can bail prior
>>> to performing the actual sampling in __perf_event_output(). This avoids
>>> the information leak, but preserves the generation of the signal.
>>>
>>> Since we don't place any sample data into the ring buffer, the signal is
>>> arguably spurious. However, a userspace ringbuffer consumer can already
>>> consume data prior to taking the associated signals, and therefore must
>>> handle spurious signals to operate correctly. Thus, this signal
>>> shouldn't be harmful.
>>
>> this could still break some of my perf_event validation tests.
>>
>> Ones that set up a sampling event for every 1M instructions, run for 100M 
>> instructions, and expect there to be 100 samples received.
> 
> Is that test reliable today?
> 
> I'd expect that at least on ARM it's not, given that events can be
> counted imprecisely, and mode filters can be applied imprecisely. So you
> might get fewer (or more) samples. I'd imagine similar is true on other
> archtiectures.
> 
> If sampling took long enough, the existing ratelimiting could come into
> effect, too.
> 
> Surely that already has some error margin?

FYI.

From my recent experience and observation (on Intel Xeon Phi) 
wakeup_events_overflow and overflow_poll tests may fail if 
/proc/sys/kernel/perf_event_max_sample_rate is low enough:

# cat /proc/sys/kernel/perf_event_max_sample_rate
6000
# abudanko/perf_event_tests/tests/overflow/wakeup_events_overflow
This tests wakeup event overflows.
Testing with wakeup_events=1.
Counts, using mmap buffer 0x7f707b0dc000
	POLL_IN : 10
	POLL_OUT: 0
	POLL_MSG: 0
	POLL_ERR: 0
	POLL_PRI: 0
	POLL_HUP: 0
	UNKNOWN : 0
Testing wakeup events overflow...                            PASSED
# abudanko/perf_event_tests/tests/overflow/overflow_poll
This tests using poll() to catch overflow.
Monitoring pid 131412 status 1407
Child has stopped due to signal 5 (Trace/breakpoint trap)
Continuing child
Returned HUP!
Counts, using mmap buffer 0x7fb3bfe04000
	POLL_IN : 10
	POLL_OUT: 0
	POLL_MSG: 0
	POLL_ERR: 0
	POLL_PRI: 0
	POLL_HUP: 1
	UNKNOWN : 0
Testing catching overflow with poll()...                     PASSED

# echo 1000 > /proc/sys/kernel/perf_event_max_sample_rate
# abudanko/perf_event_tests/tests/overflow/overflow_poll
This tests using poll() to catch overflow.
Monitoring pid 131551 status 1407
Child has stopped due to signal 5 (Trace/breakpoint trap)
Continuing child
Returned HUP!
Counts, using mmap buffer 0x7f80532df000
	POLL_IN : 9
	POLL_OUT: 0
	POLL_MSG: 0
	POLL_ERR: 0
	POLL_PRI: 0
	POLL_HUP: 1
	UNKNOWN : 0
Unexpected POLL_IN interrupt.
Testing catching overflow with poll()...                     FAILED
[root@nntpdsd52-210 ~]# abudanko/perf_event_tests/tests/overflow/overflow_poll
This tests using poll() to catch overflow.
Monitoring pid 131553 status 1407
Child has stopped due to signal 5 (Trace/breakpoint trap)
Continuing child
Returned HUP!
Counts, using mmap buffer 0x7f650952c000
	POLL_IN : 9
	POLL_OUT: 0
	POLL_MSG: 0
	POLL_ERR: 0
	POLL_PRI: 0
	POLL_HUP: 1
	UNKNOWN : 0
Unexpected POLL_IN interrupt.
Testing catching overflow with poll()...                     FAILED

> 
>> If we're so worried about info leakage, can't we just zero-out the problem 
>> address (or randomize the kernel address) rather than just pretending the 
>> interrupt didn't happen?
> 
> Making up zeroed or randomized data is going to confuse users. I can't
> imagine that real users are going to want bogus samples that they have
> to identify (somehow) in order to skip when processing the data.
> 
> I can see merit in signalling "lost" samples to userspace, so long as
> they're easily distinguished from real samples.
> 
> One option is to fake up a sample using the user regs regardless, but
> that's both fragile and surprising in other cases.
> 
> Thanks,
> Mark.
> 

Thanks,
Alexey

Thread

Re: [REGRESSION] perf/core: PMU interrupts dropped if we entered the  kernel in the "skid" region Kyle Huey <me@kylehuey.com> - 2017-06-28 03:10 +0200
  Re: [REGRESSION] perf/core: PMU interrupts dropped if we entered the  kernel in the "skid" region "Jin, Yao" <yao.jin@linux.intel.com> - 2017-06-28 04:10 +0200
    Re: [REGRESSION] perf/core: PMU interrupts dropped if we entered the  kernel in the "skid" region Kyle Huey <me@kylehuey.com> - 2017-06-28 07:00 +0200
      Re: [REGRESSION] perf/core: PMU interrupts dropped if we entered the  kernel in the "skid" region "Jin, Yao" <yao.jin@linux.intel.com> - 2017-06-28 07:40 +0200
        Re: [REGRESSION] perf/core: PMU interrupts dropped if we entered the  kernel in the "skid" region Kyle Huey <me@kylehuey.com> - 2017-06-28 09:40 +0200
      Re: [REGRESSION] perf/core: PMU interrupts dropped if we entered the  kernel in the "skid" region Mark Rutland <mark.rutland@arm.com> - 2017-06-28 12:20 +0200
        [PATCH] perf/core: generate overflow signal when samples are dropped  (WAS: Re: [REGRESSION] perf/core: PMU interrupts dropped if we entered the  kernel in the "skid" region) Mark Rutland <mark.rutland@arm.com> - 2017-06-28 13:00 +0200
          Re: [PATCH] perf/core: generate overflow signal when samples are  dropped (WAS: Re: [REGRESSION] perf/core: PMU interrupts dropped if we  entered the kernel in the "skid" region) Vince Weaver <vincent.weaver@maine.edu> - 2017-06-28 14:50 +0200
            Re: [PATCH] perf/core: generate overflow signal when samples are  dropped (WAS: Re: [REGRESSION] perf/core: PMU interrupts dropped if we  entered the kernel in the "skid" region) Mark Rutland <mark.rutland@arm.com> - 2017-06-28 15:10 +0200
              Re: [PATCH] perf/core: generate overflow signal when samples are  dropped (WAS: Re: [REGRESSION] perf/core: PMU interrupts dropped if we  entered the kernel in the "skid" region) Alexey Budankov <alexey.budankov@linux.intel.com> - 2017-06-29 10:20 +0200
                Re: [PATCH] perf/core: generate overflow signal when samples are  dropped (WAS: Re: [REGRESSION] perf/core: PMU interrupts dropped if we  entered the kernel in the "skid" region) Alexey Budankov <alexey.budankov@linux.intel.com> - 2017-06-29 10:30 +0200
          Re: [PATCH] perf/core: generate overflow signal when samples are  dropped (WAS: Re: [REGRESSION] perf/core: PMU interrupts dropped if we  entered the kernel in the "skid" region) Mark Rutland <mark.rutland@arm.com> - 2017-06-28 20:00 +0200
            Re: [PATCH] perf/core: generate overflow signal when samples are  dropped (WAS: Re: [REGRESSION] perf/core: PMU interrupts dropped if we  entered the kernel in the "skid" region) Kyle Huey <me@kylehuey.com> - 2017-06-29 01:00 +0200
              Re: [PATCH] perf/core: generate overflow signal when samples are  dropped (WAS: Re: [REGRESSION] perf/core: PMU interrupts dropped if we  entered the kernel in the "skid" region) "Jin, Yao" <yao.jin@linux.intel.com> - 2017-06-29 02:30 +0200
              Re: [PATCH] perf/core: generate overflow signal when samples are  dropped (WAS: Re: [REGRESSION] perf/core: PMU interrupts dropped if we  entered the kernel in the "skid" region) Kyle Huey <me@kylehuey.com> - 2017-06-30 19:50 +0200
            Re: [PATCH] perf/core: generate overflow signal when samples are  dropped (WAS: Re: [REGRESSION] perf/core: PMU interrupts dropped if we  entered the kernel in the "skid" region) Ingo Molnar <mingo@kernel.org> - 2017-06-29 10:20 +0200
          Re: [PATCH] perf/core: generate overflow signal when samples are  dropped (WAS: Re: [REGRESSION] perf/core: PMU interrupts dropped if we  entered the kernel in the "skid" region) Kyle Huey <me@kylehuey.com> - 2017-06-28 20:00 +0200
        Re: [REGRESSION] perf/core: PMU interrupts dropped if we entered the  kernel in the "skid" region "Robert O'Callahan" <robert@ocallahan.org> - 2017-06-28 20:00 +0200
        Re: [REGRESSION] perf/core: PMU interrupts dropped if we entered the  kernel in the "skid" region Kyle Huey <me@kylehuey.com> - 2017-06-28 20:00 +0200
          Re: [REGRESSION] perf/core: PMU interrupts dropped if we entered the  kernel in the "skid" region Mark Rutland <mark.rutland@arm.com> - 2017-06-28 20:00 +0200
        Re: [REGRESSION] perf/core: PMU interrupts dropped if we entered the  kernel in the "skid" region Mark Rutland <mark.rutland@arm.com> - 2017-06-28 20:00 +0200
        Re: [REGRESSION] perf/core: PMU interrupts dropped if we entered the  kernel in the "skid" region Kyle Huey <me@kylehuey.com> - 2017-06-28 20:00 +0200

csiph-web