Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1676304 > unrolled thread

Re: [REGRESSION] perf/core: PMU interrupts dropped if we entered the kernel in the "skid" region

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [REGRESSION] perf/core: PMU interrupts dropped if we entered the  kernel in the "skid" region Kyle Huey <me@kylehuey.com> - 2017-06-28 03:10 +0200
    Re: [REGRESSION] perf/core: PMU interrupts dropped if we entered the  kernel in the "skid" region "Jin, Yao" <yao.jin@linux.intel.com> - 2017-06-28 04:10 +0200
      Re: [REGRESSION] perf/core: PMU interrupts dropped if we entered the  kernel in the "skid" region Kyle Huey <me@kylehuey.com> - 2017-06-28 07:00 +0200
        Re: [REGRESSION] perf/core: PMU interrupts dropped if we entered the  kernel in the "skid" region "Jin, Yao" <yao.jin@linux.intel.com> - 2017-06-28 07:40 +0200
          Re: [REGRESSION] perf/core: PMU interrupts dropped if we entered the  kernel in the "skid" region Kyle Huey <me@kylehuey.com> - 2017-06-28 09:40 +0200
        Re: [REGRESSION] perf/core: PMU interrupts dropped if we entered the  kernel in the "skid" region Mark Rutland <mark.rutland@arm.com> - 2017-06-28 12:20 +0200
          [PATCH] perf/core: generate overflow signal when samples are dropped  (WAS: Re: [REGRESSION] perf/core: PMU interrupts dropped if we entered the  kernel in the "skid" region) Mark Rutland <mark.rutland@arm.com> - 2017-06-28 13:00 +0200
            Re: [PATCH] perf/core: generate overflow signal when samples are  dropped (WAS: Re: [REGRESSION] perf/core: PMU interrupts dropped if we  entered the kernel in the "skid" region) Vince Weaver <vincent.weaver@maine.edu> - 2017-06-28 14:50 +0200
              Re: [PATCH] perf/core: generate overflow signal when samples are  dropped (WAS: Re: [REGRESSION] perf/core: PMU interrupts dropped if we  entered the kernel in the "skid" region) Mark Rutland <mark.rutland@arm.com> - 2017-06-28 15:10 +0200
                Re: [PATCH] perf/core: generate overflow signal when samples are  dropped (WAS: Re: [REGRESSION] perf/core: PMU interrupts dropped if we  entered the kernel in the "skid" region) Alexey Budankov <alexey.budankov@linux.intel.com> - 2017-06-29 10:20 +0200
                  Re: [PATCH] perf/core: generate overflow signal when samples are  dropped (WAS: Re: [REGRESSION] perf/core: PMU interrupts dropped if we  entered the kernel in the "skid" region) Alexey Budankov <alexey.budankov@linux.intel.com> - 2017-06-29 10:30 +0200
            Re: [PATCH] perf/core: generate overflow signal when samples are  dropped (WAS: Re: [REGRESSION] perf/core: PMU interrupts dropped if we  entered the kernel in the "skid" region) Mark Rutland <mark.rutland@arm.com> - 2017-06-28 20:00 +0200
              Re: [PATCH] perf/core: generate overflow signal when samples are  dropped (WAS: Re: [REGRESSION] perf/core: PMU interrupts dropped if we  entered the kernel in the "skid" region) Kyle Huey <me@kylehuey.com> - 2017-06-29 01:00 +0200
                Re: [PATCH] perf/core: generate overflow signal when samples are  dropped (WAS: Re: [REGRESSION] perf/core: PMU interrupts dropped if we  entered the kernel in the "skid" region) "Jin, Yao" <yao.jin@linux.intel.com> - 2017-06-29 02:30 +0200
                Re: [PATCH] perf/core: generate overflow signal when samples are  dropped (WAS: Re: [REGRESSION] perf/core: PMU interrupts dropped if we  entered the kernel in the "skid" region) Kyle Huey <me@kylehuey.com> - 2017-06-30 19:50 +0200
              Re: [PATCH] perf/core: generate overflow signal when samples are  dropped (WAS: Re: [REGRESSION] perf/core: PMU interrupts dropped if we  entered the kernel in the "skid" region) Ingo Molnar <mingo@kernel.org> - 2017-06-29 10:20 +0200
            Re: [PATCH] perf/core: generate overflow signal when samples are  dropped (WAS: Re: [REGRESSION] perf/core: PMU interrupts dropped if we  entered the kernel in the "skid" region) Kyle Huey <me@kylehuey.com> - 2017-06-28 20:00 +0200
          Re: [REGRESSION] perf/core: PMU interrupts dropped if we entered the  kernel in the "skid" region "Robert O'Callahan" <robert@ocallahan.org> - 2017-06-28 20:00 +0200
          Re: [REGRESSION] perf/core: PMU interrupts dropped if we entered the  kernel in the "skid" region Kyle Huey <me@kylehuey.com> - 2017-06-28 20:00 +0200
            Re: [REGRESSION] perf/core: PMU interrupts dropped if we entered the  kernel in the "skid" region Mark Rutland <mark.rutland@arm.com> - 2017-06-28 20:00 +0200
          Re: [REGRESSION] perf/core: PMU interrupts dropped if we entered the  kernel in the "skid" region Mark Rutland <mark.rutland@arm.com> - 2017-06-28 20:00 +0200
          Re: [REGRESSION] perf/core: PMU interrupts dropped if we entered the  kernel in the "skid" region Kyle Huey <me@kylehuey.com> - 2017-06-28 20:00 +0200

Page 2 of 2 — ← Prev page 1 [2]

#1676995

On Wed, Jun 28, 2017 at 09:46:43AM -0700, Kyle Huey wrote:
> On Wed, Jun 28, 2017 at 3:12 AM, Mark Rutland <mark.rutland@arm.com> wrote:
> > On Tue, Jun 27, 2017 at 09:51:00PM -0700, Kyle Huey wrote:
> >> My understanding of the situation is as follows:
> >>
> >> There is some time, call it t_0, where the hardware counter overflows.
> >> The PMU triggers an interrupt, but this is not instantaneous.  Call
> >> the time when the interrupt is actually delivered t_1.  Then t_1 - t_0
> >> is the "skid".
> >>
> >> Note that if the counter is `exclude_kernel`, then at t_0 the CPU
> >> *must* be running a userspace program.  But by t_1, the CPU may be
> >> doing something else.  Your patch changed things so that if at t_1 the
> >> CPU is in the kernel, then the interrupt is discarded.  But rr has
> >> programmed the counter to deliver a signal on overflow (via F_SETSIG
> >> on the fd returned by perf_event_open).  This change results in the
> >> signal never being delivered, because the interrupt was ignored.
> >> (More accurately, the signal is delivered the *next* time the counter
> >> overflows, which is far past where we wanted to inject our
> >> asynchronous event into our tracee.
> >
> > Yes, this is a bug.
> >
> > As we're trying to avoid smapling state, I think we can move the check
> > into perf_prepare_sample() or __perf_event_output(), where that state is
> > actually sampled. I'll take a look at that momentarily.
> >
> > Just to clarify, you don't care about the sample state at all? i.e. you
> > don't need the user program counter?
> 
> Right. `sample_regs_user`, `sample_star_user`, `branch_sample_type`,
> etc are all 0.
> https://github.com/mozilla/rr/blob/cf594dd01f07d96a61409e9f41a29f78c8c51693/src/PerfCounters.cc#L194
> is what we do use.

Given that, I must be missing something.

In __perf_event_overflow(), we already bail out early if
!is_sampling_event(event), i.e. when the sample_period is 0.

Your attr has a sample_period of zero, so something must be initialising
that.

Do you always call PERF_EVENT_IOC_PERIOD, or is something in the core
fiddling with the sample period behind your back?

It seems odd that an event without any samples to take has a sample
period. I'm surprised that there's not *some* sample_type set.

> > Is that signal delivered to the tracee, or to a different process that
> > traces it? If the latter, what ensures that the task is stopped
> > sufficiently quickly?
> 
> It's delivered to the tracee (via an F_SETOWN_EX with the tracee tid).
> In practice we've found that on modern Intel hardware that the
> interrupt and resulting signal delivery delay is bounded by a
> relatively small number of counter events.

Ok.

Thanks,
Mark.

[toc] | [prev] | [next] | [standalone]

#1676998

On Wed, Jun 28, 2017 at 3:12 AM, Mark Rutland <mark.rutland@arm.com> wrote:
> On Tue, Jun 27, 2017 at 09:51:00PM -0700, Kyle Huey wrote:
>> On Tue, Jun 27, 2017 at 7:09 PM, Jin, Yao <yao.jin@linux.intel.com> wrote:
>> > Hi,
>> >
>> > In theory, the PMI interrupts in skid region should be dropped, right?
>>
>> No, why would they be dropped?
>>
>> My understanding of the situation is as follows:
>>
>> There is some time, call it t_0, where the hardware counter overflows.
>> The PMU triggers an interrupt, but this is not instantaneous.  Call
>> the time when the interrupt is actually delivered t_1.  Then t_1 - t_0
>> is the "skid".
>>
>> Note that if the counter is `exclude_kernel`, then at t_0 the CPU
>> *must* be running a userspace program.  But by t_1, the CPU may be
>> doing something else.  Your patch changed things so that if at t_1 the
>> CPU is in the kernel, then the interrupt is discarded.  But rr has
>> programmed the counter to deliver a signal on overflow (via F_SETSIG
>> on the fd returned by perf_event_open).  This change results in the
>> signal never being delivered, because the interrupt was ignored.
>> (More accurately, the signal is delivered the *next* time the counter
>> overflows, which is far past where we wanted to inject our
>> asynchronous event into our tracee.
>
> Yes, this is a bug.
>
> As we're trying to avoid smapling state, I think we can move the check
> into perf_prepare_sample() or __perf_event_output(), where that state is
> actually sampled. I'll take a look at that momentarily.
>
> Just to clarify, you don't care about the sample state at all? i.e. you
> don't need the user program counter?

Right. `sample_regs_user`, `sample_star_user`, `branch_sample_type`,
etc are all 0.
https://github.com/mozilla/rr/blob/cf594dd01f07d96a61409e9f41a29f78c8c51693/src/PerfCounters.cc#L194
is what we do use.

> Is that signal delivered to the tracee, or to a different process that
> traces it? If the latter, what ensures that the task is stopped
> sufficiently quickly?

It's delivered to the tracee (via an F_SETOWN_EX with the tracee tid).
In practice we've found that on modern Intel hardware that the
interrupt and resulting signal delivery delay is bounded by a
relatively small number of counter events.

>> It seems to me that it might be reasonable to ignore the interrupt if
>> the purpose of the interrupt is to trigger sampling of the CPUs
>> register state.  But if the interrupt will trigger some other
>> operation, such as a signal on an fd, then there's no reason to drop
>> it.
>
> Agreed. I'll try to have a patch for this soon.
>
> I just need to figure out exactly where that overflow signal is
> generated by the perf core.
>
> Thanks,
> Mark.

- Kyle

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | linux.kernel

csiph-web