Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1528644 > unrolled thread

[PATCH 00/14] export perf overheads information

Back to article view | Back to linux.kernel

Contents

  [PATCH 00/14] export perf overheads information kan.liang@intel.com - 2016-11-23 18:50 +0100
    [PATCH 01/14] perf/x86: Introduce PERF_RECORD_OVERHEAD kan.liang@intel.com - 2016-11-23 18:50 +0100
      Re: [PATCH 01/14] perf/x86: Introduce PERF_RECORD_OVERHEAD Peter Zijlstra <peterz@infradead.org> - 2016-11-23 21:20 +0100
      Re: [PATCH 01/14] perf/x86: Introduce PERF_RECORD_OVERHEAD Peter Zijlstra <peterz@infradead.org> - 2016-11-23 21:20 +0100
      Re: [PATCH 01/14] perf/x86: Introduce PERF_RECORD_OVERHEAD Jiri Olsa <jolsa@redhat.com> - 2016-11-24 00:50 +0100
        RE: [PATCH 01/14] perf/x86: Introduce PERF_RECORD_OVERHEAD "Liang, Kan" <kan.liang@intel.com> - 2016-11-24 14:50 +0100
          Re: [PATCH 01/14] perf/x86: Introduce PERF_RECORD_OVERHEAD Peter Zijlstra <peterz@infradead.org> - 2016-11-24 15:00 +0100
            RE: [PATCH 01/14] perf/x86: Introduce PERF_RECORD_OVERHEAD "Liang, Kan" <kan.liang@intel.com> - 2016-11-24 15:10 +0100
              Re: [PATCH 01/14] perf/x86: Introduce PERF_RECORD_OVERHEAD Jiri Olsa <jolsa@redhat.com> - 2016-11-24 15:30 +0100
                Re: [PATCH 01/14] perf/x86: Introduce PERF_RECORD_OVERHEAD Jiri Olsa <jolsa@redhat.com> - 2016-11-24 15:50 +0100
                RE: [PATCH 01/14] perf/x86: Introduce PERF_RECORD_OVERHEAD "Liang, Kan" <kan.liang@intel.com> - 2016-11-24 15:50 +0100
            Re: [PATCH 01/14] perf/x86: Introduce PERF_RECORD_OVERHEAD Andi Kleen <andi@firstfloor.org> - 2016-11-24 19:30 +0100
              Re: [PATCH 01/14] perf/x86: Introduce PERF_RECORD_OVERHEAD Peter Zijlstra <peterz@infradead.org> - 2016-11-24 20:00 +0100
                Re: [PATCH 01/14] perf/x86: Introduce PERF_RECORD_OVERHEAD Andi Kleen <andi@firstfloor.org> - 2016-11-24 20:10 +0100
                  Re: [PATCH 01/14] perf/x86: Introduce PERF_RECORD_OVERHEAD Peter Zijlstra <peterz@infradead.org> - 2016-11-24 20:10 +0100
    [PATCH 11/14] perf tools: record write data overhead kan.liang@intel.com - 2016-11-23 18:50 +0100
      Re: [PATCH 11/14] perf tools: record write data overhead Jiri Olsa <jolsa@redhat.com> - 2016-11-24 00:10 +0100
      Re: [PATCH 11/14] perf tools: record write data overhead Jiri Olsa <jolsa@redhat.com> - 2016-11-24 00:20 +0100
    [PATCH 04/14] perf/x86: output side-band events overhead kan.liang@intel.com - 2016-11-23 18:50 +0100
      Re: [PATCH 04/14] perf/x86: output side-band events overhead Peter Zijlstra <peterz@infradead.org> - 2016-11-23 21:10 +0100
      Re: [PATCH 04/14] perf/x86: output side-band events overhead Mark Rutland <mark.rutland@arm.com> - 2016-11-24 17:30 +0100
        RE: [PATCH 04/14] perf/x86: output side-band events overhead "Liang, Kan" <kan.liang@intel.com> - 2016-11-24 20:50 +0100
    [PATCH 07/14] perf tools: show multiplexing overhead kan.liang@intel.com - 2016-11-23 18:50 +0100
    [PATCH 08/14] perf tools: show side-band events overhead kan.liang@intel.com - 2016-11-23 18:50 +0100
    [PATCH 05/14] perf tools: handle PERF_RECORD_OVERHEAD record type kan.liang@intel.com - 2016-11-23 18:50 +0100
      Re: [PATCH 05/14] perf tools: handle PERF_RECORD_OVERHEAD record type Jiri Olsa <jolsa@redhat.com> - 2016-11-23 23:40 +0100
        Re: [PATCH 05/14] perf tools: handle PERF_RECORD_OVERHEAD record type Jiri Olsa <jolsa@redhat.com> - 2016-11-24 00:00 +0100
    [PATCH 12/14] perf tools: record elapsed time kan.liang@intel.com - 2016-11-23 18:50 +0100
    [PATCH 13/14] perf tools: warn on high overhead kan.liang@intel.com - 2016-11-23 18:50 +0100
      Re: [PATCH 13/14] perf tools: warn on high overhead Andi Kleen <andi@firstfloor.org> - 2016-11-23 21:30 +0100
        RE: [PATCH 13/14] perf tools: warn on high overhead "Liang, Kan" <kan.liang@intel.com> - 2016-11-23 23:10 +0100
    [PATCH 14/14] perf script: show overhead events kan.liang@intel.com - 2016-11-23 18:50 +0100
      Re: [PATCH 14/14] perf script: show overhead events Jiri Olsa <jolsa@redhat.com> - 2016-11-24 00:30 +0100
      Re: [PATCH 14/14] perf script: show overhead events Jiri Olsa <jolsa@redhat.com> - 2016-11-24 00:30 +0100
      Re: [PATCH 14/14] perf script: show overhead events Jiri Olsa <jolsa@redhat.com> - 2016-11-24 00:40 +0100
      Re: [PATCH 14/14] perf script: show overhead events Jiri Olsa <jolsa@redhat.com> - 2016-11-24 00:40 +0100
    [PATCH 03/14] perf/x86: output multiplexing overhead kan.liang@intel.com - 2016-11-23 18:50 +0100
      Re: [PATCH 03/14] perf/x86: output multiplexing overhead Peter Zijlstra <peterz@infradead.org> - 2016-11-23 21:10 +0100
        RE: [PATCH 03/14] perf/x86: output multiplexing overhead "Liang, Kan" <kan.liang@intel.com> - 2016-11-23 21:20 +0100
    [PATCH 10/14] perf tools: introduce PERF_RECORD_USER_OVERHEAD kan.liang@intel.com - 2016-11-23 18:50 +0100
    [PATCH 06/14] perf tools: show NMI overhead kan.liang@intel.com - 2016-11-23 18:50 +0100
      Re: [PATCH 06/14] perf tools: show NMI overhead Jiri Olsa <jolsa@redhat.com> - 2016-11-24 00:00 +0100
      Re: [PATCH 06/14] perf tools: show NMI overhead Jiri Olsa <jolsa@redhat.com> - 2016-11-24 00:00 +0100
        RE: [PATCH 06/14] perf tools: show NMI overhead "Liang, Kan" <kan.liang@intel.com> - 2016-11-24 14:40 +0100
          Re: [PATCH 06/14] perf tools: show NMI overhead Jiri Olsa <jolsa@redhat.com> - 2016-11-24 16:30 +0100
            Re: [PATCH 06/14] perf tools: show NMI overhead Namhyung Kim <namhyung@kernel.org> - 2016-11-25 00:30 +0100
              Re: [PATCH 06/14] perf tools: show NMI overhead Jiri Olsa <jolsa@redhat.com> - 2016-11-25 00:50 +0100
            Re: [PATCH 06/14] perf tools: show NMI overhead Andi Kleen <andi@firstfloor.org> - 2016-11-25 01:30 +0100
      Re: [PATCH 06/14] perf tools: show NMI overhead Jiri Olsa <jolsa@redhat.com> - 2016-11-24 00:00 +0100
    Re: [PATCH 00/14] export perf overheads information Ingo Molnar <mingo@kernel.org> - 2016-11-24 05:30 +0100

Page 1 of 3  [1] 2 3  Next page →

#1528644 — [PATCH 00/14] export perf overheads information

From: Kan Liang <kan.liang@intel.com>

Profiling brings additional overhead. High overhead may impacts the
behavior of the profiling object, impacts the accuracy of the
profiling result, and even hang the system.
Currently, perf has dynamic interrupt throttle mechanism to lower the
sample rate and overhead. But it has limitations.
 - The mechanism only focus in the overhead from NMI. However, there
   are other parts which bring big overhead. E.g, multiplexing.
 - The hint from the mechanism doesn't work on fixed period.
 - The system changes which caused by the mechanism are not recorded
   in the perf.data. Users have no idea about the overhead and its
   impact.
Acctually, any passive ways like dynamic interrupt throttle mechanism
are only palliative. The best way is to export overheads information,
provide more hints, and help the users design more proper perf command.

According to our test, there are four parts which can bring big overhead.
They include NMI handler, multiplexing handler, iterate side-band events,
and write data in file. Two new perf record type PERF_RECORD_OVERHEAD and
PERF_RECORD_USER_OVERHEAD are introduced to record the overhead
information in kernel and user space respectively.
The overhead information is the system per-CPU overhead, not per-event
overhead. The implementation takes advantage of the existing event log
mechanism.
To reduce the additional overhead from logging overhead information, the
overhead information only be output when the event is going to be
disabled or task is scheduling out.

In perf report, the overhead will be checked automatically. If the
overhead rate is larger than 10%. A warning will be displayed.
A new option is also introduced to display detial per-CPU overhead
information.

Current implementation only include four overhead sources. There could be
more in other parts. The new overhead source can be easily added as a
new type.

Kan Liang (14):
  perf/x86: Introduce PERF_RECORD_OVERHEAD
  perf/x86: output NMI overhead
  perf/x86: output multiplexing overhead
  perf/x86: output side-band events overhead
  perf tools: handle PERF_RECORD_OVERHEAD record type
  perf tools: show NMI overhead
  perf tools: show multiplexing overhead
  perf tools: show side-band events overhead
  perf tools: make get_nsecs visible for buildin files
  perf tools: introduce PERF_RECORD_USER_OVERHEAD
  perf tools: record write data overhead
  perf tools: record elapsed time
  perf tools: warn on high overhead
  perf script: show overhead events

 arch/x86/events/core.c                   |  19 ++++-
 arch/x86/events/perf_event.h             |   2 +
 include/linux/perf_event.h               |   9 ++
 include/uapi/linux/perf_event.h          |  33 ++++++++
 kernel/events/core.c                     | 139 +++++++++++++++++++++++++------
 tools/include/uapi/linux/perf_event.h    |  33 ++++++++
 tools/perf/Documentation/perf-report.txt |   3 +
 tools/perf/builtin-record.c              |  71 +++++++++++++++-
 tools/perf/builtin-report.c              |  25 ++++++
 tools/perf/builtin-sched.c               |   2 +-
 tools/perf/builtin-script.c              |  36 ++++++++
 tools/perf/builtin.h                     |   1 +
 tools/perf/util/event.c                  |  47 +++++++++++
 tools/perf/util/event.h                  |  37 ++++++++
 tools/perf/util/machine.c                |  35 ++++++++
 tools/perf/util/machine.h                |   2 +
 tools/perf/util/session.c                |  68 +++++++++++++++
 tools/perf/util/symbol.h                 |   3 +-
 tools/perf/util/tool.h                   |   1 +
 19 files changed, 535 insertions(+), 31 deletions(-)

-- 
2.5.5

[toc] | [next] | [standalone]

#1528645 — [PATCH 01/14] perf/x86: Introduce PERF_RECORD_OVERHEAD

From: Kan Liang <kan.liang@intel.com>

A new perf record is introduced to export perf overhead information to
userspace. So we can measure the overhead of sampling directly.

To output the overhead information, it takes advantage of the existing
event log mechanism. But the overhead information is the system
overhead, not per-event overhead.

Signed-off-by: Kan Liang <kan.liang@intel.com>
---
 include/linux/perf_event.h      |  4 ++++
 include/uapi/linux/perf_event.h | 29 +++++++++++++++++++++++++++
 kernel/events/core.c            | 43 +++++++++++++++++++++++++++++++++++++++++
 3 files changed, 76 insertions(+)

diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h
index 4741ecd..3a530a7 100644
--- a/include/linux/perf_event.h
+++ b/include/linux/perf_event.h
@@ -998,6 +998,10 @@ perf_event__output_id_sample(struct perf_event *event,
 extern void
 perf_log_lost_samples(struct perf_event *event, u64 lost);
 
+extern void
+perf_log_overhead(struct perf_event *event, u32 type,
+		  struct perf_overhead_entry *entry);
+
 static inline bool is_sampling_event(struct perf_event *event)
 {
 	return event->attr.sample_period != 0;
diff --git a/include/uapi/linux/perf_event.h b/include/uapi/linux/perf_event.h
index c66a485..ae571be 100644
--- a/include/uapi/linux/perf_event.h
+++ b/include/uapi/linux/perf_event.h
@@ -862,6 +862,17 @@ enum perf_event_type {
 	 */
 	PERF_RECORD_SWITCH_CPU_WIDE		= 15,
 
+	/*
+	 * Records perf overhead
+	 * struct {
+	 * 	struct perf_event_header 	header;
+	 * 	u32				type;
+	 * 	struct perf_overhead_entry	entry;
+	 * 	struct sample_id		sample_id;
+	 * };
+	 */
+	PERF_RECORD_OVERHEAD			= 16,
+
 	PERF_RECORD_MAX,			/* non-ABI */
 };
 
@@ -980,4 +991,22 @@ struct perf_branch_entry {
 		reserved:44;
 };
 
+enum perf_record_overhead_type {
+	PERF_OVERHEAD_MAX,
+};
+
+/*
+ * single overhead record layout:
+ *
+ * 	 cpu: The cpu which overhead occues
+ * 	  nr: Times of overhead happens.
+ * 	      E.g. for NMI, nr == times of NMI handler are called.
+ * 	time: Total overhead cost(ns)
+ */
+struct perf_overhead_entry {
+	__u32	cpu;
+	__u64	nr;
+	__u64	time;
+};
+
 #endif /* _UAPI_LINUX_PERF_EVENT_H */
diff --git a/kernel/events/core.c b/kernel/events/core.c
index ff230bb..590c399 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -7037,6 +7037,49 @@ static void perf_log_itrace_start(struct perf_event *event)
 	perf_output_end(&handle);
 }
 
+
+/*
+ * Record overhead logging
+ *
+ * The overhead logged here is the system overhead, not per-event overhead.
+ * This function only take advantage of the existing event log mechanism
+ * to log the overhead record.
+ *
+ */
+void perf_log_overhead(struct perf_event *event, u32 type,
+		       struct perf_overhead_entry *entry)
+{
+	struct perf_output_handle handle;
+	struct perf_sample_data sample;
+	int ret;
+
+	struct {
+		struct perf_event_header	header;
+		u32				type;
+		struct perf_overhead_entry	overhead;
+	} overhead_event = {
+		.header = {
+			.type = PERF_RECORD_OVERHEAD,
+			.misc = 0,
+			.size = sizeof(overhead_event),
+		},
+	};
+	overhead_event.type = type;
+	memcpy(&overhead_event.overhead, entry, sizeof(overhead_event.overhead));
+
+	perf_event_header__init_id(&overhead_event.header, &sample, event);
+	ret = perf_output_begin(&handle, event, overhead_event.header.size);
+
+	if (ret)
+		return;
+
+	perf_output_put(&handle, overhead_event);
+	perf_event__output_id_sample(event, &handle, &sample);
+
+	perf_output_end(&handle);
+	memset(entry, 0, sizeof(*entry));
+}
+
 /*
  * Generic event overflow handling, sampling.
  */
-- 
2.5.5

[toc] | [prev] | [next] | [standalone]

#1528752 — Re: [PATCH 01/14] perf/x86: Introduce PERF_RECORD_OVERHEAD

On Wed, Nov 23, 2016 at 04:44:39AM -0500, kan.liang@intel.com wrote:
> +/*
> + * single overhead record layout:
> + *
> + * 	 cpu: The cpu which overhead occues

This is duplicate information, its already present in sample_id when
PERF_SAMPLE_CPU, and without that we don't care.

> + * 	  nr: Times of overhead happens.
> + * 	      E.g. for NMI, nr == times of NMI handler are called.
> + * 	time: Total overhead cost(ns)
> + */
> +struct perf_overhead_entry {
> +	__u32	cpu;
> +	__u64	nr;
> +	__u64	time;
> +};

[toc] | [prev] | [next] | [standalone]

#1528753 — Re: [PATCH 01/14] perf/x86: Introduce PERF_RECORD_OVERHEAD

On Wed, Nov 23, 2016 at 04:44:39AM -0500, kan.liang@intel.com wrote:
> +struct perf_overhead_entry {
> +	__u32	cpu;
> +	__u64	nr;
> +	__u64	time;
> +};

> +void perf_log_overhead(struct perf_event *event, u32 type,
> +		       struct perf_overhead_entry *entry)
> +{
> +	struct perf_output_handle handle;
> +	struct perf_sample_data sample;
> +	int ret;
> +
> +	struct {
> +		struct perf_event_header	header;
> +		u32				type;

There's two u32 sized holes here.

> +		struct perf_overhead_entry	overhead;
> +	} overhead_event = {
> +		.header = {
> +			.type = PERF_RECORD_OVERHEAD,
> +			.misc = 0,
> +			.size = sizeof(overhead_event),
> +		},
		.type = type,
		.overhead = *event,
> +	};

> +	overhead_event.type = type;
> +	memcpy(&overhead_event.overhead, entry, sizeof(overhead_event.overhead));

Do the above named initialization instead.

> +
> +	perf_event_header__init_id(&overhead_event.header, &sample, event);
> +	ret = perf_output_begin(&handle, event, overhead_event.header.size);
> +
> +	if (ret)
> +		return;
> +
> +	perf_output_put(&handle, overhead_event);
> +	perf_event__output_id_sample(event, &handle, &sample);
> +
> +	perf_output_end(&handle);
> +	memset(entry, 0, sizeof(*entry));
> +}

[toc] | [prev] | [next] | [standalone]

#1528871 — Re: [PATCH 01/14] perf/x86: Introduce PERF_RECORD_OVERHEAD

On Wed, Nov 23, 2016 at 04:44:39AM -0500, kan.liang@intel.com wrote:
> From: Kan Liang <kan.liang@intel.com>
> 
> A new perf record is introduced to export perf overhead information to
> userspace. So we can measure the overhead of sampling directly.
> 
> To output the overhead information, it takes advantage of the existing
> event log mechanism. But the overhead information is the system
> overhead, not per-event overhead.
> 
> Signed-off-by: Kan Liang <kan.liang@intel.com>
> ---
>  include/linux/perf_event.h      |  4 ++++
>  include/uapi/linux/perf_event.h | 29 +++++++++++++++++++++++++++
>  kernel/events/core.c            | 43 +++++++++++++++++++++++++++++++++++++++++
>  3 files changed, 76 insertions(+)
> 
> diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h
> index 4741ecd..3a530a7 100644
> --- a/include/linux/perf_event.h
> +++ b/include/linux/perf_event.h
> @@ -998,6 +998,10 @@ perf_event__output_id_sample(struct perf_event *event,
>  extern void
>  perf_log_lost_samples(struct perf_event *event, u64 lost);
>  
> +extern void
> +perf_log_overhead(struct perf_event *event, u32 type,
> +		  struct perf_overhead_entry *entry);
> +
>  static inline bool is_sampling_event(struct perf_event *event)
>  {
>  	return event->attr.sample_period != 0;
> diff --git a/include/uapi/linux/perf_event.h b/include/uapi/linux/perf_event.h
> index c66a485..ae571be 100644
> --- a/include/uapi/linux/perf_event.h
> +++ b/include/uapi/linux/perf_event.h
> @@ -862,6 +862,17 @@ enum perf_event_type {
>  	 */
>  	PERF_RECORD_SWITCH_CPU_WIDE		= 15,
>  
> +	/*
> +	 * Records perf overhead
> +	 * struct {
> +	 * 	struct perf_event_header 	header;
> +	 * 	u32				type;
> +	 * 	struct perf_overhead_entry	entry;
> +	 * 	struct sample_id		sample_id;
> +	 * };
> +	 */
> +	PERF_RECORD_OVERHEAD			= 16,
> +
>  	PERF_RECORD_MAX,			/* non-ABI */
>  };

I think we should make this optional/configurable like the
rest of the aux events, like below..

jirka


---
diff --git a/include/uapi/linux/perf_event.h b/include/uapi/linux/perf_event.h
index 5e7c52278ef0..853a919bd9ae 100644
--- a/include/uapi/linux/perf_event.h
+++ b/include/uapi/linux/perf_event.h
@@ -344,7 +344,8 @@ struct perf_event_attr {
 				use_clockid    :  1, /* use @clockid for time fields */
 				context_switch :  1, /* context switch data */
 				write_backward :  1, /* Write ring buffer from end to beginning */
-				__reserved_1   : 36;
+				overhead       :  1,
+				__reserved_1   : 35;
 
 	union {
 		__u32		wakeup_events;	  /* wakeup every n events */

[toc] | [prev] | [next] | [standalone]

#1529299 — RE: [PATCH 01/14] perf/x86: Introduce PERF_RECORD_OVERHEAD

> 
> On Wed, Nov 23, 2016 at 04:44:39AM -0500, kan.liang@intel.com wrote:
> > From: Kan Liang <kan.liang@intel.com>
> >
> > A new perf record is introduced to export perf overhead information to
> > userspace. So we can measure the overhead of sampling directly.
> >
> > To output the overhead information, it takes advantage of the existing
> > event log mechanism. But the overhead information is the system
> > overhead, not per-event overhead.
> >
> > Signed-off-by: Kan Liang <kan.liang@intel.com>
> > ---
> >  include/linux/perf_event.h      |  4 ++++
> >  include/uapi/linux/perf_event.h | 29 +++++++++++++++++++++++++++
> >  kernel/events/core.c            | 43
> +++++++++++++++++++++++++++++++++++++++++
> >  3 files changed, 76 insertions(+)
> >
> > diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h
> > index 4741ecd..3a530a7 100644
> > --- a/include/linux/perf_event.h
> > +++ b/include/linux/perf_event.h
> > @@ -998,6 +998,10 @@ perf_event__output_id_sample(struct
> perf_event
> > *event,  extern void  perf_log_lost_samples(struct perf_event *event,
> > u64 lost);
> >
> > +extern void
> > +perf_log_overhead(struct perf_event *event, u32 type,
> > +		  struct perf_overhead_entry *entry);
> > +
> >  static inline bool is_sampling_event(struct perf_event *event)  {
> >  	return event->attr.sample_period != 0; diff --git
> > a/include/uapi/linux/perf_event.h b/include/uapi/linux/perf_event.h
> > index c66a485..ae571be 100644
> > --- a/include/uapi/linux/perf_event.h
> > +++ b/include/uapi/linux/perf_event.h
> > @@ -862,6 +862,17 @@ enum perf_event_type {
> >  	 */
> >  	PERF_RECORD_SWITCH_CPU_WIDE		= 15,
> >
> > +	/*
> > +	 * Records perf overhead
> > +	 * struct {
> > +	 * 	struct perf_event_header 	header;
> > +	 * 	u32				type;
> > +	 * 	struct perf_overhead_entry	entry;
> > +	 * 	struct sample_id		sample_id;
> > +	 * };
> > +	 */
> > +	PERF_RECORD_OVERHEAD			= 16,
> > +
> >  	PERF_RECORD_MAX,			/* non-ABI */
> >  };
> 
> I think we should make this optional/configurable like the rest of the aux
> events, like below..
> 

The overhead logging only happens when event is going to be disabled or
the task is scheduling out. It should not be much and expensive.

Peter,

What do you think?
Should we make it configurable?

Thanks,
Kan

> jirka
> 
> 
> ---
> diff --git a/include/uapi/linux/perf_event.h
> b/include/uapi/linux/perf_event.h index 5e7c52278ef0..853a919bd9ae
> 100644
> --- a/include/uapi/linux/perf_event.h
> +++ b/include/uapi/linux/perf_event.h
> @@ -344,7 +344,8 @@ struct perf_event_attr {
>  				use_clockid    :  1, /* use @clockid for time
> fields */
>  				context_switch :  1, /* context switch data */
>  				write_backward :  1, /* Write ring buffer
> from end to beginning */
> -				__reserved_1   : 36;
> +				overhead       :  1,
> +				__reserved_1   : 35;
> 
>  	union {
>  		__u32		wakeup_events;	  /* wakeup every n
> events */

[toc] | [prev] | [next] | [standalone]

#1529310 — Re: [PATCH 01/14] perf/x86: Introduce PERF_RECORD_OVERHEAD

On Thu, Nov 24, 2016 at 01:45:28PM +0000, Liang, Kan wrote:
> > I think we should make this optional/configurable like the rest of the aux
> > events, like below..
> > 
> 
> The overhead logging only happens when event is going to be disabled or
> the task is scheduling out. It should not be much and expensive.
> 
> Peter,
> 
> What do you think?
> Should we make it configurable?

Is there a downside to having it optional?

[toc] | [prev] | [next] | [standalone]

#1529318 — RE: [PATCH 01/14] perf/x86: Introduce PERF_RECORD_OVERHEAD

> 
> On Thu, Nov 24, 2016 at 01:45:28PM +0000, Liang, Kan wrote:
> > > I think we should make this optional/configurable like the rest of
> > > the aux events, like below..
> > >
> >
> > The overhead logging only happens when event is going to be disabled
> > or the task is scheduling out. It should not be much and expensive.
> >
> > Peter,
> >
> > What do you think?
> > Should we make it configurable?
> 
> Is there a downside to having it optional?

NO. There will be no overhead information dumped into perf.data. Just like
current implementation.

[toc] | [prev] | [next] | [standalone]

#1529329 — Re: [PATCH 01/14] perf/x86: Introduce PERF_RECORD_OVERHEAD

On Thu, Nov 24, 2016 at 01:56:51PM +0000, Liang, Kan wrote:
> 
> 
> > 
> > On Thu, Nov 24, 2016 at 01:45:28PM +0000, Liang, Kan wrote:
> > > > I think we should make this optional/configurable like the rest of
> > > > the aux events, like below..
> > > >
> > >
> > > The overhead logging only happens when event is going to be disabled
> > > or the task is scheduling out. It should not be much and expensive.
> > >
> > > Peter,
> > >
> > > What do you think?
> > > Should we make it configurable?
> > 
> > Is there a downside to having it optional?
> 
> NO. There will be no overhead information dumped into perf.data. Just like
> current implementation.

old perf tools will get unexpected events if it's not optional

[root@ibm-x3650m4-01 perf]# ./perf c2c record -ag
^C[ perf record: Woken up 1 times to write data ]
Warning:
Found 1 unknown events!

Is this an older tool processing a perf.data file generated by a more recent tool?

If that is not the case, consider reporting to linux-kernel@vger.kernel.org.

[ perf record: Captured and wrote 2.333 MB perf.data (9370 samples) ]


jirka

[toc] | [prev] | [next] | [standalone]

#1529364 — Re: [PATCH 01/14] perf/x86: Introduce PERF_RECORD_OVERHEAD

On Thu, Nov 24, 2016 at 02:39:26PM +0000, Liang, Kan wrote:
> 
> 
> > On Thu, Nov 24, 2016 at 01:56:51PM +0000, Liang, Kan wrote:
> > >
> > >
> > > >
> > > > On Thu, Nov 24, 2016 at 01:45:28PM +0000, Liang, Kan wrote:
> > > > > > I think we should make this optional/configurable like the rest
> > > > > > of the aux events, like below..
> > > > > >
> > > > >
> > > > > The overhead logging only happens when event is going to be
> > > > > disabled or the task is scheduling out. It should not be much and
> > expensive.
> > > > >
> > > > > Peter,
> > > > >
> > > > > What do you think?
> > > > > Should we make it configurable?
> > > >
> > > > Is there a downside to having it optional?
> > >
> > > NO. There will be no overhead information dumped into perf.data. Just
> > > like current implementation.
> > 
> > old perf tools will get unexpected events if it's not optional
> >
> 
> That's expected because a new record type is introduced.

but not if it's configurable.. older tools wont get it

jirka

[toc] | [prev] | [next] | [standalone]

#1529366 — RE: [PATCH 01/14] perf/x86: Introduce PERF_RECORD_OVERHEAD

> On Thu, Nov 24, 2016 at 01:56:51PM +0000, Liang, Kan wrote:
> >
> >
> > >
> > > On Thu, Nov 24, 2016 at 01:45:28PM +0000, Liang, Kan wrote:
> > > > > I think we should make this optional/configurable like the rest
> > > > > of the aux events, like below..
> > > > >
> > > >
> > > > The overhead logging only happens when event is going to be
> > > > disabled or the task is scheduling out. It should not be much and
> expensive.
> > > >
> > > > Peter,
> > > >
> > > > What do you think?
> > > > Should we make it configurable?
> > >
> > > Is there a downside to having it optional?
> >
> > NO. There will be no overhead information dumped into perf.data. Just
> > like current implementation.
> 
> old perf tools will get unexpected events if it's not optional
>

That's expected because a new record type is introduced.
 
OK, I will make it configurable. Anyway, there is nothing to lose. :)

Thanks,
Kan

> [root@ibm-x3650m4-01 perf]# ./perf c2c record -ag ^C[ perf record: Woken
> up 1 times to write data ]
> Warning:
> Found 1 unknown events!
> 
> Is this an older tool processing a perf.data file generated by a more recent
> tool?
> 
> If that is not the case, consider reporting to linux-kernel@vger.kernel.org.
> 
> [ perf record: Captured and wrote 2.333 MB perf.data (9370 samples) ]
> 
> 
> jirka

[toc] | [prev] | [next] | [standalone]

#1529610 — Re: [PATCH 01/14] perf/x86: Introduce PERF_RECORD_OVERHEAD

On Thu, Nov 24, 2016 at 02:50:43PM +0100, Peter Zijlstra wrote:
> On Thu, Nov 24, 2016 at 01:45:28PM +0000, Liang, Kan wrote:
> > > I think we should make this optional/configurable like the rest of the aux
> > > events, like below..
> > > 
> > 
> > The overhead logging only happens when event is going to be disabled or
> > the task is scheduling out. It should not be much and expensive.
> > 
> > Peter,
> > 
> > What do you think?
> > Should we make it configurable?
> 
> Is there a downside to having it optional?

It would be good to always have at least one line overhead summary in the
default output. So if someone sends you a perf report output file and it has
suspicious overhead can investigate.

-Andi

[toc] | [prev] | [next] | [standalone]

#1529621 — Re: [PATCH 01/14] perf/x86: Introduce PERF_RECORD_OVERHEAD

On Thu, Nov 24, 2016 at 10:28:22AM -0800, Andi Kleen wrote:
> On Thu, Nov 24, 2016 at 02:50:43PM +0100, Peter Zijlstra wrote:
> > On Thu, Nov 24, 2016 at 01:45:28PM +0000, Liang, Kan wrote:
> > > > I think we should make this optional/configurable like the rest of the aux
> > > > events, like below..
> > > > 
> > > 
> > > The overhead logging only happens when event is going to be disabled or
> > > the task is scheduling out. It should not be much and expensive.
> > > 
> > > Peter,
> > > 
> > > What do you think?
> > > Should we make it configurable?
> > 
> > Is there a downside to having it optional?
> 
> It would be good to always have at least one line overhead summary in the
> default output. So if someone sends you a perf report output file and it has
> suspicious overhead can investigate.

Sure, but that's a tool thing, totally irrelevant for the kernel.

[toc] | [prev] | [next] | [standalone]

#1529628 — Re: [PATCH 01/14] perf/x86: Introduce PERF_RECORD_OVERHEAD

On Thu, Nov 24, 2016 at 07:58:49PM +0100, Peter Zijlstra wrote:
> On Thu, Nov 24, 2016 at 10:28:22AM -0800, Andi Kleen wrote:
> > On Thu, Nov 24, 2016 at 02:50:43PM +0100, Peter Zijlstra wrote:
> > > On Thu, Nov 24, 2016 at 01:45:28PM +0000, Liang, Kan wrote:
> > > > > I think we should make this optional/configurable like the rest of the aux
> > > > > events, like below..
> > > > > 
> > > > 
> > > > The overhead logging only happens when event is going to be disabled or
> > > > the task is scheduling out. It should not be much and expensive.
> > > > 
> > > > Peter,
> > > > 
> > > > What do you think?
> > > > Should we make it configurable?
> > > 
> > > Is there a downside to having it optional?
> > 
> > It would be good to always have at least one line overhead summary in the
> > default output. So if someone sends you a perf report output file and it has
> > suspicious overhead can investigate.
> 
> Sure, but that's a tool thing, totally irrelevant for the kernel.

It requires enabling the overhead records by default.

If there is really concern about not supporting old tool would need a
disable instead of an enable option.

-Andi

[toc] | [prev] | [next] | [standalone]

#1529629 — Re: [PATCH 01/14] perf/x86: Introduce PERF_RECORD_OVERHEAD

On Thu, Nov 24, 2016 at 11:02:16AM -0800, Andi Kleen wrote:
> > Sure, but that's a tool thing, totally irrelevant for the kernel.
> 
> It requires enabling the overhead records by default.

by the tool, not the kernel.

[toc] | [prev] | [next] | [standalone]

#1528646 — [PATCH 11/14] perf tools: record write data overhead

From: Kan Liang <kan.liang@intel.com>

perf record writes all kinds of events to file, which brings overhead as
well. Introducing PERF_USER_WRITE_OVERHEAD to save the overhead
information.
The overhead is measured by the time taken by the write itself, not the
elapsed system time.
Caculate the total user write overhead on each CPU, and display them in
perf report

Signed-off-by: Kan Liang <kan.liang@intel.com>
---
 tools/perf/builtin-record.c | 61 +++++++++++++++++++++++++++++++++++++++++++--
 tools/perf/util/event.h     |  9 +++++++
 tools/perf/util/machine.c   |  5 ++++
 tools/perf/util/session.c   |  4 +++
 4 files changed, 77 insertions(+), 2 deletions(-)

diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index 67d2a90..492058e 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -45,6 +45,11 @@
 #include <asm/bug.h>
 #include <linux/time64.h>
 
+struct write_overhead{
+	u64	nr;
+	u64	time;
+};
+
 struct record {
 	struct perf_tool	tool;
 	struct record_opts	opts;
@@ -63,17 +68,40 @@ struct record {
 	bool			timestamp_filename;
 	bool			switch_output;
 	unsigned long long	samples;
+	struct write_overhead	overhead[MAX_NR_CPUS];
 };
 
+static u64 get_vnsecs(void)
+{
+	struct timespec ts;
+
+	clock_gettime(CLOCK_PROCESS_CPUTIME_ID, &ts);
+
+	return ts.tv_sec * NSEC_PER_SEC + ts.tv_nsec;
+}
+
 static int record__write(struct record *rec, void *bf, size_t size)
 {
+	int cpu = sched_getcpu();
+	u64 start, end;
+	int ret = 0;
+
+	start = get_vnsecs();
 	if (perf_data_file__write(rec->session->file, bf, size) < 0) {
 		pr_err("failed to write perf data, error: %m\n");
-		return -1;
+		ret = -1;
+		goto done;
 	}
 
 	rec->bytes_written += size;
-	return 0;
+done:
+	end = get_vnsecs();
+	if (cpu >= 0) {
+		rec->overhead[cpu].nr++;
+		rec->overhead[cpu].time += (end - start);
+	}
+
+	return ret;
 }
 
 static int process_synthesized_event(struct perf_tool *tool,
@@ -813,6 +841,33 @@ static int record__synthesize(struct record *rec, bool tail)
 	return err;
 }
 
+static void perf_event__synth_overhead(struct record *rec, perf_event__handler_t process)
+{
+	int cpu;
+
+	union perf_event event = {
+		.overhead = {
+			.header = {
+				.type = PERF_RECORD_USER_OVERHEAD,
+				.size = sizeof(struct perf_overhead),
+			},
+		},
+	};
+
+	event.overhead.type = PERF_USER_WRITE_OVERHEAD;
+
+	for (cpu = 0; cpu < MAX_NR_CPUS; cpu++) {
+		if (!rec->overhead[cpu].nr)
+			continue;
+
+		event.overhead.entry.cpu = cpu;
+		event.overhead.entry.nr = rec->overhead[cpu].nr;
+		event.overhead.entry.time = rec->overhead[cpu].time;
+
+		(void)process(&rec->tool, &event, NULL, NULL);
+	}
+}
+
 static int __cmd_record(struct record *rec, int argc, const char **argv)
 {
 	int err;
@@ -1074,6 +1129,8 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 		goto out_child;
 	}
 
+	perf_event__synth_overhead(rec, process_synthesized_event);
+
 	if (!quiet)
 		fprintf(stderr, "[ perf record: Woken up %ld times to write data ]\n", waking);
 
diff --git a/tools/perf/util/event.h b/tools/perf/util/event.h
index 1ef1a9d..9927cf9 100644
--- a/tools/perf/util/event.h
+++ b/tools/perf/util/event.h
@@ -272,6 +272,9 @@ enum auxtrace_error_type {
  * events on each CPU.
  * The total number of accessing side-band events handler function is stored
  * in [0], while the accumulated processing time is in [1].
+ * The total_user_write_overhead tells exactly the overhead to write data in
+ * perf record.
+ * The total write# is stored in [0], while the accumulated time is in [1].
  */
 struct events_stats {
 	u64 total_period;
@@ -283,6 +286,7 @@ struct events_stats {
 	u64 total_nmi_overhead[MAX_NR_CPUS][2];
 	u64 total_mux_overhead[MAX_NR_CPUS][2];
 	u64 total_sb_overhead[MAX_NR_CPUS][2];
+	u64 total_user_write_overhead[MAX_NR_CPUS][2];
 	u32 nr_events[PERF_RECORD_HEADER_MAX];
 	u32 nr_non_filtered_samples;
 	u32 nr_lost_warned;
@@ -493,6 +497,11 @@ struct time_conv_event {
 	u64 time_zero;
 };
 
+enum perf_user_overhead_event_type { /* above any possible kernel type */
+	PERF_USER_OVERHEAD_TYPE_START	= 100,
+	PERF_USER_WRITE_OVERHEAD	= 100,
+};
+
 struct perf_overhead {
 	struct perf_event_header	header;
 	u32				type;
diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c
index d8cde21..ce7a0ea 100644
--- a/tools/perf/util/machine.c
+++ b/tools/perf/util/machine.c
@@ -573,6 +573,11 @@ int machine__process_overhead_event(struct machine *machine __maybe_unused,
 			    event->overhead.entry.nr,
 			    event->overhead.entry.time,
 			    event->overhead.entry.cpu);
+	} else if (event->overhead.type == PERF_USER_WRITE_OVERHEAD) {
+		dump_printf(" User write nr: %llu  time: %llu cpu %u\n",
+			    event->overhead.entry.nr,
+			    event->overhead.entry.time,
+			    event->overhead.entry.cpu);
 	} else {
 		dump_printf("\tUNSUPPORT OVERHEAD TYPE 0x%x!\n", event->overhead.type);
 	}
diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index 27a5c8a..a72992b 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -1227,6 +1227,10 @@ overhead_stats_update(struct perf_tool *tool,
 			evlist->stats.total_sb_overhead[event->overhead.entry.cpu][0] += event->overhead.entry.nr;
 			evlist->stats.total_sb_overhead[event->overhead.entry.cpu][1] += event->overhead.entry.time;
 			break;
+		case PERF_USER_WRITE_OVERHEAD:
+			evlist->stats.total_user_write_overhead[event->overhead.entry.cpu][0] += event->overhead.entry.nr;
+			evlist->stats.total_user_write_overhead[event->overhead.entry.cpu][1] += event->overhead.entry.time;
+			break;
 		default:
 			break;
 		}
-- 
2.5.5

[toc] | [prev] | [next] | [standalone]

#1528838 — Re: [PATCH 11/14] perf tools: record write data overhead

On Wed, Nov 23, 2016 at 04:44:49AM -0500, kan.liang@intel.com wrote:

SNIP

>  
> +static void perf_event__synth_overhead(struct record *rec, perf_event__handler_t process)
> +{
> +	int cpu;
> +
> +	union perf_event event = {
> +		.overhead = {
> +			.header = {
> +				.type = PERF_RECORD_USER_OVERHEAD,
> +				.size = sizeof(struct perf_overhead),
> +			},
> +		},
> +	};
> +
> +	event.overhead.type = PERF_USER_WRITE_OVERHEAD;
> +
> +	for (cpu = 0; cpu < MAX_NR_CPUS; cpu++) {
> +		if (!rec->overhead[cpu].nr)
> +			continue;
> +
> +		event.overhead.entry.cpu = cpu;
> +		event.overhead.entry.nr = rec->overhead[cpu].nr;
> +		event.overhead.entry.time = rec->overhead[cpu].time;
> +
> +		(void)process(&rec->tool, &event, NULL, NULL);

plese make perf_event__synth_overhead return int and check above 'process' return value

jirka

[toc] | [prev] | [next] | [standalone]

#1528849 — Re: [PATCH 11/14] perf tools: record write data overhead

On Wed, Nov 23, 2016 at 04:44:49AM -0500, kan.liang@intel.com wrote:

SNIP

> diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c
> index d8cde21..ce7a0ea 100644
> --- a/tools/perf/util/machine.c
> +++ b/tools/perf/util/machine.c
> @@ -573,6 +573,11 @@ int machine__process_overhead_event(struct machine *machine __maybe_unused,
>  			    event->overhead.entry.nr,
>  			    event->overhead.entry.time,
>  			    event->overhead.entry.cpu);
> +	} else if (event->overhead.type == PERF_USER_WRITE_OVERHEAD) {
> +		dump_printf(" User write nr: %llu  time: %llu cpu %u\n",
> +			    event->overhead.entry.nr,
> +			    event->overhead.entry.time,
> +			    event->overhead.entry.cpu);

should this be rather in the switch ?

jirka

[toc] | [prev] | [next] | [standalone]

#1528647 — [PATCH 04/14] perf/x86: output side-band events overhead

From: Kan Liang <kan.liang@intel.com>

Iterating all events which need to receive side-band events also bring
some overhead.
Save the overhead information in task context or CPU context, whichever
context is available.

Signed-off-by: Kan Liang <kan.liang@intel.com>
---
 include/linux/perf_event.h      |  2 ++
 include/uapi/linux/perf_event.h |  1 +
 kernel/events/core.c            | 32 ++++++++++++++++++++++++++++----
 3 files changed, 31 insertions(+), 4 deletions(-)

diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h
index f72b97a..ec3cb7f 100644
--- a/include/linux/perf_event.h
+++ b/include/linux/perf_event.h
@@ -764,6 +764,8 @@ struct perf_event_context {
 #endif
 	void				*task_ctx_data; /* pmu specific data */
 	struct rcu_head			rcu_head;
+
+	struct perf_overhead_entry	sb_overhead;
 };
 
 /*
diff --git a/include/uapi/linux/perf_event.h b/include/uapi/linux/perf_event.h
index 9124c7c..5e7c522 100644
--- a/include/uapi/linux/perf_event.h
+++ b/include/uapi/linux/perf_event.h
@@ -994,6 +994,7 @@ struct perf_branch_entry {
 enum perf_record_overhead_type {
 	PERF_NMI_OVERHEAD	= 0,
 	PERF_MUX_OVERHEAD,
+	PERF_SB_OVERHEAD,
 
 	PERF_OVERHEAD_MAX,
 };
diff --git a/kernel/events/core.c b/kernel/events/core.c
index 9934059..51e9df7 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -1829,9 +1829,15 @@ event_sched_out(struct perf_event *event,
 	if (event->attr.exclusive || !cpuctx->active_oncpu)
 		cpuctx->exclusive = 0;
 
-	if (log_overhead && cpuctx->mux_overhead.nr) {
-		cpuctx->mux_overhead.cpu = smp_processor_id();
-		perf_log_overhead(event, PERF_MUX_OVERHEAD, &cpuctx->mux_overhead);
+	if (log_overhead) {
+		if (cpuctx->mux_overhead.nr) {
+			cpuctx->mux_overhead.cpu = smp_processor_id();
+			perf_log_overhead(event, PERF_MUX_OVERHEAD, &cpuctx->mux_overhead);
+		}
+		if (ctx->sb_overhead.nr) {
+			ctx->sb_overhead.cpu = smp_processor_id();
+			perf_log_overhead(event, PERF_SB_OVERHEAD, &ctx->sb_overhead);
+		}
 	}
 
 	perf_pmu_enable(event->pmu);
@@ -6133,6 +6139,14 @@ static void perf_iterate_sb_cpu(perf_iterate_f output, void *data)
 	}
 }
 
+static void
+perf_caculate_sb_overhead(struct perf_event_context *ctx,
+			  u64 time)
+{
+	ctx->sb_overhead.nr++;
+	ctx->sb_overhead.time += time;
+}
+
 /*
  * Iterate all events that need to receive side-band events.
  *
@@ -6143,9 +6157,12 @@ static void
 perf_iterate_sb(perf_iterate_f output, void *data,
 	       struct perf_event_context *task_ctx)
 {
+	struct perf_event_context *overhead_ctx = task_ctx;
 	struct perf_event_context *ctx;
+	u64 start_clock, end_clock;
 	int ctxn;
 
+	start_clock = perf_clock();
 	rcu_read_lock();
 	preempt_disable();
 
@@ -6163,12 +6180,19 @@ perf_iterate_sb(perf_iterate_f output, void *data,
 
 	for_each_task_context_nr(ctxn) {
 		ctx = rcu_dereference(current->perf_event_ctxp[ctxn]);
-		if (ctx)
+		if (ctx) {
 			perf_iterate_ctx(ctx, output, data, false);
+			if (!overhead_ctx)
+				overhead_ctx = ctx;
+		}
 	}
 done:
 	preempt_enable();
 	rcu_read_unlock();
+
+	end_clock = perf_clock();
+	if (overhead_ctx)
+		perf_caculate_sb_overhead(overhead_ctx, end_clock - start_clock);
 }
 
 /*
-- 
2.5.5

[toc] | [prev] | [next] | [standalone]

#1528741 — Re: [PATCH 04/14] perf/x86: output side-band events overhead

On Wed, Nov 23, 2016 at 04:44:42AM -0500, kan.liang@intel.com wrote:
> From: Kan Liang <kan.liang@intel.com>
> 
> Iterating all events which need to receive side-band events also bring
> some overhead.
> Save the overhead information in task context or CPU context, whichever
> context is available.
> 
> Signed-off-by: Kan Liang <kan.liang@intel.com>
> ---
>  include/linux/perf_event.h      |  2 ++
>  include/uapi/linux/perf_event.h |  1 +
>  kernel/events/core.c            | 32 ++++++++++++++++++++++++++++----
>  3 files changed, 31 insertions(+), 4 deletions(-)

Guess?

[toc] | [prev] | [next] | [standalone]

Page 1 of 3  [1] 2 3  Next page →

Back to top | Article view | linux.kernel

csiph-web