Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1346223 > unrolled thread

perf, tools: Refactor and support interval and CSV metrics

Started byAndi Kleen <andi@firstfloor.org>
First post2016-02-29 23:40 +0100
Last post2016-03-01 14:10 +0100
Articles 6 — 3 participants

Back to article view | Back to linux.kernel

Contents

  perf, tools: Refactor and support interval and CSV metrics Andi Kleen <andi@firstfloor.org> - 2016-02-29 23:40 +0100
    [PATCH 2/7] perf, tools, stat: Implement CSV metrics output Andi Kleen <andi@firstfloor.org> - 2016-02-29 23:40 +0100
      Re: [PATCH 2/7] perf, tools, stat: Implement CSV metrics output Jiri Olsa <jolsa@redhat.com> - 2016-03-01 13:40 +0100
    [PATCH 4/7] perf, tools, stat: Document CSV format in manpage Andi Kleen <andi@firstfloor.org> - 2016-02-29 23:40 +0100
      Re: [PATCH 4/7] perf, tools, stat: Document CSV format in manpage Jiri Olsa <jolsa@redhat.com> - 2016-03-01 13:40 +0100
    Re: perf, tools: Refactor and support interval and CSV metrics Arnaldo Carvalho de Melo <acme@kernel.org> - 2016-03-01 14:10 +0100

#1346223 — perf, tools: Refactor and support interval and CSV metrics

FromAndi Kleen <andi@firstfloor.org>
Date2016-02-29 23:40 +0100
Subjectperf, tools: Refactor and support interval and CSV metrics
Message-ID<r7EDT-zZ-3@gated-at.bofh.it>
Fixed last feedback.

[v5: Fix mainly bisect problems. No regressions introduced by one
patch and fixed again later. Some minor fixes in addition]
[v6: Fix running/noise printing patch.]
[v7: Reorder and merge two patches to avoid a bisect hole where unsupported was
printed as 0]
[v8: Minor fixes for review feedback. See changelog in patches.]
[v9: Fix newline bug. Add support for -A for --metric-only]
[v10: Remove extra "noise" printing (Jiri)
      Fix fields in documentation (Jiri)]
[v11: Fix manpage again. Avoid extra metric output in CSV mode.]
[v12: Move CSV metrics fields to after running/enabled/variance.
      Minor fixes.]
[v13: Address review comments. Now probe for stalled events 
      in advance to avoid empty columns or lines. Fix -A shadowing.
      Various minor changes. Drop merged patches.]
[v14: Fix empty lines with CSV metrics. Avoid one more empty column
      in metric-only.]

Currently perf stat does not support printing computed metrics for interval (-I xxx)
or CSV (-x,) mode. For example IPC or TSX metrics over time are quite useful to know.

This patch implements them. The main obstacle was that the
metrics printing was all open coded all over the metrics computation code.
The second patch refactors the metrics printing to work through call backs that
can be more easily changed. This also cleans up the metrics printing significantly.
The indentation is now handled through printf, no more need to manually count spaces.

Then based on that it implements metrics printing for CSV and interval mode,
and finally a --metric-only mode.

Example output:

% perf stat  -I1000 -a sleep 1
#          time              counts unit events                    metric                              multiplex
     1.001301370       12020.049593      task-clock (msec)                                             (100.00%)
     1.001301370              3,952      context-switches          #    0.329 K/sec                    (100.00%)
     1.001301370                 69      cpu-migrations            #    0.006 K/sec                    (100.00%)
     1.001301370                 76      page-faults               #    0.006 K/sec                  
     1.001301370        386,582,789      cycles                    #    0.032 GHz                      (100.00%)
     1.001301370        716,441,544      stalled-cycles-frontend   #  185.33% frontend cycles idle     (100.00%)
     1.001301370    <not supported>      stalled-cycles-backend   
     1.001301370        101,751,678      instructions              #    0.26  insn per cycle         
     1.001301370                                                   #    7.04  stalled cycles per insn  (100.00%)
     1.001301370         20,914,692      branches                  #    1.740 M/sec                    (100.00%)
     1.001301370          1,943,630      branch-misses             #    9.29% of all branches        

CSV mode:

% perf stat  -x, -I1000 -a sleep 1
     1.000982778,12006.549977,,task-clock,12006547787,100.00,,,,
     1.000982778,12822,,context-switches,12007100604,100.00,0.001,M/sec
     1.000982778,175,,cpu-migrations,12007180306,100.00,0.015,K/sec
     1.000982778,3404,,page-faults,12007185482,100.00,0.284,K/sec
     1.000982778,1930307489,,cycles,12007018233,100.00,0.161,GHz
     1.000982778,6971803638,,stalled-cycles-frontend,12006902870,100.00,361.18,frontend cycles idle
     1.000982778,464493941,,instructions,12006873327,100.00,0.24,insn per cycle
     1.000982778,,,,,,15.01,stalled cycles per insn
     1.000982778,86548409,,branches,12006758420,100.00,7.208,M/sec
     1.000982778,4933638,,branch-misses,12006648104,100.00,5.70,of all branches

Now includes metrics

Metric only mode:

Concicse information if you only care about computed metrics, not raw values

% perf stat --metric-only  -a -I 1000
         1.001452803 frontend cycles idle insn per cycle       stalled cycles per insn branch-misses of all branches
         1.001452803  158.91%               0.66                2.39                    2.92%
         2.002192321  180.63%               0.76                2.08                    2.96%
         3.003088282  150.59%               0.62                2.57                    2.84%
         4.004369835  196.20%               0.98                1.62                    3.79%
         5.005227314  231.98%               0.84                1.90                    4.71%


Metric only mode in CSV (flat format, easy to plot and analyze in statistical tools like JMP, R, pandas, gnuplot):

% perf stat -x, --metric-only  -a -I 1000
     1.001381652,frontend cycles idle,insn per cycle,stalled cycles per insn,branch-misses of all branches,
     1.001381652,173.32,0.83,2.09,1.73,
     2.002073343,199.47,1.07,1.60,2.14,
     3.002875524,109.52,0.22,7.83,1.63,
     4.003970059,132.10,0.17,10.85,1.51,
     5.004818754,181.60,0.22,8.87,2.22,


Available in
git://git.kernel.org/pub/scm/linux/kernel/git/ak/linux-misc-2.6 perf/stat-metrics-18

[toc] | [next] | [standalone]

#1346226 — [PATCH 2/7] perf, tools, stat: Implement CSV metrics output

FromAndi Kleen <andi@firstfloor.org>
Date2016-02-29 23:40 +0100
Subject[PATCH 2/7] perf, tools, stat: Implement CSV metrics output
Message-ID<r7EDV-zZ-45@gated-at.bofh.it>
In reply to#1346223
From: Andi Kleen <ak@linux.intel.com>

Now support CSV output for metrics. With the new output callbacks
this is relatively straight forward by creating new callbacks.

This allows to easily plot metrics from CSV files.

The new line callback needs to know the number of fields to skip them
correctly

Example output before:

% perf stat -x, true
0.200687,,task-clock,200687,100.00
0,,context-switches,200687,100.00
0,,cpu-migrations,200687,100.00
40,,page-faults,200687,100.00
730871,,cycles,203601,100.00
551056,,stalled-cycles-frontend,203601,100.00
<not supported>,,stalled-cycles-backend,0,100.00
385523,,instructions,203601,100.00
78028,,branches,203601,100.00
3946,,branch-misses,203601,100.00

After:

% perf stat -x, true
.502457,,task-clock,502457,100.00,0.485,CPUs utilized
0,,context-switches,502457,100.00,0.000,K/sec
0,,cpu-migrations,502457,100.00,0.000,K/sec
45,,page-faults,502457,100.00,0.090,M/sec
644692,,cycles,509102,100.00,1.283,GHz
423470,,stalled-cycles-frontend,509102,100.00,65.69,frontend cycles idle
<not supported>,,stalled-cycles-backend,0,100.00,,,,
492701,,instructions,509102,100.00,0.76,insn per cycle
,,,,,0.86,stalled cycles per insn
97767,,branches,509102,100.00,194.578,M/sec
4788,,branch-misses,509102,100.00,4.90,of all branches

or easier readable

perf stat  -x, -o x.csv true
[ak@tassilo hle]$ column -s, -t x.csv
0.490635                                 task-clock               490635  100.00  0.489    CPUs utilized
0                                        context-switches         490635  100.00  0.000    K/sec
0                                        cpu-migrations           490635  100.00  0.000    K/sec
45                                       page-faults              490635  100.00  0.092    M/sec
629080                                   cycles                   497698  100.00  1.282    GHz
409498                                   stalled-cycles-frontend  497698  100.00  65.09    frontend cycles idle
<not supported>                          stalled-cycles-backend   0       100.00
491424                                   instructions             497698  100.00  0.78     insn per cycle
                                                                                  0.83     stalled cycles per insn
97278                                    branches                 497698  100.00  198.270  M/sec
4569                                     branch-misses            497698  100.00  4.70     of all branches

Two new fields are added: metric value and metric name.

v2: Split out function argument changes
v3: Reenable metrics for real.
v4: Fix wrong hunk from refactoring.
v5: Remove extra "noise" printing (Jiri), but add it to the not counted case.
Print empty metrics for not counted.
v6: Avoid outputting metric on empty format.
v7: Print metric at the end
v8: Remove extra run, ena fields
v9: Avoid extra new line for unsupported counters
Signed-off-by: Andi Kleen <ak@linux.intel.com>
---
 tools/perf/builtin-stat.c     | 73 ++++++++++++++++++++++++++++++++++++++++---
 tools/perf/util/stat-shadow.c |  2 +-
 2 files changed, 70 insertions(+), 5 deletions(-)

diff --git a/tools/perf/builtin-stat.c b/tools/perf/builtin-stat.c
index 24f222d..2ffb822 100644
--- a/tools/perf/builtin-stat.c
+++ b/tools/perf/builtin-stat.c
@@ -739,6 +739,7 @@ struct outstate {
 	FILE *fh;
 	bool newline;
 	const char *prefix;
+	int  nfields;
 };
 
 #define METRIC_LEN  35
@@ -789,6 +790,43 @@ static void print_metric_std(void *ctx, const char *color, const char *fmt,
 	fprintf(out, " %-*s", METRIC_LEN - n - 1, unit);
 }
 
+static void new_line_csv(void *ctx)
+{
+	struct outstate *os = ctx;
+	int i;
+
+	fputc('\n', os->fh);
+	if (os->prefix)
+		fprintf(os->fh, "%s%s", os->prefix, csv_sep);
+	for (i = 0; i < os->nfields; i++)
+		fputs(csv_sep, os->fh);
+}
+
+static void print_metric_csv(void *ctx,
+			     const char *color __maybe_unused,
+			     const char *fmt, const char *unit, double val)
+{
+	struct outstate *os = ctx;
+	FILE *out = os->fh;
+	char buf[64], *vals, *ends;
+
+	if (unit == NULL || fmt == NULL) {
+		fprintf(out, "%s%s%s%s", csv_sep, csv_sep, csv_sep, csv_sep);
+		return;
+	}
+	snprintf(buf, sizeof(buf), fmt, val);
+	vals = buf;
+	while (isspace(*vals))
+		vals++;
+	ends = vals;
+	while (isdigit(*ends) || *ends == '.')
+		ends++;
+	*ends = 0;
+	while (isspace(*unit))
+		unit++;
+	fprintf(out, "%s%s%s%s", csv_sep, vals, csv_sep, unit);
+}
+
 static void nsec_printout(int id, int nr, struct perf_evsel *evsel, double avg)
 {
 	FILE *output = stat_config.output;
@@ -860,6 +898,22 @@ static void printout(int id, int nr, struct perf_evsel *counter, double uval,
 
 	nl = new_line_std;
 
+	if (csv_output) {
+		static int aggr_fields[] = {
+			[AGGR_GLOBAL] = 0,
+			[AGGR_THREAD] = 1,
+			[AGGR_NONE] = 1,
+			[AGGR_SOCKET] = 2,
+			[AGGR_CORE] = 2,
+		};
+
+		pm = print_metric_csv;
+		nl = new_line_csv;
+		os.nfields = 3;
+		os.nfields += aggr_fields[stat_config.aggr_mode];
+		if (counter->cgrp)
+			os.nfields++;
+	}
 	if (run == 0 || ena == 0 || counter->counts->scaled == -1) {
 		aggr_printout(counter, id, nr);
 
@@ -880,7 +934,12 @@ static void printout(int id, int nr, struct perf_evsel *counter, double uval,
 			fprintf(stat_config.output, "%s%s",
 				csv_sep, counter->cgrp->name);
 
+		if (!csv_output)
+			pm(&os, NULL, NULL, "", 0);
+		print_noise(counter, noise);
 		print_running(run, ena);
+		if (csv_output)
+			pm(&os, NULL, NULL, "", 0);
 		return;
 	}
 
@@ -893,14 +952,20 @@ static void printout(int id, int nr, struct perf_evsel *counter, double uval,
 	out.new_line = nl;
 	out.ctx = &os;
 
-	if (!csv_output)
-		perf_stat__print_shadow_stats(counter, uval,
+	if (csv_output) {
+		print_noise(counter, noise);
+		print_running(run, ena);
+	}
+
+	perf_stat__print_shadow_stats(counter, uval,
 				stat_config.aggr_mode == AGGR_GLOBAL ? 0 :
 				cpu_map__id_to_cpu(id),
 				&out);
 
-	print_noise(counter, noise);
-	print_running(run, ena);
+	if (!csv_output) {
+		print_noise(counter, noise);
+		print_running(run, ena);
+	}
 }
 
 static void print_aggr(char *prefix)
diff --git a/tools/perf/util/stat-shadow.c b/tools/perf/util/stat-shadow.c
index 4d8f185..367e220 100644
--- a/tools/perf/util/stat-shadow.c
+++ b/tools/perf/util/stat-shadow.c
@@ -310,8 +310,8 @@ void perf_stat__print_shadow_stats(struct perf_evsel *evsel,
 		total = avg_stats(&runtime_stalled_cycles_front_stats[ctx][cpu]);
 		total = max(total, avg_stats(&runtime_stalled_cycles_back_stats[ctx][cpu]));
 
-		out->new_line(ctxp);
 		if (total && avg) {
+			out->new_line(ctxp);
 			ratio = total / avg;
 			print_metric(ctxp, NULL, "%7.2f ",
 					"stalled cycles per insn",
-- 
2.5.0

[toc] | [prev] | [next] | [standalone]

#1346575 — Re: [PATCH 2/7] perf, tools, stat: Implement CSV metrics output

FromJiri Olsa <jolsa@redhat.com>
Date2016-03-01 13:40 +0100
SubjectRe: [PATCH 2/7] perf, tools, stat: Implement CSV metrics output
Message-ID<r7RKO-Hz-11@gated-at.bofh.it>
In reply to#1346226
On Mon, Feb 29, 2016 at 02:36:21PM -0800, Andi Kleen wrote:
> From: Andi Kleen <ak@linux.intel.com>
> 
> Now support CSV output for metrics. With the new output callbacks
> this is relatively straight forward by creating new callbacks.
> 
> This allows to easily plot metrics from CSV files.
> 
> The new line callback needs to know the number of fields to skip them
> correctly
> 
> Example output before:
> 
> % perf stat -x, true
> 0.200687,,task-clock,200687,100.00
> 0,,context-switches,200687,100.00
> 0,,cpu-migrations,200687,100.00
> 40,,page-faults,200687,100.00
> 730871,,cycles,203601,100.00
> 551056,,stalled-cycles-frontend,203601,100.00
> <not supported>,,stalled-cycles-backend,0,100.00
> 385523,,instructions,203601,100.00
> 78028,,branches,203601,100.00
> 3946,,branch-misses,203601,100.00
> 
> After:
> 
> % perf stat -x, true
> .502457,,task-clock,502457,100.00,0.485,CPUs utilized
> 0,,context-switches,502457,100.00,0.000,K/sec
> 0,,cpu-migrations,502457,100.00,0.000,K/sec
> 45,,page-faults,502457,100.00,0.090,M/sec
> 644692,,cycles,509102,100.00,1.283,GHz
> 423470,,stalled-cycles-frontend,509102,100.00,65.69,frontend cycles idle
> <not supported>,,stalled-cycles-backend,0,100.00,,,,
> 492701,,instructions,509102,100.00,0.76,insn per cycle
> ,,,,,0.86,stalled cycles per insn
> 97767,,branches,509102,100.00,194.578,M/sec
> 4788,,branch-misses,509102,100.00,4.90,of all branches
> 
> or easier readable
> 
> perf stat  -x, -o x.csv true
> [ak@tassilo hle]$ column -s, -t x.csv
> 0.490635                                 task-clock               490635  100.00  0.489    CPUs utilized
> 0                                        context-switches         490635  100.00  0.000    K/sec
> 0                                        cpu-migrations           490635  100.00  0.000    K/sec
> 45                                       page-faults              490635  100.00  0.092    M/sec
> 629080                                   cycles                   497698  100.00  1.282    GHz
> 409498                                   stalled-cycles-frontend  497698  100.00  65.09    frontend cycles idle
> <not supported>                          stalled-cycles-backend   0       100.00
> 491424                                   instructions             497698  100.00  0.78     insn per cycle
>                                                                                   0.83     stalled cycles per insn
> 97278                                    branches                 497698  100.00  198.270  M/sec
> 4569                                     branch-misses            497698  100.00  4.70     of all branches
> 
> Two new fields are added: metric value and metric name.

Acked-by: Jiri Olsa <jolsa@kernel.org>

thanks,
jirka

[toc] | [prev] | [next] | [standalone]

#1346227 — [PATCH 4/7] perf, tools, stat: Document CSV format in manpage

FromAndi Kleen <andi@firstfloor.org>
Date2016-02-29 23:40 +0100
Subject[PATCH 4/7] perf, tools, stat: Document CSV format in manpage
Message-ID<r7EDV-zZ-47@gated-at.bofh.it>
In reply to#1346223
From: Andi Kleen <ak@linux.intel.com>

With all the recently added fields in the perf stat CSV output
we should finally document them in the man page. Do this here.

v2: Fix fields in documentation (Jiri)
v3: fix order of fields again (Jiri)
v4: Change order again.
Signed-off-by: Andi Kleen <ak@linux.intel.com>
---
 tools/perf/Documentation/perf-stat.txt | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/tools/perf/Documentation/perf-stat.txt b/tools/perf/Documentation/perf-stat.txt
index 52ef7a9..3ae7907 100644
--- a/tools/perf/Documentation/perf-stat.txt
+++ b/tools/perf/Documentation/perf-stat.txt
@@ -211,6 +211,27 @@ $ perf stat -- make -j
 
  Wall-clock time elapsed:   719.554352 msecs
 
+CSV FORMAT
+----------
+
+With -x, perf stat is able to output a not-quite-CSV format output
+Commas in the output are not put into "". To make it easy to parse
+it is recommended to use a different character like -x \;
+
+The fields are in this order:
+
+	- optional usec time stamp in fractions of second (with -I xxx)
+	- counter value
+	- unit of the counter value or empty
+	- event name
+	- run time of counter
+	- percentage of measurement time the counter was running
+	- optional variance if multiple values are collected with -r
+	- optional metric value
+	- optional unit of metric
+
+Additional metrics may be printed with all earlier fields being empty.
+
 SEE ALSO
 --------
 linkperf:perf-top[1], linkperf:perf-list[1]
-- 
2.5.0

[toc] | [prev] | [next] | [standalone]

#1346574 — Re: [PATCH 4/7] perf, tools, stat: Document CSV format in manpage

FromJiri Olsa <jolsa@redhat.com>
Date2016-03-01 13:40 +0100
SubjectRe: [PATCH 4/7] perf, tools, stat: Document CSV format in manpage
Message-ID<r7RKO-Hz-9@gated-at.bofh.it>
In reply to#1346227
On Mon, Feb 29, 2016 at 02:36:23PM -0800, Andi Kleen wrote:
> From: Andi Kleen <ak@linux.intel.com>
> 
> With all the recently added fields in the perf stat CSV output
> we should finally document them in the man page. Do this here.
> 
> v2: Fix fields in documentation (Jiri)
> v3: fix order of fields again (Jiri)
> v4: Change order again.
> Signed-off-by: Andi Kleen <ak@linux.intel.com>
> ---
>  tools/perf/Documentation/perf-stat.txt | 21 +++++++++++++++++++++
>  1 file changed, 21 insertions(+)
> 
> diff --git a/tools/perf/Documentation/perf-stat.txt b/tools/perf/Documentation/perf-stat.txt
> index 52ef7a9..3ae7907 100644
> --- a/tools/perf/Documentation/perf-stat.txt
> +++ b/tools/perf/Documentation/perf-stat.txt
> @@ -211,6 +211,27 @@ $ perf stat -- make -j
>  
>   Wall-clock time elapsed:   719.554352 msecs
>  
> +CSV FORMAT
> +----------
> +
> +With -x, perf stat is able to output a not-quite-CSV format output
> +Commas in the output are not put into "". To make it easy to parse
> +it is recommended to use a different character like -x \;
> +
> +The fields are in this order:
> +
> +	- optional usec time stamp in fractions of second (with -I xxx)

there's also optional CPU field in case you do other than GLOBAL aggregation:

[jolsa@krava perf]$ sudo ./perf.old stat -a -A -x, kill
kill: not enough arguments
CPU0,0.629921,,task-clock,629706,100.00


[jolsa@krava perf]$ sudo ./perf.old stat -a --per-core -x, kill
kill: not enough arguments
S0-C0,2,1.168179,,task-clock,1167956,100.00


[jolsa@krava perf]$ sudo ./perf.old stat -a --per-socket -x, kill
kill: not enough arguments
S0,4,2.296581,,task-clock,2296198,100.00

thanks,
jirka

> +	- counter value
> +	- unit of the counter value or empty
> +	- event name
> +	- run time of counter
> +	- percentage of measurement time the counter was running
> +	- optional variance if multiple values are collected with -r
> +	- optional metric value
> +	- optional unit of metric
> +
> +Additional metrics may be printed with all earlier fields being empty.
> +
>  SEE ALSO
>  --------
>  linkperf:perf-top[1], linkperf:perf-list[1]
> -- 
> 2.5.0
>

[toc] | [prev] | [next] | [standalone]

#1346599

FromArnaldo Carvalho de Melo <acme@kernel.org>
Date2016-03-01 14:10 +0100
Message-ID<r7SdQ-18V-17@gated-at.bofh.it>
In reply to#1346223
Em Mon, Feb 29, 2016 at 02:36:19PM -0800, Andi Kleen escreveu:
> Fixed last feedback.
> 
> [v5: Fix mainly bisect problems. No regressions introduced by one
> patch and fixed again later. Some minor fixes in addition]
> [v6: Fix running/noise printing patch.]
> [v7: Reorder and merge two patches to avoid a bisect hole where unsupported was
> printed as 0]
> [v8: Minor fixes for review feedback. See changelog in patches.]
> [v9: Fix newline bug. Add support for -A for --metric-only]
> [v10: Remove extra "noise" printing (Jiri)
>       Fix fields in documentation (Jiri)]
> [v11: Fix manpage again. Avoid extra metric output in CSV mode.]
> [v12: Move CSV metrics fields to after running/enabled/variance.
>       Minor fixes.]
> [v13: Address review comments. Now probe for stalled events 
>       in advance to avoid empty columns or lines. Fix -A shadowing.
>       Various minor changes. Drop merged patches.]
> [v14: Fix empty lines with CSV metrics. Avoid one more empty column
>       in metric-only.]

1/7 was already merged, pushed to Ingo, applied (2,3)/7.

- Arnaldo

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web