Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1288272 > unrolled thread

[PATCHSET 00/16] perf top: Add multi-thread support (v1)

Back to article view | Back to linux.kernel

Contents

  [PATCHSET 00/16] perf top: Add multi-thread support (v1) Namhyung Kim <namhyung@kernel.org> - 2015-12-10 09:00 +0100
    [PATCH/RFC 03/16] perf top: Factor out warnings about kernel addresses and symbols Namhyung Kim <namhyung@kernel.org> - 2015-12-10 09:00 +0100
      Re: [PATCH/RFC 03/16] perf top: Factor out warnings about kernel  addresses and symbols Arnaldo Carvalho de Melo <acme@kernel.org> - 2015-12-10 20:10 +0100
        Re: [PATCH/RFC 03/16] perf top: Factor out warnings about kernel  addresses and symbols Namhyung Kim <namhyung@kernel.org> - 2015-12-14 02:50 +0100
          Re: [PATCH/RFC 03/16] perf top: Factor out warnings about kernel  addresses and symbols Namhyung Kim <namhyung@kernel.org> - 2015-12-14 03:10 +0100
    [PATCH/RFC 14/16] perf top: Separate struct perf_top_stats Namhyung Kim <namhyung@kernel.org> - 2015-12-10 09:00 +0100
    [PATCH/RFC 11/16] perf top: Implement basic parallel processing Namhyung Kim <namhyung@kernel.org> - 2015-12-10 09:00 +0100
      Re: [PATCH/RFC 11/16] perf top: Implement basic parallel processing Jiri Olsa <jolsa@redhat.com> - 2015-12-14 10:30 +0100
        Re: [PATCH/RFC 11/16] perf top: Implement basic parallel processing Jiri Olsa <jolsa@redhat.com> - 2015-12-14 10:40 +0100
          Re: [PATCH/RFC 11/16] perf top: Implement basic parallel processing Namhyung Kim <namhyung@kernel.org> - 2015-12-15 03:10 +0100
    [PATCH/RFC 01/16] perf top: Delete half-processed hist entries when exit Namhyung Kim <namhyung@kernel.org> - 2015-12-10 09:00 +0100
      RE: [PATCH/RFC 01/16] perf top: Delete half-processed hist entries  when exit 平松雅巳 / HIRAMATU，MASAMI   <masami.hiramatsu.pt@hitachi.com> - 2015-12-10 11:00 +0100
        Re: [PATCH/RFC 01/16] perf top: Delete half-processed hist entries  when exit Arnaldo Carvalho de Melo <acme@kernel.org> - 2015-12-10 20:00 +0100
      [tip:perf/core] perf top:   Delete half-processed hist entries when exit tip-bot for Namhyung Kim <tipbot@zytor.com> - 2015-12-14 09:20 +0100
    [PATCH/RFC 10/16] perf hist: Add events_stats__add() and hists__add_stats() Namhyung Kim <namhyung@kernel.org> - 2015-12-10 09:00 +0100
    [PATCH/RFC 16/16] perf tools: Skip dso front cache for multi-threaded lookup Namhyung Kim <namhyung@kernel.org> - 2015-12-10 09:00 +0100
    [PATCH/RFC 04/16] perf top: Factor out warnings in perf_top__record_precise_ip() Namhyung Kim <namhyung@kernel.org> - 2015-12-10 09:00 +0100
    [PATCH/RFC 15/16] perf top: Add --num-thread option Namhyung Kim <namhyung@kernel.org> - 2015-12-10 09:00 +0100
    [PATCH/RFC 09/16] perf tools: Update hist entry's hists pointer Namhyung Kim <namhyung@kernel.org> - 2015-12-10 09:00 +0100
      Re: [PATCH/RFC 09/16] perf tools: Update hist entry's hists pointer Jiri Olsa <jolsa@redhat.com> - 2015-12-14 00:30 +0100
        Re: [PATCH/RFC 09/16] perf tools: Update hist entry's hists pointer Namhyung Kim <namhyung@kernel.org> - 2015-12-14 03:00 +0100
      Re: [PATCH/RFC 09/16] perf tools: Update hist entry's hists pointer Jiri Olsa <jolsa@redhat.com> - 2015-12-14 00:30 +0100
    Re: [PATCHSET 00/16] perf top: Add multi-thread support (v1) Ingo Molnar <mingo@kernel.org> - 2015-12-10 09:10 +0100
      Re: [PATCHSET 00/16] perf top: Add multi-thread support (v1) Namhyung Kim <namhyung@gmail.com> - 2015-12-10 09:50 +0100
        Re: [PATCHSET 00/16] perf top: Add multi-thread support (v1) Ingo Molnar <mingo@kernel.org> - 2015-12-11 09:20 +0100
          Re: [PATCHSET 00/16] perf top: Add multi-thread support (v1) David Ahern <dsahern@gmail.com> - 2015-12-11 16:10 +0100
            Re: [PATCHSET 00/16] perf top: Add multi-thread support (v1) Namhyung Kim <namhyung@kernel.org> - 2015-12-14 02:20 +0100
            Re: [PATCHSET 00/16] perf top: Add multi-thread support (v1) Peter Zijlstra <peterz@infradead.org> - 2015-12-14 10:30 +0100
              Re: [PATCHSET 00/16] perf top: Add multi-thread support (v1) Ingo Molnar <mingo@kernel.org> - 2015-12-14 10:40 +0100
                Re: [PATCHSET 00/16] perf top: Add multi-thread support (v1) David Ahern <dsahern@gmail.com> - 2015-12-14 16:00 +0100
                  Re: [PATCHSET 00/16] perf top: Add multi-thread support (v1) Arnaldo Carvalho de Melo <acme@kernel.org> - 2015-12-14 17:30 +0100
                    Re: [PATCHSET 00/16] perf top: Add multi-thread support (v1) Peter Zijlstra <peterz@infradead.org> - 2015-12-14 17:50 +0100
                      Re: [PATCHSET 00/16] perf top: Add multi-thread support (v1) Arnaldo Carvalho de Melo <acme@kernel.org> - 2015-12-14 19:00 +0100
                Re: [PATCHSET 00/16] perf top: Add multi-thread support (v1) Namhyung Kim <namhyung@kernel.org> - 2015-12-14 17:40 +0100
                  Re: [PATCHSET 00/16] perf top: Add multi-thread support (v1) Peter Zijlstra <peterz@infradead.org> - 2015-12-14 18:00 +0100
                    Re: [PATCHSET 00/16] perf top: Add multi-thread support (v1) Namhyung Kim <namhyung@kernel.org> - 2015-12-14 18:20 +0100
              Re: [PATCHSET 00/16] perf top: Add multi-thread support (v1) David Ahern <dsahern@gmail.com> - 2015-12-14 15:50 +0100
                Re: [PATCHSET 00/16] perf top: Add multi-thread support (v1) Namhyung Kim <namhyung@kernel.org> - 2015-12-14 18:10 +0100
                  Re: [PATCHSET 00/16] perf top: Add multi-thread support (v1) Arnaldo Carvalho de Melo <acme@kernel.org> - 2015-12-14 19:00 +0100
              Re: [PATCHSET 00/16] perf top: Add multi-thread support (v1) Namhyung Kim <namhyung@kernel.org> - 2015-12-14 17:30 +0100
                Re: [PATCHSET 00/16] perf top: Add multi-thread support (v1) Peter Zijlstra <peterz@infradead.org> - 2015-12-14 17:50 +0100

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

#1290781 — Re: [PATCH/RFC 09/16] perf tools: Update hist entry's hists pointer

On Mon, Dec 14, 2015 at 12:28:12AM +0100, Jiri Olsa wrote:
> On Mon, Dec 14, 2015 at 12:23:30AM +0100, Jiri Olsa wrote:
> > On Thu, Dec 10, 2015 at 04:53:28PM +0900, Namhyung Kim wrote:
> > > When sample is processed using multi-thread, each sample is gathered on
> > > each thread's hist tree and then merged into the real hist tree.  But
> > > hist_entry->hists pointer was not updated so it could refer wrong hists
> > > resulted in missing outputs.
> > > 
> > > Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> > > ---
> > >  tools/perf/util/hist.c | 8 ++++++++
> > >  1 file changed, 8 insertions(+)
> > > 
> > > diff --git a/tools/perf/util/hist.c b/tools/perf/util/hist.c
> > > index ea4f3ad978b0..a12e5022fe04 100644
> > > --- a/tools/perf/util/hist.c
> > > +++ b/tools/perf/util/hist.c
> > > @@ -1008,6 +1008,14 @@ bool hists__collapse_insert_entry(struct hists *hists __maybe_unused,
> > 
> > there's 'struct hists *hists __maybe_unused' in hists__collapse_insert_entry def..

OK, will remove.

> 
> also hists__collapse_insert_entry is global which seems unnecessary

What do you mean?  The hists__collapse_insert_entry() will be used by
collect_worker thread.

Thanks,
Namhyung


> 
> jirka
> 
> > 
> > jirka
> > 
> > >  	}
> > >  	hists->nr_entries++;
> > >  
> > > +	/*
> > > +	 * If a hist entry is processed in multi-threaded environment,
> > > +	 * it points to a dummy local hists which was used only for
> > > +	 * intermidate processing.  So update it to a real one so that
> > > +	 * it can find the correct info later.
> > > +	 */
> > > +	he->hists = hists;
> > > +
> > >  	rb_link_node(&he->rb_node_in, parent, p);
> > >  	rb_insert_color(&he->rb_node_in, root);
> > >  	return true;
> > > -- 
> > > 2.6.2
> > > 
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1290752 — Re: [PATCH/RFC 09/16] perf tools: Update hist entry's hists pointer

On Thu, Dec 10, 2015 at 04:53:28PM +0900, Namhyung Kim wrote:
> When sample is processed using multi-thread, each sample is gathered on
> each thread's hist tree and then merged into the real hist tree.  But
> hist_entry->hists pointer was not updated so it could refer wrong hists
> resulted in missing outputs.
> 
> Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> ---
>  tools/perf/util/hist.c | 8 ++++++++
>  1 file changed, 8 insertions(+)
> 
> diff --git a/tools/perf/util/hist.c b/tools/perf/util/hist.c
> index ea4f3ad978b0..a12e5022fe04 100644
> --- a/tools/perf/util/hist.c
> +++ b/tools/perf/util/hist.c
> @@ -1008,6 +1008,14 @@ bool hists__collapse_insert_entry(struct hists *hists __maybe_unused,

there's 'struct hists *hists __maybe_unused' in hists__collapse_insert_entry def..

jirka

>  	}
>  	hists->nr_entries++;
>  
> +	/*
> +	 * If a hist entry is processed in multi-threaded environment,
> +	 * it points to a dummy local hists which was used only for
> +	 * intermidate processing.  So update it to a real one so that
> +	 * it can find the correct info later.
> +	 */
> +	he->hists = hists;
> +
>  	rb_link_node(&he->rb_node_in, parent, p);
>  	rb_insert_color(&he->rb_node_in, root);
>  	return true;
> -- 
> 2.6.2
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1288286

* Namhyung Kim <namhyung@kernel.org> wrote:

> Hello,
> 
> This patchset if an attempt to support multi-threading in perf top.
> In fact, perf top already run on two threads - a worker thread and a
> display thread.  However processing all samples with a single thread
> in a large machine can have scalability problems.
> 
> This patchset extends it to have multiple worker threads to process
> samples concurrently.  Users can control the number of threads using
>  --num-thread option.  And there's a collector thread for passing hist
> entries from worker threads to the display thread.

Could you please make the number of threads default to the number of CPUs?

Since perf top is doing one perf event per CPU anyway, that's a pretty natural 
model.

( I think 'perf record' should use per CPU threads as well to receive events, to 
  address the 'IO overload' problems with -g recording on larger CPU counts. )

Thanks,

	Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1288362

On December 10, 2015 5:01:18 PM GMT+09:00, Ingo Molnar <mingo@kernel.org> wrote:
>
>* Namhyung Kim <namhyung@kernel.org> wrote:
>
>> Hello,
>> 
>> This patchset if an attempt to support multi-threading in perf top.
>> In fact, perf top already run on two threads - a worker thread and a
>> display thread.  However processing all samples with a single thread
>> in a large machine can have scalability problems.
>> 
>> This patchset extends it to have multiple worker threads to process
>> samples concurrently.  Users can control the number of threads using
>>  --num-thread option.  And there's a collector thread for passing
>hist
>> entries from worker threads to the display thread.
>
>Could you please make the number of threads default to the number of
>CPUs?
>
>Since perf top is doing one perf event per CPU anyway, that's a pretty
>natural 
>model.
>
>( I think 'perf record' should use per CPU threads as well to receive
>events, to 
>address the 'IO overload' problems with -g recording on larger CPU
>counts. )

IIRC David said that thread per cpu seems too much
especially on a large system (like ~1024 cpu).  I have no idea 
what's the reasonable default on the system, so I chose 1/4 
of map buffers (i.e. cpus for most cases). But I think I should
take non-system-wide mode into account too.

Thanks
Namhyung

Hi Ingo,
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1289301

* Namhyung Kim <namhyung@gmail.com> wrote:

> IIRC David said that thread per cpu seems too much especially on a large system 
> (like ~1024 cpu). [...]

Too much in what fashion? For recording I think it's the fastest, most natural 
model - anything else will create cache line bounces.

For perf report, I suspect you are right, it would depend on the actual possible 
parallelism - which with time would improve I suspect.

Thanks,

	Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1289656

On 12/11/15 1:11 AM, Ingo Molnar wrote:
>
> * Namhyung Kim <namhyung@gmail.com> wrote:
>
>> IIRC David said that thread per cpu seems too much especially on a large system
>> (like ~1024 cpu). [...]
>
> Too much in what fashion? For recording I think it's the fastest, most natural
> model - anything else will create cache line bounces.

The intrusiveness of perf on the system under observation. I understand 
there are a lot of factors that go into it.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1290767

Hi David and Ingo,

On Fri, Dec 11, 2015 at 08:01:31AM -0700, David Ahern wrote:
> On 12/11/15 1:11 AM, Ingo Molnar wrote:
> >
> >* Namhyung Kim <namhyung@gmail.com> wrote:
> >
> >>IIRC David said that thread per cpu seems too much especially on a large system
> >>(like ~1024 cpu). [...]
> >
> >Too much in what fashion? For recording I think it's the fastest, most natural
> >model - anything else will create cache line bounces.
> 
> The intrusiveness of perf on the system under observation. I understand
> there are a lot of factors that go into it.

So what would be the sane default?  Do you have any other suggestion?

If there's no opinion, I'll follow Ingo's..

Thanks,
Namhyung
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1291001

On Fri, Dec 11, 2015 at 08:01:31AM -0700, David Ahern wrote:
> On 12/11/15 1:11 AM, Ingo Molnar wrote:
> >
> >* Namhyung Kim <namhyung@gmail.com> wrote:
> >
> >>IIRC David said that thread per cpu seems too much especially on a large system
> >>(like ~1024 cpu). [...]
> >
> >Too much in what fashion? For recording I think it's the fastest, most natural
> >model - anything else will create cache line bounces.
> 
> The intrusiveness of perf on the system under observation. I understand
> there are a lot of factors that go into it.

So I can see some of that, if every cpu has its own thread then every
cpu will occasionally schedule that thread. Whereas if there were less,
you'd not have that.

Still, I think it makes sense to implement it, we need the multi-file
option anyway. Once we have that, we can also implement a per-node
option, which should be a fairly simple hybrid of the two approaches.

The thing is, perf-record is really struggling on big machines.

And in an unrelated note, I absolutely detest --buildid being the
default, it makes perf-record blow chunks.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1291006

* Peter Zijlstra <peterz@infradead.org> wrote:

> On Fri, Dec 11, 2015 at 08:01:31AM -0700, David Ahern wrote:
> > On 12/11/15 1:11 AM, Ingo Molnar wrote:
> > >
> > >* Namhyung Kim <namhyung@gmail.com> wrote:
> > >
> > >>IIRC David said that thread per cpu seems too much especially on a large system
> > >>(like ~1024 cpu). [...]
> > >
> > >Too much in what fashion? For recording I think it's the fastest, most natural
> > >model - anything else will create cache line bounces.
> > 
> > The intrusiveness of perf on the system under observation. I understand
> > there are a lot of factors that go into it.
> 
> So I can see some of that, if every cpu has its own thread then every
> cpu will occasionally schedule that thread. Whereas if there were less,
> you'd not have that.
> 
> Still, I think it makes sense to implement it, we need the multi-file
> option anyway. Once we have that, we can also implement a per-node
> option, which should be a fairly simple hybrid of the two approaches.
> 
> The thing is, perf-record is really struggling on big machines.
> 
> And in an unrelated note, I absolutely detest --buildid being the
> default, it makes perf-record blow chunks.

So I'd absolutely _love_ to split up the singular perf.data into a hierarchy of 
files in a .perf directory, with a structure like this (4-core system):

	.perf/cmdline
	.perf/features
	.perf/evlist
	.perf/ring_buffers/cpu0/raw.trace
	.perf/ring_buffers/cpu1/raw.trace
	.perf/ring_buffers/cpu2/raw.trace
	.perf/ring_buffers/cpu3/raw.trace
	...

I.e. the current single file format of perf.data would be split up into individual 
files. Each CPU would get its own trace file output - any sorting and ordering 
would be done afterwards. 'perf record' itself would never by default have to do 
any of that, it's a pure recording session.

'perf archive' would still create a single file to make transport between machines 
easy.

perf.data.old would be replaced by a .perf.old directory or so.

Debugging would be easier too I think, as there's no complex perf data format 
anymore, it's all in individual (typically text, or binary dump) files in the 
.perf directory.

This would solve all the scalability problems - and would make the format more 
extensible and generally more accessible as well.

What do you think?

Thanks,

	Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1291246

On 12/14/15 2:38 AM, Ingo Molnar wrote:
>
>> And in an unrelated note, I absolutely detest --buildid being the
>> default, it makes perf-record blow chunks.

Yes, a .debug directory that gets bloated fast and being a dot-directory 
is off the primary radar. I forget about it and too often forget to add 
the option to disable it.

>
> So I'd absolutely _love_ to split up the singular perf.data into a hierarchy of
> files in a .perf directory, with a structure like this (4-core system):
>
> 	.perf/cmdline
> 	.perf/features
> 	.perf/evlist
> 	.perf/ring_buffers/cpu0/raw.trace
> 	.perf/ring_buffers/cpu1/raw.trace
> 	.perf/ring_buffers/cpu2/raw.trace
> 	.perf/ring_buffers/cpu3/raw.trace
> 	...

On a related note why a .perf directory?

>
> I.e. the current single file format of perf.data would be split up into individual
> files. Each CPU would get its own trace file output - any sorting and ordering
> would be done afterwards. 'perf record' itself would never by default have to do
> any of that, it's a pure recording session.
>
> 'perf archive' would still create a single file to make transport between machines
> easy.
>
> perf.data.old would be replaced by a .perf.old directory or so.
>
> Debugging would be easier too I think, as there's no complex perf data format
> anymore, it's all in individual (typically text, or binary dump) files in the
> .perf directory.
>
> This would solve all the scalability problems - and would make the format more
> extensible and generally more accessible as well.
>
> What do you think?

Big change to user experience.

I realize perf-archive has been around since I started using perf in 
mid-2010, but I for one never use it. I suspect it is not widely used 
(definitely not in the circles I have been involved and helped with 
perf), so suddenly requiring it is a change in user experience.

The only 2 files on the system I pull off the box are kallsyms and 
perf.data. Most of the systems where I use perf have limited symbols and 
there is nothing in .debug I need to pull of the box.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1291329

Em Mon, Dec 14, 2015 at 07:55:32AM -0700, David Ahern escreveu:
> On 12/14/15 2:38 AM, Ingo Molnar wrote:
> >
> >>And in an unrelated note, I absolutely detest --buildid being the
> >>default, it makes perf-record blow chunks.
> 
> Yes, a .debug directory that gets bloated fast and being a dot-directory is
> off the primary radar. I forget about it and too often forget to add the
> option to disable it.

Multiple things here, .debug/ should be size limited, and buildid
processing doesn't need necessarily to store things in that cache, I bet
what PeterZ is complaining about is the reprocessing of events at the
end of the session, to find out about the PERF_RECORD_MMAP* events to
then read the build-ids and insert then into the perf.data file header.

All this can be disabled by default, the downside is that when samples
get resolved to something random because the binary used in the session
was replaced by some other we will not be able to notice.

This would be solved by inserting the buildid (20-some bytes) into the
PERF_RECORD_MMAP record, but that remains to be done...
 
> >
> >So I'd absolutely _love_ to split up the singular perf.data into a hierarchy of
> >files in a .perf directory, with a structure like this (4-core system):
> >
> >	.perf/cmdline
> >	.perf/features
> >	.perf/evlist
> >	.perf/ring_buffers/cpu0/raw.trace
> >	.perf/ring_buffers/cpu1/raw.trace
> >	.perf/ring_buffers/cpu2/raw.trace
> >	.perf/ring_buffers/cpu3/raw.trace
> >	...
> 
> On a related note why a .perf directory?
> 
> >
> >I.e. the current single file format of perf.data would be split up into individual
> >files. Each CPU would get its own trace file output - any sorting and ordering
> >would be done afterwards. 'perf record' itself would never by default have to do
> >any of that, it's a pure recording session.
> >
> >'perf archive' would still create a single file to make transport between machines
> >easy.
> >
> >perf.data.old would be replaced by a .perf.old directory or so.
> >
> >Debugging would be easier too I think, as there's no complex perf data format
> >anymore, it's all in individual (typically text, or binary dump) files in the
> >.perf directory.
> >
> >This would solve all the scalability problems - and would make the format more
> >extensible and generally more accessible as well.
> >
> >What do you think?
> 
> Big change to user experience.
> 
> I realize perf-archive has been around since I started using perf in
> mid-2010, but I for one never use it. I suspect it is not widely used
> (definitely not in the circles I have been involved and helped with perf),
> so suddenly requiring it is a change in user experience.
> 
> The only 2 files on the system I pull off the box are kallsyms and
> perf.data. Most of the systems where I use perf have limited symbols and
> there is nothing in .debug I need to pull of the box.

Well, we don't have to use perf archive for that, doing what Ingo
suggests and on top of it putting it into a perf.data file that in fact
is a cpio or tarball would keep the one-file while introducing the split
up for later processing bits.

- Arnaldo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1291349

On Mon, Dec 14, 2015 at 01:26:55PM -0300, Arnaldo Carvalho de Melo wrote:
> Em Mon, Dec 14, 2015 at 07:55:32AM -0700, David Ahern escreveu:
> > On 12/14/15 2:38 AM, Ingo Molnar wrote:
> > >
> > >>And in an unrelated note, I absolutely detest --buildid being the
> > >>default, it makes perf-record blow chunks.

> Multiple things here, .debug/ should be size limited, and buildid
> processing doesn't need necessarily to store things in that cache, I bet
> what PeterZ is complaining about is the reprocessing of events at the
> end of the session, to find out about the PERF_RECORD_MMAP* events to
> then read the build-ids and insert then into the perf.data file header.
> 

Yeah, its the reprocessing that is taking forever.. On my moderately
sized system with 40 CPUs, the reprocessing is taking about as long as
the actual workload, which is tedious.

Once I figured out what was happening Jiri was quick to point out I
should be using -B.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1291398

Em Mon, Dec 14, 2015 at 05:41:30PM +0100, Peter Zijlstra escreveu:
> On Mon, Dec 14, 2015 at 01:26:55PM -0300, Arnaldo Carvalho de Melo wrote:
> > Em Mon, Dec 14, 2015 at 07:55:32AM -0700, David Ahern escreveu:
> > > On 12/14/15 2:38 AM, Ingo Molnar wrote:
> > > >
> > > >>And in an unrelated note, I absolutely detest --buildid being the
> > > >>default, it makes perf-record blow chunks.
> 
> > Multiple things here, .debug/ should be size limited, and buildid
> > processing doesn't need necessarily to store things in that cache, I bet
> > what PeterZ is complaining about is the reprocessing of events at the
> > end of the session, to find out about the PERF_RECORD_MMAP* events to
> > then read the build-ids and insert then into the perf.data file header.
> > 
> 
> Yeah, its the reprocessing that is taking forever.. On my moderately
> sized system with 40 CPUs, the reprocessing is taking about as long as
> the actual workload, which is tedious.

Right, I thought about using some dummy event for tracking mmaps, which
I think is a technique used by the Intel PT code (well, there it uses it
for sched_switches) will check...
 
> Once I figured out what was happening Jiri was quick to point out I
> should be using -B.

Right, we have to have sane behaviour by default, damn long/high freq
workloads, duh ;-)

- Arnaldo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1291339

Hi Ingo,

On Mon, Dec 14, 2015 at 10:38:41AM +0100, Ingo Molnar wrote:
> 
> * Peter Zijlstra <peterz@infradead.org> wrote:
> 
> > On Fri, Dec 11, 2015 at 08:01:31AM -0700, David Ahern wrote:
> > > On 12/11/15 1:11 AM, Ingo Molnar wrote:
> > > >
> > > >* Namhyung Kim <namhyung@gmail.com> wrote:
> > > >
> > > >>IIRC David said that thread per cpu seems too much especially on a large system
> > > >>(like ~1024 cpu). [...]
> > > >
> > > >Too much in what fashion? For recording I think it's the fastest, most natural
> > > >model - anything else will create cache line bounces.
> > > 
> > > The intrusiveness of perf on the system under observation. I understand
> > > there are a lot of factors that go into it.
> > 
> > So I can see some of that, if every cpu has its own thread then every
> > cpu will occasionally schedule that thread. Whereas if there were less,
> > you'd not have that.
> > 
> > Still, I think it makes sense to implement it, we need the multi-file
> > option anyway. Once we have that, we can also implement a per-node
> > option, which should be a fairly simple hybrid of the two approaches.
> > 
> > The thing is, perf-record is really struggling on big machines.
> > 
> > And in an unrelated note, I absolutely detest --buildid being the
> > default, it makes perf-record blow chunks.
> 
> So I'd absolutely _love_ to split up the singular perf.data into a hierarchy of 
> files in a .perf directory, with a structure like this (4-core system):
> 
> 	.perf/cmdline
> 	.perf/features
> 	.perf/evlist
> 	.perf/ring_buffers/cpu0/raw.trace
> 	.perf/ring_buffers/cpu1/raw.trace
> 	.perf/ring_buffers/cpu2/raw.trace
> 	.perf/ring_buffers/cpu3/raw.trace
> 	...
> 
> I.e. the current single file format of perf.data would be split up into individual 
> files. Each CPU would get its own trace file output - any sorting and ordering 
> would be done afterwards. 'perf record' itself would never by default have to do 
> any of that, it's a pure recording session.
> 
> 'perf archive' would still create a single file to make transport between machines 
> easy.
> 
> perf.data.old would be replaced by a .perf.old directory or so.
> 
> Debugging would be easier too I think, as there's no complex perf data format 
> anymore, it's all in individual (typically text, or binary dump) files in the 
> .perf directory.
> 
> This would solve all the scalability problems - and would make the format more 
> extensible and generally more accessible as well.
> 
> What do you think?

It requires many changes, but basically I also like the split-up since
it's easier to deal with.  IIRC there was an opinion (Andi?) regarding
single-file vs multi-file.  The file access will be better for single
file so I changed my earlier implementation to use indexed single data
file instead of multiple files.

But anyway, I'll work on perf-top first :)

Thanks,
Namhyung
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1291350

On Tue, Dec 15, 2015 at 01:38:30AM +0900, Namhyung Kim wrote:
> It requires many changes, but basically I also like the split-up since
> it's easier to deal with.  IIRC there was an opinion (Andi?) regarding
> single-file vs multi-file.  The file access will be better for single
> file so I changed my earlier implementation to use indexed single data
> file instead of multiple files.

The page-cache has a lock per inode, so by having all CPUs populate the
one file you get contention on that.

Also, I suppose you'll have to arbitrate ranges in that file for each
cpu to make it work, that too could get you some contention.

Having a file per cpu avoids all that.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1291360

On Mon, Dec 14, 2015 at 05:56:14PM +0100, Peter Zijlstra wrote:
> On Tue, Dec 15, 2015 at 01:38:30AM +0900, Namhyung Kim wrote:
> > It requires many changes, but basically I also like the split-up since
> > it's easier to deal with.  IIRC there was an opinion (Andi?) regarding
> > single-file vs multi-file.  The file access will be better for single
> > file so I changed my earlier implementation to use indexed single data
> > file instead of multiple files.
> 
> The page-cache has a lock per inode, so by having all CPUs populate the
> one file you get contention on that.
> 
> Also, I suppose you'll have to arbitrate ranges in that file for each
> cpu to make it work, that too could get you some contention.
> 
> Having a file per cpu avoids all that.

Right.  Now I recall that it was about *report* (not record) time
accessing single file vs. multi files.

At record time we should use file per cpu.  I combined them into
one with index at post-processing time in my earlier work.

Thanks for clarification!
Namhyung
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1291236

On 12/14/15 2:26 AM, Peter Zijlstra wrote:
> On Fri, Dec 11, 2015 at 08:01:31AM -0700, David Ahern wrote:
>> On 12/11/15 1:11 AM, Ingo Molnar wrote:
>>>
>>> * Namhyung Kim <namhyung@gmail.com> wrote:
>>>
>>>> IIRC David said that thread per cpu seems too much especially on a large system
>>>> (like ~1024 cpu). [...]
>>>
>>> Too much in what fashion? For recording I think it's the fastest, most natural
>>> model - anything else will create cache line bounces.
>>
>> The intrusiveness of perf on the system under observation. I understand
>> there are a lot of factors that go into it.
>
> So I can see some of that, if every cpu has its own thread then every
> cpu will occasionally schedule that thread. Whereas if there were less,
> you'd not have that.
>
> Still, I think it makes sense to implement it, we need the multi-file
> option anyway. Once we have that, we can also implement a per-node
> option, which should be a fairly simple hybrid of the two approaches.
>
> The thing is, perf-record is really struggling on big machines.

I've gone from the 1024-cpu sparc systems earlier this year down to 
small PPC and Rangeley-based switches. For both ends of the scale (and 
in between) I constantly struggle with the options to manage memory, cpu 
and disk consumption.

There definitely needs to be options (e.g., multi-threaded on/off). For 
the threading options I get the appeal for 1-thread per cpu but other 
options make sense as well -- 1 thread per core, 1 per NUMA node. perf 
has the CPU topology so should not be too difficult.

If you have 1-thread per cpu that means you are pinning the threads to 
the cpu? That brings in additional permissions problems.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1291353

Hi David,

On Mon, Dec 14, 2015 at 07:46:28AM -0700, David Ahern wrote:
> On 12/14/15 2:26 AM, Peter Zijlstra wrote:
> >On Fri, Dec 11, 2015 at 08:01:31AM -0700, David Ahern wrote:
> >>On 12/11/15 1:11 AM, Ingo Molnar wrote:
> >>>
> >>>* Namhyung Kim <namhyung@gmail.com> wrote:
> >>>
> >>>>IIRC David said that thread per cpu seems too much especially on a large system
> >>>>(like ~1024 cpu). [...]
> >>>
> >>>Too much in what fashion? For recording I think it's the fastest, most natural
> >>>model - anything else will create cache line bounces.
> >>
> >>The intrusiveness of perf on the system under observation. I understand
> >>there are a lot of factors that go into it.
> >
> >So I can see some of that, if every cpu has its own thread then every
> >cpu will occasionally schedule that thread. Whereas if there were less,
> >you'd not have that.
> >
> >Still, I think it makes sense to implement it, we need the multi-file
> >option anyway. Once we have that, we can also implement a per-node
> >option, which should be a fairly simple hybrid of the two approaches.
> >
> >The thing is, perf-record is really struggling on big machines.
> 
> I've gone from the 1024-cpu sparc systems earlier this year down to small
> PPC and Rangeley-based switches. For both ends of the scale (and in between)
> I constantly struggle with the options to manage memory, cpu and disk
> consumption.
> 
> There definitely needs to be options (e.g., multi-threaded on/off). For the
> threading options I get the appeal for 1-thread per cpu but other options
> make sense as well -- 1 thread per core, 1 per NUMA node. perf has the CPU
> topology so should not be too difficult.

I think we can use --num-thread option to control multi-threading: 1
for disabling and others for enabling.  In the current implementation,
using 1 thread still use same logic so 1 reader + 1 collector will be
created as well as 1 display thread.  Not sure it'd be better special
casing 1 thread to use different code path.

Anyway, I think it'd be nice to have per-core, per-socket and per-node
options and per-core is a good default then.

> 
> If you have 1-thread per cpu that means you are pinning the threads to the
> cpu? That brings in additional permissions problems.

Did you mean setting sched affinity?  It seems not a privileged
operation doing it for its own threads..

Thanks,
Namhyung
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1291394

Em Tue, Dec 15, 2015 at 02:06:54AM +0900, Namhyung Kim escreveu:
> On Mon, Dec 14, 2015 at 07:46:28AM -0700, David Ahern wrote:
> > If you have 1-thread per cpu that means you are pinning the threads to the
> > cpu? That brings in additional permissions problems.
> 
> Did you mean setting sched affinity?  It seems not a privileged
> operation doing it for its own threads..

Right, and we should do it (create per cpu rb consuming threads) just
for the cpus in the workload affinity mask.

- Arnaldo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1291331

Hi Peter,

On Mon, Dec 14, 2015 at 10:26:13AM +0100, Peter Zijlstra wrote:
> On Fri, Dec 11, 2015 at 08:01:31AM -0700, David Ahern wrote:
> > On 12/11/15 1:11 AM, Ingo Molnar wrote:
> > >
> > >* Namhyung Kim <namhyung@gmail.com> wrote:
> > >
> > >>IIRC David said that thread per cpu seems too much especially on a large system
> > >>(like ~1024 cpu). [...]
> > >
> > >Too much in what fashion? For recording I think it's the fastest, most natural
> > >model - anything else will create cache line bounces.
> > 
> > The intrusiveness of perf on the system under observation. I understand
> > there are a lot of factors that go into it.
> 
> So I can see some of that, if every cpu has its own thread then every
> cpu will occasionally schedule that thread. Whereas if there were less,
> you'd not have that.
> 
> Still, I think it makes sense to implement it, we need the multi-file
> option anyway. Once we have that, we can also implement a per-node
> option, which should be a fairly simple hybrid of the two approaches.
> 
> The thing is, perf-record is really struggling on big machines.

Yes, but perf-record and perf-top is different.  The perf-record
merely saves the data into file while perf-top read events and process
them at the same time without file.  So we should choose different
default IMHO.

I want to focus on perf-top for now, once it's in a good shape, I'll
work on perf record/report too.


> 
> And in an unrelated note, I absolutely detest --buildid being the
> default, it makes perf-record blow chunks.

Maybe we can add a config option?

Thanks,
Namhyung
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

Back to top | Article view | linux.kernel

csiph-web