Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1251166 > unrolled thread

Re: [PATCH 13/16] perf callchain: Switch default to 'graph,0.5,caller'

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH 13/16] perf callchain: Switch default to 'graph,0.5,caller' Brendan Gregg <brendan.d.gregg@gmail.com> - 2015-10-20 02:20 +0200
    Re: [PATCH 13/16] perf callchain: Switch default to  'graph,0.5,caller' Arnaldo Carvalho de Melo <arnaldo.melo@gmail.com> - 2015-10-20 14:10 +0200
      Re: [PATCH 13/16] perf callchain: Switch default to  'graph,0.5,caller' Frederic Weisbecker <fweisbec@gmail.com> - 2015-10-20 14:20 +0200
        Re: [PATCH 13/16] perf callchain: Switch default to  'graph,0.5,caller' Arnaldo Carvalho de Melo <arnaldo.melo@gmail.com> - 2015-10-20 15:10 +0200
          Re: [PATCH 13/16] perf callchain: Switch default to  'graph,0.5,caller' Frederic Weisbecker <fweisbec@gmail.com> - 2015-10-20 19:30 +0200
            Re: [PATCH 13/16] perf callchain: Switch default to  'graph,0.5,caller' Arnaldo Carvalho de Melo <arnaldo.melo@gmail.com> - 2015-10-20 20:50 +0200
              Re: [PATCH 13/16] perf callchain: Switch default to  'graph,0.5,caller' Namhyung Kim <namhyung@kernel.org> - 2015-10-21 03:30 +0200
                Re: [PATCH 13/16] perf callchain: Switch default to  'graph,0.5,caller' Arnaldo Carvalho de Melo <arnaldo.melo@gmail.com> - 2015-10-21 15:30 +0200
            Re: [PATCH 13/16] perf callchain: Switch default to  'graph,0.5,caller' Namhyung Kim <namhyung@kernel.org> - 2015-10-21 10:10 +0200
              Re: [PATCH 13/16] perf callchain: Switch default to 'graph,0.5,caller' "Wangnan (F)" <wangnan0@huawei.com> - 2015-10-21 14:00 +0200
              Re: [PATCH 13/16] perf callchain: Switch default to  'graph,0.5,caller' Frederic Weisbecker <fweisbec@gmail.com> - 2015-10-21 18:40 +0200
          Re: [PATCH 13/16] perf callchain: Switch default to  'graph,0.5,caller' Frederic Weisbecker <fweisbec@gmail.com> - 2015-10-21 18:30 +0200
          Re: [PATCH 13/16] perf callchain: Switch default to 'graph,0.5,caller' Brendan Gregg <brendan.d.gregg@gmail.com> - 2015-10-21 20:30 +0200
            Re: [PATCH 13/16] perf callchain: Switch default to  'graph,0.5,caller' Arnaldo Carvalho de Melo <arnaldo.melo@gmail.com> - 2015-10-21 21:30 +0200
              Re: [PATCH 13/16] perf callchain: Switch default to 'graph,0.5,caller' Brendan Gregg <brendan.d.gregg@gmail.com> - 2015-10-22 02:50 +0200
      Re: [PATCH 13/16] perf callchain: Switch default to  'graph,0.5,caller' Ingo Molnar <mingo@kernel.org> - 2015-10-21 10:10 +0200
        Re: [PATCH 13/16] perf callchain: Switch default to  'graph,0.5,caller' Arnaldo Carvalho de Melo <arnaldo.melo@gmail.com> - 2015-10-21 15:30 +0200
        Re: [PATCH 13/16] perf callchain: Switch default to 'graph,0.5,caller' Brendan Gregg <brendan.d.gregg@gmail.com> - 2015-10-21 21:20 +0200

#1251166 — Re: [PATCH 13/16] perf callchain: Switch default to 'graph,0.5,caller'

On Fri, Oct 9, 2015 at 3:25 PM, Arnaldo Carvalho de Melo
<arnaldo.melo@gmail.com> wrote:
> Em Fri, Oct 09, 2015 at 03:10:29PM -0700, Brendan Gregg escreveu:
>> On Fri, Oct 9, 2015 at 2:56 PM, Arnaldo Carvalho de Melo
>> <arnaldo.melo@gmail.com> wrote:
>> >
>> > Em Fri, Oct 09, 2015 at 01:34:33PM -0700, Brendan Gregg escreveu:
>> > > On Mon, Oct 5, 2015 at 2:03 PM, Arnaldo Carvalho de Melo
>> > > <acme@kernel.org> wrote:
>> > > >
>> > > > From: Arnaldo Carvalho de Melo <acme@redhat.com>
>> > > >
>> > > > Which is the most common default found in other similar tools.
>> > >
>> > > Interactive tools, sure, like the perf report TUI.
>> >
>> > > But this also changes the ordering of the non-interactive tools which
>> > > dump stacks: "perf report -n --stdio" and "perf script". The most
>> > > common default for dumping stacks is caller. Eg:
>> >
>> > And you use that for scripting?
>>
>> Yes; how I typically CPU profile:
>>
>> git clone https://github.com/brendangregg/FlameGraph
>> cd FlameGraph
>> perf record -F 99 -a -g -- sleep 60
>> perf script | ./stackcollapse-perf.pl | /flamegraph.pl > flame.svg
>>
>> Then open flame.svg in a browser and click around. Try it. :)
>
> Should've already, guess there are things to merge and save you some
> time :)
>
>> But it's not just scripting; We often email around "perf report -n
>> --stdio" output, or attach it to tickets, when working on an issue.
>> Easier than trying to grab the right TUI screenshot.
>
> Well, that is something we need to change... But then, are you aware of
> 'P' in the TUI? It allows you to go, expand whatever callchains you deem
> necessary and then press P and the current state of the UI will be
> dumped to a perf.hist.<N> file. I.e. no need to get a png screenshot nor
> to copy things with gpm like stuff.
> [...]

I didn't know about 'P', thanks. I still find it quicker to "perf
report -n --stdio", then browse with my eyes and the scroll wheel.
(Then use a flame graph if the output isn't trivial to understand.)

... Just to express a callee order preference differently (at least
for the text dumping outputs), it's showing what's running (first few
frames) followed by how we got there. And that's the order I'm usually
wanting to know them... I still think it's the order printed by most
other text-based tools, although GUIs (like those with tree views) are
probably more usually caller order.

Brendan
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1251618 — Re: [PATCH 13/16] perf callchain: Switch default to 'graph,0.5,caller'

Em Mon, Oct 19, 2015 at 05:16:53PM -0700, Brendan Gregg escreveu:
> On Fri, Oct 9, 2015 at 3:25 PM, Arnaldo Carvalho de Melo
> <arnaldo.melo@gmail.com> wrote:
> > Em Fri, Oct 09, 2015 at 03:10:29PM -0700, Brendan Gregg escreveu:
> >> On Fri, Oct 9, 2015 at 2:56 PM, Arnaldo Carvalho de Melo
> >> <arnaldo.melo@gmail.com> wrote:
> >> > Em Fri, Oct 09, 2015 at 01:34:33PM -0700, Brendan Gregg escreveu:
> >> > > On Mon, Oct 5, 2015 at 2:03 PM, Arnaldo Carvalho de Melo
> >> > > <acme@kernel.org> wrote:
> >> > > > From: Arnaldo Carvalho de Melo <acme@redhat.com>

> >> > > > Which is the most common default found in other similar tools.

> >> > > Interactive tools, sure, like the perf report TUI.

> >> > > But this also changes the ordering of the non-interactive tools which
> >> > > dump stacks: "perf report -n --stdio" and "perf script". The most
> >> > > common default for dumping stacks is caller. Eg:

> >> > And you use that for scripting?

> >> Yes; how I typically CPU profile:

> >> git clone https://github.com/brendangregg/FlameGraph
> >> cd FlameGraph
> >> perf record -F 99 -a -g -- sleep 60
> >> perf script | ./stackcollapse-perf.pl | /flamegraph.pl > flame.svg

> >> Then open flame.svg in a browser and click around. Try it. :)

> > Should've already, guess there are things to merge and save you some
> > time :)

> >> But it's not just scripting; We often email around "perf report -n
> >> --stdio" output, or attach it to tickets, when working on an issue.
> >> Easier than trying to grab the right TUI screenshot.

> > Well, that is something we need to change... But then, are you aware of
> > 'P' in the TUI? It allows you to go, expand whatever callchains you deem
> > necessary and then press P and the current state of the UI will be
> > dumped to a perf.hist.<N> file. I.e. no need to get a png screenshot nor
> > to copy things with gpm like stuff.
> > [...]
 
> I didn't know about 'P', thanks. I still find it quicker to "perf
> report -n --stdio", then browse with my eyes and the scroll wheel.
> (Then use a flame graph if the output isn't trivial to understand.)
 
> ... Just to express a callee order preference differently (at least
> for the text dumping outputs), it's showing what's running (first few
> frames) followed by how we got there. And that's the order I'm usually
> wanting to know them... I still think it's the order printed by most
> other text-based tools, although GUIs (like those with tree views) are
> probably more usually caller order.

So are you advocating different defaults, one for --stdio (callee),
another for --tui, --gtk (caller)?

This is all configurable via ~/.perfconfig :-\

Indeed, finding a default that is deemed adequate for most people is,
ho-hum, difficult 8-)

Ingo, what do you think?

- Arnaldo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1251620 — Re: [PATCH 13/16] perf callchain: Switch default to 'graph,0.5,caller'

On Tue, Oct 20, 2015 at 09:00:34AM -0300, Arnaldo Carvalho de Melo wrote:
> Em Mon, Oct 19, 2015 at 05:16:53PM -0700, Brendan Gregg escreveu:
> So are you advocating different defaults, one for --stdio (callee),
> another for --tui, --gtk (caller)?
> 
> This is all configurable via ~/.perfconfig :-\
> 
> Indeed, finding a default that is deemed adequate for most people is,
> ho-hum, difficult 8-)

Most uses I've seen on LKML by the past involved callee because people
mostly look at the precise point where a performance issue is.

IMHO changing that order is not a good idea. Unless many users complained
about it.

> 
> Ingo, what do you think?
> 
> - Arnaldo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1251660 — Re: [PATCH 13/16] perf callchain: Switch default to 'graph,0.5,caller'

Em Tue, Oct 20, 2015 at 02:19:50PM +0200, Frederic Weisbecker escreveu:
> On Tue, Oct 20, 2015 at 09:00:34AM -0300, Arnaldo Carvalho de Melo wrote:
> > Em Mon, Oct 19, 2015 at 05:16:53PM -0700, Brendan Gregg escreveu:
> > So are you advocating different defaults, one for --stdio (callee),
> > another for --tui, --gtk (caller)?

> > This is all configurable via ~/.perfconfig :-\

> > Indeed, finding a default that is deemed adequate for most people is,
> > ho-hum, difficult 8-)
 
> Most uses I've seen on LKML by the past involved callee because people
> mostly look at the precise point where a performance issue is.

A good chunk of that was because that was the default? 
 
> IMHO changing that order is not a good idea. Unless many users complained
> about it.

Perhaps there are not that many users of callchains because the default
is not what they're used to see?

Motivation for the change came from a video from Chandler, that
resurfaced the callchain default issue, Chandler?

Anedoctally, he tweeted about it and people seemed to like it.
 
> > Ingo, what do you think?

What about providing a hotkey, in the tui, to toggle caller/callee
views, and another hotkey to save that in ~/.perfconfig so that becomes
the new default?

In the --stdio mode we need to add a warning in the first lines,
something like:

# This is caller (or callee) based, please use --callchain ... to change it

One way or the other people will disagree, so making it easy to switch
to the preferred way would be nice to have?

- Arnaldo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1251909 — Re: [PATCH 13/16] perf callchain: Switch default to 'graph,0.5,caller'

On Tue, Oct 20, 2015 at 10:06:51AM -0300, Arnaldo Carvalho de Melo wrote:
> Em Tue, Oct 20, 2015 at 02:19:50PM +0200, Frederic Weisbecker escreveu:
> > On Tue, Oct 20, 2015 at 09:00:34AM -0300, Arnaldo Carvalho de Melo wrote:
> > > Em Mon, Oct 19, 2015 at 05:16:53PM -0700, Brendan Gregg escreveu:
> > > So are you advocating different defaults, one for --stdio (callee),
> > > another for --tui, --gtk (caller)?
> 
> > > This is all configurable via ~/.perfconfig :-\
> 
> > > Indeed, finding a default that is deemed adequate for most people is,
> > > ho-hum, difficult 8-)
>  
> > Most uses I've seen on LKML by the past involved callee because people
> > mostly look at the precise point where a performance issue is.
> 
> A good chunk of that was because that was the default?

I doubt it. When you need to find the culprit of a syscall of IRQ performance issue,
you don't care much to see __libc_start_main() / main() on the top of your callchain.

>  
> > IMHO changing that order is not a good idea. Unless many users complained
> > about it.
> 
> Perhaps there are not that many users of callchains because the default
> is not what they're used to see?
> 
> Motivation for the change came from a video from Chandler, that
> resurfaced the callchain default issue, Chandler?
> 
> Anedoctally, he tweeted about it and people seemed to like it.

Well, I would prefer to hear from regular users than random twitter followers.
I could be wrong so lets ask some users first.


> > > Ingo, what do you think?
> 
> What about providing a hotkey, in the tui, to toggle caller/callee
> views, and another hotkey to save that in ~/.perfconfig so that becomes
> the new default?

That means rebuilding the whole tree, it might be costly and events need to
be processed again.

> 
> In the --stdio mode we need to add a warning in the first lines,
> something like:
> 
> # This is caller (or callee) based, please use --callchain ... to change it
> 
> One way or the other people will disagree, so making it easy to switch
> to the preferred way would be nice to have?

Agreed.

Thanks!
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1251951 — Re: [PATCH 13/16] perf callchain: Switch default to 'graph,0.5,caller'

Em Tue, Oct 20, 2015 at 07:21:16PM +0200, Frederic Weisbecker escreveu:
> On Tue, Oct 20, 2015 at 10:06:51AM -0300, Arnaldo Carvalho de Melo wrote:
> > Em Tue, Oct 20, 2015 at 02:19:50PM +0200, Frederic Weisbecker escreveu:
> > > On Tue, Oct 20, 2015 at 09:00:34AM -0300, Arnaldo Carvalho de Melo wrote:
> > > > Em Mon, Oct 19, 2015 at 05:16:53PM -0700, Brendan Gregg escreveu:
> > > > So are you advocating different defaults, one for --stdio (callee),
> > > > another for --tui, --gtk (caller)?
> > 
> > > > This is all configurable via ~/.perfconfig :-\
> > 
> > > > Indeed, finding a default that is deemed adequate for most people is,
> > > > ho-hum, difficult 8-)
> >  
> > > Most uses I've seen on LKML by the past involved callee because people
> > > mostly look at the precise point where a performance issue is.
> > 
> > A good chunk of that was because that was the default?
> 
> I doubt it. When you need to find the culprit of a syscall of IRQ performance issue,
> you don't care much to see __libc_start_main() / main() on the top of your callchain.

And I noticed some other shortcoming of this ordering (caller) the stack
closer to userspace in a typical perf.data with callchains (perf record
-g) can have bogus addresses (-fomit-frame-pointer) :-\
 
> > > IMHO changing that order is not a good idea. Unless many users complained
> > > about it.
> > 
> > Perhaps there are not that many users of callchains because the default
> > is not what they're used to see?
> > 
> > Motivation for the change came from a video from Chandler, that
> > resurfaced the callchain default issue, Chandler?
> > 
> > Anedoctally, he tweeted about it and people seemed to like it.
> 
> Well, I would prefer to hear from regular users than random twitter followers.
> I could be wrong so lets ask some users first.

That is why I put the "anedoctaly" :-)
 
> > > > Ingo, what do you think?
> > 
> > What about providing a hotkey, in the tui, to toggle caller/callee
> > views, and another hotkey to save that in ~/.perfconfig so that becomes
> > the new default?
 
> That means rebuilding the whole tree, it might be costly and events need to
> be processed again.

I never tried to reimplement that with going to/from caller/caller in
mind, should do at some point, but yeah, reprocessing all entries can be
way costly.
 
> > In the --stdio mode we need to add a warning in the first lines,
> > something like:
> > 
> > # This is caller (or callee) based, please use --callchain ... to change it
> > 
> > One way or the other people will disagree, so making it easy to switch
> > to the preferred way would be nice to have?
> 
> Agreed.

I'll probably revert this change in default and provide some way to
change it dynamically, just waiting a some more time for more comments.

- Arnaldo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1252403 — Re: [PATCH 13/16] perf callchain: Switch default to 'graph,0.5,caller'

Hi Arnaldo,

On Tue, Oct 20, 2015 at 03:44:04PM -0300, Arnaldo Carvalho de Melo wrote:
> Em Tue, Oct 20, 2015 at 07:21:16PM +0200, Frederic Weisbecker escreveu:
> > On Tue, Oct 20, 2015 at 10:06:51AM -0300, Arnaldo Carvalho de Melo wrote:
> > > Em Tue, Oct 20, 2015 at 02:19:50PM +0200, Frederic Weisbecker escreveu:
> > > > On Tue, Oct 20, 2015 at 09:00:34AM -0300, Arnaldo Carvalho de Melo wrote:
> > > > > Em Mon, Oct 19, 2015 at 05:16:53PM -0700, Brendan Gregg escreveu:
> > > > > So are you advocating different defaults, one for --stdio (callee),
> > > > > another for --tui, --gtk (caller)?
> > > 
> > > > > This is all configurable via ~/.perfconfig :-\
> > > 
> > > > > Indeed, finding a default that is deemed adequate for most people is,
> > > > > ho-hum, difficult 8-)
> > >  
> > > > Most uses I've seen on LKML by the past involved callee because people
> > > > mostly look at the precise point where a performance issue is.
> > > 
> > > A good chunk of that was because that was the default?
> > 
> > I doubt it. When you need to find the culprit of a syscall of IRQ performance issue,
> > you don't care much to see __libc_start_main() / main() on the top of your callchain.
> 
> And I noticed some other shortcoming of this ordering (caller) the stack
> closer to userspace in a typical perf.data with callchains (perf record
> -g) can have bogus addresses (-fomit-frame-pointer) :-\

Maybe we need to add an option to ignore unresolved callchains?


>  
> > > > IMHO changing that order is not a good idea. Unless many users complained
> > > > about it.
> > > 
> > > Perhaps there are not that many users of callchains because the default
> > > is not what they're used to see?
> > > 
> > > Motivation for the change came from a video from Chandler, that
> > > resurfaced the callchain default issue, Chandler?
> > > 
> > > Anedoctally, he tweeted about it and people seemed to like it.
> > 
> > Well, I would prefer to hear from regular users than random twitter followers.
> > I could be wrong so lets ask some users first.
> 
> That is why I put the "anedoctaly" :-)
>  
> > > > > Ingo, what do you think?
> > > 
> > > What about providing a hotkey, in the tui, to toggle caller/callee
> > > views, and another hotkey to save that in ~/.perfconfig so that becomes
> > > the new default?
>  
> > That means rebuilding the whole tree, it might be costly and events need to
> > be processed again.
> 
> I never tried to reimplement that with going to/from caller/caller in
> mind, should do at some point, but yeah, reprocessing all entries can be
> way costly.

Right.  Anyway it seems useful to update config dynamically.  I really
need to take a look at Taeung's config patchset..


>  
> > > In the --stdio mode we need to add a warning in the first lines,
> > > something like:
> > > 
> > > # This is caller (or callee) based, please use --callchain ... to change it
> > > 
> > > One way or the other people will disagree, so making it easy to switch
> > > to the preferred way would be nice to have?
> > 
> > Agreed.
> 
> I'll probably revert this change in default and provide some way to
> change it dynamically, just waiting a some more time for more comments.

Agreed.

Thanks,
Namhyung
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1252816 — Re: [PATCH 13/16] perf callchain: Switch default to 'graph,0.5,caller'

Em Wed, Oct 21, 2015 at 10:21:36AM +0900, Namhyung Kim escreveu:
> On Tue, Oct 20, 2015 at 03:44:04PM -0300, Arnaldo Carvalho de Melo wrote:
> > Em Tue, Oct 20, 2015 at 07:21:16PM +0200, Frederic Weisbecker escreveu:
> > > On Tue, Oct 20, 2015 at 10:06:51AM -0300, Arnaldo Carvalho de Melo wrote:
> > And I noticed some other shortcoming of this ordering (caller) the stack
> > closer to userspace in a typical perf.data with callchains (perf record
> > -g) can have bogus addresses (-fomit-frame-pointer) :-\
 
> Maybe we need to add an option to ignore unresolved callchains?

We have:

    -U, --hide-unresolved
                          Only display entries resolved to a symbol

But I think the problem here is different, its not about unresolved
stuff, its about invalid addresses, no?

- Arnaldo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1252526 — Re: [PATCH 13/16] perf callchain: Switch default to 'graph,0.5,caller'

Hi Frederic,

On Tue, Oct 20, 2015 at 07:21:16PM +0200, Frederic Weisbecker wrote:
> On Tue, Oct 20, 2015 at 10:06:51AM -0300, Arnaldo Carvalho de Melo wrote:
> > Em Tue, Oct 20, 2015 at 02:19:50PM +0200, Frederic Weisbecker escreveu:
> > > On Tue, Oct 20, 2015 at 09:00:34AM -0300, Arnaldo Carvalho de Melo wrote:
> > > > Em Mon, Oct 19, 2015 at 05:16:53PM -0700, Brendan Gregg escreveu:
> > > > So are you advocating different defaults, one for --stdio (callee),
> > > > another for --tui, --gtk (caller)?
> > 
> > > > This is all configurable via ~/.perfconfig :-\
> > 
> > > > Indeed, finding a default that is deemed adequate for most people is,
> > > > ho-hum, difficult 8-)
> >  
> > > Most uses I've seen on LKML by the past involved callee because people
> > > mostly look at the precise point where a performance issue is.
> > 
> > A good chunk of that was because that was the default?
> 
> I doubt it. When you need to find the culprit of a syscall of IRQ performance issue,
> you don't care much to see __libc_start_main() / main() on the top of your callchain.
> 
> >  
> > > IMHO changing that order is not a good idea. Unless many users complained
> > > about it.
> > 
> > Perhaps there are not that many users of callchains because the default
> > is not what they're used to see?
> > 
> > Motivation for the change came from a video from Chandler, that
> > resurfaced the callchain default issue, Chandler?
> > 
> > Anedoctally, he tweeted about it and people seemed to like it.
> 
> Well, I would prefer to hear from regular users than random twitter followers.
> I could be wrong so lets ask some users first.

Just a question.  Do you often use --children and/or '--g caller' options?

I guess that for most kernel developers, --children is not that useful
as you said.  But I think it can be useful for many userspace
developers and with '-g caller' it can be even more useful. :)

When '-g caller' is used, the callchains shown in a (self) entry are
less important IMHO.  However callchains in entries generated by
--children will show which functions are called by the entry (since
it's reversed!) and will be more important.

Thanks,
Namhyung
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1252763

On 2015/10/21 16:09, Namhyung Kim wrote:
> Hi Frederic,
>
> On Tue, Oct 20, 2015 at 07:21:16PM +0200, Frederic Weisbecker wrote:
>> On Tue, Oct 20, 2015 at 10:06:51AM -0300, Arnaldo Carvalho de Melo wrote:
>>> Em Tue, Oct 20, 2015 at 02:19:50PM +0200, Frederic Weisbecker escreveu:
>>>> On Tue, Oct 20, 2015 at 09:00:34AM -0300, Arnaldo Carvalho de Melo wrote:
>>>>> Em Mon, Oct 19, 2015 at 05:16:53PM -0700, Brendan Gregg escreveu:
>>>>> So are you advocating different defaults, one for --stdio (callee),
>>>>> another for --tui, --gtk (caller)?
>>>>> This is all configurable via ~/.perfconfig :-\
>>>>> Indeed, finding a default that is deemed adequate for most people is,
>>>>> ho-hum, difficult 8-)
>>>   
>>>> Most uses I've seen on LKML by the past involved callee because people
>>>> mostly look at the precise point where a performance issue is.
>>> A good chunk of that was because that was the default?
>> I doubt it. When you need to find the culprit of a syscall of IRQ performance issue,
>> you don't care much to see __libc_start_main() / main() on the top of your callchain.
>>
>>>   
>>>> IMHO changing that order is not a good idea. Unless many users complained
>>>> about it.
>>> Perhaps there are not that many users of callchains because the default
>>> is not what they're used to see?
>>>
>>> Motivation for the change came from a video from Chandler, that
>>> resurfaced the callchain default issue, Chandler?
>>>
>>> Anedoctally, he tweeted about it and people seemed to like it.
>> Well, I would prefer to hear from regular users than random twitter followers.
>> I could be wrong so lets ask some users first.
> Just a question.  Do you often use --children and/or '--g caller' options?

For me, I always use --no-children. However, I think it is because
I have used to --no-children and no one teach me how to utilize the
additional information --children provided. In case when result
of --no-children hard to explain I use Brendan's flame graph tool.

Thank you.

> I guess that for most kernel developers, --children is not that useful
> as you said.  But I think it can be useful for many userspace
> developers and with '-g caller' it can be even more useful. :)
>
> When '-g caller' is used, the callchains shown in a (self) entry are
> less important IMHO.  However callchains in entries generated by
> --children will show which functions are called by the entry (since
> it's reversed!) and will be more important.
>
> Thanks,
> Namhyung


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1253046 — Re: [PATCH 13/16] perf callchain: Switch default to 'graph,0.5,caller'

On Wed, Oct 21, 2015 at 05:09:08PM +0900, Namhyung Kim wrote:
> On Tue, Oct 20, 2015 at 07:21:16PM +0200, Frederic Weisbecker wrote:
> > Well, I would prefer to hear from regular users than random twitter followers.
> > I could be wrong so lets ask some users first.
> 
> Just a question.  Do you often use --children and/or '--g caller' options?

I must confess I haven't used perf much for some time now. So I never
really used --children but I'm fairly convinced it's a very useful option.

> 
> I guess that for most kernel developers, --children is not that useful
> as you said.

Not sure about that. It provides a good overview.

> But I think it can be useful for many userspace
> developers and with '-g caller' it can be even more useful. :)
> 
> When '-g caller' is used, the callchains shown in a (self) entry are
> less important IMHO.  However callchains in entries generated by
> --children will show which functions are called by the entry (since
> it's reversed!) and will be more important.

Right and in fact the ever growing adoption of perf by userland developers
makes we start to understand why people are interested in caller based
callchains. Callee-based zoom into low level details that many people are not
interested in.

I think that if people could filter the callchain to only keep the part
concerning their binary, callee may suddenly become interesting for them.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1253032 — Re: [PATCH 13/16] perf callchain: Switch default to 'graph,0.5,caller'

On Wed, Oct 21, 2015 at 02:21:12AM +0000, Chandler Carruth wrote:
> On Tue, Oct 20, 2015 at 3:06 AM Arnaldo Carvalho de Melo <
> arnaldo.melo@gmail.com> wrote:
> 
> > > IMHO changing that order is not a good idea. Unless many users complained
> > > about it.
> >
> > Perhaps there are not that many users of callchains because the default
> > is not what they're used to see?
> >
> > Motivation for the change came from a video from Chandler, that
> > resurfaced the callchain default issue, Chandler?
> >
> 
> So, first and foremost, thanks for fixing some of my gripes about the
> usability of the perf tool, I'm super excited about the changes you're
> making, even if this one isn't among them.
> 
> I think the default of caller vs. callee is probably the hardest judgement
> call to make about the right defaults. I can see it going both ways.
> 
> When profiling my *system*, or a diverse group of programs or tasks, I
> often find callee useful. Were I a kernel developer, I suspect callee would
> be *dramatically* more common than caller.
> 
> For me, what makes the caller view much more frequently desired is that I'm
> usually profiling a fairly isolated application, or benchmark for an
> isolated library. While I always start off with some more system-level
> performance problem, I rarely need a detailed profile to get a reasonable
> idea of what subsystem to stare at, and then I spend days looking at a
> relatively isolated reproduction.

I understand it that way: callee based is good when you look for a specific issue
to resolve and caller based is better when you want an overview of an object.

That makes sense.

> 
> Anyways, for profiling user-land applications, I suspect from my
> conversations with users that "caller" is the more common expectation.

I wonder what would be the result if people were to use callchains that only involve
the user part. Maybe they prefer caller based because they don't care about
the kernel part.

> > What about providing a hotkey, in the tui, to toggle caller/callee
> > views, and another hotkey to save that in ~/.perfconfig so that becomes
> > the new default?
> >
> 
> OMG, being able to toggle between caller and callee in the tui would be
> *awesome*. Regardless of which default you end up with, I'd love to have
> this feature.

Indeed it could be interesting.

I we want that toggling to be fast enough, we need to process both callee and
caller trees on hists processing, and not rebuild the entire tree each time we
toggle (which would be costly). That's fairly possible to do and it might not
even impact much the loading time if we do this in multithread.

Thanks!
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1253108

On Tue, Oct 20, 2015 at 7:21 PM, Chandler Carruth <chandlerc@gmail.com> wrote:
> Since Arnaldo asked, I thought I should actually try to respond specifically
> to the question of why I favor the 'caller' view as the default.
>
> On Tue, Oct 20, 2015 at 3:06 AM Arnaldo Carvalho de Melo
> <arnaldo.melo@gmail.com> wrote:
>>
>> > IMHO changing that order is not a good idea. Unless many users
>> > complained
>> > about it.
>>
>> Perhaps there are not that many users of callchains because the default
>> is not what they're used to see?
>>
>> Motivation for the change came from a video from Chandler, that
>> resurfaced the callchain default issue, Chandler?
>
>
> So, first and foremost, thanks for fixing some of my gripes about the
> usability of the perf tool, I'm super excited about the changes you're
> making, even if this one isn't among them.
>
> I think the default of caller vs. callee is probably the hardest judgement
> call to make about the right defaults. I can see it going both ways.
>
> When profiling my *system*, or a diverse group of programs or tasks, I often
> find callee useful. Were I a kernel developer, I suspect callee would be
> *dramatically* more common than caller.
>
> For me, what makes the caller view much more frequently desired is that I'm
> usually profiling a fairly isolated application, or benchmark for an
> isolated library. While I always start off with some more system-level
> performance problem, I rarely need a detailed profile to get a reasonable
> idea of what subsystem to stare at, and then I spend days looking at a
> relatively isolated reproduction.
>
> Anyways, for profiling user-land applications, I suspect from my
> conversations with users that "caller" is the more common expectation.
[...]

I would have said callee is the more common expectation, certainly for
system profilers. I quickly checked various tools to see what their
defaults are:

callee:

gdb
lldb
pstack
jstack
perf
stap
ktap
dtrace
kernel oops message
jvm crash
node.js/v8 crash

caller:

python traceback


Python was the only one I knew off-hand that is caller by default (and
it includes the text "most recent call last", suggesting the
developers thought it was necessary to point out that it was
different). Do you know what tools/profilers/debuggers these users are
using?

I really think people will be surprised if by default perf prints
stacks differently to gdb, lldb, oops messages, etc, etc. That may be
true for a specific developer community (eg, Python), but not for
system profilers.

Brendan
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1253149 — Re: [PATCH 13/16] perf callchain: Switch default to 'graph,0.5,caller'

Em Wed, Oct 21, 2015 at 11:28:54AM -0700, Brendan Gregg escreveu:
> On Tue, Oct 20, 2015 at 7:21 PM, Chandler Carruth <chandlerc@gmail.com> wrote:
> > Since Arnaldo asked, I thought I should actually try to respond specifically
> > to the question of why I favor the 'caller' view as the default.
> >
> > On Tue, Oct 20, 2015 at 3:06 AM Arnaldo Carvalho de Melo
> > <arnaldo.melo@gmail.com> wrote:
> >>
> >> > IMHO changing that order is not a good idea. Unless many users
> >> > complained
> >> > about it.
> >>
> >> Perhaps there are not that many users of callchains because the default
> >> is not what they're used to see?
> >>
> >> Motivation for the change came from a video from Chandler, that
> >> resurfaced the callchain default issue, Chandler?
> >
> >
> > So, first and foremost, thanks for fixing some of my gripes about the
> > usability of the perf tool, I'm super excited about the changes you're
> > making, even if this one isn't among them.
> >
> > I think the default of caller vs. callee is probably the hardest judgement
> > call to make about the right defaults. I can see it going both ways.
> >
> > When profiling my *system*, or a diverse group of programs or tasks, I often
> > find callee useful. Were I a kernel developer, I suspect callee would be
> > *dramatically* more common than caller.
> >
> > For me, what makes the caller view much more frequently desired is that I'm
> > usually profiling a fairly isolated application, or benchmark for an
> > isolated library. While I always start off with some more system-level
> > performance problem, I rarely need a detailed profile to get a reasonable
> > idea of what subsystem to stare at, and then I spend days looking at a
> > relatively isolated reproduction.
> >
> > Anyways, for profiling user-land applications, I suspect from my
> > conversations with users that "caller" is the more common expectation.
> [...]
> 
> I would have said callee is the more common expectation, certainly for
> system profilers. I quickly checked various tools to see what their
> defaults are:
> 
> callee:
> 
> gdb
> lldb
> pstack
> jstack
> perf
> stap
> ktap
> dtrace
> kernel oops message
> jvm crash
> node.js/v8 crash
> 
> caller:
> 
> python traceback

sysprof -> http://sysprof.com/screen-shot-4.png
 
> 
> Python was the only one I knew off-hand that is caller by default (and
> it includes the text "most recent call last", suggesting the
> developers thought it was necessary to point out that it was
> different). Do you know what tools/profilers/debuggers these users are
> using?
> 
> I really think people will be surprised if by default perf prints
> stacks differently to gdb, lldb, oops messages, etc, etc. That may be
> true for a specific developer community (eg, Python), but not for
> system profilers.
> 
> Brendan
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1253376

On Wed, Oct 21, 2015 at 12:23 PM, Arnaldo Carvalho de Melo
<arnaldo.melo@gmail.com> wrote:
> Em Wed, Oct 21, 2015 at 11:28:54AM -0700, Brendan Gregg escreveu:
>> On Tue, Oct 20, 2015 at 7:21 PM, Chandler Carruth <chandlerc@gmail.com> wrote:
>> > Since Arnaldo asked, I thought I should actually try to respond specifically
>> > to the question of why I favor the 'caller' view as the default.
>> >
>> > On Tue, Oct 20, 2015 at 3:06 AM Arnaldo Carvalho de Melo
>> > <arnaldo.melo@gmail.com> wrote:
>> >>
>> >> > IMHO changing that order is not a good idea. Unless many users
>> >> > complained
>> >> > about it.
>> >>
>> >> Perhaps there are not that many users of callchains because the default
>> >> is not what they're used to see?
>> >>
>> >> Motivation for the change came from a video from Chandler, that
>> >> resurfaced the callchain default issue, Chandler?
>> >
>> >
>> > So, first and foremost, thanks for fixing some of my gripes about the
>> > usability of the perf tool, I'm super excited about the changes you're
>> > making, even if this one isn't among them.
>> >
>> > I think the default of caller vs. callee is probably the hardest judgement
>> > call to make about the right defaults. I can see it going both ways.
>> >
>> > When profiling my *system*, or a diverse group of programs or tasks, I often
>> > find callee useful. Were I a kernel developer, I suspect callee would be
>> > *dramatically* more common than caller.
>> >
>> > For me, what makes the caller view much more frequently desired is that I'm
>> > usually profiling a fairly isolated application, or benchmark for an
>> > isolated library. While I always start off with some more system-level
>> > performance problem, I rarely need a detailed profile to get a reasonable
>> > idea of what subsystem to stare at, and then I spend days looking at a
>> > relatively isolated reproduction.
>> >
>> > Anyways, for profiling user-land applications, I suspect from my
>> > conversations with users that "caller" is the more common expectation.
>> [...]
>>
>> I would have said callee is the more common expectation, certainly for
>> system profilers. I quickly checked various tools to see what their
>> defaults are:
>>
>> callee:
>>
>> gdb
>> lldb
>> pstack
>> jstack
>> perf
>> stap
>> ktap
>> dtrace
>> kernel oops message
>> jvm crash
>> node.js/v8 crash
>>
>> caller:
>>
>> python traceback
>
> sysprof -> http://sysprof.com/screen-shot-4.png

Right, GUIs with tree views do this.

I've thought about it more, and how does the following 3 changes sound?:

1. perf top: change to caller default

- Since it's dumping a tree view, this ordering is intuitive, and this
is the convention from other interactive GUIs that do tree views.
Optionally add a hotkey to change it live. Could also add -G from perf
report (--inverted) to easily call the old behavior.

2. perf report: change to caller default

- I'd like to leave it callee, but as Ingo said, "top" and "report"
should be unified. I can always run -G, or -g, or edit ~/.perfconfig.

3. perf script: leave callee default.

- Since it's dumping a full stack, that's the convention with other
tools that dump full stacks (as I listed), and I think that's useful
ordering anyway when reading a full stack: show me event context
first, followed by history next. "perf script" should have -G and -g
added, to adjust behavior.

I'd also like different UIs from perf report (folded and json), but
that's a different thread. :)

Brendan
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1252533 — Re: [PATCH 13/16] perf callchain: Switch default to 'graph,0.5,caller'

* Arnaldo Carvalho de Melo <arnaldo.melo@gmail.com> wrote:

> Indeed, finding a default that is deemed adequate for most people is, ho-hum, 
> difficult 8-)
> 
> Ingo, what do you think?

So I think the new default is a lot more intuitive, because it starts with a high 
level top-down 'overview' kind of output, allowing one to drill down to the 
details.

That's what most other tools do as well by default, and people migrating to perf 
found it pretty confusing that we do a bottom-up kind of callgraph.

To people using perf for the first time it does not matter nearly as much, but 
intuitiveness is still a factor.

The new default requires good call-chains and good symbol resolution to be 
maximally effective, but we want perfect callchains and symbols in any case, 
right?

Thanks,

	Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1252809 — Re: [PATCH 13/16] perf callchain: Switch default to 'graph,0.5,caller'

Em Wed, Oct 21, 2015 at 10:06:45AM +0200, Ingo Molnar escreveu:
> * Arnaldo Carvalho de Melo <arnaldo.melo@gmail.com> wrote:
> > Indeed, finding a default that is deemed adequate for most people is, ho-hum, 
> > difficult 8-)

> > Ingo, what do you think?
 
> So I think the new default is a lot more intuitive, because it starts with a high 
> level top-down 'overview' kind of output, allowing one to drill down to the 
> details.
 
> That's what most other tools do as well by default, and people migrating to perf 
> found it pretty confusing that we do a bottom-up kind of callgraph.
 
> To people using perf for the first time it does not matter nearly as much, but 
> intuitiveness is still a factor.
 
> The new default requires good call-chains and good symbol resolution to be 
> maximally effective, but we want perfect callchains and symbols in any case, 
> right?

Sure, but in hindsight, we could've fixed those problems before flipping
the switch :-)

And also we didn't care about it that much because for callee based
callchains they are far from the focus point.

- Arnaldo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1253141

On Wed, Oct 21, 2015 at 1:06 AM, Ingo Molnar <mingo@kernel.org> wrote:
>
> * Arnaldo Carvalho de Melo <arnaldo.melo@gmail.com> wrote:
>
>> Indeed, finding a default that is deemed adequate for most people is, ho-hum,
>> difficult 8-)
>>
>> Ingo, what do you think?
>
> So I think the new default is a lot more intuitive, because it starts with a high
> level top-down 'overview' kind of output, allowing one to drill down to the
> details.
>

Yes, it may be more intuitive.

> That's what most other tools do as well by default, and people migrating to perf
> found it pretty confusing that we do a bottom-up kind of callgraph.
> [...]

But I don't think it's commonplace. Which other tools do this? Java
Flight Recorder? GUIs? Certainly not text-based system profilers or
debuggers (I made a list in another email).

Although, these are usually printing entire stacks. So the most alien
change would be to flip the "perf script" order (please don't do
this). You could argue that the perf report/top tree view is not
commonplace from other text-based profilers or debuggers, so changing
it won't be as confusing. In fact, it's like a sideways flame graph
(except it doesn't fit on the screen!).

Brendan
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web