Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1345219 > unrolled thread

[RFC] perf: probe_finder: continue if atleast one probe point found

Started byJoel Fernandes <joel.opensrc@gmail.com>
First post2016-02-28 12:20 +0100
Last post2016-03-01 11:40 +0100
Articles 7 — 5 participants

Back to article view | Back to linux.kernel

Contents

  [RFC] perf: probe_finder: continue if atleast one probe point found Joel Fernandes <joel.opensrc@gmail.com> - 2016-02-28 12:20 +0100
    Re: [RFC] perf: probe_finder: continue if atleast one probe point  found Taeung Song <treeze.taeung@gmail.com> - 2016-02-28 12:40 +0100
      Re: [RFC] perf: probe_finder: continue if atleast one probe point  found Arnaldo Carvalho de Melo <acme@kernel.org> - 2016-02-29 15:20 +0100
        RE: [RFC] perf: probe_finder: continue if atleast one probe point  found 平松雅巳 / HIRAMATU，MASAMI   <masami.hiramatsu.pt@hitachi.com> - 2016-03-01 04:20 +0100
          RE: [RFC] perf: probe_finder: continue if atleast one probe point  found 平松雅巳 / HIRAMATU，MASAMI   <masami.hiramatsu.pt@hitachi.com> - 2016-03-01 05:30 +0100
            Re: [RFC] perf: probe_finder: continue if atleast one probe point found Joel Fernandes <agnel.joel@gmail.com> - 2016-03-01 11:40 +0100
            Re: [RFC] perf: probe_finder: continue if atleast one probe point found Joel Fernandes <agnel.joel@gmail.com> - 2016-03-01 11:40 +0100

#1345219 — [RFC] perf: probe_finder: continue if atleast one probe point found

FromJoel Fernandes <joel.opensrc@gmail.com>
Date2016-02-28 12:20 +0100
Subject[RFC] perf: probe_finder: continue if atleast one probe point found
Message-ID<r77yh-1rP-9@gated-at.bofh.it>
Sometimes for inline functions, perf probe can fail such as if an arguments are
requested at probe points. This is probably because when the compiler inlines,
for some instances it optimizes away arguments. Either way, the DWARF has
missing arguments for certain probe points of inline functions causing 'perf
probe' to fail. I noticed this when probing the C library that ships with my
distribution. With the following patch I am successfully able to record kprobe
events with arguments.

Test
----
 ./perf probe \
   -k ./vmlinux -s ./ -x /lib/x86_64-linux-gnu/libc.so.6 -a 'malloc bytes' -v

Without the patch
-----------------
Matched function: __libc_malloc
found inline addr: 0x831a6
Probe point found: malloc_atfork+150
Searching 'bytes' variable in context.
Converting variable bytes into trace event.
bytes type is long unsigned int.
found inline addr: 0x844a0
Probe point found: __libc_malloc+0
Searching 'bytes' variable in context.
Converting variable bytes into trace event.
bytes type is long unsigned int.
found inline addr: 0x8463b
Probe point found: __malloc_get_state+11
Searching 'bytes' variable in context.
Failed to find 'bytes' in this function.
An error occurred in debuginfo analysis (-2).
  Error: Failed to add events. Reason: No such file or directory (Code: -2)

With the patch
--------------
Open Debuginfo file: /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.21.so
Try to find probe point from debuginfo.
Symbol malloc address found : 844a0
Matched function: __libc_malloc
found inline addr: 0x831a6
Probe point found: malloc_atfork+150
Searching 'bytes' variable in context.
Converting variable bytes into trace event.
bytes type is long unsigned int.
found inline addr: 0x844a0
Probe point found: __libc_malloc+0
Searching 'bytes' variable in context.
Converting variable bytes into trace event.
bytes type is long unsigned int.
found inline addr: 0x8463b
Probe point found: __malloc_get_state+11
Searching 'bytes' variable in context.
Failed to find 'bytes' in this function.
Probe point error, ignoring. Atleast one probe point found
found inline addr: 0x84bba
Probe point found: __libc_realloc+410
Searching 'bytes' variable in context.
Failed to find 'bytes' in this function.
Probe point error, ignoring. Atleast one probe point found
found inline addr: 0x84d08
...
Probe point error, ignoring. Atleast one probe point found
Found 3 probe_trace_events.
Opening /sys/kernel/debug/tracing//uprobe_events write=1
Writing event: p:probe_libc/malloc /lib/x86_64-linux-gnu/libc-2.21.so:0x831a6 bytes=%bp:u64
Writing event: p:probe_libc/malloc_1 /lib/x86_64-linux-gnu/libc-2.21.so:0x844a0 bytes=%di:u64
Writing event: p:probe_libc/malloc_2 /lib/x86_64-linux-gnu/libc-2.21.so:0x85af7 bytes=%di:u64
Added new events:
  probe_libc:malloc    (on malloc in /lib/x86_64-linux-gnu/libc-2.21.so with bytes)
  probe_libc:malloc_1  (on malloc in /lib/x86_64-linux-gnu/libc-2.21.so with bytes)
  probe_libc:malloc_2  (on malloc in /lib/x86_64-linux-gnu/libc-2.21.so with bytes)

Signed-off-by: Joel Fernandes <agnel.joel@gmail.com>
---
 tools/perf/util/probe-finder.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/tools/perf/util/probe-finder.c b/tools/perf/util/probe-finder.c
index 4ce5c5e..3ac9481 100644
--- a/tools/perf/util/probe-finder.c
+++ b/tools/perf/util/probe-finder.c
@@ -1255,6 +1255,10 @@ end:
 	if (ret) {
 		clear_probe_trace_event(tev);
 		tf->ntevs--;
+		if (tf->ntevs != 0) {
+			pr_debug("Ignoring error as atleast one probe point found.\n");
+			ret = 0;
+		}
 	}
 	free(args);
 	return ret;
-- 
2.5.0

[toc] | [next] | [standalone]

#1345224 — Re: [RFC] perf: probe_finder: continue if atleast one probe point found

FromTaeung Song <treeze.taeung@gmail.com>
Date2016-02-28 12:40 +0100
SubjectRe: [RFC] perf: probe_finder: continue if atleast one probe point found
Message-ID<r77RD-1De-7@gated-at.bofh.it>
In reply to#1345219
Hi, Joel

perf subsystem's maintainers are
Arnaldo Carvalho de Melo <acme@kernel.org>
Peter Zijlstra <peterz@infradead.org>
Ingo Molnar <mingo@redhat.com>

(or Namhyung Kim <namhyung@kernel.org>, Jiri Olsa <jolsa@kernel.org>)

AFAIK,
When sending some patches, it would be better to add them into To: or Cc:


Thanks,
Taeung

On 02/28/2016 08:14 PM, Joel Fernandes wrote:
> Sometimes for inline functions, perf probe can fail such as if an arguments are
> requested at probe points. This is probably because when the compiler inlines,
> for some instances it optimizes away arguments. Either way, the DWARF has
> missing arguments for certain probe points of inline functions causing 'perf
> probe' to fail. I noticed this when probing the C library that ships with my
> distribution. With the following patch I am successfully able to record kprobe
> events with arguments.
>
> Test
> ----
>   ./perf probe \
>     -k ./vmlinux -s ./ -x /lib/x86_64-linux-gnu/libc.so.6 -a 'malloc bytes' -v
>
> Without the patch
> -----------------
> Matched function: __libc_malloc
> found inline addr: 0x831a6
> Probe point found: malloc_atfork+150
> Searching 'bytes' variable in context.
> Converting variable bytes into trace event.
> bytes type is long unsigned int.
> found inline addr: 0x844a0
> Probe point found: __libc_malloc+0
> Searching 'bytes' variable in context.
> Converting variable bytes into trace event.
> bytes type is long unsigned int.
> found inline addr: 0x8463b
> Probe point found: __malloc_get_state+11
> Searching 'bytes' variable in context.
> Failed to find 'bytes' in this function.
> An error occurred in debuginfo analysis (-2).
>    Error: Failed to add events. Reason: No such file or directory (Code: -2)
>
> With the patch
> --------------
> Open Debuginfo file: /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.21.so
> Try to find probe point from debuginfo.
> Symbol malloc address found : 844a0
> Matched function: __libc_malloc
> found inline addr: 0x831a6
> Probe point found: malloc_atfork+150
> Searching 'bytes' variable in context.
> Converting variable bytes into trace event.
> bytes type is long unsigned int.
> found inline addr: 0x844a0
> Probe point found: __libc_malloc+0
> Searching 'bytes' variable in context.
> Converting variable bytes into trace event.
> bytes type is long unsigned int.
> found inline addr: 0x8463b
> Probe point found: __malloc_get_state+11
> Searching 'bytes' variable in context.
> Failed to find 'bytes' in this function.
> Probe point error, ignoring. Atleast one probe point found
> found inline addr: 0x84bba
> Probe point found: __libc_realloc+410
> Searching 'bytes' variable in context.
> Failed to find 'bytes' in this function.
> Probe point error, ignoring. Atleast one probe point found
> found inline addr: 0x84d08
> ...
> Probe point error, ignoring. Atleast one probe point found
> Found 3 probe_trace_events.
> Opening /sys/kernel/debug/tracing//uprobe_events write=1
> Writing event: p:probe_libc/malloc /lib/x86_64-linux-gnu/libc-2.21.so:0x831a6 bytes=%bp:u64
> Writing event: p:probe_libc/malloc_1 /lib/x86_64-linux-gnu/libc-2.21.so:0x844a0 bytes=%di:u64
> Writing event: p:probe_libc/malloc_2 /lib/x86_64-linux-gnu/libc-2.21.so:0x85af7 bytes=%di:u64
> Added new events:
>    probe_libc:malloc    (on malloc in /lib/x86_64-linux-gnu/libc-2.21.so with bytes)
>    probe_libc:malloc_1  (on malloc in /lib/x86_64-linux-gnu/libc-2.21.so with bytes)
>    probe_libc:malloc_2  (on malloc in /lib/x86_64-linux-gnu/libc-2.21.so with bytes)
>
> Signed-off-by: Joel Fernandes <agnel.joel@gmail.com>
> ---
>   tools/perf/util/probe-finder.c | 4 ++++
>   1 file changed, 4 insertions(+)
>
> diff --git a/tools/perf/util/probe-finder.c b/tools/perf/util/probe-finder.c
> index 4ce5c5e..3ac9481 100644
> --- a/tools/perf/util/probe-finder.c
> +++ b/tools/perf/util/probe-finder.c
> @@ -1255,6 +1255,10 @@ end:
>   	if (ret) {
>   		clear_probe_trace_event(tev);
>   		tf->ntevs--;
> +		if (tf->ntevs != 0) {
> +			pr_debug("Ignoring error as atleast one probe point found.\n");
> +			ret = 0;
> +		}
>   	}
>   	free(args);
>   	return ret;
>

[toc] | [prev] | [next] | [standalone]

#1345859 — Re: [RFC] perf: probe_finder: continue if atleast one probe point found

FromArnaldo Carvalho de Melo <acme@kernel.org>
Date2016-02-29 15:20 +0100
SubjectRe: [RFC] perf: probe_finder: continue if atleast one probe point found
Message-ID<r7wQ2-483-7@gated-at.bofh.it>
In reply to#1345224
Em Sun, Feb 28, 2016 at 03:49:44AM -0800, Joel Fernandes escreveu:
> Taeung,
> 
> Thanks for that. As this is an RFC, I have added them now to this email
> thread and when I post the final patch, I'll include them in CC.
> 
> All,
> Please provide your comments on my patch.

Masami, are you ok with this patch?

- Arnaldo
 
> Thanks,
> Joel
> 
> On Sun, Feb 28, 2016 at 3:35 AM, Taeung Song <treeze.taeung@gmail.com>
> wrote:
> 
> > Hi, Joel
> >
> > perf subsystem's maintainers are
> > Arnaldo Carvalho de Melo <acme@kernel.org>
> > Peter Zijlstra <peterz@infradead.org>
> > Ingo Molnar <mingo@redhat.com>
> >
> > (or Namhyung Kim <namhyung@kernel.org>, Jiri Olsa <jolsa@kernel.org>)
> >
> > AFAIK,
> > When sending some patches, it would be better to add them into To: or Cc:
> >
> >
> > Thanks,
> > Taeung
> >
> >
> > On 02/28/2016 08:14 PM, Joel Fernandes wrote:
> >
> >> Sometimes for inline functions, perf probe can fail such as if an
> >> arguments are
> >> requested at probe points. This is probably because when the compiler
> >> inlines,
> >> for some instances it optimizes away arguments. Either way, the DWARF has
> >> missing arguments for certain probe points of inline functions causing
> >> 'perf
> >> probe' to fail. I noticed this when probing the C library that ships with
> >> my
> >> distribution. With the following patch I am successfully able to record
> >> kprobe
> >> events with arguments.
> >>
> >> Test
> >> ----
> >>   ./perf probe \
> >>     -k ./vmlinux -s ./ -x /lib/x86_64-linux-gnu/libc.so.6 -a 'malloc
> >> bytes' -v
> >>
> >> Without the patch
> >> -----------------
> >> Matched function: __libc_malloc
> >> found inline addr: 0x831a6
> >> Probe point found: malloc_atfork+150
> >> Searching 'bytes' variable in context.
> >> Converting variable bytes into trace event.
> >> bytes type is long unsigned int.
> >> found inline addr: 0x844a0
> >> Probe point found: __libc_malloc+0
> >> Searching 'bytes' variable in context.
> >> Converting variable bytes into trace event.
> >> bytes type is long unsigned int.
> >> found inline addr: 0x8463b
> >> Probe point found: __malloc_get_state+11
> >> Searching 'bytes' variable in context.
> >> Failed to find 'bytes' in this function.
> >> An error occurred in debuginfo analysis (-2).
> >>    Error: Failed to add events. Reason: No such file or directory (Code:
> >> -2)
> >>
> >> With the patch
> >> --------------
> >> Open Debuginfo file: /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.21.so
> >> Try to find probe point from debuginfo.
> >> Symbol malloc address found : 844a0
> >> Matched function: __libc_malloc
> >> found inline addr: 0x831a6
> >> Probe point found: malloc_atfork+150
> >> Searching 'bytes' variable in context.
> >> Converting variable bytes into trace event.
> >> bytes type is long unsigned int.
> >> found inline addr: 0x844a0
> >> Probe point found: __libc_malloc+0
> >> Searching 'bytes' variable in context.
> >> Converting variable bytes into trace event.
> >> bytes type is long unsigned int.
> >> found inline addr: 0x8463b
> >> Probe point found: __malloc_get_state+11
> >> Searching 'bytes' variable in context.
> >> Failed to find 'bytes' in this function.
> >> Probe point error, ignoring. Atleast one probe point found
> >> found inline addr: 0x84bba
> >> Probe point found: __libc_realloc+410
> >> Searching 'bytes' variable in context.
> >> Failed to find 'bytes' in this function.
> >> Probe point error, ignoring. Atleast one probe point found
> >> found inline addr: 0x84d08
> >> ...
> >> Probe point error, ignoring. Atleast one probe point found
> >> Found 3 probe_trace_events.
> >> Opening /sys/kernel/debug/tracing//uprobe_events write=1
> >> Writing event: p:probe_libc/malloc /lib/x86_64-linux-gnu/libc-2.21.so:0x831a6
> >> bytes=%bp:u64
> >> Writing event: p:probe_libc/malloc_1 /lib/x86_64-linux-gnu/libc-2.21.so:0x844a0
> >> bytes=%di:u64
> >> Writing event: p:probe_libc/malloc_2 /lib/x86_64-linux-gnu/libc-2.21.so:0x85af7
> >> bytes=%di:u64
> >> Added new events:
> >>    probe_libc:malloc    (on malloc in /lib/x86_64-linux-gnu/libc-2.21.so
> >> with bytes)
> >>    probe_libc:malloc_1  (on malloc in /lib/x86_64-linux-gnu/libc-2.21.so
> >> with bytes)
> >>    probe_libc:malloc_2  (on malloc in /lib/x86_64-linux-gnu/libc-2.21.so
> >> with bytes)
> >>
> >> Signed-off-by: Joel Fernandes <agnel.joel@gmail.com>
> >> ---
> >>   tools/perf/util/probe-finder.c | 4 ++++
> >>   1 file changed, 4 insertions(+)
> >>
> >> diff --git a/tools/perf/util/probe-finder.c
> >> b/tools/perf/util/probe-finder.c
> >> index 4ce5c5e..3ac9481 100644
> >> --- a/tools/perf/util/probe-finder.c
> >> +++ b/tools/perf/util/probe-finder.c
> >> @@ -1255,6 +1255,10 @@ end:
> >>         if (ret) {
> >>                 clear_probe_trace_event(tev);
> >>                 tf->ntevs--;
> >> +               if (tf->ntevs != 0) {
> >> +                       pr_debug("Ignoring error as atleast one probe
> >> point found.\n");
> >> +                       ret = 0;
> >> +               }
> >>         }
> >>         free(args);
> >>         return ret;
> >>
> >>

[toc] | [prev] | [next] | [standalone]

#1346340 — RE: [RFC] perf: probe_finder: continue if atleast one probe point found

From平松雅巳 / HIRAMATU，MASAMI <masami.hiramatsu.pt@hitachi.com>
Date2016-03-01 04:20 +0100
SubjectRE: [RFC] perf: probe_finder: continue if atleast one probe point found
Message-ID<r7J0R-3vn-1@gated-at.bofh.it>
In reply to#1345859
Hi,

>From: Arnaldo Carvalho de Melo [mailto:acme@kernel.org]
>
>Em Sun, Feb 28, 2016 at 03:49:44AM -0800, Joel Fernandes escreveu:
>> Taeung,
>>
>> Thanks for that. As this is an RFC, I have added them now to this email
>> thread and when I post the final patch, I'll include them in CC.
>>
>> All,
>> Please provide your comments on my patch.
>
>Masami, are you ok with this patch?

Hmm, I think this is not enough. I concider that the case if users
expect probing on "__malloc_get_state+11", just ignoring it may
confuse or mislead them that perf-probe has a bug not to find it.

IOW, this was by design that perf probe refuses all probe point
if it hits any error.

I think we have 2 options, one is adding an option (or reuse --force)
to ignore error on finding probe points (but this MUST show the error
message by pr_error.) The other is extending perf probe's argument
format so that it can accept local-vars which may not exist, e.g.
'malloc *bytes' or 'malloc {bytes,}' :)

Thank you,


>
>- Arnaldo
>
>> Thanks,
>> Joel
>>
>> On Sun, Feb 28, 2016 at 3:35 AM, Taeung Song <treeze.taeung@gmail.com>
>> wrote:
>>
>> > Hi, Joel
>> >
>> > perf subsystem's maintainers are
>> > Arnaldo Carvalho de Melo <acme@kernel.org>
>> > Peter Zijlstra <peterz@infradead.org>
>> > Ingo Molnar <mingo@redhat.com>
>> >
>> > (or Namhyung Kim <namhyung@kernel.org>, Jiri Olsa <jolsa@kernel.org>)
>> >
>> > AFAIK,
>> > When sending some patches, it would be better to add them into To: or Cc:
>> >
>> >
>> > Thanks,
>> > Taeung
>> >
>> >
>> > On 02/28/2016 08:14 PM, Joel Fernandes wrote:
>> >
>> >> Sometimes for inline functions, perf probe can fail such as if an
>> >> arguments are
>> >> requested at probe points. This is probably because when the compiler
>> >> inlines,
>> >> for some instances it optimizes away arguments. Either way, the DWARF has
>> >> missing arguments for certain probe points of inline functions causing
>> >> 'perf
>> >> probe' to fail. I noticed this when probing the C library that ships with
>> >> my
>> >> distribution. With the following patch I am successfully able to record
>> >> kprobe
>> >> events with arguments.
>> >>
>> >> Test
>> >> ----
>> >>   ./perf probe \
>> >>     -k ./vmlinux -s ./ -x /lib/x86_64-linux-gnu/libc.so.6 -a 'malloc
>> >> bytes' -v
>> >>
>> >> Without the patch
>> >> -----------------
>> >> Matched function: __libc_malloc
>> >> found inline addr: 0x831a6
>> >> Probe point found: malloc_atfork+150
>> >> Searching 'bytes' variable in context.
>> >> Converting variable bytes into trace event.
>> >> bytes type is long unsigned int.
>> >> found inline addr: 0x844a0
>> >> Probe point found: __libc_malloc+0
>> >> Searching 'bytes' variable in context.
>> >> Converting variable bytes into trace event.
>> >> bytes type is long unsigned int.
>> >> found inline addr: 0x8463b
>> >> Probe point found: __malloc_get_state+11
>> >> Searching 'bytes' variable in context.
>> >> Failed to find 'bytes' in this function.
>> >> An error occurred in debuginfo analysis (-2).
>> >>    Error: Failed to add events. Reason: No such file or directory (Code:
>> >> -2)
>> >>
>> >> With the patch
>> >> --------------
>> >> Open Debuginfo file: /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.21.so
>> >> Try to find probe point from debuginfo.
>> >> Symbol malloc address found : 844a0
>> >> Matched function: __libc_malloc
>> >> found inline addr: 0x831a6
>> >> Probe point found: malloc_atfork+150
>> >> Searching 'bytes' variable in context.
>> >> Converting variable bytes into trace event.
>> >> bytes type is long unsigned int.
>> >> found inline addr: 0x844a0
>> >> Probe point found: __libc_malloc+0
>> >> Searching 'bytes' variable in context.
>> >> Converting variable bytes into trace event.
>> >> bytes type is long unsigned int.
>> >> found inline addr: 0x8463b
>> >> Probe point found: __malloc_get_state+11
>> >> Searching 'bytes' variable in context.
>> >> Failed to find 'bytes' in this function.
>> >> Probe point error, ignoring. Atleast one probe point found
>> >> found inline addr: 0x84bba
>> >> Probe point found: __libc_realloc+410
>> >> Searching 'bytes' variable in context.
>> >> Failed to find 'bytes' in this function.
>> >> Probe point error, ignoring. Atleast one probe point found
>> >> found inline addr: 0x84d08
>> >> ...
>> >> Probe point error, ignoring. Atleast one probe point found
>> >> Found 3 probe_trace_events.
>> >> Opening /sys/kernel/debug/tracing//uprobe_events write=1
>> >> Writing event: p:probe_libc/malloc /lib/x86_64-linux-gnu/libc-2.21.so:0x831a6
>> >> bytes=%bp:u64
>> >> Writing event: p:probe_libc/malloc_1 /lib/x86_64-linux-gnu/libc-2.21.so:0x844a0
>> >> bytes=%di:u64
>> >> Writing event: p:probe_libc/malloc_2 /lib/x86_64-linux-gnu/libc-2.21.so:0x85af7
>> >> bytes=%di:u64
>> >> Added new events:
>> >>    probe_libc:malloc    (on malloc in /lib/x86_64-linux-gnu/libc-2.21.so
>> >> with bytes)
>> >>    probe_libc:malloc_1  (on malloc in /lib/x86_64-linux-gnu/libc-2.21.so
>> >> with bytes)
>> >>    probe_libc:malloc_2  (on malloc in /lib/x86_64-linux-gnu/libc-2.21.so
>> >> with bytes)
>> >>
>> >> Signed-off-by: Joel Fernandes <agnel.joel@gmail.com>
>> >> ---
>> >>   tools/perf/util/probe-finder.c | 4 ++++
>> >>   1 file changed, 4 insertions(+)
>> >>
>> >> diff --git a/tools/perf/util/probe-finder.c
>> >> b/tools/perf/util/probe-finder.c
>> >> index 4ce5c5e..3ac9481 100644
>> >> --- a/tools/perf/util/probe-finder.c
>> >> +++ b/tools/perf/util/probe-finder.c
>> >> @@ -1255,6 +1255,10 @@ end:
>> >>         if (ret) {
>> >>                 clear_probe_trace_event(tev);
>> >>                 tf->ntevs--;
>> >> +               if (tf->ntevs != 0) {
>> >> +                       pr_debug("Ignoring error as atleast one probe
>> >> point found.\n");
>> >> +                       ret = 0;
>> >> +               }
>> >>         }
>> >>         free(args);
>> >>         return ret;
>> >>
>> >>

[toc] | [prev] | [next] | [standalone]

#1346357 — RE: [RFC] perf: probe_finder: continue if atleast one probe point found

From平松雅巳 / HIRAMATU，MASAMI <masami.hiramatsu.pt@hitachi.com>
Date2016-03-01 05:30 +0100
SubjectRE: [RFC] perf: probe_finder: continue if atleast one probe point found
Message-ID<r7K6B-4aZ-9@gated-at.bofh.it>
In reply to#1346340
BTW,

./perf probe  \
  -k ./vmlinux -s ./ -x /lib/x86_64-linux-gnu/libc.so.6 -a 'malloc $params'

might help your case. $params is expanded to function parameters automatically
and if there is no parameters, it is just ignored :)

Thank you,

>From: 平松雅巳 / HIRAMATU，MASAMI [mailto:masami.hiramatsu.pt@hitachi.com]
>
>Hi,
>
>>From: Arnaldo Carvalho de Melo [mailto:acme@kernel.org]
>>
>>Em Sun, Feb 28, 2016 at 03:49:44AM -0800, Joel Fernandes escreveu:
>>> Taeung,
>>>
>>> Thanks for that. As this is an RFC, I have added them now to this email
>>> thread and when I post the final patch, I'll include them in CC.
>>>
>>> All,
>>> Please provide your comments on my patch.
>>
>>Masami, are you ok with this patch?
>
>Hmm, I think this is not enough. I concider that the case if users
>expect probing on "__malloc_get_state+11", just ignoring it may
>confuse or mislead them that perf-probe has a bug not to find it.
>
>IOW, this was by design that perf probe refuses all probe point
>if it hits any error.
>
>I think we have 2 options, one is adding an option (or reuse --force)
>to ignore error on finding probe points (but this MUST show the error
>message by pr_error.) The other is extending perf probe's argument
>format so that it can accept local-vars which may not exist, e.g.
>'malloc *bytes' or 'malloc {bytes,}' :)
>
>Thank you,
>
>
>>
>>- Arnaldo
>>
>>> Thanks,
>>> Joel
>>>
>>> On Sun, Feb 28, 2016 at 3:35 AM, Taeung Song <treeze.taeung@gmail.com>
>>> wrote:
>>>
>>> > Hi, Joel
>>> >
>>> > perf subsystem's maintainers are
>>> > Arnaldo Carvalho de Melo <acme@kernel.org>
>>> > Peter Zijlstra <peterz@infradead.org>
>>> > Ingo Molnar <mingo@redhat.com>
>>> >
>>> > (or Namhyung Kim <namhyung@kernel.org>, Jiri Olsa <jolsa@kernel.org>)
>>> >
>>> > AFAIK,
>>> > When sending some patches, it would be better to add them into To: or Cc:
>>> >
>>> >
>>> > Thanks,
>>> > Taeung
>>> >
>>> >
>>> > On 02/28/2016 08:14 PM, Joel Fernandes wrote:
>>> >
>>> >> Sometimes for inline functions, perf probe can fail such as if an
>>> >> arguments are
>>> >> requested at probe points. This is probably because when the compiler
>>> >> inlines,
>>> >> for some instances it optimizes away arguments. Either way, the DWARF has
>>> >> missing arguments for certain probe points of inline functions causing
>>> >> 'perf
>>> >> probe' to fail. I noticed this when probing the C library that ships with
>>> >> my
>>> >> distribution. With the following patch I am successfully able to record
>>> >> kprobe
>>> >> events with arguments.
>>> >>
>>> >> Test
>>> >> ----
>>> >>   ./perf probe \
>>> >>     -k ./vmlinux -s ./ -x /lib/x86_64-linux-gnu/libc.so.6 -a 'malloc
>>> >> bytes' -v
>>> >>
>>> >> Without the patch
>>> >> -----------------
>>> >> Matched function: __libc_malloc
>>> >> found inline addr: 0x831a6
>>> >> Probe point found: malloc_atfork+150
>>> >> Searching 'bytes' variable in context.
>>> >> Converting variable bytes into trace event.
>>> >> bytes type is long unsigned int.
>>> >> found inline addr: 0x844a0
>>> >> Probe point found: __libc_malloc+0
>>> >> Searching 'bytes' variable in context.
>>> >> Converting variable bytes into trace event.
>>> >> bytes type is long unsigned int.
>>> >> found inline addr: 0x8463b
>>> >> Probe point found: __malloc_get_state+11
>>> >> Searching 'bytes' variable in context.
>>> >> Failed to find 'bytes' in this function.
>>> >> An error occurred in debuginfo analysis (-2).
>>> >>    Error: Failed to add events. Reason: No such file or directory (Code:
>>> >> -2)
>>> >>
>>> >> With the patch
>>> >> --------------
>>> >> Open Debuginfo file: /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.21.so
>>> >> Try to find probe point from debuginfo.
>>> >> Symbol malloc address found : 844a0
>>> >> Matched function: __libc_malloc
>>> >> found inline addr: 0x831a6
>>> >> Probe point found: malloc_atfork+150
>>> >> Searching 'bytes' variable in context.
>>> >> Converting variable bytes into trace event.
>>> >> bytes type is long unsigned int.
>>> >> found inline addr: 0x844a0
>>> >> Probe point found: __libc_malloc+0
>>> >> Searching 'bytes' variable in context.
>>> >> Converting variable bytes into trace event.
>>> >> bytes type is long unsigned int.
>>> >> found inline addr: 0x8463b
>>> >> Probe point found: __malloc_get_state+11
>>> >> Searching 'bytes' variable in context.
>>> >> Failed to find 'bytes' in this function.
>>> >> Probe point error, ignoring. Atleast one probe point found
>>> >> found inline addr: 0x84bba
>>> >> Probe point found: __libc_realloc+410
>>> >> Searching 'bytes' variable in context.
>>> >> Failed to find 'bytes' in this function.
>>> >> Probe point error, ignoring. Atleast one probe point found
>>> >> found inline addr: 0x84d08
>>> >> ...
>>> >> Probe point error, ignoring. Atleast one probe point found
>>> >> Found 3 probe_trace_events.
>>> >> Opening /sys/kernel/debug/tracing//uprobe_events write=1
>>> >> Writing event: p:probe_libc/malloc /lib/x86_64-linux-gnu/libc-2.21.so:0x831a6
>>> >> bytes=%bp:u64
>>> >> Writing event: p:probe_libc/malloc_1 /lib/x86_64-linux-gnu/libc-2.21.so:0x844a0
>>> >> bytes=%di:u64
>>> >> Writing event: p:probe_libc/malloc_2 /lib/x86_64-linux-gnu/libc-2.21.so:0x85af7
>>> >> bytes=%di:u64
>>> >> Added new events:
>>> >>    probe_libc:malloc    (on malloc in /lib/x86_64-linux-gnu/libc-2.21.so
>>> >> with bytes)
>>> >>    probe_libc:malloc_1  (on malloc in /lib/x86_64-linux-gnu/libc-2.21.so
>>> >> with bytes)
>>> >>    probe_libc:malloc_2  (on malloc in /lib/x86_64-linux-gnu/libc-2.21.so
>>> >> with bytes)
>>> >>
>>> >> Signed-off-by: Joel Fernandes <agnel.joel@gmail.com>
>>> >> ---
>>> >>   tools/perf/util/probe-finder.c | 4 ++++
>>> >>   1 file changed, 4 insertions(+)
>>> >>
>>> >> diff --git a/tools/perf/util/probe-finder.c
>>> >> b/tools/perf/util/probe-finder.c
>>> >> index 4ce5c5e..3ac9481 100644
>>> >> --- a/tools/perf/util/probe-finder.c
>>> >> +++ b/tools/perf/util/probe-finder.c
>>> >> @@ -1255,6 +1255,10 @@ end:
>>> >>         if (ret) {
>>> >>                 clear_probe_trace_event(tev);
>>> >>                 tf->ntevs--;
>>> >> +               if (tf->ntevs != 0) {
>>> >> +                       pr_debug("Ignoring error as atleast one probe
>>> >> point found.\n");
>>> >> +                       ret = 0;
>>> >> +               }
>>> >>         }
>>> >>         free(args);
>>> >>         return ret;
>>> >>
>>> >>

[toc] | [prev] | [next] | [standalone]

#1346523

FromJoel Fernandes <agnel.joel@gmail.com>
Date2016-03-01 11:40 +0100
Message-ID<r7PSF-7T6-3@gated-at.bofh.it>
In reply to#1346357
Sorry about the line wrapping, I sent this email through g-mail.
Shouldn't have any issue when I send the final patch through git
send-email.

Thanks,
Joel

On Tue, Mar 1, 2016 at 2:34 AM, Joel Fernandes <agnel.joel@gmail.com> wrote:
> Hi Hiramatu,
>
> On Mon, Feb 29, 2016 at 8:29 PM, 平松雅巳 / HIRAMATU，MASAMI
> <masami.hiramatsu.pt@hitachi.com> wrote:
>> BTW,
>>
>> ./perf probe  \
>>   -k ./vmlinux -s ./ -x /lib/x86_64-linux-gnu/libc.so.6 -a 'malloc $params'
>>
>> might help your case. $params is expanded to function parameters automatically
>> and if there is no parameters, it is just ignored :)
>
> Thanks, I agree this is a better approach.
>
> How about an informative error letting the user know about it? Is the
> below Ok? I can send a patch.
>
> ----x8--------
> diff --git a/tools/perf/util/probe-finder.c b/tools/perf/util/probe-finder.c
> index 4ce5c5e..21f282a 100644
> --- a/tools/perf/util/probe-finder.c
> +++ b/tools/perf/util/probe-finder.c
> @@ -1247,8 +1247,12 @@ static int add_probe_trace_event(Dwarf_Die
> *sc_die, struct probe_finder *pf)
>                 pf->tvar = &tev->args[i];
>                 /* Variable should be found from scope DIE */
>                 ret = find_variable(sc_die, pf);
> -               if (ret != 0)
> +               if (ret != 0) {
> +                       pr_err("Variable couldn't be found, if probe
> is on inline function,"
> +                              "try '%s' or '%s' instead.\n",
> +                              PROBE_ARG_PARAMS, PROBE_ARG_VARS);
>                         break;
> +               }
>         }

[toc] | [prev] | [next] | [standalone]

#1346524

FromJoel Fernandes <agnel.joel@gmail.com>
Date2016-03-01 11:40 +0100
Message-ID<r7PSF-7T6-5@gated-at.bofh.it>
In reply to#1346357
Hi Hiramatu,

On Mon, Feb 29, 2016 at 8:29 PM, 平松雅巳 / HIRAMATU，MASAMI
<masami.hiramatsu.pt@hitachi.com> wrote:
> BTW,
>
> ./perf probe  \
>   -k ./vmlinux -s ./ -x /lib/x86_64-linux-gnu/libc.so.6 -a 'malloc $params'
>
> might help your case. $params is expanded to function parameters automatically
> and if there is no parameters, it is just ignored :)

Thanks, I agree this is a better approach.

How about an informative error letting the user know about it? Is the
below Ok? I can send a patch.

----x8--------
diff --git a/tools/perf/util/probe-finder.c b/tools/perf/util/probe-finder.c
index 4ce5c5e..21f282a 100644
--- a/tools/perf/util/probe-finder.c
+++ b/tools/perf/util/probe-finder.c
@@ -1247,8 +1247,12 @@ static int add_probe_trace_event(Dwarf_Die
*sc_die, struct probe_finder *pf)
                pf->tvar = &tev->args[i];
                /* Variable should be found from scope DIE */
                ret = find_variable(sc_die, pf);
-               if (ret != 0)
+               if (ret != 0) {
+                       pr_err("Variable couldn't be found, if probe
is on inline function,"
+                              "try '%s' or '%s' instead.\n",
+                              PROBE_ARG_PARAMS, PROBE_ARG_VARS);
                        break;
+               }
        }

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web