Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1376806 > unrolled thread
| Started by | Arnaldo Carvalho de Melo <acme@kernel.org> |
|---|---|
| First post | 2016-04-12 15:40 +0200 |
| Last post | 2016-04-13 03:10 +0200 |
| Articles | 2 — 2 participants |
Back to article view | Back to linux.kernel
zeros in frame pointer callchains Arnaldo Carvalho de Melo <acme@kernel.org> - 2016-04-12 15:40 +0200
Re: zeros in frame pointer callchains Namhyung Kim <namhyung@kernel.org> - 2016-04-13 03:10 +0200
| From | Arnaldo Carvalho de Melo <acme@kernel.org> |
|---|---|
| Date | 2016-04-12 15:40 +0200 |
| Subject | zeros in frame pointer callchains |
| Message-ID | <rn6HU-89g-29@gated-at.bofh.it> |
Hi,
Namhyung, while working on supporting callchains in 'perf trace' I
noticed something I think we discussed at some point, zeros in frame pointer
callchains...
Oh well figured it out, its probably userspace not having frame
pointers, grrr anyway, keeping the rest of the message, for reference,
but now I think we need to have some expressiveness to say that for that
particular event we want to use perf_event_attr.exclude_callchain_user...
First add a special probe point that 'perf trace' uses, if present:
# perf probe 'vfs_getname=getname_flags:72 pathname=filename:string'
Then lets ask to see just when this thing hits, no syscall
printing whatsoever ('trace' == 'perf trace'), sometimes just a few
zeros at the top, sometimes a long series, need to investigate how to
chop those off in the kernel, not to use the ring buffer needlessly,
will start chopping these in userspace (in perf trace):
# trace --no-sys --ev probe:vfs_getname/call-graph=fp/
34033.533 probe:vfs_getname:(ffffffff8124c06c) pathname="/usr/share/pixmaps")
getname_flags+0xfe20009d ([kernel.kallsyms])
user_path_at_empty+0xfe200023 ([kernel.kallsyms])
vfs_fstatat+0xfe200066 ([kernel.kallsyms])
SYSC_newstat+0xfe20002e ([kernel.kallsyms])
sys_newstat+0xfe20000e ([kernel.kallsyms])
entry_SYSCALL_64_fastpath+0xfe20001a ([kernel.kallsyms])
__xstat64+0xffff011e90b98015 (/usr/lib64/libc-2.22.so)
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
36001.681 probe:vfs_getname:(ffffffff8124c06c) pathname="/var/tmp")
getname_flags+0xfe20009d ([kernel.kallsyms])
user_path_at_empty+0xfe200023 ([kernel.kallsyms])
vfs_fstatat+0xfe200066 ([kernel.kallsyms])
SYSC_newstat+0xfe20002e ([kernel.kallsyms])
sys_newstat+0xfe20000e ([kernel.kallsyms])
entry_SYSCALL_64_fastpath+0xfe20001a ([kernel.kallsyms])
__xstat64+0xffff00b816a28015 (/usr/lib64/libc-2.22.so)
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
36001.696 probe:vfs_getname:(ffffffff8124c06c) pathname="/var/tmp")
getname_flags+0xfe20009d ([kernel.kallsyms])
user_path_at_empty+0xfe200023 ([kernel.kallsyms])
sys_access+0xfe2000b4 ([kernel.kallsyms])
entry_SYSCALL_64_fastpath+0xfe20001a ([kernel.kallsyms])
__GI___access+0xffff00b816a28007 (/usr/lib64/libc-2.22.so)
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
8254.597 probe:vfs_getname:(ffffffff8124c06c) pathname="/etc/localtime")
getname_flags+0xfe20009d ([kernel.kallsyms])
user_path_at_empty+0xfe200023 ([kernel.kallsyms])
vfs_fstatat+0xfe200066 ([kernel.kallsyms])
SYSC_newstat+0xfe20002e ([kernel.kallsyms])
sys_newstat+0xfe20000e ([kernel.kallsyms])
entry_SYSCALL_64_fastpath+0xfe20001a ([kernel.kallsyms])
__xstat64+0xffff0104f5696015 (/usr/lib64/libc-2.22.so)
[0] ([unknown])
[0] ([unknown])
9001.567 probe:vfs_getname:(ffffffff8124c06c) pathname="/etc/firewalld/ipsets")
getname_flags+0xfe20009d ([kernel.kallsyms])
user_path_at_empty+0xfe200023 ([kernel.kallsyms])
sys_inotify_add_watch+0xfe200098 ([kernel.kallsyms])
entry_SYSCALL_64_fastpath+0xfe20001a ([kernel.kallsyms])
inotify_add_watch+0xffff008bb9432007 (/usr/lib64/libc-2.22.so)
9450.181 probe:vfs_getname:(ffffffff8124c06c) pathname="/root")
getname_flags+0xfe20009d ([kernel.kallsyms])
user_path_at_empty+0xfe200023 ([kernel.kallsyms])
sys_inotify_add_watch+0xfe200098 ([kernel.kallsyms])
entry_SYSCALL_64_fastpath+0xfe20001a ([kernel.kallsyms])
inotify_add_watch+0xffff01b7aa4f2007 (/usr/lib64/libc-2.22.so)
[toc] | [next] | [standalone]
| From | Namhyung Kim <namhyung@kernel.org> |
|---|---|
| Date | 2016-04-13 03:10 +0200 |
| Message-ID | <rnhtE-oq-3@gated-at.bofh.it> |
| In reply to | #1376806 |
Hi Arnaldo,
On Tue, Apr 12, 2016 at 10:38:47AM -0300, Arnaldo Carvalho de Melo wrote:
> Hi,
>
> Namhyung, while working on supporting callchains in 'perf trace' I
> noticed something I think we discussed at some point, zeros in frame pointer
> callchains...
>
> Oh well figured it out, its probably userspace not having frame
> pointers, grrr anyway, keeping the rest of the message, for reference,
> but now I think we need to have some expressiveness to say that for that
> particular event we want to use perf_event_attr.exclude_callchain_user...
>
> First add a special probe point that 'perf trace' uses, if present:
>
> # perf probe 'vfs_getname=getname_flags:72 pathname=filename:string'
>
> Then lets ask to see just when this thing hits, no syscall
> printing whatsoever ('trace' == 'perf trace'), sometimes just a few
> zeros at the top, sometimes a long series, need to investigate how to
> chop those off in the kernel, not to use the ring buffer needlessly,
> will start chopping these in userspace (in perf trace):
My solution was to look at /proc/sys/vm/mmap_min_addr [1]. For kernel
it'd be simply discarding callchains after seeing an address below the
mmap_min_addr. For userspace, it requires to carry the mmap_min_addr
via feature bit or something. Or Maybe we can just discard addresses
under 4096 for simplicity?
[1] https://lkml.org/lkml/2015/11/26/40
Thanks,
Namhyung
>
> # trace --no-sys --ev probe:vfs_getname/call-graph=fp/
>
> 34033.533 probe:vfs_getname:(ffffffff8124c06c) pathname="/usr/share/pixmaps")
> getname_flags+0xfe20009d ([kernel.kallsyms])
> user_path_at_empty+0xfe200023 ([kernel.kallsyms])
> vfs_fstatat+0xfe200066 ([kernel.kallsyms])
> SYSC_newstat+0xfe20002e ([kernel.kallsyms])
> sys_newstat+0xfe20000e ([kernel.kallsyms])
> entry_SYSCALL_64_fastpath+0xfe20001a ([kernel.kallsyms])
> __xstat64+0xffff011e90b98015 (/usr/lib64/libc-2.22.so)
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> 36001.681 probe:vfs_getname:(ffffffff8124c06c) pathname="/var/tmp")
> getname_flags+0xfe20009d ([kernel.kallsyms])
> user_path_at_empty+0xfe200023 ([kernel.kallsyms])
> vfs_fstatat+0xfe200066 ([kernel.kallsyms])
> SYSC_newstat+0xfe20002e ([kernel.kallsyms])
> sys_newstat+0xfe20000e ([kernel.kallsyms])
> entry_SYSCALL_64_fastpath+0xfe20001a ([kernel.kallsyms])
> __xstat64+0xffff00b816a28015 (/usr/lib64/libc-2.22.so)
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> 36001.696 probe:vfs_getname:(ffffffff8124c06c) pathname="/var/tmp")
> getname_flags+0xfe20009d ([kernel.kallsyms])
> user_path_at_empty+0xfe200023 ([kernel.kallsyms])
> sys_access+0xfe2000b4 ([kernel.kallsyms])
> entry_SYSCALL_64_fastpath+0xfe20001a ([kernel.kallsyms])
> __GI___access+0xffff00b816a28007 (/usr/lib64/libc-2.22.so)
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> 8254.597 probe:vfs_getname:(ffffffff8124c06c) pathname="/etc/localtime")
> getname_flags+0xfe20009d ([kernel.kallsyms])
> user_path_at_empty+0xfe200023 ([kernel.kallsyms])
> vfs_fstatat+0xfe200066 ([kernel.kallsyms])
> SYSC_newstat+0xfe20002e ([kernel.kallsyms])
> sys_newstat+0xfe20000e ([kernel.kallsyms])
> entry_SYSCALL_64_fastpath+0xfe20001a ([kernel.kallsyms])
> __xstat64+0xffff0104f5696015 (/usr/lib64/libc-2.22.so)
> [0] ([unknown])
> [0] ([unknown])
> 9001.567 probe:vfs_getname:(ffffffff8124c06c) pathname="/etc/firewalld/ipsets")
> getname_flags+0xfe20009d ([kernel.kallsyms])
> user_path_at_empty+0xfe200023 ([kernel.kallsyms])
> sys_inotify_add_watch+0xfe200098 ([kernel.kallsyms])
> entry_SYSCALL_64_fastpath+0xfe20001a ([kernel.kallsyms])
> inotify_add_watch+0xffff008bb9432007 (/usr/lib64/libc-2.22.so)
> 9450.181 probe:vfs_getname:(ffffffff8124c06c) pathname="/root")
> getname_flags+0xfe20009d ([kernel.kallsyms])
> user_path_at_empty+0xfe200023 ([kernel.kallsyms])
> sys_inotify_add_watch+0xfe200098 ([kernel.kallsyms])
> entry_SYSCALL_64_fastpath+0xfe20001a ([kernel.kallsyms])
> inotify_add_watch+0xffff01b7aa4f2007 (/usr/lib64/libc-2.22.so)
[toc] | [prev] | [standalone]
Back to top | Article view | linux.kernel
csiph-web