Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1312308 > unrolled thread

Re: [RFC] perf record: missing buildid for callstack modules

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [RFC] perf record: missing buildid for callstack modules Namhyung Kim <namhyung@kernel.org> - 2016-01-19 16:00 +0100
    Re: [RFC] perf record: missing buildid for callstack modules Peter Zijlstra <peterz@infradead.org> - 2016-01-19 16:30 +0100
      Re: [RFC] perf record: missing buildid for callstack modules Arnaldo Carvalho de Melo <acme@kernel.org> - 2016-01-19 16:50 +0100

#1312308 — Re: [RFC] perf record: missing buildid for callstack modules

On Wed, Jan 13, 2016 at 10:57:38AM +0100, Ingo Molnar wrote:
> 
> * Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
> 
> > Em Tue, Jan 12, 2016 at 11:39:43AM +0100, Ingo Molnar escreveu:
> > > But perf tooling cares very much: it can lead to subtle bugs and bad data if we 
> > > display a profile with the wrong DSO or binary. 'Bad' profiles resulting out of 
> > > binary mismatch can be very convincing and can send developers down the wrong path 
> > > for hours. I'd expect my tooling to not do that.
> >  
> > > Path names alone (the thing that exec() cares about) are not unique enough to 
> > > identify the binary that was profiled. So we need a content hash - hence the 
> > > build-ID.
> >  
> > > Can you suggest a better solution than a build-time calculated content hash?
> >  
> > > As for binary formats that suck and don't allow for a content hash: we do our 
> > > best, but of course the risk of data mismatch is there. We could perhaps cache the 
> > > binary inode's mtime field to at least produce a 'profile data is older than 
> > > binary/DSO modification date!' warning. (Which check won't catch all cases, like 
> > > cross-system profiling data matches.)
> > 
> > So, we could think of this as: binary formats that want to aid
> > observability tools to:
> > 
> > 1) Detect mismatches in contents for DSOs present at recording time to
> >    those to be used at analysis time.
> > 
> > 2) Find symtabs, DSO binary contents, CFI tables, present in the DSO
> >    where samples were taken.
> > 
> > Using mtime, as suggested in other messages will help with #1, but not
> > with #2.
> 
> But but ... why is #2 a problem with mtime? If we have an out of date record in 
> the perf.data, then the perf.data is uninteresting in 99% of the usecases! It's 
> out of date, most likely because the binary the developer is working on got 
> rebuilt, or the system got upgraded - in both cases the developer does not care 
> about the old records anymore...

We have 'perf diff' command which compares old and new performance
results of a same program.  People can use it to see how much improved
in the new version than the baseline.  In this case, the old binary
should be found from the old perf.data.

Thanks,
Namhyung


> 
> What matters is #1, to detect mismatches, to be a reliable tool. Once we've 
> detected that, we can inform the user and our job is mostly done.
> 
> But reliable != perfect time machine. Really, #2 is a second, third order concern 
> that should never cause slowdowns on the magnitude that Peter is complaining 
> about!
> 
> I realize that there might be special workflows (such as system-wide monitoring) 
> where collecting at recording time might be useful, but those are not the common 
> case - and they should not slow down the common case.
> 
> > Checking for inefficiencies in the current approach of
> > right-after-recording post-processing looking for PERF_RECORD_MMAPs,
> > Adrian suggested something here, also disabling the saving into
> > ~/.debug/ will help, collecting numbers would be great.
> 
> I think Peter mentioned a number: the kernel build time almost _doubles_ with 
> this. That's clearly unacceptable.
> 
> > But the mtime thing also requires traversing the whole perf.data
> > contents looking for those paths in PERF_RECORD_MMAP records.
> 
> But why? Why cannot we do it at perf report time, when we will parse them anyway?
> 
> Thanks,
> 
> 	Ingo

[toc] | [next] | [standalone]

#1312327

On Tue, Jan 19, 2016 at 11:56:40PM +0900, Namhyung Kim wrote:

> > But but ... why is #2 a problem with mtime? If we have an out of date record in 
> > the perf.data, then the perf.data is uninteresting in 99% of the usecases! It's 
> > out of date, most likely because the binary the developer is working on got 
> > rebuilt, or the system got upgraded - in both cases the developer does not care 
> > about the old records anymore...
> 
> We have 'perf diff' command which compares old and new performance
> results of a same program.  People can use it to see how much improved
> in the new version than the baseline.  In this case, the old binary
> should be found from the old perf.data.

Just means they'll have to use perf-archive or whatnot before that
works. Making the regular perf-record dead slow just so that a few more
complex workloads work doesn't make sense.

[toc] | [prev] | [next] | [standalone]

#1312340

Em Tue, Jan 19, 2016 at 04:27:42PM +0100, Peter Zijlstra escreveu:
> On Tue, Jan 19, 2016 at 11:56:40PM +0900, Namhyung Kim wrote:
> 
> > > But but ... why is #2 a problem with mtime? If we have an out of date record in 
> > > the perf.data, then the perf.data is uninteresting in 99% of the usecases! It's 
> > > out of date, most likely because the binary the developer is working on got 
> > > rebuilt, or the system got upgraded - in both cases the developer does not care 
> > > about the old records anymore...

> > We have 'perf diff' command which compares old and new performance
> > results of a same program.  People can use it to see how much improved
> > in the new version than the baseline.  In this case, the old binary
> > should be found from the old perf.data.

> Just means they'll have to use perf-archive or whatnot before that
> works. Making the regular perf-record dead slow just so that a few more
> complex workloads work doesn't make sense.

And perf-diff will be able to find the right binaries in most cases, as
we'll end up using that mtime to pick the right binary for the baseline
and the other perf.data files used.

So, if both files are reachable via the usual path (/usr/lib/debug,
/bin/, /lib64, ~/.debug/, etc) it'll work.

For users wanting the convenience of auto-saving binaries + its sources,
then they will have to incur the cost of postprocessing the just
generated perf.data file to do that, and make sure debuginfo is enabled,
for having the sources embedded into those fat binaries.

With the current situation, without any disambiguation mechanism, we
_need_ to traverse the perf.data file to find that disambiguation bit,
the build-id (we could'be been using the mtime, but having to traverse
it all we may as well use something better, the build-id), and while
doing that we could as well do a hardlink into ~/.debug/ for the DSOs
found in PERF_RECORD_MMAP(2) meta events (a copy if that hardlink isn't
possible, more overhead).

Now we'll use that mtime, which should be good enough for both figuring
out if a file can be used to even choose among various entries for a
same pathname. And we'll cover the case where a DSO is replaced during
a record session, as newer PERF_RECORD_MMAP3 emitted after the update
will have a different mtime, yay!

The extra, content based verification to double check that is the real
DSO used, which we're not using anyway right now (but could) will not be
possible, i.e. regenerate the build-id from the DSO contents to check
that it is really what was present at the record phase.

But for those super stringent needs, the door is open to add the
content-based cookie at the end of the PERF_RECORD_MMAP3, as we have in
it the filename-len, etc.

So now, with PERF_RECORD_MMAP3 the onus of using this all is back at
where it belongs, away from kernel developers and into tools/perf/
developers, tooling should use build-ids if available and if not,
fallback to mtime, if even that is not available (older kernels) then
warn the user and hope the pathname is enough.

- Arnaldo

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web