RE: [PATCH RFC V4 5/6] perf top: switch to backward overwrite mode

From	"Liang, Kan" <kan.liang@intel.com>
Newsgroups	linux.kernel
Subject	RE: [PATCH RFC V4 5/6] perf top: switch to backward overwrite mode
Date	2017-10-02 22:50 +0200
Message-ID	<uwfP3-ur-9@gated-at.bofh.it> (permalink)
References	<uv4M1-7tm-3@gated-at.bofh.it> <uv4M1-7tm-1@gated-at.bofh.it> <uvvVU-8vi-7@gated-at.bofh.it>
Organization	linux.* mail to news gateway

Show all headers | View raw

> On Fri, Sep 29, 2017 at 07:47:56AM -0700, kan.liang@intel.com wrote:
> > From: Kan Liang <kan.liang@intel.com>
> >
> > perf_top__mmap_read has severe performance issue in Knights
> > Landing/Mill, when monitoring in heavy load system. It costs several
> > minutes to finish, which is unacceptable.
> >
> > perf top was overwrite mode. But it is changed to non overwrite mode
> > since commit 93fc64f14472 ("perf top: Switch to non overwrite mode").
> > For non overwrite mode, it tries to read everything in the ring buffer
> > and does not check the messup. Once there are lots of samples
> > delivered shortly, the processing time could be very long.
> > Knights Landing/Mill as a manycore processor contains a large number
> > of small cores. Because of the huge core number, it will generated
> > lots of samples in a heavy load system. Also, since the huge sample#,
> > the mmap writer probably bite the tail and mess up the samples.
> >
> > Also, to avoid the problems which is described in commit 9ecda41acb97
> > ("perf/core: Add ::write_backward attribute to perf event"), switch to
> > backward overwrite mode.
> > Pausing the ring-buffer during perf_top__mmap_read to ensure the
> > ring-buffer is stable.
> > There would be some records lost in backward overwrite mode. Removing
> > the lost events checking.
> 
> I'm getting perf top hogging the cpu completely with this change
> 

I think I find the root cause of the cpu hogging.
perf_mmap__read_catchup discards the md->prev from previous mmap_read.
Current mmap_read doesn't know which data has already been processed by
previous mmap_read. So it has to go through all the valid data in the ring buffer,
even most of the data has been processed by previous mmap_read.

Also, it looks perf record has the similar issue.
The previous location will be discarded as well in backward overwrite mode.
That will be an issue when --overwrite and --switch-output are enabled.
The new output will always include the old data in the previous output, which
should be wrong.

I think I will rewrite the perf_mmap__read_backward and perf_mmap__read_catchup
to fix this issue in a separate thread. Those functions should be common backward
mmap_read functions for all tools and tests.

BTW, are you OK with patch 1-4?
Those patches multithreading the machine__synthesize_threads, which is
irrelevant with the overwrite mode.
I think they can be merged separately.

Thanks,
Kan

Back to linux.kernel | Previous | Next — Previous in thread | Find similar | Unroll thread

Thread

[PATCH RFC V4 5/6] perf top: switch to backward overwrite mode kan.liang@intel.com - 2017-09-29 16:50 +0200
  Re: [PATCH RFC V4 5/6] perf top: switch to backward overwrite mode Jiri Olsa <jolsa@redhat.com> - 2017-09-30 21:50 +0200
    Re: [PATCH RFC V4 5/6] perf top: switch to backward overwrite mode "acme@kernel.org" <acme@kernel.org> - 2017-10-02 22:50 +0200
    RE: [PATCH RFC V4 5/6] perf top: switch to backward overwrite mode "Liang, Kan" <kan.liang@intel.com> - 2017-10-02 22:50 +0200

csiph-web