Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1370793 > unrolled thread

[PATCH 0/3] mm: support bigger cache workingsets and protect against writes

Back to article view | Back to linux.kernel

Contents

  [PATCH 0/3] mm: support bigger cache workingsets and protect against writes Johannes Weiner <hannes@cmpxchg.org> - 2016-04-04 19:20 +0200
    [PATCH 2/3] mm: filemap: only do access activations on reads Johannes Weiner <hannes@cmpxchg.org> - 2016-04-04 19:20 +0200
      Re: [PATCH 2/3] mm: filemap: only do access activations on reads Andrew Morton <akpm@linux-foundation.org> - 2016-04-04 23:30 +0200
        Re: [PATCH 2/3] mm: filemap: only do access activations on reads Rik van Riel <riel@redhat.com> - 2016-04-04 23:40 +0200
          Re: [PATCH 2/3] mm: filemap: only do access activations on reads Andrew Morton <akpm@linux-foundation.org> - 2016-04-05 00:00 +0200
          Re: [PATCH 2/3] mm: filemap: only do access activations on reads Johannes Weiner <hannes@cmpxchg.org> - 2016-04-05 20:00 +0200
        Re: [PATCH 2/3] mm: filemap: only do access activations on reads Johannes Weiner <hannes@cmpxchg.org> - 2016-04-05 00:50 +0200
    Re: [PATCH 0/3] mm: support bigger cache workingsets and protect  against writes Andres Freund <andres@anarazel.de> - 2016-04-04 21:00 +0200

#1370793 — [PATCH 0/3] mm: support bigger cache workingsets and protect against writes

Hi,

this is a follow-up to http://www.spinics.net/lists/linux-mm/msg101739.html
where Andres reported his database workingset being pushed out by the
minimum size enforcement of the inactive file list - currently 50% of cache
- as well as repeatedly written file pages that are never actually read.

Two changes fell out of the discussions. The first change observes that
pages that are only ever written don't benefit from caching beyond what the
writeback cache does for partial page writes, and so we shouldn't promote
them to the active file list where they compete with pages whose cached data
is actually accessed repeatedly. This change comes in two patches - one for
in-cache write accesses and one for refaults triggered by writes, neither of
which should promote a cache page.

Second, with the refault detection we don't need to set 50% of the cache
aside for used-once cache anymore since we can detect frequently used pages
even when they are evicted between accesses. We can allow the active list to
be bigger and thus protect a bigger workingset that isn't challenged by
streamers. Depending on the access patterns, this can increase major faults
during workingset transitions for better performance during stable phases.

Andres, I tried reproducing your postgres scenario, but I could never get
the WAL to interfere even with wal_log = hot_standby mode. It's a 8G
machine, I set shared_buffers = 2GB, ran pgbench -i -s 290, and then -c 32
-j 32 -M prepared -t 150000. Any input on how to trigger the thrashing you
observed would be appreciated. But it would be great if you could test these
patches on your known-problematic setup as well.

Thanks!

 include/linux/memcontrol.h |  25 -----------
 mm/filemap.c               |   8 +++-
 mm/page_alloc.c            |  44 ------------------
 mm/vmscan.c                | 104 +++++++++++++++++--------------------------
 4 files changed, 48 insertions(+), 133 deletions(-)

[toc] | [next] | [standalone]

#1370794 — [PATCH 2/3] mm: filemap: only do access activations on reads

Andres Freund observed that his database workload is struggling with
the transaction journal creating pressure on frequently read pages.

Access patterns like transaction journals frequently write the same
pages over and over, but in the majority of cases those pages are
never read back. There are no caching benefits to be had for those
pages, so activating them and having them put pressure on pages that
do benefit from caching is a bad choice.

Leave page activations to read accesses and don't promote pages based
on writes alone.

It could be said that partially written pages do contain cache-worthy
data, because even if *userspace* does not access the unwritten part,
the kernel still has to read it from the filesystem for correctness.
However, a counter argument is that these pages enjoy at least *some*
protection over other inactive file pages through the writeback cache,
in the sense that dirty pages are written back with a delay and cache
reclaim leaves them alone until they have been written back to
disk. Should that turn out to be insufficient and we see increased
read IO from partial writes under memory pressure, we can always go
back and update grab_cache_page_write_begin() to take (pos, len) so
that it can tell partial writes from pages that don't need partial
reads. But for now, keep it simple.

Reported-by: Andres Freund <andres@anarazel.de>
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
---
 mm/filemap.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mm/filemap.c b/mm/filemap.c
index ca33816..edfec5e 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -2579,7 +2579,7 @@ struct page *grab_cache_page_write_begin(struct address_space *mapping,
 					pgoff_t index, unsigned flags)
 {
 	struct page *page;
-	int fgp_flags = FGP_LOCK|FGP_ACCESSED|FGP_WRITE|FGP_CREAT;
+	int fgp_flags = FGP_LOCK|FGP_WRITE|FGP_CREAT;
 
 	if (flags & AOP_FLAG_NOFS)
 		fgp_flags |= FGP_NOFS;
-- 
2.8.0

[toc] | [prev] | [next] | [standalone]

#1370906 — Re: [PATCH 2/3] mm: filemap: only do access activations on reads

On Mon,  4 Apr 2016 13:13:37 -0400 Johannes Weiner <hannes@cmpxchg.org> wrote:

> Andres Freund observed that his database workload is struggling with
> the transaction journal creating pressure on frequently read pages.
> 
> Access patterns like transaction journals frequently write the same
> pages over and over, but in the majority of cases those pages are
> never read back. There are no caching benefits to be had for those
> pages, so activating them and having them put pressure on pages that
> do benefit from caching is a bad choice.

Read-after-write is a pretty common pattern: temporary files for
example.  What are the opportunities for regressions here?

Did you consider providing userspace with a way to hint "this file is
probably write-then-not-read"?

[toc] | [prev] | [next] | [standalone]

#1370910 — Re: [PATCH 2/3] mm: filemap: only do access activations on reads

[Multipart message — attachments visible in raw view] — view raw

On Mon, 2016-04-04 at 14:22 -0700, Andrew Morton wrote:
> On Mon,  4 Apr 2016 13:13:37 -0400 Johannes Weiner <hannes@cmpxchg.or
> g> wrote:
> 
> > 
> > Andres Freund observed that his database workload is struggling
> > with
> > the transaction journal creating pressure on frequently read pages.
> > 
> > Access patterns like transaction journals frequently write the same
> > pages over and over, but in the majority of cases those pages are
> > never read back. There are no caching benefits to be had for those
> > pages, so activating them and having them put pressure on pages
> > that
> > do benefit from caching is a bad choice.
> Read-after-write is a pretty common pattern: temporary files for
> example.  What are the opportunities for regressions here?
> 
> Did you consider providing userspace with a way to hint "this file is
> probably write-then-not-read"?

I suspect the opportunity for regressions is fairly small,
considering that temporary files usually have a very short
life span, and will likely be read-after-written before they
get evicted from the inactive list.

As for hinting, I suspect it may make sense to differentiate
between whole page and partial page writes, where partial
page writes use FGP_ACCESSED, and whole page writes do not,
under the assumption that if we write a partial page, there
may be a higher chance that other parts of the page get
accessed again for other writes (or reads).

I do not know whether that assumption holds :)

-- 
All Rights Reversed.

[toc] | [prev] | [next] | [standalone]

#1370928 — Re: [PATCH 2/3] mm: filemap: only do access activations on reads

On Mon, 04 Apr 2016 17:39:47 -0400 Rik van Riel <riel@redhat.com> wrote:

> On Mon, 2016-04-04 at 14:22 -0700, Andrew Morton wrote:
> > On Mon,____4 Apr 2016 13:13:37 -0400 Johannes Weiner <hannes@cmpxchg.or
> > g> wrote:
> > 
> > > 
> > > Andres Freund observed that his database workload is struggling
> > > with
> > > the transaction journal creating pressure on frequently read pages.
> > > 
> > > Access patterns like transaction journals frequently write the same
> > > pages over and over, but in the majority of cases those pages are
> > > never read back. There are no caching benefits to be had for those
> > > pages, so activating them and having them put pressure on pages
> > > that
> > > do benefit from caching is a bad choice.
> > Read-after-write is a pretty common pattern: temporary files for
> > example.____What are the opportunities for regressions here?
> > 
> > Did you consider providing userspace with a way to hint "this file is
> > probably write-then-not-read"?
> 
> I suspect the opportunity for regressions is fairly small,
> considering that temporary files usually have a very short
> life span, and will likely be read-after-written before they
> get evicted from the inactive list.

The opportunity for regressions in the current code is fairly small,
but Andres found one :( If there's any possibility at all, someone will
hit it.

One possible way to move forward is to write testcases to deliberately
hit the predicted problem, gain an understanding of how hard it is to
hit, how bad the effects are.

> As for hinting, I suspect it may make sense to differentiate
> between whole page and partial page writes, where partial
> page writes use FGP_ACCESSED, and whole page writes do not,
> under the assumption that if we write a partial page, there
> may be a higher chance that other parts of the page get
> accessed again for other writes (or reads).

hm, the FGP_foo documentation is a mess.  There's some placed randomly
at pagecache_get_page() and FGP_WRITE got missed altogether.

The ext4 journal would be a decent (but not very significant) candidate
for a "this is never read from" interface.  I guess the fs could
manually deactivate (or even free?) the pages.

[toc] | [prev] | [next] | [standalone]

#1371840 — Re: [PATCH 2/3] mm: filemap: only do access activations on reads

On Mon, Apr 04, 2016 at 05:39:47PM -0400, Rik van Riel wrote:
> As for hinting, I suspect it may make sense to differentiate
> between whole page and partial page writes, where partial
> page writes use FGP_ACCESSED, and whole page writes do not,
> under the assumption that if we write a partial page, there
> may be a higher chance that other parts of the page get
> accessed again for other writes (or reads).

The writeback cache should handle at least the multiple subpage writes
case.

What I find a little weird about counting accesses from partial writes
only is when a write covers a full page and then parts of the next. We
would cache only a small piece of what's likely one coherent chunk.

Or when a user writes out several pages in a loop of subpage chunks.

This will get even worse to program against when we start having page
cache transparently backed by pages of different sizes.

Because of that I think it'd be better to apply LRU aging decisions
based on type of access rather based on specific request sizes.

[toc] | [prev] | [next] | [standalone]

#1370940 — Re: [PATCH 2/3] mm: filemap: only do access activations on reads

On Mon, Apr 04, 2016 at 02:22:33PM -0700, Andrew Morton wrote:
> On Mon,  4 Apr 2016 13:13:37 -0400 Johannes Weiner <hannes@cmpxchg.org> wrote:
> 
> > Andres Freund observed that his database workload is struggling with
> > the transaction journal creating pressure on frequently read pages.
> > 
> > Access patterns like transaction journals frequently write the same
> > pages over and over, but in the majority of cases those pages are
> > never read back. There are no caching benefits to be had for those
> > pages, so activating them and having them put pressure on pages that
> > do benefit from caching is a bad choice.
> 
> Read-after-write is a pretty common pattern: temporary files for
> example.  What are the opportunities for regressions here?

The read(s) following the write will call mark_page_accessed() and so
promote the pages if their data is in fact repeatedly accessed. That
makes sense, because the writes really don't say anything about the
cache-worthiness. One write followed by one read shouldn't mean the
data is strongly benefiting from being cached. Only multiple reads.

What complicates that a little bit is that when the multiple reads do
happen on write-instantiated pages, the pages might have already been
aged somewhat in between, whereas fresh-faulting reads start counting
accesses from the head of the LRU right away. If both have re-use
distances shorter than memory, the LRU offset of pages instantiated by
writes could push the second access past eviction.

In that case, they would likely get picked up by refault detection and
promoted after all. So it would be one more IO, but nothing permanent.

This is also somewhat compensated by the dirty cache delaying reclaim
and giving these pages another round-trip anyway - unless dirty limits
cause the pages to be written back before they reach the LRU tail.

It's really hard to tell whether that would even be an issue since it
depends on whether a workload matching those parameters even exist. A
synthetic test doesn't really say us much about that. I think all we
can do here is decide whether the cache semantics make logical sense.

One thing I proposed in the thread that would compensate for the LRU
offset of write-instantiated pages would be to set PageReferenced on
these pages but never call mark_page_accessed() from the write. This
wouldn't be perfect because the distance between write and read does
not necessarily predict the distance between the subsequent reads, but
it would mean that the first read would promote the pages, whereas
repeatedly written files would never be activated or refault-activate.

Would that make sense? Is there something I'm missing?

> Did you consider providing userspace with a way to hint "this file is
> probably write-then-not-read"?

Yes, but I'm not too confident in that working out :(

[toc] | [prev] | [next] | [standalone]

#1370822 — Re: [PATCH 0/3] mm: support bigger cache workingsets and protect against writes

Hi Johannes,

On 2016-04-04 13:13:35 -0400, Johannes Weiner wrote:
> this is a follow-up to http://www.spinics.net/lists/linux-mm/msg101739.html
> where Andres reported his database workingset being pushed out by the
> minimum size enforcement of the inactive file list - currently 50% of cache
> - as well as repeatedly written file pages that are never actually read.

Thanks for following up!


> Andres, I tried reproducing your postgres scenario, but I could never get
> the WAL to interfere even with wal_log = hot_standby mode. It's a 8G
> machine, I set shared_buffers = 2GB, ran pgbench -i -s 290, and then -c 32
> -j 32 -M prepared -t 150000. Any input on how to trigger the thrashing you
> observed would be appreciated. But it would be great if you could test these
> patches on your known-problematic setup as well.

I'm unfortunately in the process of moving to the US (as in, I'm packing
boxes), so I can't get back to you just now. I'll try ASAP (early next
week).

Regards,

Andres

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web