Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1741027 > unrolled thread

Re: [PATCH] mm: Make count list_lru_one::nr_items lockless

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH] mm: Make count list_lru_one::nr_items lockless Andrew Morton <akpm@linux-foundation.org> - 2017-09-27 23:20 +0200
    Re: [PATCH] mm: Make count list_lru_one::nr_items lockless Andrew Morton <akpm@linux-foundation.org> - 2017-09-28 23:10 +0200

#1741027 — Re: [PATCH] mm: Make count list_lru_one::nr_items lockless

On Tue, 19 Sep 2017 18:06:33 +0300 Kirill Tkhai <ktkhai@virtuozzo.com> wrote:

> During the reclaiming slab of a memcg, shrink_slab iterates
> over all registered shrinkers in the system, and tries to count
> and consume objects related to the cgroup. In case of memory
> pressure, this behaves bad: I observe high system time and
> time spent in list_lru_count_one() for many processes on RHEL7
> kernel (collected via $perf record --call-graph fp -j k -a):
> 
> 0,50%  nixstatsagent  [kernel.vmlinux]  [k] _raw_spin_lock                [k] _raw_spin_lock
> 0,26%  nixstatsagent  [kernel.vmlinux]  [k] shrink_slab                   [k] shrink_slab
> 0,23%  nixstatsagent  [kernel.vmlinux]  [k] super_cache_count             [k] super_cache_count
> 0,15%  nixstatsagent  [kernel.vmlinux]  [k] __list_lru_count_one.isra.2   [k] _raw_spin_lock
> 0,15%  nixstatsagent  [kernel.vmlinux]  [k] list_lru_count_one            [k] __list_lru_count_one.isra.2
> 
> 0,94%  mysqld         [kernel.vmlinux]  [k] _raw_spin_lock                [k] _raw_spin_lock
> 0,57%  mysqld         [kernel.vmlinux]  [k] shrink_slab                   [k] shrink_slab
> 0,51%  mysqld         [kernel.vmlinux]  [k] super_cache_count             [k] super_cache_count
> 0,32%  mysqld         [kernel.vmlinux]  [k] __list_lru_count_one.isra.2   [k] _raw_spin_lock
> 0,32%  mysqld         [kernel.vmlinux]  [k] list_lru_count_one            [k] __list_lru_count_one.isra.2
> 
> 0,73%  sshd           [kernel.vmlinux]  [k] _raw_spin_lock                [k] _raw_spin_lock
> 0,35%  sshd           [kernel.vmlinux]  [k] shrink_slab                   [k] shrink_slab
> 0,32%  sshd           [kernel.vmlinux]  [k] super_cache_count             [k] super_cache_count
> 0,21%  sshd           [kernel.vmlinux]  [k] __list_lru_count_one.isra.2   [k] _raw_spin_lock
> 0,21%  sshd           [kernel.vmlinux]  [k] list_lru_count_one            [k] __list_lru_count_one.isra.2
> 
> This patch aims to make super_cache_count() (and other functions,
> which count LRU nr_items) more effective.
> It allows list_lru_node::memcg_lrus to be RCU-accessed, and makes
> __list_lru_count_one() count nr_items lockless to minimize
> overhead introduced by locking operation, and to make parallel
> reclaims more scalable.

And...  what were the effects of the patch?  Did you not run the same
performance tests after applying it?

[toc] | [next] | [standalone]

#1741814

On Thu, 28 Sep 2017 10:48:55 +0300 Kirill Tkhai <ktkhai@virtuozzo.com> wrote:

> >> This patch aims to make super_cache_count() (and other functions,
> >> which count LRU nr_items) more effective.
> >> It allows list_lru_node::memcg_lrus to be RCU-accessed, and makes
> >> __list_lru_count_one() count nr_items lockless to minimize
> >> overhead introduced by locking operation, and to make parallel
> >> reclaims more scalable.
> > 
> > And...  what were the effects of the patch?  Did you not run the same
> > performance tests after applying it?
> 
> I've just detected the such high usage of shrink slab on production node. It's rather
> difficult to make it use another kernel, than it uses, only kpatches are possible.
> So, I haven't estimated how it acts on node's performance.
> On test node I see, that the patch obviously removes raw_spin_lock from perf profile.
> So, it's a little bit untested in this way.

Well that's a problem.  The patch increases list_lru.o text size by a
lot (4800->5696) which will have a cost.  And we don't have proof that
any benefit is worth that cost.  It shouldn't be too hard to cook up a
synthetic test to trigger memcg slab reclaim and then run a
before-n-after benchmark?

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web