Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1698111 > unrolled thread

[PATCH 0/3] memdelay: memory health metric for systems and workloads

Back to article view | Back to linux.kernel

Contents

  [PATCH 0/3] memdelay: memory health metric for systems and workloads Johannes Weiner <hannes@cmpxchg.org> - 2017-07-27 17:40 +0200
    Re: [PATCH 0/3] memdelay: memory health metric for systems and  workloads Andrew Morton <akpm@linux-foundation.org> - 2017-07-27 22:50 +0200
      Re: [PATCH 0/3] memdelay: memory health metric for systems and  workloads Johannes Weiner <hannes@cmpxchg.org> - 2017-07-28 21:50 +0200
    Re: [PATCH 0/3] memdelay: memory health metric for systems and  workloads Mike Galbraith <efault@gmx.de> - 2017-07-29 04:50 +0200
      Re: [PATCH 0/3] memdelay: memory health metric for systems and  workloads Mike Galbraith <efault@gmx.de> - 2017-07-29 05:30 +0200
      Re: [PATCH 0/3] memdelay: memory health metric for systems and  workloads Mike Galbraith <efault@gmx.de> - 2017-07-29 08:40 +0200

#1698111 — [PATCH 0/3] memdelay: memory health metric for systems and workloads

This patch series implements a fine-grained metric for memory
health. It builds on top of the refault detection code to quantify the
time lost on VM events that occur exclusively due a lack of memory and
maps it into a percentage of lost walltime for the system and cgroups.

Rationale

When presented with a Linux system or container executing a workload,
it's hard to judge the health of its memory situation.

The statistics exported by the memory management subsystem can reveal
smoking guns: page reclaim activity, major faults and refaults can be
indicative of an unhealthy memory situation. But they don't actually
quantify the cost a memory shortage imposes on the system or workload.

How bad is it when 2000 pages are refaulting each second? If the data
is stored contiguously on a fast flash drive, it might be okay. If the
data is spread out all over a rotating disk, it could be a problem -
unless the CPUs are still fully utilized, in which case adding memory
wouldn't make things move faster, but instead wait for CPU time.

A previous attempt to provide a health signal from the VM was the
vmpressure interface, 70ddf637eebe ("memcg: add memory.pressure_level
events"). This derives its pressure levels from recently observed
reclaim efficiency. As pages are scanned but not reclaimed, the ratio
is translated into levels of low, medium, and critical pressure.

However, the vmpressure scale is too coarse for today's systems. The
accuracy relies on storage being relatively slow compared to how fast
the CPU can go through the LRUs, so that when LRU scan cycles outstrip
IO completion rates the reclaim code runs into pages that are still
reading from disk. But as solid state devices close this speed gap,
and memory sizes are in the hundreds of gigabytes, this effect has
almost completely disappeared. By the time the reclaim scanner runs
into in-flight pages, the tasks in the system already spend a
significant part of their runtime waiting for refaulting pages. The
vmpressure range is compressed into the split second before OOM and
misses large, practically relevant parts of the pressure spectrum.

Knowing the exact time penalty that the kernel's paging activity is
imposing on a workload is a powerful tool. It allows users to finetune
a workload to available memory, but also detect and quantify minute
regressions and improvements in the reclaim and caching algorithms.

Structure

The first patch cleans up the different loadavg callsites and macros
as the memdelay averages are going to be tracked using these.

The second patch adds a distinction between page cache transitions
(inactive list refaults) and page cache thrashing (active list
refaults), since only the latter are unproductive refaults.

The third patch finally adds the memdelay accounting and interface:
its scheduler side identifies productive and unproductive task states,
and the VM side aggregates them into system and cgroup domain states
and calculates moving averages of the time spent in each state.

 arch/powerpc/platforms/cell/spufs/sched.c |   3 -
 arch/s390/appldata/appldata_os.c          |   4 -
 drivers/cpuidle/governors/menu.c          |   4 -
 fs/proc/array.c                           |   8 +
 fs/proc/base.c                            |   2 +
 fs/proc/internal.h                        |   2 +
 fs/proc/loadavg.c                         |   3 -
 include/linux/cgroup.h                    |  14 ++
 include/linux/memcontrol.h                |  14 ++
 include/linux/memdelay.h                  | 174 +++++++++++++++++
 include/linux/mmzone.h                    |   1 +
 include/linux/page-flags.h                |   5 +-
 include/linux/sched.h                     |  10 +-
 include/linux/sched/loadavg.h             |   3 +
 include/linux/swap.h                      |   2 +-
 include/trace/events/mmflags.h            |   1 +
 kernel/cgroup/cgroup.c                    |   4 +-
 kernel/debug/kdb/kdb_main.c               |   7 +-
 kernel/fork.c                             |   4 +
 kernel/sched/Makefile                     |   2 +-
 kernel/sched/core.c                       |  20 ++
 kernel/sched/memdelay.c                   | 112 +++++++++++
 mm/Makefile                               |   2 +-
 mm/compaction.c                           |   4 +
 mm/filemap.c                              |  18 +-
 mm/huge_memory.c                          |   1 +
 mm/memcontrol.c                           |  25 +++
 mm/memdelay.c                             | 289 ++++++++++++++++++++++++++++
 mm/migrate.c                              |   2 +
 mm/page_alloc.c                           |  11 +-
 mm/swap_state.c                           |   1 +
 mm/vmscan.c                               |  10 +
 mm/vmstat.c                               |   1 +
 mm/workingset.c                           |  98 ++++++----
 34 files changed, 792 insertions(+), 69 deletions(-)

[toc] | [next] | [standalone]

#1698304 — Re: [PATCH 0/3] memdelay: memory health metric for systems and workloads

On Thu, 27 Jul 2017 11:30:07 -0400 Johannes Weiner <hannes@cmpxchg.org> wrote:

> This patch series implements a fine-grained metric for memory
> health.

I assume some Documentation/ is forthcoming.

Consuming another page flag hurts.  What's our current status there?

I'd be interested in seeing some usage examples.  Perhaps anecdotes
where "we observed problem X so we used memdelay in manner Y and saw
result Z".

I assume that some userspace code which utilizes this interface exists
already.  What's the long-term plan here?  systemd changes?

[toc] | [prev] | [next] | [standalone]

#1699041 — Re: [PATCH 0/3] memdelay: memory health metric for systems and workloads

Hi Andrew,

On Thu, Jul 27, 2017 at 01:43:25PM -0700, Andrew Morton wrote:
> On Thu, 27 Jul 2017 11:30:07 -0400 Johannes Weiner <hannes@cmpxchg.org> wrote:
> 
> > This patch series implements a fine-grained metric for memory
> > health.
> 
> I assume some Documentation/ is forthcoming.

Yep, I'll describe the interface and how to use this more extensively.

> Consuming another page flag hurts.  What's our current status there?

I would say we can make it 64-bit only, but I also need this refault
distinction flag in the LRU balancing patches [1] to apply pressure on
anon pages only when the page cache is actually thrashing, not when
it's just transitioning to another workingset. So let's see...

20 flags are always defined.

21 if you have an MMU.

23 with the zone bits for DMA, Normal, HighMem, Movable.

29 with the sparsemem section bits,

30 if PAE is enabled.

On that config, NUMA gets 2 bits for 4 nodes. If I take the 31st bit,
it'd be left with 2 possible nodes. If that's not enough, that system
can switch to discontigmem and regain the 6 or 7 sparsemem bits.

> I'd be interested in seeing some usage examples.  Perhaps anecdotes
> where "we observed problem X so we used memdelay in manner Y and saw
> result Z".

The very first thing that had me look at this was the pathological
behavior of memory pressure after switching my systems from spinning
disks to SSDs. Just like vmpressure, the OOM killer depends on reclaim
efficiency dropping through the floor - but that doesn't really happen
anymore. Sometimes my systems lock up for seconds, sometimes minutes,
or until I hard-reset them. The page cache, including executables, is
thrashing like crazy while reclaim efficiency hovers around 100%.

The same happens at FB data centers, where we lose machines during
peak times with no kernel-side remedy for recovering this livelock.

The OOM killer really needs to be hooked up to a portable measure of
thrashing impact rather than an inability of the VM to recycle pages.
I think expressing this cutoff in terms of unproductive time makes the
most sense: e.g. 60%+ of the last 10 seconds of elapsed walltime the
system was doing nothing but waiting for refaults or reclaiming; time
to kill something to free up memory and reduce access frequencies.

But even before OOM, we need to know when we start packing machines
and containers too tightly in terms of memory. Free pages don't mean
anything because of the page cache, and the refault rate on its own
doesn't tell you anything about throughput or latency deterioration.

A recurring scenario for me is that somebody has a machine running a
workload with peaks of 100% CPU, 100% IO, bursts of refaults and a
slowdown in the application. What resource is really lacking here? A
lack of memory can result in high CPU and IO times, but it could also
be mostly the application's own appetite for those resources. The
memdelay code shows us how much of the slowdown is caused by memory.

Figuring this out with tracing and profiling is *sometimes* possible,
but takes a ridiculous amount of effort and a reproducible workload.
In many cases it's not an option due to the scale we're dealing with.

For example, we have large pools of machines that run some hundred
jobs whose peak activity depends in part on factors out of our
control, such as user activity. When the peaks of several jobs align,
their individual throughput and latency goes down, and like above we
see the CPU, IO, latency spikes. Separating out how much of that is
due to memory then feeds into the job scheduler, which adjusts the job
placement, cgroup limits etc. accordingly throughout the pool.

Another thing is detecting regressions. Kernel developers tend to run
handpicked, idempotent A/B tests, on single machines, to detect
walltime impact of VM changes. That's often not very representative of
real applications. By tracking memdelay trends averaged over thousands
of machines that run similar workloads, we can tell whether a kernel
upgrade introduced a VM regression that matters to real applications
down to sub-percent walltime impact fairly easily and reliably.

Even with workloads that have their own clear throughput metrics to
detect regressions, knowing where to look makes finding problems
easier, which makes upgrades faster, which means we can run yet more
recent kernels :)

> I assume that some userspace code which utilizes this interface exists
> already.  What's the long-term plan here?  systemd changes?

We're putting it into our custom job scheduler/load balancers and
fleet monitoring infrastructure to track capacity and regressions.

System health monitoring tools like top, atop etc. can incorporate
this in their summaries as well as per-task statistics.

Things like systemd-cgtop that give container overviews can as well.

And as mentioned above, IMO the OOM killer is a prime candidate for
being an in-kernel user of this.

Thanks

[1] https://lwn.net/Articles/690079/

[toc] | [prev] | [next] | [standalone]

#1699206 — Re: [PATCH 0/3] memdelay: memory health metric for systems and workloads

On Thu, 2017-07-27 at 11:30 -0400, Johannes Weiner wrote:
> 
> Structure
> 
> The first patch cleans up the different loadavg callsites and macros
> as the memdelay averages are going to be tracked using these.
> 
> The second patch adds a distinction between page cache transitions
> (inactive list refaults) and page cache thrashing (active list
> refaults), since only the latter are unproductive refaults.
> 
> The third patch finally adds the memdelay accounting and interface:
> its scheduler side identifies productive and unproductive task states,
> and the VM side aggregates them into system and cgroup domain states
> and calculates moving averages of the time spent in each state.

What tree is this against?  ttwu asm delta says "measure me".

	-Mike

[toc] | [prev] | [next] | [standalone]

#1699207 — Re: [PATCH 0/3] memdelay: memory health metric for systems and workloads

On Sat, 2017-07-29 at 04:48 +0200, Mike Galbraith wrote:
> On Thu, 2017-07-27 at 11:30 -0400, Johannes Weiner wrote:
> > 
> > Structure
> > 
> > The first patch cleans up the different loadavg callsites and macros
> > as the memdelay averages are going to be tracked using these.
> > 
> > The second patch adds a distinction between page cache transitions
> > (inactive list refaults) and page cache thrashing (active list
> > refaults), since only the latter are unproductive refaults.
> > 
> > The third patch finally adds the memdelay accounting and interface:
> > its scheduler side identifies productive and unproductive task states,
> > and the VM side aggregates them into system and cgroup domain states
> > and calculates moving averages of the time spent in each state.
> 
> What tree is this against?  ttwu asm delta says "measure me".

(mm/master.. gee)

[toc] | [prev] | [next] | [standalone]

#1699239 — Re: [PATCH 0/3] memdelay: memory health metric for systems and workloads

On Sat, 2017-07-29 at 04:48 +0200, Mike Galbraith wrote:
> ttwu asm delta says "measure me".

q/d measurement with pipe-test

+cgroup_disable=memory
2.241926 usecs/loop -- avg 2.242376 891.9 KHz  1.000
+patchset
2.284428 usecs/loop -- avg 2.357621 848.3 KHz   .951

-cgroup_disable=memory
2.257433 usecs/loop -- avg 2.327356 859.3 KHz  1.000
+patchset
2.394804 usecs/loop -- avg 2.404556 831.8 KHz   .967

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web