Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1613243 > unrolled thread

Re: [RFC 0/1] add support for reclaiming priorities per mem cgroup

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [RFC 0/1] add support for reclaiming priorities per mem cgroup Johannes Weiner <hannes@cmpxchg.org> - 2017-03-30 18:00 +0200
    Re: [RFC 0/1] add support for reclaiming priorities per mem cgroup Shakeel Butt <shakeelb@google.com> - 2017-03-30 18:50 +0200
    Re: [RFC 0/1] add support for reclaiming priorities per mem cgroup Tim Murray <timmurray@google.com> - 2017-03-30 21:50 +0200
      Re: [RFC 0/1] add support for reclaiming priorities per mem cgroup Tim Murray <timmurray@google.com> - 2017-03-31 00:00 +0200

#1613243 — Re: [RFC 0/1] add support for reclaiming priorities per mem cgroup

Hi Tim,

On Fri, Mar 17, 2017 at 04:16:35PM -0700, Tim Murray wrote:
> Hi all,
> 
> I've been working to improve Android's memory management and drop lowmemorykiller from the kernel, and I'd like to get some feedback on a small patch with a lot of side effects. 
> 
> Currently, when an Android device is under memory pressure, one of three things will happen from kswapd:
> 
> 1. Compress an anonymous page to ZRAM.
> 2. Evict a file page.
> 3. Kill a process via lowmemorykiller.
> 
> The first two are cheap and per-page, the third is relatively cheap in the short term, frees many pages, and may cause power and performance penalties later on when the process has to be started again. For lots of reasons, I'd like a better balance between reclamation and killing on Android.
> 
> One of the nice things about Android from an optimization POV is that the execution model is more constrained than a generic Linux machine. There are only a limited number of processes that need to execute quickly for the device to appear to have good performance, and a userspace daemon (called ActivityManagerService) knows exactly what those processes are at any given time. We've made use of that in the past via cpusets and schedtune to limit the CPU resources available to background processes, and I think we can apply the same concept to memory.
> 
> This patch adds a new tunable to mem cgroups, memory.priority. A mem cgroup with a non-zero priority will not be eligible for scanning until the scan_control's priority is greater than zero. Once the mem cgroup is eligible for scanning, the priority acts as a bias to reduce the number of pages that should be scanned.
> 
> We've seen cases on Android where the global LRU isn't sufficient. For example, notifications in Android are rendered as part of a separate process that runs infrequently. However, when a notification appears and the user slides down the notification tray, we'll often see dropped frames due to page faults if there has been severe memory pressure. There are similar issues with other persistent processes.
> 
> The goal on an Android device is to aggressively evict from very low-priority background tasks that are likely to be killed anyway, since this will reduce the likelihood of lowmemorykiller running in the first place. It will still evict some from foreground and persistent processes, but it should help ensure that background processes are effectively reduced to the size of their heaps before evicting from more critical tasks. This should mean fewer background processes end up killed, which should improve performance and power on Android across the board (since it costs significantly less to page things back in than to replay the entirety of application startup).

In cgroup2, we've added a memory.low knob, where groups within their
memory.low setting are not reclaimed.

You can set that knob on foreground groups to the amount of memory
they need to function properly, and set it to 0 on background groups.

Have you tried doing that?

> The follow-on that I'm also experimenting with is how to improve vmpressure such that userspace can have some idea when low-priority memory cgroups are about as small as they can get. The correct time for Android to kill a background process under memory pressure is when there is evidence that a process has to be killed in order to alleviate memory pressure. If the device is below the low memory watermark and we know that there's probably no way to reclaim any more from background processes, then a userspace daemon should kill one or more background processes to fix that. Per-cgroup priority could be the first step toward that information.

Memory pressure is a wider-reaching issue, something I've been working
on for a while.

Both vmpressure and priority levels are based on reclaim efficiency,
which is problematic on solid state storage because page reads have
very low latency. It's rare that pages are still locked from the
read-in by the time reclaim gets to them on the LRU, so efficiency
tends to stay at 100%, until the system is essentially livelocked.

On solid state storage, the bigger problem when you don't have enough
memory is that you can reclaim just fine but wait a significant amount
of time to refault the recently evicted pages, i.e. on thrashing.

A more useful metric for memory pressure at this point is quantifying
that time you spend thrashing: time the job spends in direct reclaim
and on the flipside time the job waits for recently evicted pages to
come back. Combined, that gives you a good measure of overhead from
memory pressure; putting that in relation to a useful baseline of
meaningful work done gives you a portable scale of how effictively
your job is running.

I'm working on that right now, hopefully I'll have something useful
soon.

[toc] | [next] | [standalone]

#1613291

> A more useful metric for memory pressure at this point is quantifying
> that time you spend thrashing: time the job spends in direct reclaim
> and on the flipside time the job waits for recently evicted pages to
> come back. Combined, that gives you a good measure of overhead from
> memory pressure; putting that in relation to a useful baseline of
> meaningful work done gives you a portable scale of how effictively
> your job is running.
>
> I'm working on that right now, hopefully I'll have something useful
> soon.

Johannes, is the work you are doing only about file pages or will it
equally apply to anon pages as well?

[toc] | [prev] | [next] | [standalone]

#1613414

On Thu, Mar 30, 2017 at 8:51 AM, Johannes Weiner <hannes@cmpxchg.org> wrote:
> In cgroup2, we've added a memory.low knob, where groups within their
> memory.low setting are not reclaimed.
>
> You can set that knob on foreground groups to the amount of memory
> they need to function properly, and set it to 0 on background groups.
>
> Have you tried doing that?

I have not, but I'm trying to get that working now to evaluate it on Android.

However, based on other experiences, I don't think it will work well.
We've experimented a lot with different limits in different places
(Java heap limits, hard_reclaim, soft_reclaim) at different times in
the process lifecycle, and the problem has always been that there's no
way for us to know what limit is reasonable. memory.low will have the
same problem. If memory.low is higher than the actual working set of a
foreground process, the system wastes memory (eg, file pages loaded
during app startup that are never used again won't be reclaimed under
pressure). If memory.low is less than the actual working set,
foreground processes will still get hit by thrashing.

Another issue is that the working set varies tremendously from app to
app. An email client's working set may be 1/10 or 1/20 of a camera
running a computational photography pipeline with multiple captures in
flight. I can imagine a case where it makes sense for a foreground
application to take 50-75% of a device's physical memory (the camera
case or something similar), but I hope that's an extreme outlier
compared to most apps on the system. However, high-memory apps are
often the most performance-sensitive, so reclaim is more likely to
cause problems.

As a result, I think there's still a need for relative priority
between mem cgroups, not just an absolute limit.

Does that make sense?

> Both vmpressure and priority levels are based on reclaim efficiency,
> which is problematic on solid state storage because page reads have
> very low latency. It's rare that pages are still locked from the
> read-in by the time reclaim gets to them on the LRU, so efficiency
> tends to stay at 100%, until the system is essentially livelocked.
>
> On solid state storage, the bigger problem when you don't have enough
> memory is that you can reclaim just fine but wait a significant amount
> of time to refault the recently evicted pages, i.e. on thrashing.
>
> A more useful metric for memory pressure at this point is quantifying
> that time you spend thrashing: time the job spends in direct reclaim
> and on the flipside time the job waits for recently evicted pages to
> come back. Combined, that gives you a good measure of overhead from
> memory pressure; putting that in relation to a useful baseline of
> meaningful work done gives you a portable scale of how effictively
> your job is running.

This sounds fantastic, and it matches the behavior I've seen around
pagecache thrashing on Android.

On Android, I think there are three different times where userspace
would do something useful for memory:

1. scan priority is creeping up, scanned/reclaim ratio is getting
worse, system is exhibiting signs of approaching severe memory
pressure. userspace should probably kill something if it's got
something it can kill cheaply.
2. direct reclaim is happening, system is thrashing, things are bad.
userspace should aggressively kill non-critical processes because
performance has already gotten worse.
3. something's gone horribly wrong, oom_killer is imminent: userspace
should kill everything it possibly can to keep the system stable.

My vmpressure experiments have focused on #1 because it integrates
nicely with memcg priorities. However, it doesn't seem like a good
approach for #2 or #3. Time spent thrashing sounds ideal for #2. I'm
not sure what to do for #3. The current critical vmpressure event
hasn't been that successful in avoiding oom-killer (on 3.18, at
least)--I've been able to get oom-killer to trigger without a
vmpressure event.

Assuming that memcg priorities are reasonable, would you be open to
using scan priority info as a vmpressure signal for a low amount of
memory pressure?

[toc] | [prev] | [next] | [standalone]

#1613507

On Thu, Mar 30, 2017 at 12:40 PM, Tim Murray <timmurray@google.com> wrote:
> The current critical vmpressure event
> hasn't been that successful in avoiding oom-killer (on 3.18, at
> least)--I've been able to get oom-killer to trigger without a
> vmpressure event.

Looked at this some more, and this is almost certainly because
vmpressure relies on workqueues. Scheduling delay from CFS workqueues
would explain vmpressure latency that results in oom-killer running
long before the critical vmpressure notification is received in
userspace, even if userspace is running as FIFO. We regularly see
10ms+ latency on workqueues, even when an Android device isn't heavily
loaded.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web