Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1732654 > unrolled thread

Detecting page cache trashing state

Back to article view | Back to linux.kernel

Contents

  Detecting page cache trashing state Taras Kondratiuk <takondra@cisco.com> - 2017-09-15 02:30 +0200
    Re: Detecting page cache trashing state Zdenek Kabelac <zkabelac@redhat.com> - 2017-09-15 14:00 +0200
    Re: Detecting page cache trashing state Daniel Walker <danielwa@cisco.com> - 2017-09-15 16:40 +0200
      Re: Detecting page cache trashing state Taras Kondratiuk <takondra@cisco.com> - 2017-09-15 18:50 +0200
        Re: Detecting page cache trashing state Daniel Walker <danielwa@cisco.com> - 2017-09-15 19:40 +0200
    Re: Detecting page cache trashing state Michal Hocko <mhocko@kernel.org> - 2017-09-15 16:40 +0200
      Re: Detecting page cache trashing state Taras Kondratiuk <takondra@cisco.com> - 2017-09-15 19:30 +0200
      Re: Detecting page cache trashing state vcaputo@pengaru.com - 2017-09-15 23:20 +0200
        Re: Detecting page cache trashing state Taras Kondratiuk <takondra@cisco.com> - 2017-09-16 01:50 +0200
        Re: Detecting page cache trashing state Michal Hocko <mhocko@kernel.org> - 2017-09-18 08:00 +0200

#1732654 — Detecting page cache trashing state

Hi

In our devices under low memory conditions we often get into a trashing
state when system spends most of the time re-reading pages of .text
sections from a file system (squashfs in our case). Working set doesn't
fit into available page cache, so it is expected. The issue is that
OOM killer doesn't get triggered because there is still memory for
reclaiming. System may stuck in this state for a quite some time and
usually dies because of watchdogs.

We are trying to detect such trashing state early to take some
preventive actions. It should be a pretty common issue, but for now we
haven't find any existing VM/IO statistics that can reliably detect such
state.

Most of metrics provide absolute values: number/rate of page faults,
rate of IO operations, number of stolen pages, etc. For a specific
device configuration we can determine threshold values for those
parameters that will detect trashing state, but it is not feasible for
hundreds of device configurations.

We are looking for some relative metric like "percent of CPU time spent
handling major page faults". With such relative metric we could use a
common threshold across all devices. For now we have added such metric
to /proc/stat in our kernel, but we would like to find some mechanism
available in upstream kernel.

Has somebody faced similar issue? How are you solving it?

[toc] | [next] | [standalone]

#1732839

Dne 15.9.2017 v 02:16 Taras Kondratiuk napsal(a):
> Hi
> 
> In our devices under low memory conditions we often get into a trashing
> state when system spends most of the time re-reading pages of .text
> sections from a file system (squashfs in our case). Working set doesn't
> fit into available page cache, so it is expected. The issue is that
> OOM killer doesn't get triggered because there is still memory for
> reclaiming. System may stuck in this state for a quite some time and
> usually dies because of watchdogs.
> 
> We are trying to detect such trashing state early to take some
> preventive actions. It should be a pretty common issue, but for now we
> haven't find any existing VM/IO statistics that can reliably detect such
> state.
> 
> Most of metrics provide absolute values: number/rate of page faults,
> rate of IO operations, number of stolen pages, etc. For a specific
> device configuration we can determine threshold values for those
> parameters that will detect trashing state, but it is not feasible for
> hundreds of device configurations.
> 
> We are looking for some relative metric like "percent of CPU time spent
> handling major page faults". With such relative metric we could use a
> common threshold across all devices. For now we have added such metric
> to /proc/stat in our kernel, but we would like to find some mechanism
> available in upstream kernel.
> 
> Has somebody faced similar issue? How are you solving it?
> 
Hi

Well I witness this when running Firefox & Thunderbird on my desktop for a 
while on just 4G RAM machine till these 2app eat all free RAM...

It gets to the position (when I open new tab) that mouse hardly moves - 
kswapd eats  CPU  (I've no swap in fact - so likely just page-caching).

The only 'quick' solution for me as desktop user is to manually invoke OOM
with SYSRQ+F key -  and I'm also wondering why the system is not reacting 
better.  In most cases it kills one of those 2 - but sometime it kills whole 
Xsession...


Regards

Zdenek

[toc] | [prev] | [next] | [standalone]

#1732913

On 09/14/2017 05:16 PM, Taras Kondratiuk wrote:
> Hi
>
> In our devices under low memory conditions we often get into a trashing
> state when system spends most of the time re-reading pages of .text
> sections from a file system (squashfs in our case). Working set doesn't
> fit into available page cache, so it is expected. The issue is that
> OOM killer doesn't get triggered because there is still memory for
> reclaiming. System may stuck in this state for a quite some time and
> usually dies because of watchdogs.
>
> We are trying to detect such trashing state early to take some
> preventive actions. It should be a pretty common issue, but for now we
> haven't find any existing VM/IO statistics that can reliably detect such
> state.
>
> Most of metrics provide absolute values: number/rate of page faults,
> rate of IO operations, number of stolen pages, etc. For a specific
> device configuration we can determine threshold values for those
> parameters that will detect trashing state, but it is not feasible for
> hundreds of device configurations.
>
> We are looking for some relative metric like "percent of CPU time spent
> handling major page faults". With such relative metric we could use a
> common threshold across all devices. For now we have added such metric
> to /proc/stat in our kernel, but we would like to find some mechanism
> available in upstream kernel.
>
> Has somebody faced similar issue? How are you solving it?


Did you make any attempt to tune swappiness ?

Documentation/sysctl/vm.txt

swappiness

This control is used to define how aggressive the kernel will swap
memory pages.  Higher values will increase agressiveness, lower values
decrease the amount of swap.

The default value is 60.
=======================================================

Since your using squashfs I would guess that's going to act like swap. 
The default tune of 60 is most likely for x86 servers which may not be a 
good value for some other device.


Daniel

[toc] | [prev] | [next] | [standalone]

#1732949

Quoting Daniel Walker (2017-09-15 07:22:27)
> On 09/14/2017 05:16 PM, Taras Kondratiuk wrote:
> > Hi
> >
> > In our devices under low memory conditions we often get into a trashing
> > state when system spends most of the time re-reading pages of .text
> > sections from a file system (squashfs in our case). Working set doesn't
> > fit into available page cache, so it is expected. The issue is that
> > OOM killer doesn't get triggered because there is still memory for
> > reclaiming. System may stuck in this state for a quite some time and
> > usually dies because of watchdogs.
> >
> > We are trying to detect such trashing state early to take some
> > preventive actions. It should be a pretty common issue, but for now we
> > haven't find any existing VM/IO statistics that can reliably detect such
> > state.
> >
> > Most of metrics provide absolute values: number/rate of page faults,
> > rate of IO operations, number of stolen pages, etc. For a specific
> > device configuration we can determine threshold values for those
> > parameters that will detect trashing state, but it is not feasible for
> > hundreds of device configurations.
> >
> > We are looking for some relative metric like "percent of CPU time spent
> > handling major page faults". With such relative metric we could use a
> > common threshold across all devices. For now we have added such metric
> > to /proc/stat in our kernel, but we would like to find some mechanism
> > available in upstream kernel.
> >
> > Has somebody faced similar issue? How are you solving it?
> 
> 
> Did you make any attempt to tune swappiness ?
> 
> Documentation/sysctl/vm.txt
> 
> swappiness
> 
> This control is used to define how aggressive the kernel will swap
> memory pages.  Higher values will increase agressiveness, lower values
> decrease the amount of swap.
> 
> The default value is 60.
> =======================================================
> 
> Since your using squashfs I would guess that's going to act like swap. 
> The default tune of 60 is most likely for x86 servers which may not be a 
> good value for some other device.

Swap is disabled in our systems, so anonymous pages can't be evicted.
As per my understanding swappiness tune is irrelevant.

Even with enabled swap swappiness tune can't help much in this case. If
working set doesn't fit into available page cache we will hit the same
trashing state.

[toc] | [prev] | [next] | [standalone]

#1732979

On 09/15/2017 09:38 AM, Taras Kondratiuk wrote:
> Quoting Daniel Walker (2017-09-15 07:22:27)
>> On 09/14/2017 05:16 PM, Taras Kondratiuk wrote:
>>> Hi
>>>
>>> In our devices under low memory conditions we often get into a trashing
>>> state when system spends most of the time re-reading pages of .text
>>> sections from a file system (squashfs in our case). Working set doesn't
>>> fit into available page cache, so it is expected. The issue is that
>>> OOM killer doesn't get triggered because there is still memory for
>>> reclaiming. System may stuck in this state for a quite some time and
>>> usually dies because of watchdogs.
>>>
>>> We are trying to detect such trashing state early to take some
>>> preventive actions. It should be a pretty common issue, but for now we
>>> haven't find any existing VM/IO statistics that can reliably detect such
>>> state.
>>>
>>> Most of metrics provide absolute values: number/rate of page faults,
>>> rate of IO operations, number of stolen pages, etc. For a specific
>>> device configuration we can determine threshold values for those
>>> parameters that will detect trashing state, but it is not feasible for
>>> hundreds of device configurations.
>>>
>>> We are looking for some relative metric like "percent of CPU time spent
>>> handling major page faults". With such relative metric we could use a
>>> common threshold across all devices. For now we have added such metric
>>> to /proc/stat in our kernel, but we would like to find some mechanism
>>> available in upstream kernel.
>>>
>>> Has somebody faced similar issue? How are you solving it?
>>
>> Did you make any attempt to tune swappiness ?
>>
>> Documentation/sysctl/vm.txt
>>
>> swappiness
>>
>> This control is used to define how aggressive the kernel will swap
>> memory pages.  Higher values will increase agressiveness, lower values
>> decrease the amount of swap.
>>
>> The default value is 60.
>> =======================================================
>>
>> Since your using squashfs I would guess that's going to act like swap.
>> The default tune of 60 is most likely for x86 servers which may not be a
>> good value for some other device.
> Swap is disabled in our systems, so anonymous pages can't be evicted.
> As per my understanding swappiness tune is irrelevant.
>
> Even with enabled swap swappiness tune can't help much in this case. If
> working set doesn't fit into available page cache we will hit the same
> trashing state.


I think it's our lack of understanding of how the VM works. If the 
system has no swap, then the system shouldn't start evicting pages 
unless you have %100 memory utilization, then the only place for those 
pages to go is back into the backing store, squashfs in this case.

What your suggesting is that there is still free memory, which means 
something must be evicting page more aggressively then waiting till %100 
utilization. Maybe someone more knownlegable about the VM subsystem can 
clear this up.

Daniel

[toc] | [prev] | [next] | [standalone]

#1732914

On Thu 14-09-17 17:16:27, Taras Kondratiuk wrote:
> Hi
> 
> In our devices under low memory conditions we often get into a trashing
> state when system spends most of the time re-reading pages of .text
> sections from a file system (squashfs in our case). Working set doesn't
> fit into available page cache, so it is expected. The issue is that
> OOM killer doesn't get triggered because there is still memory for
> reclaiming. System may stuck in this state for a quite some time and
> usually dies because of watchdogs.
> 
> We are trying to detect such trashing state early to take some
> preventive actions. It should be a pretty common issue, but for now we
> haven't find any existing VM/IO statistics that can reliably detect such
> state.
> 
> Most of metrics provide absolute values: number/rate of page faults,
> rate of IO operations, number of stolen pages, etc. For a specific
> device configuration we can determine threshold values for those
> parameters that will detect trashing state, but it is not feasible for
> hundreds of device configurations.
> 
> We are looking for some relative metric like "percent of CPU time spent
> handling major page faults". With such relative metric we could use a
> common threshold across all devices. For now we have added such metric
> to /proc/stat in our kernel, but we would like to find some mechanism
> available in upstream kernel.
> 
> Has somebody faced similar issue? How are you solving it?

Yes this is a pain point for a _long_ time. And we still do not have a
good answer upstream. Johannes has been playing in this area [1].
The main problem is that our OOM detection logic is based on the ability
to reclaim memory to allocate new memory. And that is pretty much true
for the pagecache when you are trashing. So we do not know that
basically whole time is spent refaulting the memory back and forth.
We do have some refault stats for the page cache but that is not
integrated to the oom detection logic because this is really a
non-trivial problem to solve without triggering early oom killer
invocations.

[1] http://lkml.kernel.org/r/20170727153010.23347-1-hannes@cmpxchg.org
-- 
Michal Hocko
SUSE Labs

[toc] | [prev] | [next] | [standalone]

#1732976

Quoting Michal Hocko (2017-09-15 07:36:19)
> On Thu 14-09-17 17:16:27, Taras Kondratiuk wrote:
> > Hi
> > 
> > In our devices under low memory conditions we often get into a trashing
> > state when system spends most of the time re-reading pages of .text
> > sections from a file system (squashfs in our case). Working set doesn't
> > fit into available page cache, so it is expected. The issue is that
> > OOM killer doesn't get triggered because there is still memory for
> > reclaiming. System may stuck in this state for a quite some time and
> > usually dies because of watchdogs.
> > 
> > We are trying to detect such trashing state early to take some
> > preventive actions. It should be a pretty common issue, but for now we
> > haven't find any existing VM/IO statistics that can reliably detect such
> > state.
> > 
> > Most of metrics provide absolute values: number/rate of page faults,
> > rate of IO operations, number of stolen pages, etc. For a specific
> > device configuration we can determine threshold values for those
> > parameters that will detect trashing state, but it is not feasible for
> > hundreds of device configurations.
> > 
> > We are looking for some relative metric like "percent of CPU time spent
> > handling major page faults". With such relative metric we could use a
> > common threshold across all devices. For now we have added such metric
> > to /proc/stat in our kernel, but we would like to find some mechanism
> > available in upstream kernel.
> > 
> > Has somebody faced similar issue? How are you solving it?
> 
> Yes this is a pain point for a _long_ time. And we still do not have a
> good answer upstream. Johannes has been playing in this area [1].
> The main problem is that our OOM detection logic is based on the ability
> to reclaim memory to allocate new memory. And that is pretty much true
> for the pagecache when you are trashing. So we do not know that
> basically whole time is spent refaulting the memory back and forth.
> We do have some refault stats for the page cache but that is not
> integrated to the oom detection logic because this is really a
> non-trivial problem to solve without triggering early oom killer
> invocations.
> 
> [1] http://lkml.kernel.org/r/20170727153010.23347-1-hannes@cmpxchg.org

Thanks Michal. memdelay looks promising. We will check it.

[toc] | [prev] | [next] | [standalone]

#1733096

On Fri, Sep 15, 2017 at 04:36:19PM +0200, Michal Hocko wrote:
> On Thu 14-09-17 17:16:27, Taras Kondratiuk wrote:
> > Hi
> > 
> > In our devices under low memory conditions we often get into a trashing
> > state when system spends most of the time re-reading pages of .text
> > sections from a file system (squashfs in our case). Working set doesn't
> > fit into available page cache, so it is expected. The issue is that
> > OOM killer doesn't get triggered because there is still memory for
> > reclaiming. System may stuck in this state for a quite some time and
> > usually dies because of watchdogs.
> > 
> > We are trying to detect such trashing state early to take some
> > preventive actions. It should be a pretty common issue, but for now we
> > haven't find any existing VM/IO statistics that can reliably detect such
> > state.
> > 
> > Most of metrics provide absolute values: number/rate of page faults,
> > rate of IO operations, number of stolen pages, etc. For a specific
> > device configuration we can determine threshold values for those
> > parameters that will detect trashing state, but it is not feasible for
> > hundreds of device configurations.
> > 
> > We are looking for some relative metric like "percent of CPU time spent
> > handling major page faults". With such relative metric we could use a
> > common threshold across all devices. For now we have added such metric
> > to /proc/stat in our kernel, but we would like to find some mechanism
> > available in upstream kernel.
> > 
> > Has somebody faced similar issue? How are you solving it?
> 
> Yes this is a pain point for a _long_ time. And we still do not have a
> good answer upstream. Johannes has been playing in this area [1].
> The main problem is that our OOM detection logic is based on the ability
> to reclaim memory to allocate new memory. And that is pretty much true
> for the pagecache when you are trashing. So we do not know that
> basically whole time is spent refaulting the memory back and forth.
> We do have some refault stats for the page cache but that is not
> integrated to the oom detection logic because this is really a
> non-trivial problem to solve without triggering early oom killer
> invocations.
> 
> [1] http://lkml.kernel.org/r/20170727153010.23347-1-hannes@cmpxchg.org

For desktop users running without swap, couldn't we just provide a kernel
setting which marks all executable pages as unevictable when first faulted
in?  Then at least thrashing within the space occupied by executables and
shared libraries before eventual OOM would be avoided, and only the
remaining file-backed non-executable pages would be thrashable.

On my swapless laptops I'd much rather have OOM killer kick in immediately
rather than wait for a few minutes of thrashing to pass while the bogged
down system crawls through depleting what's left of technically reclaimable
memory.  It's much improved on modern SSDs, but still annoying.

Regards,
Vito Caputo

[toc] | [prev] | [next] | [standalone]

#1733165

Quoting vcaputo@pengaru.com (2017-09-15 14:20:28)
> On Fri, Sep 15, 2017 at 04:36:19PM +0200, Michal Hocko wrote:
> > On Thu 14-09-17 17:16:27, Taras Kondratiuk wrote:
> > > Hi
> > > 
> > > In our devices under low memory conditions we often get into a trashing
> > > state when system spends most of the time re-reading pages of .text
> > > sections from a file system (squashfs in our case). Working set doesn't
> > > fit into available page cache, so it is expected. The issue is that
> > > OOM killer doesn't get triggered because there is still memory for
> > > reclaiming. System may stuck in this state for a quite some time and
> > > usually dies because of watchdogs.
> > > 
> > > We are trying to detect such trashing state early to take some
> > > preventive actions. It should be a pretty common issue, but for now we
> > > haven't find any existing VM/IO statistics that can reliably detect such
> > > state.
> > > 
> > > Most of metrics provide absolute values: number/rate of page faults,
> > > rate of IO operations, number of stolen pages, etc. For a specific
> > > device configuration we can determine threshold values for those
> > > parameters that will detect trashing state, but it is not feasible for
> > > hundreds of device configurations.
> > > 
> > > We are looking for some relative metric like "percent of CPU time spent
> > > handling major page faults". With such relative metric we could use a
> > > common threshold across all devices. For now we have added such metric
> > > to /proc/stat in our kernel, but we would like to find some mechanism
> > > available in upstream kernel.
> > > 
> > > Has somebody faced similar issue? How are you solving it?
> > 
> > Yes this is a pain point for a _long_ time. And we still do not have a
> > good answer upstream. Johannes has been playing in this area [1].
> > The main problem is that our OOM detection logic is based on the ability
> > to reclaim memory to allocate new memory. And that is pretty much true
> > for the pagecache when you are trashing. So we do not know that
> > basically whole time is spent refaulting the memory back and forth.
> > We do have some refault stats for the page cache but that is not
> > integrated to the oom detection logic because this is really a
> > non-trivial problem to solve without triggering early oom killer
> > invocations.
> > 
> > [1] http://lkml.kernel.org/r/20170727153010.23347-1-hannes@cmpxchg.org
> 
> For desktop users running without swap, couldn't we just provide a kernel
> setting which marks all executable pages as unevictable when first faulted
> in?  Then at least thrashing within the space occupied by executables and
> shared libraries before eventual OOM would be avoided, and only the
> remaining file-backed non-executable pages would be thrashable.
> 
> On my swapless laptops I'd much rather have OOM killer kick in immediately
> rather than wait for a few minutes of thrashing to pass while the bogged
> down system crawls through depleting what's left of technically reclaimable
> memory.  It's much improved on modern SSDs, but still annoying.

Usually a significant part of executable is used rarely or only once
during initialization. Pinning all executable pages forever will waste
a lot of memory.

[toc] | [prev] | [next] | [standalone]

#1733659

On Fri 15-09-17 14:20:28, vcaputo@pengaru.com wrote:
> On Fri, Sep 15, 2017 at 04:36:19PM +0200, Michal Hocko wrote:
> > On Thu 14-09-17 17:16:27, Taras Kondratiuk wrote:
> > > Hi
> > > 
> > > In our devices under low memory conditions we often get into a trashing
> > > state when system spends most of the time re-reading pages of .text
> > > sections from a file system (squashfs in our case). Working set doesn't
> > > fit into available page cache, so it is expected. The issue is that
> > > OOM killer doesn't get triggered because there is still memory for
> > > reclaiming. System may stuck in this state for a quite some time and
> > > usually dies because of watchdogs.
> > > 
> > > We are trying to detect such trashing state early to take some
> > > preventive actions. It should be a pretty common issue, but for now we
> > > haven't find any existing VM/IO statistics that can reliably detect such
> > > state.
> > > 
> > > Most of metrics provide absolute values: number/rate of page faults,
> > > rate of IO operations, number of stolen pages, etc. For a specific
> > > device configuration we can determine threshold values for those
> > > parameters that will detect trashing state, but it is not feasible for
> > > hundreds of device configurations.
> > > 
> > > We are looking for some relative metric like "percent of CPU time spent
> > > handling major page faults". With such relative metric we could use a
> > > common threshold across all devices. For now we have added such metric
> > > to /proc/stat in our kernel, but we would like to find some mechanism
> > > available in upstream kernel.
> > > 
> > > Has somebody faced similar issue? How are you solving it?
> > 
> > Yes this is a pain point for a _long_ time. And we still do not have a
> > good answer upstream. Johannes has been playing in this area [1].
> > The main problem is that our OOM detection logic is based on the ability
> > to reclaim memory to allocate new memory. And that is pretty much true
> > for the pagecache when you are trashing. So we do not know that
> > basically whole time is spent refaulting the memory back and forth.
> > We do have some refault stats for the page cache but that is not
> > integrated to the oom detection logic because this is really a
> > non-trivial problem to solve without triggering early oom killer
> > invocations.
> > 
> > [1] http://lkml.kernel.org/r/20170727153010.23347-1-hannes@cmpxchg.org
> 
> For desktop users running without swap, couldn't we just provide a kernel
> setting which marks all executable pages as unevictable when first faulted
> in?

This could result in the immediate DoS vector and you could see trashing
elsewhere. In fact we already do protect executable pages and reclaim
them later (see page_check_references).

I am afraid that the only way to resolve the trashing behavior is to
release a larger amount of memory because shifting the reclaim priority
will just push the suboptimal behavior somewhere else. In order to do
that we really have to detect that the working set doesn't fit into
memory and refaults are predominating system activity.
-- 
Michal Hocko
SUSE Labs

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web