Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #227301 > unrolled thread

OOM-killer not being involked under memory pressure

Back to article view | Back to linux.debian.user

Contents

  OOM-killer not being involked under memory pressure Pariksheet Nanda <pariksheet.nanda@protonmail.com> - 2020-09-24 17:40 +0200
    Re: OOM-killer not being involked under memory pressure Stefan Monnier <monnier@iro.umontreal.ca> - 2020-09-24 20:10 +0200
      Re: OOM-killer not being involked under memory pressure Pariksheet Nanda <pariksheet.nanda@protonmail.com> - 2020-09-25 01:40 +0200
        Re: OOM-killer not being involked under memory pressure Pariksheet Nanda <pariksheet.nanda@protonmail.com> - 2020-09-25 02:00 +0200
          Re: OOM-killer not being involked under memory pressure Pariksheet Nanda <pariksheet.nanda@protonmail.com> - 2020-09-25 02:10 +0200
            Re: OOM-killer not being involked under memory pressure Linux-Fan <Ma_Sys.ma@web.de> - 2020-09-25 03:00 +0200
              Re: OOM-killer not being involked under memory pressure Pariksheet Nanda <pariksheet.nanda@protonmail.com> - 2020-09-25 21:20 +0200
        Re: OOM-killer not being involked under memory pressure Charles Curley <charlescurley@charlescurley.com> - 2020-09-25 15:50 +0200
          Re: OOM-killer not being involked under memory pressure Tixy <tixy@yxit.co.uk> - 2020-09-25 18:30 +0200
          Re: OOM-killer not being involked under memory pressure Brian <ad44@cityscape.co.uk> - 2020-09-25 20:10 +0200
          Re: OOM-killer not being involked under memory pressure David Wright <deblis@lionunicorn.co.uk> - 2020-09-26 04:50 +0200
      Re: OOM-killer not being involked under memory pressure Stefan Monnier <monnier@iro.umontreal.ca> - 2020-09-25 03:50 +0200
        Re: OOM-killer not being involked under memory pressure Pariksheet Nanda <pariksheet.nanda@protonmail.com> - 2020-09-25 21:40 +0200

#227301 — OOM-killer not being involked under memory pressure

[Multipart message — attachments visible in raw view] — view raw

Hello,
First post to this list, so go easy on me!
Every few days, my desktop runs out of RAM, and this usually happens while web browsing. I wait for as much as an hour and a half, but just see the screen as frozen the way it was at the time of the hang. 8 GB RAM is installed in it, and it's running Debian 10. My experience with other Linux distributions has been that the out-of-memory killer works as expected, but I set vm.oom_kill_allocating_task = 1 in sysctl to avoid it - yes, I also save the setting persistently in a file /etc/sysctl.d/60-oom-killer.conf. I'm back to using the default 0 setting. I can reproduce the problem using:
stress -m 4 --vm-bytes 1000M
My persistent journald logging shows me no evidence of memory pressure being registered by the kernel or the oom-killer being involved leading up to the hang, so I'm not sure what to do next.
Pariksheet

[toc] | [next] | [standalone]

#227309

Hi,

> Every few days, my desktop runs out of RAM, and this usually happens while
> web browsing.

What exactly makes you think the problem is that it ran out of RAM?

> I wait for as much as an hour and a half, but just see the
> screen as frozen the way it was at the time of the hang. 8 GB RAM is
> installed in it, and it's running Debian 10.

I do regular email+web on 3GB machines on a regular basis, so if your
machine "swaps to death" despite 8GB of RAM, either you're visiting web
sites which use up a lot more RAM than the ones I visit (that's
definitely possible), or you're hitting a bug that may be unrelated to
the amount of free RAM.

If your problem is due to a lack of free RAM, I'd expect the OOM to do
its job at some point and I'd also expect the machine get slower
somewhat gradually.  So, before hanging, does your machine "feel slower
and slower and slower"?

My crystal ball says that you're not running out of RAM, but you're
hitting a nasty bug instead.  I hope I'm wrong.


        Stefan

[toc] | [prev] | [next] | [standalone]

#227313

Hi Stefan,

>> Every few days, my desktop runs out of RAM, and this usually happens while
>> web browsing.
>
> What exactly makes you think the problem is that it ran out of RAM?
--snip--
> If your problem is due to a lack of free RAM, I'd expect the OOM to do
> its job at some point and I'd also expect the machine get slower
> somewhat gradually. So, before hanging, does your machine "feel slower
> and slower and slower"?

Good question!

The hanging behavior is like a step function: the computer goes from being fully responsive to completely unresponsive; I don't don't see the swap bar in the System Load Monitor increase, which is strange.  I have experienced slow swap behavior before, but usually there I have intermittent control to Ctrl+Q programs and recover out of the slowness at time scales on the order of a few tens of seconds.

Your thought about swap inspired me to check whether my swap partition is functional.  I don't know how to empirically test that swap works and only know to reading /etc/fstab where I do see that I have a swap mount point present.  `apropos swap` led me to check systemd's swap.target which I also see is active, and also see the corresponding swap volume in systemctl.  This resembles my laptop setup where I know the swap partition works there, from the rare occasions where I seeing its swap bar move in the XFCE System Load Monitor panel plugin.


>> I wait for as much as an hour and a half, but just see the
>> screen as frozen the way it was at the time of the hang. 8 GB RAM is
>> installed in it, and it's running Debian 10.
>
> I do regular email+web on 3GB machines on a regular basis, so if your
> machine "swaps to death" despite 8GB of RAM, either you're visiting web
> sites which use up a lot more RAM than the ones I visit (that's
> definitely possible), or you're hitting a bug that may be unrelated to
> the amount of free RAM.

This is a great insight.  I've hit my share of graphics card bugs over the years.  A few years ago I was bitten by an Intel IOMMU related graphical bug on my laptop which I worked around with a kernel parameter tweak.  In earlier years, similar story with nvidia and nouveau drivers.

What's also unusual about this desktop is it's my first attempt using encrypted ZFS on Linux for personal work, and I've noticed that ZFS uses a lot of RAM.  Even when I have no significant applications running in my desktop I suspect it can grow to use as much as half my RAM, but because the ZFS kernel processes register RAM usage as 0 in the process managers like htop, etc, it's usually difficult to directly quantify the effect and I only suspect it's heavy memory use from previous experience using XFCE without ZFS.  I have a setup nearly identical to https://openzfs.github.io/openzfs-docs/Getting%20Started/Debian/Debian%20Buster%20Root%20on%20ZFS.html and am using the latest zfs version 0.8.4-2~bpo10+1.


> My crystal ball says that you're not running out of RAM, but you're
> hitting a nasty bug instead. I hope I'm wrong.

I think it might be prudent to seek help from the ZFS folks to see what they say.  Maybe something is misconfigured with using the ZFS swap partition and there might be some ZFS related logging I could collect to better understand what's going on.

Perhaps there's a ZFS bug or misconfiguration that happens before the OOM-killer has the chance to be involked.  The encryption support is a relatively new feature and I wonder how it behaves with swap.

Thank you for all your thoughtful questions and insights!


> Stefan

Pariksheet

[toc] | [prev] | [next] | [standalone]

#227314

Hi Stefan,

>> My crystal ball says that you're not running out of RAM, but you're
>> hitting a nasty bug instead. I hope I'm wrong.
>
> Perhaps there's a ZFS bug or misconfiguration that happens before the OOM-killer has the chance to be involked. The encryption support is a relatively new feature and I wonder how it behaves with swap.

I just checked my other server which has ZFS on root without encryption, and see that I did not enable swap at all on that machine.  So I'll disable swap, thrash the RAM with `stress`, and then hopefully the OOM-killer works like it does on that machine.


>> Stefan
>
> Pariksheet

Pariksheet

[toc] | [prev] | [next] | [standalone]

#227315

> I just checked my other server which has ZFS on root without encryption, and see that I did not enable swap at all on that machine. So I'll disable swap, thrash the RAM with`stress`, and then hopefully the OOM-killer works like it does on that machine.

Yay!  Indeed disabling swap allowed the OOM-killer to work.

Pariksheet


$ sudo swapoff -av
swapoff /dev/zd0
$ stress -m 4 --vm-bytes 2000M
stress: info: [16295] dispatching hogs: 0 cpu, 0 io, 4 vm, 0 hdd
stress: FAIL: [16295] (415) <-- worker 16299 got signal 9
stress: WARN: [16295] (417) now reaping child worker processes
stress: FAIL: [16295] (451) failed run completed in 2s
$ sudo journalctl -rt kernel | head
-- Logs begin at Tue 2020-06-23 13:23:36 EDT, end at Thu 2020-09-24 20:06:09 EDT. --
Sep 24 20:02:12 xm1 kernel: oom_reaper: reaped process 16838 (stress), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Sep 24 20:02:12 xm1 kernel: Killed process 16838 (stress) total-vm:2051852kB, anon-rss:1345260kB, file-rss:0kB, shmem-rss:0kB
Sep 24 20:02:12 xm1 kernel: Out of memory: Kill process 16838 (stress) score 166 or sacrifice child
Sep 24 20:02:12 xm1 kernel: [  16840]     0 16840     1143       39    53248        0             0 nc
Sep 24 20:02:12 xm1 kernel: [  16839]  1000 16839   512963   323517  2637824        0             0 stress
Sep 24 20:02:12 xm1 kernel: [  16838]  1000 16838   512963   336315  2740224        0             0 stress
Sep 24 20:02:12 xm1 kernel: [  16837]  1000 16837   512963   308763  2523136        0             0 stress
Sep 24 20:02:12 xm1 kernel: [  16836]  1000 16836   512963   313283  2555904        0             0 stress
Sep 24 20:02:12 xm1 kernel: [  16835]  1000 16835      962       20    40960        0             0 stress

[toc] | [prev] | [next] | [standalone]

#227316

[Multipart message — attachments visible in raw view] — view raw

Pariksheet Nanda writes:

> > I just checked my other server which has ZFS on root without encryption,
> and see that I did not enable swap at all on that machine. So I'll disable
> swap, thrash the RAM with`stress`, and then hopefully the OOM-killer works
> like it does on that machine.
>
> Yay!  Indeed disabling swap allowed the OOM-killer to work.
>
> Pariksheet

Hello,

glad you could solve the problem and thanks for sharing the solution.

I am planning on using ZFS, too, so would you mind a follow-up question
about your setup: Is your swap on a ZFS volume?

I have heard there are bugs with Swap-on-ZFS which may cause lockups similar
to what you describe:

https://github.com/openzfs/zfs/issues/342
https://github.com/openzfs/zfs/issues/7734

Thanks in advance
Linux-Fan

[...]

[toc] | [prev] | [next] | [standalone]

#227366

Hi Linux-Fan,
>>> I just checked my other server which has ZFS on root without encryption,
>>> and see that I did not enable swap at all on that machine. So I'll disable
>>> swap, thrash the RAM with `stress`, and then hopefully the OOM-killer works
>>> like it does on that machine.
>>
>> Yay! Indeed disabling swap allowed the OOM-killer to work.
>
> I am planning on using ZFS, too, so would you mind a follow-up question
> about your setup: Is your swap on a ZFS volume?

Yes, the swap was on the ZFS volume, from my following the ZFS on Linux guide linked earlier.


> I have heard there are bugs with Swap-on-ZFS which may cause lockups similar
> to what you describe:
>
> https://github.com/openzfs/zfs/issues/342
> https://github.com/openzfs/zfs/issues/7734

Thank you for the links!  I started reading the zfs-discuss mailing list and completely neglected the GitHub tracker.


>>> Pariksheet
>> Pariksheet
> Linux-Fan
Pariksheet

[toc] | [prev] | [next] | [standalone]

#227344

On Thu, 24 Sep 2020 23:37:52 +0000
Pariksheet Nanda <pariksheet.nanda@protonmail.com> wrote:

> I don't know how to empirically test that swap works

Try free. E.g.:

root@jhegaala:~# free
              total        used        free      shared  buff/cache   available
Mem:           7860        5095         780         597        1985        1958
Swap:         15623        2426       13197
Total:        23484        7521       13977
root@jhegaala:~# 

If swap is not enabled, the Swap line won't be there.

Or try htop.

-- 
Does anybody read signatures any more?

https://charlescurley.com
https://charlescurley.com/blog/

[toc] | [prev] | [next] | [standalone]

#227358

On Fri, 2020-09-25 at 07:41 -0600, Charles Curley wrote:
> On Thu, 24 Sep 2020 23:37:52 +0000
> Pariksheet Nanda <pariksheet.nanda@protonmail.com> wrote:
> 
> > I don't know how to empirically test that swap works
> 
> Try free. E.g.:
> 
> root@jhegaala:~# free
>               total        used        free      shared  buff/cache   available
> Mem:           7860        5095         780         597        1985        1958
> Swap:         15623        2426       13197
> Total:        23484        7521       13977
> root@jhegaala:~# 
> 
> If swap is not enabled, the Swap line won't be there.

Or show zero for the total size, that's what it does for me. (I'm not
the OP, just commenting).

-- 
Tixy

[toc] | [prev] | [next] | [standalone]

#227364

On Fri 25 Sep 2020 at 07:41:10 -0600, Charles Curley wrote:

> On Thu, 24 Sep 2020 23:37:52 +0000
> Pariksheet Nanda <pariksheet.nanda@protonmail.com> wrote:
> 
> > I don't know how to empirically test that swap works
> 
> Try free. E.g.:
> 
> root@jhegaala:~# free
>               total        used        free      shared  buff/cache   available
> Mem:           7860        5095         780         597        1985        1958
> Swap:         15623        2426       13197
> Total:        23484        7521       13977
> root@jhegaala:~# 
> 
> If swap is not enabled, the Swap line won't be there.

Are you sure?

brian@5730cups:~$ free
              total        used        free      shared  buff/cache   available                                           
Mem:         928100       48764      790348        6520       88988      762644                                           
Swap:             0           0           0

-- 
Brian.

[toc] | [prev] | [next] | [standalone]

#227381

On Fri 25 Sep 2020 at 07:41:10 (-0600), Charles Curley wrote:
> On Thu, 24 Sep 2020 23:37:52 +0000 Pariksheet Nanda <pariksheet.nanda@protonmail.com> wrote:
> 
> > I don't know how to empirically test that swap works
> 
> Try free. E.g.:
> 
> root@jhegaala:~# free
>               total        used        free      shared  buff/cache   available
> Mem:           7860        5095         780         597        1985        1958
> Swap:         15623        2426       13197
> Total:        23484        7521       13977
> root@jhegaala:~# 
> 
> If swap is not enabled, the Swap line won't be there.
> 
> Or try htop.

$ /sbin/swapon --noheadings --show=name,type,size
/dev/dm-0 partition 499M

Cheers,
David.

[toc] | [prev] | [next] | [standalone]

#227317

> I neglected to mention that I run XFCE with its System Load Monitor panel
> plugin having a RAM indicator bar that climbs and maxes out right before the
> hang.  RAM is something I often need to monitor because I run scientific
> computing programs which can also exhaust my RAM if I'm not careful.

OK, the climb you describe does justify your suspicion that it's related
to RAM usage.  It's not a proof, but it's a strong indication.

> The hanging behavior is like a step function: the computer goes from being
> fully responsive to completely unresponsive;

That's very much *unlike* a normal "out of RAM" situation, OTOH.
Normally what happens is that the OS starts to shuffle things around
(throwing out cached data, moving other to swap, etc...) making the
machine slower and slower.

The step function sounds much more like a bug such as a deadlock.

> I don't don't see the swap bar in the System Load Monitor increase,
> which is strange.  I have experienced slow swap behavior before, but
> usually there I have intermittent control to Ctrl+Q programs and
> recover out of the slowness at time scales on the order of a few tens
> of seconds.

That's what happens for "normal out of RAM" situations, indeed.

> Your thought about swap inspired me to check whether my swap partition is
> functional.  I don't know how to empirically test that swap works and only
> know to reading /etc/fstab where I do see that I have a swap mount point
> present.  `apropos swap` led me to check systemd's swap.target which I also
> see is active, and also see the corresponding swap volume in systemctl.
> This resembles my laptop setup where I know the swap partition works there,
> from the rare occasions where I seeing its swap bar move in the XFCE System
> Load Monitor panel plugin.

You could run a `memtester` process and tell it to test, say 6GB, so you
the kernel only has 2GB left to play with and it will be forced to push
stuff to swap, which you should then see in the output of `free`.

But I suspect in your case the details of *how* you get into the "out of
RAM" are relevant.  

> This is a great insight.  I've hit my share of graphics card bugs over the
> years.  A few years ago I was bitten by an Intel IOMMU related graphical bug
> on my laptop which I worked around with a kernel parameter tweak.
> In earlier years, similar story with nvidia and nouveau drivers.

Indeed, graphics card bugs often display the step function, because if
the rest of the system may keep working (at least for a while), you
can't really "see" it (unless you manage to connect into the machine
via the network).

> What's also unusual about this desktop is it's my first attempt using ZFS on

I have no experience or even much knowledge about ZFS, sorry.
You might want to try and set that same machine up with an ext4
filesystem instead temporarily to see if you can reproduce the problem
even without the use of ZFS (depending on how ZFS is used and your disk
setup, it might be possible to do it easily, without having to
reinstall (which could result in a sufficiently different system that
it'd then be hard to convince oneself that the only difference is
ZFS-vs-ext4)).


        Stefan

[toc] | [prev] | [next] | [standalone]

#227367

>> The hanging behavior is like a step function: the computer goes from being
>> fully responsive to completely unresponsive;
>
> That's very much unlike a normal "out of RAM" situation, OTOH.
> Normally what happens is that the OS starts to shuffle things around
> (throwing out cached data, moving other to swap, etc...) making the
> machine slower and slower.
>
> The step function sounds much more like a bug such as a deadlock.

Thank you for clearing up my misunderstanding of the observed behavior!  Indeed, Linux-Fan linked to the swap deadlock issue with ZFS.


> You could run a`memtester` process and tell it to test, say 6GB, so you
> the kernel only has 2GB left to play with and it will be forced to push
> stuff to swap, which you should then see in the output of `free`.

Thank you to you, Charles, and Tixy, for the suggestion to use `free`; I had only previously used it to check RAM usage and did not know it's usefulness to also check swap.  As Tixy mentions, `free` does show zeros now that I have swap disabled:

$ free
              total        used        free      shared  buff/cache   available
Mem:        8100708     4136324     2972808      197340      991576     3597312
Swap:             0           0           0


> You might want to try and set that same machine up with an ext4
> filesystem instead temporarily to see if you can reproduce the problem
> even without the use of ZFS (depending on how ZFS is used and your disk
> setup, it might be possible to do it easily, without having to
> reinstall (which could result in a sufficiently different system that
> it'd then be hard to convince oneself that the only difference is
> ZFS-vs-ext4)).

I actually was using ext4 for a few years on this machine before switching to ZFS in June and it behaved well, right up to the point where it didn't - I have had a few long power outages in my rural area, and running LUKS on LVM with ext4 after 1 particular power outage made it irrecoverable with my abilities.  Which is why I switched to using encrypted ZFS.  Though I in no way wish to suggest encrypted ZFS is more reliable than LUKS on LVM with ext4; minus the encryption, my colleagues have had fewer instances of disk corription with ZFS and so I'm experimenting with it.


> Stefan

Pariksheet

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web