Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #246707 > unrolled thread

Out of memory killer misconfigured?

Started bypiorunz <piorunz@gmx.com>
First post2022-03-29 11:40 +0200
Last post2022-04-20 12:30 +0200
Articles 6 on this page of 26 — 9 participants

Back to article view | Back to linux.debian.user

Contents

  Out of memory killer misconfigured? piorunz <piorunz@gmx.com> - 2022-03-29 11:40 +0200
    Re: Out of memory killer misconfigured? Sven Hoexter <sven@stormbind.net> - 2022-03-29 12:10 +0200
      Re: Out of memory killer misconfigured? piorunz <piorunz@gmx.com> - 2022-03-29 20:40 +0200
        Re: Out of memory killer misconfigured? Tixy <tixy@yxit.co.uk> - 2022-03-30 10:20 +0200
          Re: Out of memory killer misconfigured? piorunz <piorunz@gmx.com> - 2022-03-31 19:00 +0200
            Re: Out of memory killer misconfigured? Jonathan Dowland <jon+debian-user@dow.land> - 2022-04-20 12:30 +0200
              Re: Out of memory killer misconfigured? piorunz <piorunz@gmx.com> - 2022-04-20 17:30 +0200
                Re: Out of memory killer misconfigured? Jonathan Dowland <jon+debian-user@dow.land> - 2022-04-20 18:20 +0200
                  Re: Out of memory killer misconfigured? piorunz <piorunz@gmx.com> - 2022-04-20 19:50 +0200
                    Re: Out of memory killer misconfigured? David Wright <deblis@lionunicorn.co.uk> - 2022-04-20 20:40 +0200
      Re: Out of memory killer misconfigured? piorunz <piorunz@gmx.com> - 2022-03-29 20:50 +0200
        Re: Out of memory killer misconfigured? Greg Wooledge <greg@wooledge.org> - 2022-03-29 21:20 +0200
          Re: Out of memory killer misconfigured? piorunz <piorunz@gmx.com> - 2022-03-29 22:10 +0200
            Re: Out of memory killer misconfigured? Greg Wooledge <greg@wooledge.org> - 2022-03-29 22:20 +0200
      Re: Out of memory killer misconfigured? piorunz <piorunz@gmx.com> - 2022-03-29 21:00 +0200
        Re: Out of memory killer misconfigured? Nicholas Geovanis <nickgeovanis@gmail.com> - 2022-03-29 21:20 +0200
        Re: Out of memory killer misconfigured? <tomas@tuxteam.de> - 2022-04-01 08:10 +0200
          Re: Out of memory killer misconfigured? piorunz <piorunz@gmx.com> - 2022-04-15 03:10 +0200
            Re: Out of memory killer misconfigured? <tomas@tuxteam.de> - 2022-04-15 08:00 +0200
              Re: Out of memory killer misconfigured? piorunz <piorunz@gmx.com> - 2022-04-15 12:10 +0200
                Re: Out of memory killer misconfigured? <tomas@tuxteam.de> - 2022-04-15 12:20 +0200
                  Re: Out of memory killer misconfigured? piorunz <piorunz@gmx.com> - 2022-04-18 22:10 +0200
                    Re: Out of memory killer misconfigured? Tim Woodall <debianuser@woodall.me.uk> - 2022-04-19 17:50 +0200
                      Re: Out of memory killer misconfigured? <tomas@tuxteam.de> - 2022-04-19 18:10 +0200
                        Re: Out of memory killer misconfigured? Nicholas Geovanis <nickgeovanis@gmail.com> - 2022-04-19 22:10 +0200
    Re: Out of memory killer misconfigured? Jonathan Dowland <jon+debian-user@dow.land> - 2022-04-20 12:30 +0200

Page 2 of 2 — ← Prev page 1 [2]

#247232

From<tomas@tuxteam.de>
Date2022-04-15 12:20 +0200
Message-ID<Ecrdv-8T4E-7@gated-at.bofh.it>
In reply to#247231

[Multipart message — attachments visible in raw view] — view raw

On Fri, Apr 15, 2022 at 11:03:07AM +0100, piorunz wrote:
> On 15/04/2022 06:53, tomas@tuxteam.de wrote:
> > If you want to learn more about that, the Linux MM ("memory
> > management") people have set up a wiki for that:
> > 
> >    https://linux-mm.org/OOM_Killer
> > 
> > Enjoy:)
> > 
> > (and yes, on my box, more memory-hungry processes have a higher
> > /proc/<pid>/oom_score, so they seem to incur a higher risk of
> > being killed. The browser lies somewhat because it spawns quite
> > a few processes, but as a product of the propaganda industry,
> > lying is its second nature;-)
> 
> Yes I understand that from my point of view everything is bad and I am
> not objective.
> However, I see how things are broken and that is indeed obvious.
> I seen before my own eyes how Linux killed KDE session including Xorg
> but left 10x wine's exe processes with 8GB RAM each as a last. Entire
> tree with parent process was using all of the memory that would be 50+
> GB. How this process tree is not having HIGHEST POSSIBLE OOM score and
> is not being killed in first microsecond of OOM situation is beyond my
> understanding.

- At the "low level": you are talking about "process trees".
 Perhaps that's the problem. Perhaps, towards the OOM score,
 all those processes count as individual processes. Note that
 the OOM killer is just the last straw the system grabs to
 try to avoid sinking. Perhaps it's the wrong instrument for
 your case and you need to look into limits, or cgroup memory
 policies.

- At the "higher level": "you see how things are broken" for
 you. You are still (in my opinion) committing the error of
 generalising your problem /before/ you have spent enough
 time investigating it. That might make it more difficult
 finding a satisfying solution :-) Plus, don't assume random
 kernel programmers aren't smart. They usually are. And they
 usually spend quite a bit of effort until a patch of this
 kind goes in.

Cheers
-- 
t

[toc] | [prev] | [next] | [standalone]

#247346

Frompiorunz <piorunz@gmx.com>
Date2022-04-18 22:10 +0200
Message-ID<EdFR7-9Dbo-1@gated-at.bofh.it>
In reply to#247232
On 15/04/2022 11:14, tomas@tuxteam.de wrote:

Thanks for your reply, I appreciate your patience.

> - At the "low level": you are talking about "process trees".
>   Perhaps that's the problem. Perhaps, towards the OOM score,
>   all those processes count as individual processes.

Each 8GB individual process should be killed absolutely first, not
entire KDE desktop and Xorg. That's obvious.

> Note that
>   the OOM killer is just the last straw the system grabs to
>   try to avoid sinking. Perhaps it's the wrong instrument for
>   your case and you need to look into limits, or cgroup memory
>   policies.

I look from desktop perspective. OS (Linux) runs my desktop and manage
all programs. When one programs eats too much memory, program gets
killed. That is default behaviour or any mature operating system, even
in Windows 2000 era we had this. I don't understand why Linux, using all
its tools available to itself, is not handling this. I highlight tools
available to it, not to me - I should not need to know which internal
Linux tool kills misbehaving programs and how it works to use my desktop.

> - At the "higher level": "you see how things are broken" for
>   you. You are still (in my opinion) committing the error of
>   generalising your problem /before/ you have spent enough
>   time investigating it. That might make it more difficult
>   finding a satisfying solution :-)

Yes, I look absolutely from higher, desktop level. Current defaults are
broken. OOM is not working, and it's killing entire system instead of
one offending application. That's a bug.

> Plus, don't assume random
>   kernel programmers aren't smart. They usually are. And they
>   usually spend quite a bit of effort until a patch of this
>   kind goes in.

Of course they are very smart. However, in this case, and in hundreds of
similar cases which are being reported as bugs every year, defaults, or
other mechanisms, needs to be adjusted so Denial of Service is not
happening when application takes all memory. Right now, that is exactly
what's happening. System-wide DoS in one second, entire system nuked and
halted when one Wine application claims too much memory.

>
> Cheers


--
With kindest regards, Piotr.

⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org/
⠈⠳⣄⠀⠀⠀⠀

[toc] | [prev] | [next] | [standalone]

#247358

FromTim Woodall <debianuser@woodall.me.uk>
Date2022-04-19 17:50 +0200
Message-ID<EdYh3-9Ogb-3@gated-at.bofh.it>
In reply to#247346
On Mon, 18 Apr 2022, piorunz wrote:

>
> I look from desktop perspective. OS (Linux) runs my desktop and manage
> all programs. When one programs eats too much memory, program gets
> killed. That is default behaviour or any mature operating system, even
> in Windows 2000 era we had this. I don't understand why Linux, using all
> its tools available to itself, is not handling this. I highlight tools
> available to it, not to me - I should not need to know which internal
> Linux tool kills misbehaving programs and how it works to use my desktop.
>

Because not every machine that has the linux kernel installed runs a
desktop - or even if it does, not everybody who kicks off a long running
process in the background wants that process to be killed if the system
starts getting memory starved.

There is no one right answer to this. If the "right way" is obvious to
you in your use case then you can patch the kernel or configure the oom
killer.

https://www.oracle.com/in/technical-resources/articles/it-infrastructure/dev-oom-killer.html

I've not read that page in detail but to summarize "if the kernel is
going to kill our one critical process then we might as well reboot
anyway" and explains how to make the critical process less likely to be
killed. You could do the same with your desktop environment.

Tim.

[toc] | [prev] | [next] | [standalone]

#247361

From<tomas@tuxteam.de>
Date2022-04-19 18:10 +0200
Message-ID<EdYAp-9OC8-7@gated-at.bofh.it>
In reply to#247358

[Multipart message — attachments visible in raw view] — view raw

On Tue, Apr 19, 2022 at 04:44:36PM +0100, Tim Woodall wrote:
> On Mon, 18 Apr 2022, piorunz wrote:
> 
> > 
> > I look from desktop perspective. OS (Linux) runs my desktop and manage
> > all programs [...]

> Because not every machine that has the linux kernel installed runs a
> desktop [...]

As I already said: I think the OOM killer is the wrong tool for this
job. Once that fires, all bets are up. Its job is to give the sys
admin/system a chance to shut down cleanly, not much more.

Some resource manager (ulimits, control groups [1], what have you)
seems more appropriate. It's up to the desktop environment folks
or to the sysadmin to set them up properly, of course.

As for why the OOM killer is not triggering the was piorunz expects,
no idea. The scores and the knobs to regulate them are well-known...

@piorunz: you could start the browser with a worse score so it
gets killed earlier if that suits better your use case.

Cheers
-- 
t

[toc] | [prev] | [next] | [standalone]

#247367

FromNicholas Geovanis <nickgeovanis@gmail.com>
Date2022-04-19 22:10 +0200
Message-ID<Ee2kF-9QRN-5@gated-at.bofh.it>
In reply to#247361

[Multipart message — attachments visible in raw view] — view raw

On Tue, Apr 19, 2022, 11:08 AM <tomas@tuxteam.de> wrote:

> On Tue, Apr 19, 2022 at 04:44:36PM +0100, Tim Woodall wrote:
> > On Mon, 18 Apr 2022, piorunz wrote:
> >
> > >
> > > I look from desktop perspective. OS (Linux) runs my desktop and manage
> > > all programs [...]
>
> > Because not every machine that has the linux kernel installed runs a
> > desktop [...]
>
> As I already said: I think the OOM killer is the wrong tool for this
> job. Once that fires, all bets are up. Its job is to give the sys
> admin/system a chance to shut down cleanly, not much more.
>

I had not heard this before but...

cgroup awareness of OOM killer
Linux Kernel 4.19 (October 2018) introduced cgroup awareness of OOM killer
implementation which adds an ability to kill a cgroup as a single unit and
so guarantee the integrity of the workload.

That's from
https://en.m.wikipedia.org/wiki/Cgroups

Of course you still need to identify and configure your cgroups
effectively. It seems overdue to make the two tools cooperate but as others
pointed out, they have different origins. And it's another one of those
things handled differently in the data center from the desktop.

Some resource manager (ulimits, control groups [1], what have you)
> seems more appropriate. It's up to the desktop environment folks
> or to the sysadmin to set them up properly, of course.
>
> As for why the OOM killer is not triggering the was piorunz expects,
> no idea. The scores and the knobs to regulate them are well-known...
>
> @piorunz: you could start the browser with a worse score so it
> gets killed earlier if that suits better your use case.
>
> Cheers
> --
> t
>

[toc] | [prev] | [next] | [standalone]

#247373

FromJonathan Dowland <jon+debian-user@dow.land>
Date2022-04-20 12:30 +0200
Message-ID<EefKV-9YMZ-1@gated-at.bofh.it>
In reply to#246707
On Tue, Mar 29, 2022 at 10:34:19AM +0100, piorunz wrote:
>Instead of killing ONE 7.5 million-worth pagetable process, Linux is
>killing everything else! KDE activity manager killed. Then it goes on to
>kill EVERYTHING in the system:

Perhaps your wine processes had their oom_adj (etc) values tweaked to
make the algorithm not consider them (/proc/<PID>/oom_{adj,score,score_adj})?
I can imagine some ill-advised launcher script for windows software on
linux doing this.

-- 
Please do not CC me for listmail.

👱🏻	Jonathan Dowland
✎	 jmtd@debian.org
🔗	https://jmtd.net

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | linux.debian.user

csiph-web