Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1400594 > unrolled thread

Re: [PATCH] mm: add config option to select the initial overcommit mode

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH] mm: add config option to select the initial overcommit  mode Michal Hocko <mhocko@kernel.org> - 2016-05-13 10:10 +0200
    Re: [PATCH] mm: add config option to select the initial overcommit  mode Mason <slash.tmp@free.fr> - 2016-05-13 10:50 +0200
      Re: [PATCH] mm: add config option to select the initial overcommit  mode Michal Hocko <mhocko@kernel.org> - 2016-05-13 12:00 +0200
        Re: [PATCH] mm: add config option to select the initial overcommit  mode Mason <slash.tmp@free.fr> - 2016-05-13 12:20 +0200
          Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-13 12:50 +0200
          Re: [PATCH] mm: add config option to select the initial overcommit  mode Michal Hocko <mhocko@kernel.org> - 2016-05-13 13:50 +0200
            Re: [PATCH] mm: add config option to select the initial overcommit  mode Mason <slash.tmp@free.fr> - 2016-05-13 14:20 +0200
              Re: [PATCH] mm: add config option to select the initial overcommit  mode Michal Hocko <mhocko@kernel.org> - 2016-05-13 16:10 +0200
                Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-13 16:20 +0200
                Re: [PATCH] mm: add config option to select the initial overcommit  mode One Thousand Gnomes <gnomes@lxorguk.ukuu.org.uk> - 2016-05-13 17:10 +0200
                  Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-13 17:40 +0200
                    Re: [PATCH] mm: add config option to select the initial overcommit  mode One Thousand Gnomes <gnomes@lxorguk.ukuu.org.uk> - 2016-05-13 17:50 +0200
                      Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-17 10:30 +0200
                        Re: [PATCH] mm: add config option to select the initial overcommit  mode Michal Hocko <mhocko@kernel.org> - 2016-05-17 11:00 +0200
                          Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-17 18:20 +0200
                            Re: [PATCH] mm: add config option to select the initial overcommit  mode "Austin S. Hemmelgarn" <ahferroin7@gmail.com> - 2016-05-17 19:30 +0200
                              Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-18 17:20 +0200
                                Re: [PATCH] mm: add config option to select the initial overcommit  mode "Austin S. Hemmelgarn" <ahferroin7@gmail.com> - 2016-05-18 18:30 +0200
                            Re: [PATCH] mm: add config option to select the initial overcommit  mode Michal Hocko <mhocko@kernel.org> - 2016-05-17 22:20 +0200
                              Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-18 17:20 +0200
                                Re: [PATCH] mm: add config option to select the initial overcommit  mode Michal Hocko <mhocko@kernel.org> - 2016-05-19 09:20 +0200
                    Re: [PATCH] mm: add config option to select the initial overcommit  mode "Austin S. Hemmelgarn" <ahferroin7@gmail.com> - 2016-05-13 19:10 +0200
          Re: [PATCH] mm: add config option to select the initial overcommit  mode "Austin S. Hemmelgarn" <ahferroin7@gmail.com> - 2016-05-13 15:30 +0200
      Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-13 12:00 +0200
        Re: [PATCH] mm: add config option to select the initial overcommit  mode Michal Hocko <mhocko@kernel.org> - 2016-05-13 14:10 +0200
          Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-13 14:40 +0200
            Re: [PATCH] mm: add config option to select the initial overcommit  mode "Austin S. Hemmelgarn" <ahferroin7@gmail.com> - 2016-05-13 15:20 +0200
              Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-13 15:40 +0200
                Re: [PATCH] mm: add config option to select the initial overcommit  mode "Austin S. Hemmelgarn" <ahferroin7@gmail.com> - 2016-05-13 16:20 +0200
                  Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-13 16:30 +0200
                    Re: [PATCH] mm: add config option to select the initial overcommit  mode "Austin S. Hemmelgarn" <ahferroin7@gmail.com> - 2016-05-13 17:10 +0200
                Re: [PATCH] mm: add config option to select the initial overcommit  mode One Thousand Gnomes <gnomes@lxorguk.ukuu.org.uk> - 2016-05-13 17:10 +0200
                  Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-13 17:20 +0200
                    Re: [PATCH] mm: add config option to select the initial overcommit  mode Michal Hocko <mhocko@kernel.org> - 2016-05-13 17:30 +0200
              Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-13 15:40 +0200
                Re: [PATCH] mm: add config option to select the initial overcommit  mode "Austin S. Hemmelgarn" <ahferroin7@gmail.com> - 2016-05-13 16:00 +0200
                  Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-13 16:40 +0200
                    Re: [PATCH] mm: add config option to select the initial overcommit  mode Michal Hocko <mhocko@kernel.org> - 2016-05-13 17:00 +0200
                    Re: [PATCH] mm: add config option to select the initial overcommit  mode "Austin S. Hemmelgarn" <ahferroin7@gmail.com> - 2016-05-13 17:20 +0200
            Re: [PATCH] mm: add config option to select the initial overcommit  mode Michal Hocko <mhocko@kernel.org> - 2016-05-13 17:00 +0200
              Re: [PATCH] mm: add config option to select the initial overcommit  mode Mason <slash.tmp@free.fr> - 2016-05-13 17:00 +0200
                Re: [PATCH] mm: add config option to select the initial overcommit  mode One Thousand Gnomes <gnomes@lxorguk.ukuu.org.uk> - 2016-05-13 17:20 +0200
                  Re: [PATCH] mm: add config option to select the initial overcommit  mode Michal Hocko <mhocko@kernel.org> - 2016-05-13 17:30 +0200
                  Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-13 17:40 +0200
              Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-13 17:20 +0200
                Re: [PATCH] mm: add config option to select the initial overcommit  mode One Thousand Gnomes <gnomes@lxorguk.ukuu.org.uk> - 2016-05-13 17:50 +0200
                  Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-23 15:20 +0200

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

#1403472

On Wed 18-05-16 17:18:45, Sebastian Frias wrote:
> Hi Michal,
> 
> On 05/17/2016 10:16 PM, Michal Hocko wrote:
> > On Tue 17-05-16 18:16:58, Sebastian Frias wrote:
[...]
> > The global OOM means there is _no_ memory at all. Many kernel
> > operations will need some memory to do something useful. Let's say you
> > would want to do an educated guess about who to kill - most proc APIs
> > will need to allocate. And this is just a beginning. Things are getting
> > really nasty when you get deeper and deeper. E.g. the OOM killer has to
> > give the oom victim access to memory reserves so that the task can exit
> > because that path needs to allocate as well. 
> 
> Really? I would have thought that once that SIGKILL is sent, the
> victim process is not expected to do anything else and thus its
> memory could be claimed immediately.  Or the OOM-killer is more of a
> OOM-terminator? (i.e.: sends SIGTERM)

Well, the path to exit is not exactly trivial. Resources have to be
released and that requires memory sometimes. E.g. exit_robust_list
needs to access the futex and that in turn means a page fault if the
memory was swapped out...
 
> >So even if you wanted to
> > give userspace some chance to resolve the OOM situation you would either
> > need some special API to tell "this process is really special and it can
> > access memory reserves and it has an absolute priority etc." or have a
> > in kernel fallback to do something or your system could lockup really
> > easily.
> > 
> 
> I see, so basically at least two cgroups would be needed, one reserved
> for handling the OOM situation through some API and another for the
> "rest of the system".  Basically just like the 5% reserved for 'root'
> on filesystems.

If you want to handle memcg OOM then you can use memory.oom_control (see
Documentation/cgroup-v1/memory.txt for more information) and have the
oom handler outside of that memcg.

> Do you think that would work?

But handling the _global_ oom from userspace is just insane with the
current kernel implementation. It just cannot work reliably.
-- 
Michal Hocko
SUSE Labs

[toc] | [prev] | [next] | [standalone]

#1400906

On 2016-05-13 11:37, Sebastian Frias wrote:
> Hi Alan,
>
> On 05/13/2016 05:04 PM, One Thousand Gnomes wrote:
>>>> Perhaps Sebastian's choice could be made to depend on CONFIG_EMBEDDED,
>>>> rather than CONFIG_EXPERT?
>>>
>>> Even if the overcommit behavior is different on those systems the
>>> primary question hasn't been answered yet. Why cannot this be done from
>>> the userspace? In other words what wouldn't work properly?
>>
>> Most allocations in C have no mechanism to report failure.
>>
>> Stakc expansion failure is not reportable. Copy on write failure is not
>> reportable and so on.
>
> But wouldn't those affect a given process at at time?
> Does that means that the OOM-killer is woken up to kill process X when those situations arise on process Y?
Barring memory cgroups, if you have hit an OOM condition, it impacts the 
entire system.  Some process other than the one which first hit the 
failure may get killed, but every process will fail allocations until 
the situation is resolved.

[toc] | [prev] | [next] | [standalone]

#1400793

On 2016-05-13 06:18, Mason wrote:
> On 13/05/2016 11:52, Michal Hocko wrote:
>> On Fri 13-05-16 10:44:30, Mason wrote:
>>> On 13/05/2016 10:04, Michal Hocko wrote:
>>>
>>>> On Tue 10-05-16 13:56:30, Sebastian Frias wrote:
>>>> [...]
>>>>> NOTE: I understand that the overcommit mode can be changed dynamically thru
>>>>> sysctl, but on embedded systems, where we know in advance that overcommit
>>>>> will be disabled, there's no reason to postpone such setting.
>>>>
>>>> To be honest I am not particularly happy about yet another config
>>>> option. At least not without a strong reason (the one above doesn't
>>>> sound that way). The config space is really large already.
>>>> So why a later initialization matters at all? Early userspace shouldn't
>>>> consume too much address space to blow up later, no?
>>>
>>> One thing I'm not quite clear on is: why was the default set
>>> to over-commit on?
>>
>> Because many applications simply rely on large and sparsely used address
>> space, I guess.
>
> What kind of applications are we talking about here?
>
> Server apps? Client apps? Supercomputer apps?
>
> I heard some HPC software use large sparse matrices, but is it a common
> idiom to request large allocations, only to use a fraction of it?
Just looking at my laptop right now, I count the number of processes 
which have a RSS which is more than 25% of their allocated memory to be 
about 15-20 out of ~170 processes and ~360 threads.  Somewhat 
unsurprisingly, most of the ones that fit this are highly purpose 
specific (cachefilesd, syslogd, etc), and the only ones whose RSS is 
within 1% of their allocated memory are BOINC applications (distributed 
and/or scientific computing apps tend to be really good about efficient 
usage of memory, even when they use sparse matrices).  There are in fact 
a lot of 'normal' daemons that do this (sshd on my system for example 
has 460k resident and 28.5M allocated, atd has 122k resident and 12.6M 
allocated, acpid has 120k resident and 4.2M allocated).

[toc] | [prev] | [next] | [standalone]

#1400660

Hi,

On 05/13/2016 10:44 AM, Mason wrote:
> On 13/05/2016 10:04, Michal Hocko wrote:
> 
>> On Tue 10-05-16 13:56:30, Sebastian Frias wrote:
>> [...]
>>> NOTE: I understand that the overcommit mode can be changed dynamically thru
>>> sysctl, but on embedded systems, where we know in advance that overcommit
>>> will be disabled, there's no reason to postpone such setting.
>>
>> To be honest I am not particularly happy about yet another config
>> option. At least not without a strong reason (the one above doesn't
>> sound that way). The config space is really large already.
>> So why a later initialization matters at all? Early userspace shouldn't
>> consume too much address space to blow up later, no?

By the way, do you know what's the rationale to allow this setting to be controlled by the userspace dynamically?
Was it for testing only?

> 
> One thing I'm not quite clear on is: why was the default set
> to over-commit on?

Indeed, I was hoping we could throw some light into that.
My patch had another note:

   "NOTE2: I tried to track down the history of overcommit but back then there
were no single patches apparently and the patch that appears to have
introduced the first overcommit mode (OVERCOMMIT_ALWAYS) is commit
9334eab8a36f ("Import 2.1.27"). OVERCOMMIT_NEVER was introduced with commit
502bff0685b2 ("[PATCH] strict overcommit").
My understanding is that prior to commit 9334eab8a36f ("Import 2.1.27")
there was no overcommit, is that correct?"

It'd be nice to know more about why was overcommit introduced.
Furthermore, it looks like allowing overcommit and the introduction of the OOM-killer has given rise to lots of other options to try to tame the OOM-killer.
Without context, that may seem like a form of "feature creep" around it.
Moreover, it makes Linux behave differently from let's say Solaris.

   https://www.win.tue.nl/~aeb/linux/lk/lk-9.html#ss9.6

Hopefully this discussion could clear some of this up and maybe result in more documentation around this subject.

> 
> I suppose the biggest use-case is when a "large" process forks
> only to exec microseconds later into a "small" process, it would
> be silly to refuse that fork. But isn't that what the COW
> optimization addresses, without the need for over-commit?
> 
> Another issue with overcommit=on is that some programmers seem
> to take for granted that "allocations will never fail" and so
> neglect to handle malloc == NULL conditions gracefully.
> 
> I tried to run LTP with overcommit off, and I vaguely recall that
> I had more failures than with overcommit on. (Perhaps only those
> tests that tickle the dreaded OOM assassin.)

From what I remember, one of the LTP maintainers said that it is highly unlikely people test (or run LTP for that matter) with different settings for overcommit.

Years ago, while using MacOS X, a long running process apparently took all the memory over night.
The next day when I checked the computer I saw a dialog that said something like (I don't remember the exact wording) "process X has been paused due to lack of memory (or is requesting too much memory, I don't remember). If you think this is not normal you can terminate process X, otherwise you can terminate other processes to free memory and unpause process X to continue" and then some options to proceed.

If left unattended (thus the dialog unanswered), the computer would still work, all other processes were left intact and only the "offending" process was paused.
Arguably, if the "offending" process is just left paused, it takes the memory away from other processes, and if it was a server, maybe it wouldn't have enough memory to reply to requests.
On the server world I can thus understand that some setting could indicate that when the situation arises, the "dialog" is automatically dismissed with some default action, like "terminate the offending process".

To me it seems really strange for the "OOM-killer" to exist.
It has happened to me that it kills my terminals or editors, how can people deal with random processes being killed?
Doesn't it bother anybody?

Best regards,

Sebastian

[toc] | [prev] | [next] | [standalone]

#1400744

On Fri 13-05-16 11:52:30, Sebastian Frias wrote:
> Hi,
> 
> On 05/13/2016 10:44 AM, Mason wrote:
> > On 13/05/2016 10:04, Michal Hocko wrote:
> > 
> >> On Tue 10-05-16 13:56:30, Sebastian Frias wrote:
> >> [...]
> >>> NOTE: I understand that the overcommit mode can be changed dynamically thru
> >>> sysctl, but on embedded systems, where we know in advance that overcommit
> >>> will be disabled, there's no reason to postpone such setting.
> >>
> >> To be honest I am not particularly happy about yet another config
> >> option. At least not without a strong reason (the one above doesn't
> >> sound that way). The config space is really large already.
> >> So why a later initialization matters at all? Early userspace shouldn't
> >> consume too much address space to blow up later, no?
> 
> By the way, do you know what's the rationale to allow this setting to
> be controlled by the userspace dynamically?  Was it for testing only?

Dunno, but I guess the default might be just too benevolent for some
specific workloads which are not so wasteful to their address space
and the strict overcommit is really helpful for them.

OVERCOMMIT_ALWAYS is certainly useful for testing.

> > One thing I'm not quite clear on is: why was the default set
> > to over-commit on?
> 
> Indeed, I was hoping we could throw some light into that.
> My patch had another note:

I cannot really tell because this was way before my time but I guess the
reason was that userspace is usually very address space hungry while the
actual memory consumption is not that bad. See my other email.

>    "NOTE2: I tried to track down the history of overcommit but back then there
> were no single patches apparently and the patch that appears to have
> introduced the first overcommit mode (OVERCOMMIT_ALWAYS) is commit
> 9334eab8a36f ("Import 2.1.27"). OVERCOMMIT_NEVER was introduced with commit
> 502bff0685b2 ("[PATCH] strict overcommit").
> My understanding is that prior to commit 9334eab8a36f ("Import 2.1.27")
> there was no overcommit, is that correct?"
> 
> It'd be nice to know more about why was overcommit introduced.
> Furthermore, it looks like allowing overcommit and the introduction of the OOM-killer has given rise to lots of other options to try to tame the OOM-killer.
> Without context, that may seem like a form of "feature creep" around it.
> Moreover, it makes Linux behave differently from let's say Solaris.
> 
>    https://www.win.tue.nl/~aeb/linux/lk/lk-9.html#ss9.6

Well, those are some really strong statements which do not really
reflect the reality of the linux userspace. I am not going to argue with
those points because it doesn't make much sense. Yes in an ideal world
everybody consumes only so much he needs. Well the real life is a bit
different...

> Hopefully this discussion could clear some of this up and maybe result
> in more documentation around this subject.

What kind of documentation would help?
Documentation/vm/overcommit-accounting seems to be pretty much extensive
about all available modes including things to be aware of.
 
> > I suppose the biggest use-case is when a "large" process forks
> > only to exec microseconds later into a "small" process, it would
> > be silly to refuse that fork. But isn't that what the COW
> > optimization addresses, without the need for over-commit?
> > 
> > Another issue with overcommit=on is that some programmers seem
> > to take for granted that "allocations will never fail" and so
> > neglect to handle malloc == NULL conditions gracefully.
> > 
> > I tried to run LTP with overcommit off, and I vaguely recall that
> > I had more failures than with overcommit on. (Perhaps only those
> > tests that tickle the dreaded OOM assassin.)
> 
> From what I remember, one of the LTP maintainers said that it is
> highly unlikely people test (or run LTP for that matter) with
> different settings for overcommit.

Yes this is sad and the result of a excessive configuration space.
That's why I was pushing back to adding yet another one without having
really good reasons...

> Years ago, while using MacOS X, a long running process apparently took
> all the memory over night.  The next day when I checked the computer
> I saw a dialog that said something like (I don't remember the exact
> wording) "process X has been paused due to lack of memory (or is
> requesting too much memory, I don't remember). If you think this is
> not normal you can terminate process X, otherwise you can terminate
> other processes to free memory and unpause process X to continue" and
> then some options to proceed.
>
> If left unattended (thus the dialog unanswered), the computer would
> still work, all other processes were left intact and only the
> "offending" process was paused.  Arguably, if the "offending" process
> is just left paused, it takes the memory away from other processes,
> and if it was a server, maybe it wouldn't have enough memory to reply
> to requests.  On the server world I can thus understand that some
> setting could indicate that when the situation arises, the "dialog" is
> automatically dismissed with some default action, like "terminate the
> offending process".

Not sure what you are trying to tell here but it seems like killing such
a leaking task is a better option as the memory can be reused for others
rather than keep it blocked for an unbounded amount of time.

> To me it seems really strange for the "OOM-killer" to exist.  It has
> happened to me that it kills my terminals or editors, how can people
> deal with random processes being killed?  Doesn't it bother anybody?

Killing random tasks is definitely a misbehavior and it happened a lot
in the past when heuristics were based on multiple metrics (including
the run time etc.). Things have changed considerably since then and
seeing random tasks being selected shouldn't happen all that often and
if it happens it should be reported, understood and fixed.

-- 
Michal Hocko
SUSE Labs

[toc] | [prev] | [next] | [standalone]

#1400769

Hi Michal,

On 05/13/2016 02:00 PM, Michal Hocko wrote:
> On Fri 13-05-16 11:52:30, Sebastian Frias wrote:
>>
>> By the way, do you know what's the rationale to allow this setting to
>> be controlled by the userspace dynamically?  Was it for testing only?
> 
> Dunno, but I guess the default might be just too benevolent for some
> specific workloads which are not so wasteful to their address space
> and the strict overcommit is really helpful for them.
> 

Exactly. That's why I was wondering what is the history behind enabling it by default.

> OVERCOMMIT_ALWAYS is certainly useful for testing.
> 
>>> One thing I'm not quite clear on is: why was the default set
>>> to over-commit on?
>>
>> Indeed, I was hoping we could throw some light into that.
>> My patch had another note:
> 
> I cannot really tell because this was way before my time but I guess the
> reason was that userspace is usually very address space hungry while the
> actual memory consumption is not that bad. See my other email.

Yes, I saw that, thanks for the example.
It's just that it feels like the default value is there to deal with (what it should be?) very specific cases, right?

>> It'd be nice to know more about why was overcommit introduced.
>> Furthermore, it looks like allowing overcommit and the introduction of the OOM-killer has given rise to lots of other options to try to tame the OOM-killer.
>> Without context, that may seem like a form of "feature creep" around it.
>> Moreover, it makes Linux behave differently from let's say Solaris.
>>
>>    https://www.win.tue.nl/~aeb/linux/lk/lk-9.html#ss9.6
> 
> Well, those are some really strong statements which do not really
> reflect the reality of the linux userspace. I am not going to argue with
> those points because it doesn't make much sense. Yes in an ideal world
> everybody consumes only so much he needs. Well the real life is a bit
> different...

:-)
I see, so basically it is a sort of workaround.

Anyway, in the embedded world the memory and system requirements are usually controlled.

Would you agree to the option if it was dependent on CONFIG_EMBEDDED? Or if it was a hidden option?
(I understand though that it wouldn't affect the size of config space)

> 
>> Hopefully this discussion could clear some of this up and maybe result
>> in more documentation around this subject.
> 
> What kind of documentation would help?

Well, mostly the history of this setting, why it was introduced, etc. more or less what we are discussing here.
Because honestly, killing random processes does not seems like a straightforward idea, ie: it is not obvious.
Like I was saying, without context, such behaviour looks a bit crazy.

>>
>> From what I remember, one of the LTP maintainers said that it is
>> highly unlikely people test (or run LTP for that matter) with
>> different settings for overcommit.
> 
> Yes this is sad and the result of a excessive configuration space.
> That's why I was pushing back to adding yet another one without having
> really good reasons...

Well, a more urgent problem would be that in that case overcommit=never is not really well tested.

> 
>> Years ago, while using MacOS X, a long running process apparently took
>> all the memory over night.  The next day when I checked the computer
>> I saw a dialog that said something like (I don't remember the exact
>> wording) "process X has been paused due to lack of memory (or is
>> requesting too much memory, I don't remember). If you think this is
>> not normal you can terminate process X, otherwise you can terminate
>> other processes to free memory and unpause process X to continue" and
>> then some options to proceed.
>>
>> If left unattended (thus the dialog unanswered), the computer would
>> still work, all other processes were left intact and only the
>> "offending" process was paused.  Arguably, if the "offending" process
>> is just left paused, it takes the memory away from other processes,
>> and if it was a server, maybe it wouldn't have enough memory to reply
>> to requests.  On the server world I can thus understand that some
>> setting could indicate that when the situation arises, the "dialog" is
>> automatically dismissed with some default action, like "terminate the
>> offending process".
> 
> Not sure what you are trying to tell here but it seems like killing such
> a leaking task is a better option as the memory can be reused for others
> rather than keep it blocked for an unbounded amount of time.

My point is that it seems to be possible to deal with such conditions in a more controlled way, ie: a way that is less random and less abrupt.

> 
>> To me it seems really strange for the "OOM-killer" to exist.  It has
>> happened to me that it kills my terminals or editors, how can people
>> deal with random processes being killed?  Doesn't it bother anybody?
> 
> Killing random tasks is definitely a misbehavior and it happened a lot
> in the past when heuristics were based on multiple metrics (including
> the run time etc.). Things have changed considerably since then and
> seeing random tasks being selected shouldn't happen all that often and
> if it happens it should be reported, understood and fixed.
> 

Well, it's hard to report, since it is essentially the result of a dynamic system.
I could assume it killed terminals with a long history buffer, or editors with many buffers (or big buffers).
Actually when it happened, I just turned overcommit off. I just checked and is on again on my desktop, probably forgot to make it a permanent setting.

In the end, no processes is a good candidate for termination.
What works for you may not work for me, that's the whole point, there's a heuristic (which conceptually can never be perfect), yet the mere fact that some process has to be killed is somewhat chilling.
I mean, all running processes are supposedly there and running for a reason.

Best regards,

Sebastian

[toc] | [prev] | [next] | [standalone]

#1400787

On 2016-05-13 08:39, Sebastian Frias wrote:
> On 05/13/2016 02:00 PM, Michal Hocko wrote:
>> On Fri 13-05-16 11:52:30, Sebastian Frias wrote:
>>>
>>> From what I remember, one of the LTP maintainers said that it is
>>> highly unlikely people test (or run LTP for that matter) with
>>> different settings for overcommit.
>>
>> Yes this is sad and the result of a excessive configuration space.
>> That's why I was pushing back to adding yet another one without having
>> really good reasons...
>
> Well, a more urgent problem would be that in that case overcommit=never is not really well tested.
I know more people who use overcommit=never than overcommit=always.  I 
use it myself on all my personal systems, but I also allocate 
significant amounts of swap space (usually 64G, but I also have a big 
disks in my systems and don't often hit swap), don't use Java, and 
generally don't use a lot of the more wasteful programs either (many of 
them on desktop systems tend to be stuff like office software).  I know 
a number of people who use overcommit=never on their servers and give 
them a decent amount of swap space (and again, don't use Java).
>
>>
>>> Years ago, while using MacOS X, a long running process apparently took
>>> all the memory over night.  The next day when I checked the computer
>>> I saw a dialog that said something like (I don't remember the exact
>>> wording) "process X has been paused due to lack of memory (or is
>>> requesting too much memory, I don't remember). If you think this is
>>> not normal you can terminate process X, otherwise you can terminate
>>> other processes to free memory and unpause process X to continue" and
>>> then some options to proceed.
>>>
>>> If left unattended (thus the dialog unanswered), the computer would
>>> still work, all other processes were left intact and only the
>>> "offending" process was paused.  Arguably, if the "offending" process
>>> is just left paused, it takes the memory away from other processes,
>>> and if it was a server, maybe it wouldn't have enough memory to reply
>>> to requests.  On the server world I can thus understand that some
>>> setting could indicate that when the situation arises, the "dialog" is
>>> automatically dismissed with some default action, like "terminate the
>>> offending process".
>>
>> Not sure what you are trying to tell here but it seems like killing such
>> a leaking task is a better option as the memory can be reused for others
>> rather than keep it blocked for an unbounded amount of time.
>
> My point is that it seems to be possible to deal with such conditions in a more controlled way, ie: a way that is less random and less abrupt.
There's an option for the OOM-killer to just kill the allocating task 
instead of using the scoring heuristic.  This is about as deterministic 
as things can get though.
>
>>
>>> To me it seems really strange for the "OOM-killer" to exist.  It has
>>> happened to me that it kills my terminals or editors, how can people
>>> deal with random processes being killed?  Doesn't it bother anybody?
>>
>> Killing random tasks is definitely a misbehavior and it happened a lot
>> in the past when heuristics were based on multiple metrics (including
>> the run time etc.). Things have changed considerably since then and
>> seeing random tasks being selected shouldn't happen all that often and
>> if it happens it should be reported, understood and fixed.
>>
>
> Well, it's hard to report, since it is essentially the result of a dynamic system.
> I could assume it killed terminals with a long history buffer, or editors with many buffers (or big buffers).
> Actually when it happened, I just turned overcommit off. I just checked and is on again on my desktop, probably forgot to make it a permanent setting.
>
> In the end, no processes is a good candidate for termination.
> What works for you may not work for me, that's the whole point, there's a heuristic (which conceptually can never be perfect), yet the mere fact that some process has to be killed is somewhat chilling.
> I mean, all running processes are supposedly there and running for a reason.
OTOH, just because something is there for a reason doesn't mean it's 
doing what it's supposed to be.  Bugs happen, including memory leaks, 
and if something is misbehaving enough that it impacts the rest of the 
system, it really should be dealt with.

This brings to mind a complex bug involving Tor and GCC whereby building 
certain (old) versions of Tor with certain (old) versions of GCC with 
-Os would cause an infinite loop in GCC.  You obviously have GCC running 
for a reason, but that doesn't mean that it's doing what it should be.

[toc] | [prev] | [next] | [standalone]

#1400797

Hi Austin,

On 05/13/2016 03:11 PM, Austin S. Hemmelgarn wrote:
> On 2016-05-13 08:39, Sebastian Frias wrote:
>>
>> My point is that it seems to be possible to deal with such conditions in a more controlled way, ie: a way that is less random and less abrupt.
> There's an option for the OOM-killer to just kill the allocating task instead of using the scoring heuristic.  This is about as deterministic as things can get though.

By the way, why does it has to "kill" anything in that case?
I mean, shouldn't it just tell the allocating task that there's not enough memory by letting malloc return NULL?

Best regards,

Sebastian

[toc] | [prev] | [next] | [standalone]

#1400822

On 2016-05-13 09:34, Sebastian Frias wrote:
> Hi Austin,
>
> On 05/13/2016 03:11 PM, Austin S. Hemmelgarn wrote:
>> On 2016-05-13 08:39, Sebastian Frias wrote:
>>>
>>> My point is that it seems to be possible to deal with such conditions in a more controlled way, ie: a way that is less random and less abrupt.
>> There's an option for the OOM-killer to just kill the allocating task instead of using the scoring heuristic.  This is about as deterministic as things can get though.
>
> By the way, why does it has to "kill" anything in that case?
> I mean, shouldn't it just tell the allocating task that there's not enough memory by letting malloc return NULL?
In theory, that's a great idea.  In practice though, it only works if:
1. The allocating task correctly handles malloc() (or whatever other 
function it uses) returning NULL, which a number of programs don't.
2. The task actually has fallback options for memory limits.  Many 
programs that do handle getting a NULL pointer from malloc() handle it 
by exiting anyway, so there's not as much value in this case.
3. There isn't a memory leak somewhere on the system.  Killing the 
allocating task doesn't help much if this is the case of course.

You have to keep in mind though, that on a properly provisioned system, 
the only situations where the OOM killer should be invoked are when 
there's a memory leak, or when someone is intentionally trying to DoS 
the system through memory exhaustion.  If you're hitting the OOM killer 
for any other reason than those or a kernel bug, then you just need more 
memory or more swap space.

[toc] | [prev] | [next] | [standalone]

#1400832

Hi Austin,

On 05/13/2016 04:14 PM, Austin S. Hemmelgarn wrote:
> On 2016-05-13 09:34, Sebastian Frias wrote:
>> Hi Austin,
>>
>> On 05/13/2016 03:11 PM, Austin S. Hemmelgarn wrote:
>>> On 2016-05-13 08:39, Sebastian Frias wrote:
>>>>
>>>> My point is that it seems to be possible to deal with such conditions in a more controlled way, ie: a way that is less random and less abrupt.
>>> There's an option for the OOM-killer to just kill the allocating task instead of using the scoring heuristic.  This is about as deterministic as things can get though.
>>
>> By the way, why does it has to "kill" anything in that case?
>> I mean, shouldn't it just tell the allocating task that there's not enough memory by letting malloc return NULL?
> In theory, that's a great idea.  In practice though, it only works if:
> 1. The allocating task correctly handles malloc() (or whatever other function it uses) returning NULL, which a number of programs don't.
> 2. The task actually has fallback options for memory limits.  Many programs that do handle getting a NULL pointer from malloc() handle it by exiting anyway, so there's not as much value in this case.
> 3. There isn't a memory leak somewhere on the system.  Killing the allocating task doesn't help much if this is the case of course.

Well, the thing is that the current behaviour, i.e.: overcommiting, does not improves the quality of those programs.
I mean, what incentive do they have to properly handle situations 1, 2?

Also, if there's a memory leak, the termination of any task, whether it is the allocating task or something random, does not help either, the system will eventually go down, right?

> 
> You have to keep in mind though, that on a properly provisioned system, the only situations where the OOM killer should be invoked are when there's a memory leak, or when someone is intentionally trying to DoS the system through memory exhaustion. 

Exactly, the DoS attack is another reason why the OOM-killer does not seem a good idea, at least compared to just letting malloc return NULL and let the program fail.

>If you're hitting the OOM killer for any other reason than those or a kernel bug, then you just need more memory or more swap space.
> 

Indeed.

Best regards,

Sebastian

[toc] | [prev] | [next] | [standalone]

#1400852

On 2016-05-13 10:23, Sebastian Frias wrote:
> Hi Austin,
>
> On 05/13/2016 04:14 PM, Austin S. Hemmelgarn wrote:
>> On 2016-05-13 09:34, Sebastian Frias wrote:
>>> Hi Austin,
>>>
>>> On 05/13/2016 03:11 PM, Austin S. Hemmelgarn wrote:
>>>> On 2016-05-13 08:39, Sebastian Frias wrote:
>>>>>
>>>>> My point is that it seems to be possible to deal with such conditions in a more controlled way, ie: a way that is less random and less abrupt.
>>>> There's an option for the OOM-killer to just kill the allocating task instead of using the scoring heuristic.  This is about as deterministic as things can get though.
>>>
>>> By the way, why does it has to "kill" anything in that case?
>>> I mean, shouldn't it just tell the allocating task that there's not enough memory by letting malloc return NULL?
>> In theory, that's a great idea.  In practice though, it only works if:
>> 1. The allocating task correctly handles malloc() (or whatever other function it uses) returning NULL, which a number of programs don't.
>> 2. The task actually has fallback options for memory limits.  Many programs that do handle getting a NULL pointer from malloc() handle it by exiting anyway, so there's not as much value in this case.
>> 3. There isn't a memory leak somewhere on the system.  Killing the allocating task doesn't help much if this is the case of course.
>
> Well, the thing is that the current behaviour, i.e.: overcommiting, does not improves the quality of those programs.
> I mean, what incentive do they have to properly handle situations 1, 2?
Overcommit got introduced because of these, not the other way around. 
It's not forcing them to change, but it's also a core concept in any 
modern virtual memory based OS, and that's not ever going to change either.

You also have to keep in mind that most apps aren't doing this 
intentionally.  There are three general reasons they do this:
1. They don't know how much memory they will need, so they guess high 
because malloc() is computationally expensive.  This is technically 
intentional, but it's also something that can't be avoided in some cases 
  Dropbox is a perfect example of this taken way too far (they also take 
the concept of a thread pool too far).
2. The program has a lot of code that isn't frequently run.  It makes no 
sense to keep code that isn't used in RAM, so it gets either dropped (if 
it's unmodified), or it gets swapped out.  Most of the programs that I 
see on my system fall into this category (acpid  for example just sleeps 
until an ACPI event happens, so it usually won't have most of it's code 
in memory on a busy system).
3. The application wants to do it's own memory management.  This is 
common on a lot of HPC apps and some high performance server software.
>
> Also, if there's a memory leak, the termination of any task, whether it is the allocating task or something random, does not help either, the system will eventually go down, right?
If the memory leak is in the kernel, then yes, the OOM killer won't 
help, period.  But if the memory leak is in userspace, and the OOM 
killer kills the task with the leak (which it usually will if you don't 
have it set to kill the allocating task), then it may have just saved 
the system from crashing completely.  Yes some user may lose some 
unsaved work, but they would lose that data anyway if the system 
crashes, and they can probably still use the rest of the system.
>> You have to keep in mind though, that on a properly provisioned system, the only situations where the OOM killer should be invoked are when there's a memory leak, or when someone is intentionally trying to DoS the system through memory exhaustion.
>
> Exactly, the DoS attack is another reason why the OOM-killer does not seem a good idea, at least compared to just letting malloc return NULL and let the program fail.
Because of overcommit, it's possible for the allocation to succeed, but 
the subsequent access to fail.  At that point, you're way past malloc() 
returning, and you have to do something.

Also, returning NULL on a failed malloc() provides zero protection 
against all but the most brain-dead memory exhaustion based DoS attacks. 
  The general core of a memory exhaustion DoS against a local system 
follows a simple three step procedure:
     1. Try to allocate a small chunk of memory (less than or equal to 
page size)
     2. If the allocation succeeded, write to the first byte of that 
chunk of memory, forcing actual allocation
     3. Repeat indefinitely from step 1
Step 2 is the crucial part here, if you don't write to the memory, it 
will only eat up your own virtual address space.  If you don't check for 
a NULL pointer and skip writing, you get a segfault.  If the OOM killer 
isn't invoked in such a situation, then this will just eat up all the 
free system memory, and then _keep running_ and eat up all the other 
memory as it's freed by other things exiting due to lack of memory.

[toc] | [prev] | [next] | [standalone]

#1400850

On Fri, 13 May 2016 15:34:52 +0200
Sebastian Frias <sf84@laposte.net> wrote:

> Hi Austin,
> 
> On 05/13/2016 03:11 PM, Austin S. Hemmelgarn wrote:
> > On 2016-05-13 08:39, Sebastian Frias wrote:  
> >>
> >> My point is that it seems to be possible to deal with such conditions in a more controlled way, ie: a way that is less random and less abrupt.  
> > There's an option for the OOM-killer to just kill the allocating task instead of using the scoring heuristic.  This is about as deterministic as things can get though.  
> 
> By the way, why does it has to "kill" anything in that case?
> I mean, shouldn't it just tell the allocating task that there's not enough memory by letting malloc return NULL?

Just turn off overcommit and it will do that. With overcommit disabled
the kernel will not hand out address space in excess of memory plus swap.

Alan

[toc] | [prev] | [next] | [standalone]

#1400858

Hi Alan,

On 05/13/2016 05:01 PM, One Thousand Gnomes wrote:
> On Fri, 13 May 2016 15:34:52 +0200
> Sebastian Frias <sf84@laposte.net> wrote:
> 
>> Hi Austin,
>>
>> On 05/13/2016 03:11 PM, Austin S. Hemmelgarn wrote:
>>> On 2016-05-13 08:39, Sebastian Frias wrote:  
>>>>
>>>> My point is that it seems to be possible to deal with such conditions in a more controlled way, ie: a way that is less random and less abrupt.  
>>> There's an option for the OOM-killer to just kill the allocating task instead of using the scoring heuristic.  This is about as deterministic as things can get though.  
>>
>> By the way, why does it has to "kill" anything in that case?
>> I mean, shouldn't it just tell the allocating task that there's not enough memory by letting malloc return NULL?
> 
> Just turn off overcommit and it will do that. With overcommit disabled
> the kernel will not hand out address space in excess of memory plus swap.

I think I'm confused.
Michal just said:

   "And again, overcommit=never doesn't imply no-OOM. It just makes it less
likely. The kernel can consume quite some unreclaimable memory as well."

which I understand as the OOM-killer will still lurk around and could still wake up.

Will overcommit=never totally disable the OOM-Killer or not?

Best regards,

Sebastian

[toc] | [prev] | [next] | [standalone]

#1400861

On Fri 13-05-16 17:15:26, Sebastian Frias wrote:
> Hi Alan,
> 
> On 05/13/2016 05:01 PM, One Thousand Gnomes wrote:
> > On Fri, 13 May 2016 15:34:52 +0200
> > Sebastian Frias <sf84@laposte.net> wrote:
> > 
> >> Hi Austin,
> >>
> >> On 05/13/2016 03:11 PM, Austin S. Hemmelgarn wrote:
> >>> On 2016-05-13 08:39, Sebastian Frias wrote:  
> >>>>
> >>>> My point is that it seems to be possible to deal with such conditions in a more controlled way, ie: a way that is less random and less abrupt.  
> >>> There's an option for the OOM-killer to just kill the allocating task instead of using the scoring heuristic.  This is about as deterministic as things can get though.  
> >>
> >> By the way, why does it has to "kill" anything in that case?
> >> I mean, shouldn't it just tell the allocating task that there's not enough memory by letting malloc return NULL?
> > 
> > Just turn off overcommit and it will do that. With overcommit disabled
> > the kernel will not hand out address space in excess of memory plus swap.
> 
> I think I'm confused.
> Michal just said:
> 
>    "And again, overcommit=never doesn't imply no-OOM. It just makes it less
> likely. The kernel can consume quite some unreclaimable memory as well."
> 
> which I understand as the OOM-killer will still lurk around and could still wake up.
> 
> Will overcommit=never totally disable the OOM-Killer or not?

Please have a look at __vm_enough_memory and which allocations are
accounted. There are lots of those in kernel which are not accounted so
the OOM killer still might be invoked if there is an excessive in kernel
unreclaimable memory consumer.

-- 
Michal Hocko
SUSE Labs

[toc] | [prev] | [next] | [standalone]

#1400799

Hi Austin,

On 05/13/2016 03:11 PM, Austin S. Hemmelgarn wrote:
> On 2016-05-13 08:39, Sebastian Frias wrote:
>> Well, a more urgent problem would be that in that case overcommit=never is not really well tested.
> I know more people who use overcommit=never than overcommit=always.  I use it myself on all my personal systems, but I also allocate significant amounts of swap space (usually 64G, but I also have a big disks in my systems and don't often hit swap), don't use Java, and generally don't use a lot of the more wasteful programs either (many of them on desktop systems tend to be stuff like office software).  I know a number of people who use overcommit=never on their servers and give them a decent amount of swap space (and again, don't use Java).

Then I'll look into LTP and the issues it has when overcommit=never.

>>
>> My point is that it seems to be possible to deal with such conditions in a more controlled way, ie: a way that is less random and less abrupt.
> There's an option for the OOM-killer to just kill the allocating task instead of using the scoring heuristic.  This is about as deterministic as things can get though.

I didn't see that in Documentation/vm/overcommit-accounting or am I looking in the wrong place?

>>
>> Well, it's hard to report, since it is essentially the result of a dynamic system.
>> I could assume it killed terminals with a long history buffer, or editors with many buffers (or big buffers).
>> Actually when it happened, I just turned overcommit off. I just checked and is on again on my desktop, probably forgot to make it a permanent setting.
>>
>> In the end, no processes is a good candidate for termination.
>> What works for you may not work for me, that's the whole point, there's a heuristic (which conceptually can never be perfect), yet the mere fact that some process has to be killed is somewhat chilling.
>> I mean, all running processes are supposedly there and running for a reason.
> OTOH, just because something is there for a reason doesn't mean it's doing what it's supposed to be.  Bugs happen, including memory leaks, and if something is misbehaving enough that it impacts the rest of the system, it really should be dealt with.

Exactly, it's just that in this case, the system is deciding how to deal with the situation by itself.

> 
> This brings to mind a complex bug involving Tor and GCC whereby building certain (old) versions of Tor with certain (old) versions of GCC with -Os would cause an infinite loop in GCC.  You obviously have GCC running for a reason, but that doesn't mean that it's doing what it should be.

I'm not sure if I followed the analogy/example, but are you saying that the OOM-killer killed GCC in your example?
This seems an odd example though, I mean, shouldn't the guy in front of the computer notice the loop and kill GCC by himself?

Best regards,

Sebastian

[toc] | [prev] | [next] | [standalone]

#1400810

On 2016-05-13 09:32, Sebastian Frias wrote:
> Hi Austin,
>
> On 05/13/2016 03:11 PM, Austin S. Hemmelgarn wrote:
>> On 2016-05-13 08:39, Sebastian Frias wrote:
>>> Well, a more urgent problem would be that in that case overcommit=never is not really well tested.
>> I know more people who use overcommit=never than overcommit=always.  I use it myself on all my personal systems, but I also allocate significant amounts of swap space (usually 64G, but I also have a big disks in my systems and don't often hit swap), don't use Java, and generally don't use a lot of the more wasteful programs either (many of them on desktop systems tend to be stuff like office software).  I know a number of people who use overcommit=never on their servers and give them a decent amount of swap space (and again, don't use Java).
>
> Then I'll look into LTP and the issues it has when overcommit=never.
>
>>>
>>> My point is that it seems to be possible to deal with such conditions in a more controlled way, ie: a way that is less random and less abrupt.
>> There's an option for the OOM-killer to just kill the allocating task instead of using the scoring heuristic.  This is about as deterministic as things can get though.
>
> I didn't see that in Documentation/vm/overcommit-accounting or am I looking in the wrong place?
It's controlled by a sysctl value, so it's listed in 
Documentation/sysctl/vm.txt
The relevant sysctl is vm.oom_kill_allocating_task
>
>>>
>>> Well, it's hard to report, since it is essentially the result of a dynamic system.
>>> I could assume it killed terminals with a long history buffer, or editors with many buffers (or big buffers).
>>> Actually when it happened, I just turned overcommit off. I just checked and is on again on my desktop, probably forgot to make it a permanent setting.
>>>
>>> In the end, no processes is a good candidate for termination.
>>> What works for you may not work for me, that's the whole point, there's a heuristic (which conceptually can never be perfect), yet the mere fact that some process has to be killed is somewhat chilling.
>>> I mean, all running processes are supposedly there and running for a reason.
>> OTOH, just because something is there for a reason doesn't mean it's doing what it's supposed to be.  Bugs happen, including memory leaks, and if something is misbehaving enough that it impacts the rest of the system, it really should be dealt with.
>
> Exactly, it's just that in this case, the system is deciding how to deal with the situation by itself.
On a busy server where uptime is critical, you can't wait for someone to 
notice and handle it manually, you need the issue resolved ASAP.  Now, 
this won't always kill the correct thing, but if it's due to a memory 
leak, it often will work like it should.
>
>>
>> This brings to mind a complex bug involving Tor and GCC whereby building certain (old) versions of Tor with certain (old) versions of GCC with -Os would cause an infinite loop in GCC.  You obviously have GCC running for a reason, but that doesn't mean that it's doing what it should be.
>
> I'm not sure if I followed the analogy/example, but are you saying that the OOM-killer killed GCC in your example?
> This seems an odd example though, I mean, shouldn't the guy in front of the computer notice the loop and kill GCC by himself?
No, I didn't mean as an example of the OOM killer, I just meant as an 
example of software not doing what it should.  It's not as easy to find 
an example for the OOM killer, so I don't really have a good example. 
The general concept is the same though, the only difference is there 
isn't a kernel protection against infinite loops (because they aren't 
always bugs, while memory leaks and similar are).

[toc] | [prev] | [next] | [standalone]

#1400837

Hi Austin,

On 05/13/2016 03:51 PM, Austin S. Hemmelgarn wrote:
> On 2016-05-13 09:32, Sebastian Frias wrote:
>> I didn't see that in Documentation/vm/overcommit-accounting or am I looking in the wrong place?
> It's controlled by a sysctl value, so it's listed in Documentation/sysctl/vm.txt
> The relevant sysctl is vm.oom_kill_allocating_task

Thanks, I just read that.
Does not look like a replacement for overcommit=never though.

>>
>>>>
>>>> Well, it's hard to report, since it is essentially the result of a dynamic system.
>>>> I could assume it killed terminals with a long history buffer, or editors with many buffers (or big buffers).
>>>> Actually when it happened, I just turned overcommit off. I just checked and is on again on my desktop, probably forgot to make it a permanent setting.
>>>>
>>>> In the end, no processes is a good candidate for termination.
>>>> What works for you may not work for me, that's the whole point, there's a heuristic (which conceptually can never be perfect), yet the mere fact that some process has to be killed is somewhat chilling.
>>>> I mean, all running processes are supposedly there and running for a reason.
>>> OTOH, just because something is there for a reason doesn't mean it's doing what it's supposed to be.  Bugs happen, including memory leaks, and if something is misbehaving enough that it impacts the rest of the system, it really should be dealt with.
>>
>> Exactly, it's just that in this case, the system is deciding how to deal with the situation by itself.
> On a busy server where uptime is critical, you can't wait for someone to notice and handle it manually, you need the issue resolved ASAP.  Now, this won't always kill the correct thing, but if it's due to a memory leak, it often will work like it should.

The keyword is "'often' will work as expected".
So you are saying that it will kill a program leaking memory in what, like 90% of the cases?
I'm not sure if I would setup a server with critical uptime to have the OOM-killer enabled, do you think that'd be a good idea?

Anyway, as a side note, I just want to say thank you guys for having this discussion.
I think it is an interesting thread and hopefully it will advance the "knowledge" about this setting.

>>
>>>
>>> This brings to mind a complex bug involving Tor and GCC whereby building certain (old) versions of Tor with certain (old) versions of GCC with -Os would cause an infinite loop in GCC.  You obviously have GCC running for a reason, but that doesn't mean that it's doing what it should be.
>>
>> I'm not sure if I followed the analogy/example, but are you saying that the OOM-killer killed GCC in your example?
>> This seems an odd example though, I mean, shouldn't the guy in front of the computer notice the loop and kill GCC by himself?
> No, I didn't mean as an example of the OOM killer, I just meant as an example of software not doing what it should.  It's not as easy to find an example for the OOM killer, so I don't really have a good example. The general concept is the same though, the only difference is there isn't a kernel protection against infinite loops (because they aren't always bugs, while memory leaks and similar are).

So how does the kernel knows that a process is "leaking memory" as opposed to just "using lots of memory"? (wouldn't that be comparable to answering how does the kernel knows the difference between an infinite loop and one that is not?)

Best regards,

Sebastian

[toc] | [prev] | [next] | [standalone]

#1400842

On Fri 13-05-16 16:35:20, Sebastian Frias wrote:
> Hi Austin,
> 
> On 05/13/2016 03:51 PM, Austin S. Hemmelgarn wrote:
> > On 2016-05-13 09:32, Sebastian Frias wrote:
> >> I didn't see that in Documentation/vm/overcommit-accounting or am I looking in the wrong place?
> > It's controlled by a sysctl value, so it's listed in Documentation/sysctl/vm.txt
> > The relevant sysctl is vm.oom_kill_allocating_task
> 
> Thanks, I just read that.
> Does not look like a replacement for overcommit=never though.

No this is just an OOM strategy. I wouldn't recommend it though because
the behavior might be really time dependant - unlike the regular OOM
killer strategy to select the largest memory consumer.

And again, overcommit=never doesn't imply no-OOM. It just makes it less
likely. The kernel can consume quite some unreclaimable memory as well.
-- 
Michal Hocko
SUSE Labs

[toc] | [prev] | [next] | [standalone]

#1400855

On 2016-05-13 10:35, Sebastian Frias wrote:
> Hi Austin,
>
> On 05/13/2016 03:51 PM, Austin S. Hemmelgarn wrote:
>> On 2016-05-13 09:32, Sebastian Frias wrote:
>>>
>>>>>
>>>>> Well, it's hard to report, since it is essentially the result of a dynamic system.
>>>>> I could assume it killed terminals with a long history buffer, or editors with many buffers (or big buffers).
>>>>> Actually when it happened, I just turned overcommit off. I just checked and is on again on my desktop, probably forgot to make it a permanent setting.
>>>>>
>>>>> In the end, no processes is a good candidate for termination.
>>>>> What works for you may not work for me, that's the whole point, there's a heuristic (which conceptually can never be perfect), yet the mere fact that some process has to be killed is somewhat chilling.
>>>>> I mean, all running processes are supposedly there and running for a reason.
>>>> OTOH, just because something is there for a reason doesn't mean it's doing what it's supposed to be.  Bugs happen, including memory leaks, and if something is misbehaving enough that it impacts the rest of the system, it really should be dealt with.
>>>
>>> Exactly, it's just that in this case, the system is deciding how to deal with the situation by itself.
>> On a busy server where uptime is critical, you can't wait for someone to notice and handle it manually, you need the issue resolved ASAP.  Now, this won't always kill the correct thing, but if it's due to a memory leak, it often will work like it should.
>
> The keyword is "'often' will work as expected".
> So you are saying that it will kill a program leaking memory in what, like 90% of the cases?
If the program leaking memory has the highest memory consumption, it 
will be the one that gets killed.  If not, then it will eventually be 
the one with the highest memory consumption and be killed (usually 
pretty quickly if it's leaking memory fast).
> I'm not sure if I would setup a server with critical uptime to have the OOM-killer enabled, do you think that'd be a good idea?
It really depends.  If you've got a setup with a bunch of web-servers 
behind a couple of load balancers which are set up in a HA 
configuration, I absolutely would run with the OOM killer enabled on 
everything.  There are in fact very few cases I wouldn't run with it 
enabled, as it's almost always better on a server to be able to actually 
log in to see what's wrong than to have to deal with resource exhaustion.

Most of the servers where I work are set to panic on OOM instead of 
killing something, because if we hit an OOM condition it's either a bug 
or a DoS attack, and either case needs to be noticed immediately, and 
taking out the entire system is the most reliable way to make sure it 
gets noticed.
>
> Anyway, as a side note, I just want to say thank you guys for having this discussion.
> I think it is an interesting thread and hopefully it will advance the "knowledge" about this setting.
>
>>>
>>>>
>>>> This brings to mind a complex bug involving Tor and GCC whereby building certain (old) versions of Tor with certain (old) versions of GCC with -Os would cause an infinite loop in GCC.  You obviously have GCC running for a reason, but that doesn't mean that it's doing what it should be.
>>>
>>> I'm not sure if I followed the analogy/example, but are you saying that the OOM-killer killed GCC in your example?
>>> This seems an odd example though, I mean, shouldn't the guy in front of the computer notice the loop and kill GCC by himself?
>> No, I didn't mean as an example of the OOM killer, I just meant as an example of software not doing what it should.  It's not as easy to find an example for the OOM killer, so I don't really have a good example. The general concept is the same though, the only difference is there isn't a kernel protection against infinite loops (because they aren't always bugs, while memory leaks and similar are).
>
> So how does the kernel knows that a process is "leaking memory" as opposed to just "using lots of memory"? (wouldn't that be comparable to answering how does the kernel knows the difference between an infinite loop and one that is not?)
It doesn't, it sees who's using the most RAM and kills that first.  If 
something is leaking memory, it will eventually kill that and you should 
have a working system again if you have process supervision.

There are three cases where it won't kill the task with the largest 
memory consumption:
1. You have /proc/sys/vm/panic_on_oom set to 1, which will cause the 
kernel to panic instead of killing a single task.
2. You have /proc/sys/vm/oom_kill_allocating_task set to 1, in which 
case it will kill whatever triggered the fault that caused the OOM 
condition.
3. You have adjusted the OOM score for tasks via /proc.  The score 
normally scales with memory usage, but it's possible to set it higher 
for specific tasks.  Many of the public distributed computing platforms 
(like BOINC) use this to cause their applications to be the first target 
for the OOM killer.

[toc] | [prev] | [next] | [standalone]

#1400843

On Fri 13-05-16 14:39:01, Sebastian Frias wrote:
> Hi Michal,
> 
> On 05/13/2016 02:00 PM, Michal Hocko wrote:
> > On Fri 13-05-16 11:52:30, Sebastian Frias wrote:
[...]
> >> Indeed, I was hoping we could throw some light into that.
> >> My patch had another note:
> > 
> > I cannot really tell because this was way before my time but I guess the
> > reason was that userspace is usually very address space hungry while the
> > actual memory consumption is not that bad. See my other email.
> 
> Yes, I saw that, thanks for the example.
> It's just that it feels like the default value is there to deal with
> (what it should be?) very specific cases, right?

The default should cover the most use cases. If you can prove that the
vast majority of embeded systems are different and would _benefit_ from
a different default I wouldn't be opposed to change the default there.

> >> It'd be nice to know more about why was overcommit introduced.
> >> Furthermore, it looks like allowing overcommit and the introduction
> >> of the OOM-killer has given rise to lots of other options to try to
> >> tame the OOM-killer.
> >> Without context, that may seem like a form of "feature creep" around it.
> >> Moreover, it makes Linux behave differently from let's say Solaris.
> >>
> >>    https://www.win.tue.nl/~aeb/linux/lk/lk-9.html#ss9.6
> > 
> > Well, those are some really strong statements which do not really
> > reflect the reality of the linux userspace. I am not going to argue with
> > those points because it doesn't make much sense. Yes in an ideal world
> > everybody consumes only so much he needs. Well the real life is a bit
> > different...
> 
> :-)
> I see, so basically it is a sort of workaround.

No it is not a workaround. It is just serving the purpose of the
operating system. The allow using the HW as much as possible to the
existing userspace. You cannot expect userspace will change just because
we do not like the overcommiting the memory with all the fallouts.

> Anyway, in the embedded world the memory and system requirements are
> usually controlled.

OK, but even when it is controlled does it suffer in any way just
because of the default setting? Do you see OOM killer invocation
when the overcommit would prevent from that?

> Would you agree to the option if it was dependent on
> CONFIG_EMBEDDED? Or if it was a hidden option?
> (I understand though that it wouldn't affect the size of config space)

It could be done in the code and make the default depending on the
existing config. But first try to think about what would be an advantage
of such a change.
 
> >> Hopefully this discussion could clear some of this up and maybe result
> >> in more documentation around this subject.
> > 
> > What kind of documentation would help?
> 
> Well, mostly the history of this setting, why it was introduced, etc.
> more or less what we are discussing here.  Because honestly, killing
> random processes does not seems like a straightforward idea, ie: it is
> not obvious.  Like I was saying, without context, such behaviour looks
> a bit crazy.

But we are not killing a random process. The semantic is quite clear. We
are trying to kill the biggest memory hog and if it has some children
try to sacrifice them to save as much work as possible.

> >> From what I remember, one of the LTP maintainers said that it is
> >> highly unlikely people test (or run LTP for that matter) with
> >> different settings for overcommit.
> > 
> > Yes this is sad and the result of a excessive configuration space.
> > That's why I was pushing back to adding yet another one without having
> > really good reasons...
> 
> Well, a more urgent problem would be that in that case
> overcommit=never is not really well tested.

This is a problem of the userspace and am really skeptical that a change
in default would make any existing bugs going away. It is more likely we
will see reports that ENOMEM has been returned even though there is
pletny of memory available.

[...]

> > Killing random tasks is definitely a misbehavior and it happened a lot
> > in the past when heuristics were based on multiple metrics (including
> > the run time etc.). Things have changed considerably since then and
> > seeing random tasks being selected shouldn't happen all that often and
> > if it happens it should be reported, understood and fixed.
> > 
> 
> Well, it's hard to report, since it is essentially the result of a
> dynamic system.

Each oom killer invocation will provide a detailed report which will
help MM developers to debug what went wrong and why.

-- 
Michal Hocko
SUSE Labs

[toc] | [prev] | [next] | [standalone]

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

Back to top | Article view | linux.kernel

csiph-web