Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1400594 > unrolled thread

Re: [PATCH] mm: add config option to select the initial overcommit mode

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH] mm: add config option to select the initial overcommit  mode Michal Hocko <mhocko@kernel.org> - 2016-05-13 10:10 +0200
    Re: [PATCH] mm: add config option to select the initial overcommit  mode Mason <slash.tmp@free.fr> - 2016-05-13 10:50 +0200
      Re: [PATCH] mm: add config option to select the initial overcommit  mode Michal Hocko <mhocko@kernel.org> - 2016-05-13 12:00 +0200
        Re: [PATCH] mm: add config option to select the initial overcommit  mode Mason <slash.tmp@free.fr> - 2016-05-13 12:20 +0200
          Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-13 12:50 +0200
          Re: [PATCH] mm: add config option to select the initial overcommit  mode Michal Hocko <mhocko@kernel.org> - 2016-05-13 13:50 +0200
            Re: [PATCH] mm: add config option to select the initial overcommit  mode Mason <slash.tmp@free.fr> - 2016-05-13 14:20 +0200
              Re: [PATCH] mm: add config option to select the initial overcommit  mode Michal Hocko <mhocko@kernel.org> - 2016-05-13 16:10 +0200
                Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-13 16:20 +0200
                Re: [PATCH] mm: add config option to select the initial overcommit  mode One Thousand Gnomes <gnomes@lxorguk.ukuu.org.uk> - 2016-05-13 17:10 +0200
                  Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-13 17:40 +0200
                    Re: [PATCH] mm: add config option to select the initial overcommit  mode One Thousand Gnomes <gnomes@lxorguk.ukuu.org.uk> - 2016-05-13 17:50 +0200
                      Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-17 10:30 +0200
                        Re: [PATCH] mm: add config option to select the initial overcommit  mode Michal Hocko <mhocko@kernel.org> - 2016-05-17 11:00 +0200
                          Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-17 18:20 +0200
                            Re: [PATCH] mm: add config option to select the initial overcommit  mode "Austin S. Hemmelgarn" <ahferroin7@gmail.com> - 2016-05-17 19:30 +0200
                              Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-18 17:20 +0200
                                Re: [PATCH] mm: add config option to select the initial overcommit  mode "Austin S. Hemmelgarn" <ahferroin7@gmail.com> - 2016-05-18 18:30 +0200
                            Re: [PATCH] mm: add config option to select the initial overcommit  mode Michal Hocko <mhocko@kernel.org> - 2016-05-17 22:20 +0200
                              Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-18 17:20 +0200
                                Re: [PATCH] mm: add config option to select the initial overcommit  mode Michal Hocko <mhocko@kernel.org> - 2016-05-19 09:20 +0200
                    Re: [PATCH] mm: add config option to select the initial overcommit  mode "Austin S. Hemmelgarn" <ahferroin7@gmail.com> - 2016-05-13 19:10 +0200
          Re: [PATCH] mm: add config option to select the initial overcommit  mode "Austin S. Hemmelgarn" <ahferroin7@gmail.com> - 2016-05-13 15:30 +0200
      Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-13 12:00 +0200
        Re: [PATCH] mm: add config option to select the initial overcommit  mode Michal Hocko <mhocko@kernel.org> - 2016-05-13 14:10 +0200
          Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-13 14:40 +0200
            Re: [PATCH] mm: add config option to select the initial overcommit  mode "Austin S. Hemmelgarn" <ahferroin7@gmail.com> - 2016-05-13 15:20 +0200
              Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-13 15:40 +0200
                Re: [PATCH] mm: add config option to select the initial overcommit  mode "Austin S. Hemmelgarn" <ahferroin7@gmail.com> - 2016-05-13 16:20 +0200
                  Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-13 16:30 +0200
                    Re: [PATCH] mm: add config option to select the initial overcommit  mode "Austin S. Hemmelgarn" <ahferroin7@gmail.com> - 2016-05-13 17:10 +0200
                Re: [PATCH] mm: add config option to select the initial overcommit  mode One Thousand Gnomes <gnomes@lxorguk.ukuu.org.uk> - 2016-05-13 17:10 +0200
                  Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-13 17:20 +0200
                    Re: [PATCH] mm: add config option to select the initial overcommit  mode Michal Hocko <mhocko@kernel.org> - 2016-05-13 17:30 +0200
              Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-13 15:40 +0200
                Re: [PATCH] mm: add config option to select the initial overcommit  mode "Austin S. Hemmelgarn" <ahferroin7@gmail.com> - 2016-05-13 16:00 +0200
                  Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-13 16:40 +0200
                    Re: [PATCH] mm: add config option to select the initial overcommit  mode Michal Hocko <mhocko@kernel.org> - 2016-05-13 17:00 +0200
                    Re: [PATCH] mm: add config option to select the initial overcommit  mode "Austin S. Hemmelgarn" <ahferroin7@gmail.com> - 2016-05-13 17:20 +0200
            Re: [PATCH] mm: add config option to select the initial overcommit  mode Michal Hocko <mhocko@kernel.org> - 2016-05-13 17:00 +0200
              Re: [PATCH] mm: add config option to select the initial overcommit  mode Mason <slash.tmp@free.fr> - 2016-05-13 17:00 +0200
                Re: [PATCH] mm: add config option to select the initial overcommit  mode One Thousand Gnomes <gnomes@lxorguk.ukuu.org.uk> - 2016-05-13 17:20 +0200
                  Re: [PATCH] mm: add config option to select the initial overcommit  mode Michal Hocko <mhocko@kernel.org> - 2016-05-13 17:30 +0200
                  Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-13 17:40 +0200
              Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-13 17:20 +0200
                Re: [PATCH] mm: add config option to select the initial overcommit  mode One Thousand Gnomes <gnomes@lxorguk.ukuu.org.uk> - 2016-05-13 17:50 +0200
                  Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-23 15:20 +0200

Page 1 of 3  [1] 2 3  Next page →

#1400594 — Re: [PATCH] mm: add config option to select the initial overcommit mode

On Tue 10-05-16 13:56:30, Sebastian Frias wrote:
[...]
> NOTE: I understand that the overcommit mode can be changed dynamically thru
> sysctl, but on embedded systems, where we know in advance that overcommit
> will be disabled, there's no reason to postpone such setting.

To be honest I am not particularly happy about yet another config
option. At least not without a strong reason (the one above doesn't
sound that way). The config space is really large already.
So why a later initialization matters at all? Early userspace shouldn't
consume too much address space to blow up later, no?

-- 
Michal Hocko
SUSE Labs

[toc] | [next] | [standalone]

#1400619

On 13/05/2016 10:04, Michal Hocko wrote:

> On Tue 10-05-16 13:56:30, Sebastian Frias wrote:
> [...]
>> NOTE: I understand that the overcommit mode can be changed dynamically thru
>> sysctl, but on embedded systems, where we know in advance that overcommit
>> will be disabled, there's no reason to postpone such setting.
> 
> To be honest I am not particularly happy about yet another config
> option. At least not without a strong reason (the one above doesn't
> sound that way). The config space is really large already.
> So why a later initialization matters at all? Early userspace shouldn't
> consume too much address space to blow up later, no?

One thing I'm not quite clear on is: why was the default set
to over-commit on?

I suppose the biggest use-case is when a "large" process forks
only to exec microseconds later into a "small" process, it would
be silly to refuse that fork. But isn't that what the COW
optimization addresses, without the need for over-commit?

Another issue with overcommit=on is that some programmers seem
to take for granted that "allocations will never fail" and so
neglect to handle malloc == NULL conditions gracefully.

I tried to run LTP with overcommit off, and I vaguely recall that
I had more failures than with overcommit on. (Perhaps only those
tests that tickle the dreaded OOM assassin.)

Regards.

[toc] | [prev] | [next] | [standalone]

#1400658

On Fri 13-05-16 10:44:30, Mason wrote:
> On 13/05/2016 10:04, Michal Hocko wrote:
> 
> > On Tue 10-05-16 13:56:30, Sebastian Frias wrote:
> > [...]
> >> NOTE: I understand that the overcommit mode can be changed dynamically thru
> >> sysctl, but on embedded systems, where we know in advance that overcommit
> >> will be disabled, there's no reason to postpone such setting.
> > 
> > To be honest I am not particularly happy about yet another config
> > option. At least not without a strong reason (the one above doesn't
> > sound that way). The config space is really large already.
> > So why a later initialization matters at all? Early userspace shouldn't
> > consume too much address space to blow up later, no?
> 
> One thing I'm not quite clear on is: why was the default set
> to over-commit on?

Because many applications simply rely on large and sparsely used address
space, I guess. That's why the default is GUESS where we ignore the
cumulative charges and simply check the current state and blow up only
when the current request is way too large.
-- 
Michal Hocko
SUSE Labs

[toc] | [prev] | [next] | [standalone]

#1400681

On 13/05/2016 11:52, Michal Hocko wrote:
> On Fri 13-05-16 10:44:30, Mason wrote:
>> On 13/05/2016 10:04, Michal Hocko wrote:
>>
>>> On Tue 10-05-16 13:56:30, Sebastian Frias wrote:
>>> [...]
>>>> NOTE: I understand that the overcommit mode can be changed dynamically thru
>>>> sysctl, but on embedded systems, where we know in advance that overcommit
>>>> will be disabled, there's no reason to postpone such setting.
>>>
>>> To be honest I am not particularly happy about yet another config
>>> option. At least not without a strong reason (the one above doesn't
>>> sound that way). The config space is really large already.
>>> So why a later initialization matters at all? Early userspace shouldn't
>>> consume too much address space to blow up later, no?
>>
>> One thing I'm not quite clear on is: why was the default set
>> to over-commit on?
> 
> Because many applications simply rely on large and sparsely used address
> space, I guess.

What kind of applications are we talking about here?

Server apps? Client apps? Supercomputer apps?

I heard some HPC software use large sparse matrices, but is it a common
idiom to request large allocations, only to use a fraction of it?

If you'll excuse the slight trolling, I'm sure many applications don't
expect being randomly zapped by the OOM killer ;-)

> That's why the default is GUESS where we ignore the cumulative
> charges and simply check the current state and blow up only when
> the current request is way too large.

I wouldn't call denying a request "blowing up". Application will
receive NULL, and is supposed to handle it gracefully.

"Blowing up" is receiving SIGKILL because another process happened
to allocate too much memory.

Regards.

[toc] | [prev] | [next] | [standalone]

#1400696

Hi,

On 05/13/2016 12:18 PM, Mason wrote:
> On 13/05/2016 11:52, Michal Hocko wrote:
>> On Fri 13-05-16 10:44:30, Mason wrote:
>>> On 13/05/2016 10:04, Michal Hocko wrote:
>>>
>>>> On Tue 10-05-16 13:56:30, Sebastian Frias wrote:
>>>> [...]
>>>>> NOTE: I understand that the overcommit mode can be changed dynamically thru
>>>>> sysctl, but on embedded systems, where we know in advance that overcommit
>>>>> will be disabled, there's no reason to postpone such setting.
>>>>
>>>> To be honest I am not particularly happy about yet another config
>>>> option. At least not without a strong reason (the one above doesn't
>>>> sound that way). The config space is really large already.
>>>> So why a later initialization matters at all? Early userspace shouldn't
>>>> consume too much address space to blow up later, no?
>>>
>>> One thing I'm not quite clear on is: why was the default set
>>> to over-commit on?
>>
>> Because many applications simply rely on large and sparsely used address
>> space, I guess.
> 
> What kind of applications are we talking about here?
> 
> Server apps? Client apps? Supercomputer apps?
> 
> I heard some HPC software use large sparse matrices, but is it a common
> idiom to request large allocations, only to use a fraction of it?
> 

Let's say there are specific applications that require overcommit.
Shouldn't overcommit be changed for those specific circumstances?
In other words, why is overcommit=GUESS default for everybody?

> If you'll excuse the slight trolling, I'm sure many applications don't
> expect being randomly zapped by the OOM killer ;-)
> 
>> That's why the default is GUESS where we ignore the cumulative
>> charges and simply check the current state and blow up only when
>> the current request is way too large.
> 
> I wouldn't call denying a request "blowing up". Application will
> receive NULL, and is supposed to handle it gracefully.
> 
> "Blowing up" is receiving SIGKILL because another process happened
> to allocate too much memory.

I agree.
Furthermore, the "blow up when the current request is too large" is more complex than that due to delay between the allocation and the time when the system realises it cannot honour the promise, there must be lots of code/heuristics involved there.
Anyway, it'd be nice to understand the real history behind overcommit (as I stated earlier, my understanding of the history is that in the early days there was no overcommit) and why it is there by default if only specific applications would require it.

Best regards,

Sebastian

[toc] | [prev] | [next] | [standalone]

#1400723

On Fri 13-05-16 12:18:54, Mason wrote:
> On 13/05/2016 11:52, Michal Hocko wrote:
> > On Fri 13-05-16 10:44:30, Mason wrote:
> >> On 13/05/2016 10:04, Michal Hocko wrote:
> >>
> >>> On Tue 10-05-16 13:56:30, Sebastian Frias wrote:
> >>> [...]
> >>>> NOTE: I understand that the overcommit mode can be changed dynamically thru
> >>>> sysctl, but on embedded systems, where we know in advance that overcommit
> >>>> will be disabled, there's no reason to postpone such setting.
> >>>
> >>> To be honest I am not particularly happy about yet another config
> >>> option. At least not without a strong reason (the one above doesn't
> >>> sound that way). The config space is really large already.
> >>> So why a later initialization matters at all? Early userspace shouldn't
> >>> consume too much address space to blow up later, no?
> >>
> >> One thing I'm not quite clear on is: why was the default set
> >> to over-commit on?
> > 
> > Because many applications simply rely on large and sparsely used address
> > space, I guess.
> 
> What kind of applications are we talking about here?
> 
> Server apps? Client apps? Supercomputer apps?

It is all over the place. But some are worse than others (e.g. try to
run some larger java application).

Anyway, this is my laptop where I do not run anything really special
(xfce, browser, few consoles, git, mutt):
$ grep Commit /proc/meminfo 
CommitLimit:     3497288 kB
Committed_AS:    3560804 kB

I am running with the default overcommit setup so I do not care about
the limit but the Committed_AS will tell you how much is actually
committed. I am definitelly not out of memory:
$ free
              total        used        free      shared  buff/cache   available
Mem:        3922584     1724120      217336      105264     1981128     2036164
Swap:       1535996      386364     1149632

If you check the rss/vsize ratio of your processes (which is not precise
but give at least some clue) then you will see that I am quite below 10% on
my system in average:
$ ps -ao vsize,rss -ax | awk '{if ($1+0>0) printf "%d\n", $2*100/$1 }' | calc_min_max.awk 
min: 0.00 max: 44.00 avg: 6.16 std: 7.85 nr: 120

> I heard some HPC software use large sparse matrices, but is it a common
> idiom to request large allocations, only to use a fraction of it?
> 
> If you'll excuse the slight trolling, I'm sure many applications don't
> expect being randomly zapped by the OOM killer ;-)

No, neither banks (and their customers) are prepared for a default
aren't they ;).

But more seriously. Overcommit is simply a reality these days. It would
be quite naive to think that enabling the overcommit protection would
guarantee that no OOM will trigger. The kernel can consume a lot of
memory as well which might be unreclaimable.
 
> > That's why the default is GUESS where we ignore the cumulative
> > charges and simply check the current state and blow up only when
> > the current request is way too large.
> 
> I wouldn't call denying a request "blowing up". Application will
> receive NULL, and is supposed to handle it gracefully.

Sure they will handle ENOMEM (in better case) but in reality it would
basically mean that they will fail eventually because there is hardly a
fallback. And it really sucks to fail with "Not enough memory" when you
check and your memory is mostly free/reclaimable (see the example above
from my running system).

-- 
Michal Hocko
SUSE Labs

[toc] | [prev] | [next] | [standalone]

#1400750

On 13/05/2016 13:44, Michal Hocko wrote:

> Anyway, this is my laptop where I do not run anything really special
> (xfce, browser, few consoles, git, mutt):
> $ grep Commit /proc/meminfo
> CommitLimit:     3497288 kB
> Committed_AS:    3560804 kB
> 
> I am running with the default overcommit setup so I do not care about
> the limit but the Committed_AS will tell you how much is actually
> committed. I am definitelly not out of memory:
> $ free
>               total        used        free      shared  buff/cache   available
> Mem:        3922584     1724120      217336      105264     1981128     2036164
> Swap:       1535996      386364     1149632

I see. Thanks for the data point.

I had a different type of system in mind.
256 to 512 MB of RAM, no swap.
Perhaps Sebastian's choice could be made to depend on CONFIG_EMBEDDED,
rather than CONFIG_EXPERT?

Regards.

[toc] | [prev] | [next] | [standalone]

#1400818

On Fri 13-05-16 14:15:35, Mason wrote:
> On 13/05/2016 13:44, Michal Hocko wrote:
> 
> > Anyway, this is my laptop where I do not run anything really special
> > (xfce, browser, few consoles, git, mutt):
> > $ grep Commit /proc/meminfo
> > CommitLimit:     3497288 kB
> > Committed_AS:    3560804 kB
> > 
> > I am running with the default overcommit setup so I do not care about
> > the limit but the Committed_AS will tell you how much is actually
> > committed. I am definitelly not out of memory:
> > $ free
> >               total        used        free      shared  buff/cache   available
> > Mem:        3922584     1724120      217336      105264     1981128     2036164
> > Swap:       1535996      386364     1149632
> 
> I see. Thanks for the data point.
> 
> I had a different type of system in mind.
> 256 to 512 MB of RAM, no swap.
> Perhaps Sebastian's choice could be made to depend on CONFIG_EMBEDDED,
> rather than CONFIG_EXPERT?

Even if the overcommit behavior is different on those systems the
primary question hasn't been answered yet. Why cannot this be done from
the userspace? In other words what wouldn't work properly?

-- 
Michal Hocko
SUSE Labs

[toc] | [prev] | [next] | [standalone]

#1400821

Hi Michal,

On 05/13/2016 04:01 PM, Michal Hocko wrote:
> On Fri 13-05-16 14:15:35, Mason wrote:
>> On 13/05/2016 13:44, Michal Hocko wrote:
>>
>>> Anyway, this is my laptop where I do not run anything really special
>>> (xfce, browser, few consoles, git, mutt):
>>> $ grep Commit /proc/meminfo
>>> CommitLimit:     3497288 kB
>>> Committed_AS:    3560804 kB
>>>
>>> I am running with the default overcommit setup so I do not care about
>>> the limit but the Committed_AS will tell you how much is actually
>>> committed. I am definitelly not out of memory:
>>> $ free
>>>               total        used        free      shared  buff/cache   available
>>> Mem:        3922584     1724120      217336      105264     1981128     2036164
>>> Swap:       1535996      386364     1149632
>>
>> I see. Thanks for the data point.
>>
>> I had a different type of system in mind.
>> 256 to 512 MB of RAM, no swap.
>> Perhaps Sebastian's choice could be made to depend on CONFIG_EMBEDDED,
>> rather than CONFIG_EXPERT?
> 
> Even if the overcommit behavior is different on those systems the
> primary question hasn't been answered yet. Why cannot this be done from
> the userspace? In other words what wouldn't work properly?
> 

You are right, and I said that since the beginning, nothing prevents the userspace from doing it.

But it'd be interesting to know the history of this option, for example, why it is left for userspace.
Are there systems that dynamically change this setting?

Best regards,

Sebastian

[toc] | [prev] | [next] | [standalone]

#1400848

> > Perhaps Sebastian's choice could be made to depend on CONFIG_EMBEDDED,
> > rather than CONFIG_EXPERT?  
> 
> Even if the overcommit behavior is different on those systems the
> primary question hasn't been answered yet. Why cannot this be done from
> the userspace? In other words what wouldn't work properly?

Most allocations in C have no mechanism to report failure.

Stakc expansion failure is not reportable. Copy on write failure is not
reportable and so on.

Alan
 

[toc] | [prev] | [next] | [standalone]

#1400871

Hi Alan,

On 05/13/2016 05:04 PM, One Thousand Gnomes wrote:
>>> Perhaps Sebastian's choice could be made to depend on CONFIG_EMBEDDED,
>>> rather than CONFIG_EXPERT?  
>>
>> Even if the overcommit behavior is different on those systems the
>> primary question hasn't been answered yet. Why cannot this be done from
>> the userspace? In other words what wouldn't work properly?
> 
> Most allocations in C have no mechanism to report failure.
> 
> Stakc expansion failure is not reportable. Copy on write failure is not
> reportable and so on.

But wouldn't those affect a given process at at time?
Does that means that the OOM-killer is woken up to kill process X when those situations arise on process Y?

Also, under what conditions would copy-on-write fail?

Best regards,

Sebastian

[toc] | [prev] | [next] | [standalone]

#1400872

> But wouldn't those affect a given process at at time?
> Does that means that the OOM-killer is woken up to kill process X when those situations arise on process Y?

Not sure I understand the question.

> Also, under what conditions would copy-on-write fail?

When you have no memory or swap pages free and you touch a COW page that
is currently shared. At that point there is no resource to back to the
copy so something must die - either the process doing the copy or
something else.

Alan

[toc] | [prev] | [next] | [standalone]

#1402193

Hi Alan,

On 05/13/2016 05:43 PM, One Thousand Gnomes wrote:
>> But wouldn't those affect a given process at at time?
>> Does that means that the OOM-killer is woken up to kill process X when those situations arise on process Y?
> 
> Not sure I understand the question.

I'm sorry for the "at at time" typo.
What I meant was that situations you described "Stakc expansion failure is not reportable. Copy on write failure is not reportable and so on.", should affect one process at the time, in that case:
1) either process X with the COW failure happens could die
2) either random process Y dies so that COW failure on process X can be handled.

Do you know why was 2) chosen over 1)?

> 
>> Also, under what conditions would copy-on-write fail?
> 
> When you have no memory or swap pages free and you touch a COW page that
> is currently shared. At that point there is no resource to back to the
> copy so something must die - either the process doing the copy or
> something else.

Exactly, and why does "killing something else" makes more sense (or was chosen over) "killing the process doing the copy"?

Best regards,

Sebastian

[toc] | [prev] | [next] | [standalone]

#1402205

On Tue 17-05-16 10:24:20, Sebastian Frias wrote:
[...]
> >> Also, under what conditions would copy-on-write fail?
> > 
> > When you have no memory or swap pages free and you touch a COW page that
> > is currently shared. At that point there is no resource to back to the
> > copy so something must die - either the process doing the copy or
> > something else.
> 
> Exactly, and why does "killing something else" makes more sense (or
> was chosen over) "killing the process doing the copy"?

Because that "something else" is usually a memory hog and so chances are
that the out of memory situation will get resolved. If you kill "process
doing the copy" then you might end up just not getting any memory back
because that might be a little forked process which doesn't own all that
much memory on its own. That would leave you in the oom situation for a
long time until somebody actually sitting on some memory happens to ask
for CoW... See the difference?
-- 
Michal Hocko
SUSE Labs

[toc] | [prev] | [next] | [standalone]

#1402450

Hi Michal,

On 05/17/2016 10:57 AM, Michal Hocko wrote:
> On Tue 17-05-16 10:24:20, Sebastian Frias wrote:
> [...]
>>>> Also, under what conditions would copy-on-write fail?
>>>
>>> When you have no memory or swap pages free and you touch a COW page that
>>> is currently shared. At that point there is no resource to back to the
>>> copy so something must die - either the process doing the copy or
>>> something else.
>>
>> Exactly, and why does "killing something else" makes more sense (or
>> was chosen over) "killing the process doing the copy"?
> 
> Because that "something else" is usually a memory hog and so chances are
> that the out of memory situation will get resolved. If you kill "process
> doing the copy" then you might end up just not getting any memory back
> because that might be a little forked process which doesn't own all that
> much memory on its own. That would leave you in the oom situation for a
> long time until somebody actually sitting on some memory happens to ask
> for CoW... See the difference?
> 

I see the difference, your answer seems a bit like the one from Austin, basically:
- killing a process is a sort of kernel protection attempting to deal "automatically" with some situation, like deciding what is a 'memory hog', or what is 'in infinite loop', "usually" in a correct way.
It seems there's people who think its better to avoid having to take such decisions and/or they should be decided by the user, because "usually" != "always".
And people who see that as a nice thing but complex thing to do.
In this thread we've tried to explain why this heuristic (and/or OOM-killer) is/was needed and/or its history, which has been very enlightening by the way.

From reading Documentation/cgroup-v1/memory.txt (and from a few replies here talking about cgroups), it looks like the OOM-killer is still being actively discussed, well, there's also "cgroup-v2".
My understanding is that cgroup's memory control will pause processes in a given cgroup until the OOM situation is solved for that cgroup, right?
If that is right, it means that there is indeed a way to deal with an OOM situation (stack expansion, COW failure, 'memory hog', etc.) in a better way than the OOM-killer, right?
In which case, do you guys know if there is a way to make the whole system behave as if it was inside a cgroup? (*)

Best regards,

Sebastian


(*): I tried setting up a simple test but failed, so I think I need more reading :-)

[toc] | [prev] | [next] | [standalone]

#1402484

On 2016-05-17 12:16, Sebastian Frias wrote:
> Hi Michal,
>
> On 05/17/2016 10:57 AM, Michal Hocko wrote:
>> On Tue 17-05-16 10:24:20, Sebastian Frias wrote:
>> [...]
>>>>> Also, under what conditions would copy-on-write fail?
>>>>
>>>> When you have no memory or swap pages free and you touch a COW page that
>>>> is currently shared. At that point there is no resource to back to the
>>>> copy so something must die - either the process doing the copy or
>>>> something else.
>>>
>>> Exactly, and why does "killing something else" makes more sense (or
>>> was chosen over) "killing the process doing the copy"?
>>
>> Because that "something else" is usually a memory hog and so chances are
>> that the out of memory situation will get resolved. If you kill "process
>> doing the copy" then you might end up just not getting any memory back
>> because that might be a little forked process which doesn't own all that
>> much memory on its own. That would leave you in the oom situation for a
>> long time until somebody actually sitting on some memory happens to ask
>> for CoW... See the difference?
>>
>
> I see the difference, your answer seems a bit like the one from Austin, basically:
> - killing a process is a sort of kernel protection attempting to deal "automatically" with some situation, like deciding what is a 'memory hog', or what is 'in infinite loop', "usually" in a correct way.
> It seems there's people who think its better to avoid having to take such decisions and/or they should be decided by the user, because "usually" != "always".
FWIW, it's really easy to see what's using a lot of memory, it's 
impossible to tell if something is stuck in an infinite loop without 
looking deep into the process state and possibly even at the source code 
(and even then it can be almost impossible to be certain).  This is why 
we have a OOM-Killer, and not a infinite-loop-killer.

Again I reiterate, if a system is properly provisioned (that is, if you 
have put in enough RAM and possibly swap space to do what you want to 
use it for), the only reason the OOM-killer should be invoked is due to 
a bug.  The non-default overcommit options still have the same issues 
they just change how and when they happen (overcommit=never will fire 
sooner, overcommit=always will fire later), and also can impact memory 
allocation performance (I have numbers somewhere that I can't find right 
now that demonstrated that overcommit=never gave more deterministic and 
(on average) marginally better malloc() performance, and simple logic 
would suggest that overcommit=always would make malloc() perform better 
too).
> And people who see that as a nice thing but complex thing to do.
> In this thread we've tried to explain why this heuristic (and/or OOM-killer) is/was needed and/or its history, which has been very enlightening by the way.
>
> From reading Documentation/cgroup-v1/memory.txt (and from a few replies here talking about cgroups), it looks like the OOM-killer is still being actively discussed, well, there's also "cgroup-v2".
> My understanding is that cgroup's memory control will pause processes in a given cgroup until the OOM situation is solved for that cgroup, right?
> If that is right, it means that there is indeed a way to deal with an OOM situation (stack expansion, COW failure, 'memory hog', etc.) in a better way than the OOM-killer, right?
> In which case, do you guys know if there is a way to make the whole system behave as if it was inside a cgroup? (*)
No, not with the process freeze behavior, because getting the group 
running again requires input from an external part of the system, which 
by definition doesn't exist if the group is the entire system; and, 
because our GUI isn't built into the kernel, we can't pause things and 
pop up a little dialog asking the user what to do to resolve the issue.

[toc] | [prev] | [next] | [standalone]

#1403071

Hi Austin,

On 05/17/2016 07:29 PM, Austin S. Hemmelgarn wrote:
>> I see the difference, your answer seems a bit like the one from Austin, basically:
>> - killing a process is a sort of kernel protection attempting to deal "automatically" with some situation, like deciding what is a 'memory hog', or what is 'in infinite loop', "usually" in a correct way.
>> It seems there's people who think its better to avoid having to take such decisions and/or they should be decided by the user, because "usually" != "always".
> FWIW, it's really easy to see what's using a lot of memory, it's impossible to tell if something is stuck in an infinite loop without looking deep into the process state and possibly even at the source code (and even then it can be almost impossible to be certain).  This is why we have a OOM-Killer, and not a infinite-loop-killer.
> 
> Again I reiterate, if a system is properly provisioned (that is, if you have put in enough RAM and possibly swap space to do what you want to use it for), the only reason the OOM-killer should be invoked is due to a bug. 

Are you sure that's the only possible reason?
I mean, what if somebody keeps opening tabs on Firefox?
If malloc() returned NULL maybe Firefox could say "hey, you have too many tabs open, please close some to free memory".

> The non-default overcommit options still have the same issues they just change how and when they happen (overcommit=never will fire sooner, overcommit=always will fire later), and also can impact memory allocation performance (I have numbers somewhere that I can't find right now that demonstrated that overcommit=never gave more deterministic and (on average) marginally better malloc() performance, and simple logic would suggest that overcommit=always would make malloc() perform better too).
>> And people who see that as a nice thing but complex thing to do.
>> In this thread we've tried to explain why this heuristic (and/or OOM-killer) is/was needed and/or its history, which has been very enlightening by the way.
>>
>> From reading Documentation/cgroup-v1/memory.txt (and from a few replies here talking about cgroups), it looks like the OOM-killer is still being actively discussed, well, there's also "cgroup-v2".
>> My understanding is that cgroup's memory control will pause processes in a given cgroup until the OOM situation is solved for that cgroup, right?
>> If that is right, it means that there is indeed a way to deal with an OOM situation (stack expansion, COW failure, 'memory hog', etc.) in a better way than the OOM-killer, right?
>> In which case, do you guys know if there is a way to make the whole system behave as if it was inside a cgroup? (*)
> No, not with the process freeze behavior, because getting the group running again requires input from an external part of the system, which by definition doesn't exist if the group is the entire system; 

Do you mean that it pauses all processes in the cgroup?
I thought it would pause on a case-by-case basis, like first process to reach the limit gets paused, and so on.

Honestly I thought it would work a bit like the filesystems, where 'root' usually has 5% reserved, so that a process (or processes) filling the disk does not disrupt the system to the point of preventing 'root' from performing administrative actions.

That makes me think, why is disk space handled differently than memory in this case? I mean, why is disk space exhaustion handled differently than memory exhaustion?
We could imagine that both resources are required for proper system and process operation, so if OOM-killer is there to attempt to keep the system working at all costs (even if that means sacrificing processes), why isn't there an OOFS-killer (out-of-free-space killer)?

>and, because our GUI isn't built into the kernel, we can't pause things and pop up a little dialog asking the user what to do to resolve the issue.

:-) Yeah, I was thinking that could be handled with the cgroups' notification system + the reserved space (like on filesystems)
Maybe I was too optimistic (naive or just plain ignorant) about this.

Best regards,

Sebastian

[toc] | [prev] | [next] | [standalone]

#1403138

On 2016-05-18 11:19, Sebastian Frias wrote:
> Hi Austin,
>
> On 05/17/2016 07:29 PM, Austin S. Hemmelgarn wrote:
>>> I see the difference, your answer seems a bit like the one from Austin, basically:
>>> - killing a process is a sort of kernel protection attempting to deal "automatically" with some situation, like deciding what is a 'memory hog', or what is 'in infinite loop', "usually" in a correct way.
>>> It seems there's people who think its better to avoid having to take such decisions and/or they should be decided by the user, because "usually" != "always".
>> FWIW, it's really easy to see what's using a lot of memory, it's impossible to tell if something is stuck in an infinite loop without looking deep into the process state and possibly even at the source code (and even then it can be almost impossible to be certain).  This is why we have a OOM-Killer, and not a infinite-loop-killer.
>>
>> Again I reiterate, if a system is properly provisioned (that is, if you have put in enough RAM and possibly swap space to do what you want to use it for), the only reason the OOM-killer should be invoked is due to a bug.
>
> Are you sure that's the only possible reason?
> I mean, what if somebody keeps opening tabs on Firefox?
> If malloc() returned NULL maybe Firefox could say "hey, you have too many tabs open, please close some to free memory".
That's an application issue, and I'm pretty sure that most browsers do 
mention this.  That also falls within normal usage for a desktop system 
(somewhat, if you're opening more than a few dozen tabs, you're asking 
for trouble for other reasons too).
>
>> The non-default overcommit options still have the same issues they just change how and when they happen (overcommit=never will fire sooner, overcommit=always will fire later), and also can impact memory allocation performance (I have numbers somewhere that I can't find right now that demonstrated that overcommit=never gave more deterministic and (on average) marginally better malloc() performance, and simple logic would suggest that overcommit=always would make malloc() perform better too).
>>> And people who see that as a nice thing but complex thing to do.
>>> In this thread we've tried to explain why this heuristic (and/or OOM-killer) is/was needed and/or its history, which has been very enlightening by the way.
>>>
>>> From reading Documentation/cgroup-v1/memory.txt (and from a few replies here talking about cgroups), it looks like the OOM-killer is still being actively discussed, well, there's also "cgroup-v2".
>>> My understanding is that cgroup's memory control will pause processes in a given cgroup until the OOM situation is solved for that cgroup, right?
>>> If that is right, it means that there is indeed a way to deal with an OOM situation (stack expansion, COW failure, 'memory hog', etc.) in a better way than the OOM-killer, right?
>>> In which case, do you guys know if there is a way to make the whole system behave as if it was inside a cgroup? (*)
>> No, not with the process freeze behavior, because getting the group running again requires input from an external part of the system, which by definition doesn't exist if the group is the entire system;
>
> Do you mean that it pauses all processes in the cgroup?
> I thought it would pause on a case-by-case basis, like first process to reach the limit gets paused, and so on.
>
> Honestly I thought it would work a bit like the filesystems, where 'root' usually has 5% reserved, so that a process (or processes) filling the disk does not disrupt the system to the point of preventing 'root' from performing administrative actions.
>
> That makes me think, why is disk space handled differently than memory in this case? I mean, why is disk space exhaustion handled differently than memory exhaustion?
> We could imagine that both resources are required for proper system and process operation, so if OOM-killer is there to attempt to keep the system working at all costs (even if that means sacrificing processes), why isn't there an OOFS-killer (out-of-free-space killer)?
There are actually sysctl's for this, vm/{admin,user}_reserve_kbytes. 
The admin one is system-wide and provides a reserve for users with 
CAP_SYS_ADMIN.  The user one is per-process and prevents a process from 
allocating beyond a specific point, and is intended for overcommit=never 
mode.

That said, there are a couple of reasons that disk space and memory are 
handled differently:
1. The kernel needs RAM to function, it does not need disk space to 
function.  In other words, if we have no free RAM, the system is 
guaranteed to be unusable, but if we have no disk space, the system may 
or may not still be usable.
2. Freeing disk space is usually an easy decision for the user, figuring 
out what to kill to free RAM is not.
3. Most end users have at least a basic understanding of disk space 
being finite, while they don't necessarily have a similar understanding 
of memory being finite (note that I'm not talking about sysadmins and 
similar, I"m talking about people's grandmothers, and people who have no 
low-level background with computers, and people like some of my friends 
who still have trouble understanding the difference between memory and 
persistent storage)
>
>> and, because our GUI isn't built into the kernel, we can't pause things and pop up a little dialog asking the user what to do to resolve the issue.
>
> :-) Yeah, I was thinking that could be handled with the cgroups' notification system + the reserved space (like on filesystems)
> Maybe I was too optimistic (naive or just plain ignorant) about this.
Ideally, we would have something that could check against some watermark 
and notify like Windows does when virtual memory is getting low (most 
people never see this, because they let windows manage the page file, 
which means it just gleefully allocates whatever it needs on disk).  I 
don't know of a way to do that right now without polling though, and 
that level of inefficiency should ideally be avoided.

[toc] | [prev] | [next] | [standalone]

#1402571

On Tue 17-05-16 18:16:58, Sebastian Frias wrote:
[...]
> From reading Documentation/cgroup-v1/memory.txt (and from a few
> replies here talking about cgroups), it looks like the OOM-killer is
> still being actively discussed, well, there's also "cgroup-v2".
> My understanding is that cgroup's memory control will pause processes
> in a given cgroup until the OOM situation is solved for that cgroup,
> right?

It will be blocked waiting either for some external action which would
result in OOM codition going away or any other charge release. You have
to configure memcg for that though. The default behavior is to invoke
the same OOM killer algorithm which is just reduced to tasks from the
memcg (hierarchy).

> If that is right, it means that there is indeed a way to deal
> with an OOM situation (stack expansion, COW failure, 'memory hog',
> etc.) in a better way than the OOM-killer, right?
> In which case, do you guys know if there is a way to make the whole
> system behave as if it was inside a cgroup? (*)

No it is not. You have to realize that the system wide and the memcg OOM
situations are quite different. There is usually quite some memory free
when you hit the memcg OOM so the administrator can actually do
something. The global OOM means there is _no_ memory at all. Many kernel
operations will need some memory to do something useful. Let's say you
would want to do an educated guess about who to kill - most proc APIs
will need to allocate. And this is just a beginning. Things are getting
really nasty when you get deeper and deeper. E.g. the OOM killer has to
give the oom victim access to memory reserves so that the task can exit
because that path needs to allocate as well. So even if you wanted to
give userspace some chance to resolve the OOM situation you would either
need some special API to tell "this process is really special and it can
access memory reserves and it has an absolute priority etc." or have a
in kernel fallback to do something or your system could lockup really
easily.
-- 
Michal Hocko
SUSE Labs

[toc] | [prev] | [next] | [standalone]

#1403073

Hi Michal,

On 05/17/2016 10:16 PM, Michal Hocko wrote:
> On Tue 17-05-16 18:16:58, Sebastian Frias wrote:
> [...]
>> From reading Documentation/cgroup-v1/memory.txt (and from a few
>> replies here talking about cgroups), it looks like the OOM-killer is
>> still being actively discussed, well, there's also "cgroup-v2".
>> My understanding is that cgroup's memory control will pause processes
>> in a given cgroup until the OOM situation is solved for that cgroup,
>> right?
> 
> It will be blocked waiting either for some external action which would
> result in OOM codition going away or any other charge release. You have
> to configure memcg for that though. The default behavior is to invoke
> the same OOM killer algorithm which is just reduced to tasks from the
> memcg (hierarchy).

Ok, I see, thanks!

> 
>> If that is right, it means that there is indeed a way to deal
>> with an OOM situation (stack expansion, COW failure, 'memory hog',
>> etc.) in a better way than the OOM-killer, right?
>> In which case, do you guys know if there is a way to make the whole
>> system behave as if it was inside a cgroup? (*)
> 
> No it is not. You have to realize that the system wide and the memcg OOM
> situations are quite different. There is usually quite some memory free
> when you hit the memcg OOM so the administrator can actually do
> something. 

Ok, so it works like the 5% reserved for 'root' on filesystems?

>The global OOM means there is _no_ memory at all. Many kernel
> operations will need some memory to do something useful. Let's say you
> would want to do an educated guess about who to kill - most proc APIs
> will need to allocate. And this is just a beginning. Things are getting
> really nasty when you get deeper and deeper. E.g. the OOM killer has to
> give the oom victim access to memory reserves so that the task can exit
> because that path needs to allocate as well. 

Really? I would have thought that once that SIGKILL is sent, the victim process is not expected to do anything else and thus its memory could be claimed immediately.
Or the OOM-killer is more of a OOM-terminator? (i.e.: sends SIGTERM)

>So even if you wanted to
> give userspace some chance to resolve the OOM situation you would either
> need some special API to tell "this process is really special and it can
> access memory reserves and it has an absolute priority etc." or have a
> in kernel fallback to do something or your system could lockup really
> easily.
> 

I see, so basically at least two cgroups would be needed, one reserved for handling the OOM situation through some API and another for the "rest of the system".
Basically just like the 5% reserved for 'root' on filesystems.
Do you think that would work?

Best regards,

Sebastian

[toc] | [prev] | [next] | [standalone]

Page 1 of 3  [1] 2 3  Next page →

Back to top | Article view | linux.kernel

csiph-web