Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1400594 > unrolled thread

Re: [PATCH] mm: add config option to select the initial overcommit mode

Started byMichal Hocko <mhocko@kernel.org>
First post2016-05-13 10:10 +0200
Last post2016-05-23 15:20 +0200
Articles 7 on this page of 47 — 5 participants

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH] mm: add config option to select the initial overcommit  mode Michal Hocko <mhocko@kernel.org> - 2016-05-13 10:10 +0200
    Re: [PATCH] mm: add config option to select the initial overcommit  mode Mason <slash.tmp@free.fr> - 2016-05-13 10:50 +0200
      Re: [PATCH] mm: add config option to select the initial overcommit  mode Michal Hocko <mhocko@kernel.org> - 2016-05-13 12:00 +0200
        Re: [PATCH] mm: add config option to select the initial overcommit  mode Mason <slash.tmp@free.fr> - 2016-05-13 12:20 +0200
          Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-13 12:50 +0200
          Re: [PATCH] mm: add config option to select the initial overcommit  mode Michal Hocko <mhocko@kernel.org> - 2016-05-13 13:50 +0200
            Re: [PATCH] mm: add config option to select the initial overcommit  mode Mason <slash.tmp@free.fr> - 2016-05-13 14:20 +0200
              Re: [PATCH] mm: add config option to select the initial overcommit  mode Michal Hocko <mhocko@kernel.org> - 2016-05-13 16:10 +0200
                Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-13 16:20 +0200
                Re: [PATCH] mm: add config option to select the initial overcommit  mode One Thousand Gnomes <gnomes@lxorguk.ukuu.org.uk> - 2016-05-13 17:10 +0200
                  Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-13 17:40 +0200
                    Re: [PATCH] mm: add config option to select the initial overcommit  mode One Thousand Gnomes <gnomes@lxorguk.ukuu.org.uk> - 2016-05-13 17:50 +0200
                      Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-17 10:30 +0200
                        Re: [PATCH] mm: add config option to select the initial overcommit  mode Michal Hocko <mhocko@kernel.org> - 2016-05-17 11:00 +0200
                          Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-17 18:20 +0200
                            Re: [PATCH] mm: add config option to select the initial overcommit  mode "Austin S. Hemmelgarn" <ahferroin7@gmail.com> - 2016-05-17 19:30 +0200
                              Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-18 17:20 +0200
                                Re: [PATCH] mm: add config option to select the initial overcommit  mode "Austin S. Hemmelgarn" <ahferroin7@gmail.com> - 2016-05-18 18:30 +0200
                            Re: [PATCH] mm: add config option to select the initial overcommit  mode Michal Hocko <mhocko@kernel.org> - 2016-05-17 22:20 +0200
                              Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-18 17:20 +0200
                                Re: [PATCH] mm: add config option to select the initial overcommit  mode Michal Hocko <mhocko@kernel.org> - 2016-05-19 09:20 +0200
                    Re: [PATCH] mm: add config option to select the initial overcommit  mode "Austin S. Hemmelgarn" <ahferroin7@gmail.com> - 2016-05-13 19:10 +0200
          Re: [PATCH] mm: add config option to select the initial overcommit  mode "Austin S. Hemmelgarn" <ahferroin7@gmail.com> - 2016-05-13 15:30 +0200
      Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-13 12:00 +0200
        Re: [PATCH] mm: add config option to select the initial overcommit  mode Michal Hocko <mhocko@kernel.org> - 2016-05-13 14:10 +0200
          Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-13 14:40 +0200
            Re: [PATCH] mm: add config option to select the initial overcommit  mode "Austin S. Hemmelgarn" <ahferroin7@gmail.com> - 2016-05-13 15:20 +0200
              Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-13 15:40 +0200
                Re: [PATCH] mm: add config option to select the initial overcommit  mode "Austin S. Hemmelgarn" <ahferroin7@gmail.com> - 2016-05-13 16:20 +0200
                  Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-13 16:30 +0200
                    Re: [PATCH] mm: add config option to select the initial overcommit  mode "Austin S. Hemmelgarn" <ahferroin7@gmail.com> - 2016-05-13 17:10 +0200
                Re: [PATCH] mm: add config option to select the initial overcommit  mode One Thousand Gnomes <gnomes@lxorguk.ukuu.org.uk> - 2016-05-13 17:10 +0200
                  Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-13 17:20 +0200
                    Re: [PATCH] mm: add config option to select the initial overcommit  mode Michal Hocko <mhocko@kernel.org> - 2016-05-13 17:30 +0200
              Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-13 15:40 +0200
                Re: [PATCH] mm: add config option to select the initial overcommit  mode "Austin S. Hemmelgarn" <ahferroin7@gmail.com> - 2016-05-13 16:00 +0200
                  Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-13 16:40 +0200
                    Re: [PATCH] mm: add config option to select the initial overcommit  mode Michal Hocko <mhocko@kernel.org> - 2016-05-13 17:00 +0200
                    Re: [PATCH] mm: add config option to select the initial overcommit  mode "Austin S. Hemmelgarn" <ahferroin7@gmail.com> - 2016-05-13 17:20 +0200
            Re: [PATCH] mm: add config option to select the initial overcommit  mode Michal Hocko <mhocko@kernel.org> - 2016-05-13 17:00 +0200
              Re: [PATCH] mm: add config option to select the initial overcommit  mode Mason <slash.tmp@free.fr> - 2016-05-13 17:00 +0200
                Re: [PATCH] mm: add config option to select the initial overcommit  mode One Thousand Gnomes <gnomes@lxorguk.ukuu.org.uk> - 2016-05-13 17:20 +0200
                  Re: [PATCH] mm: add config option to select the initial overcommit  mode Michal Hocko <mhocko@kernel.org> - 2016-05-13 17:30 +0200
                  Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-13 17:40 +0200
              Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-13 17:20 +0200
                Re: [PATCH] mm: add config option to select the initial overcommit  mode One Thousand Gnomes <gnomes@lxorguk.ukuu.org.uk> - 2016-05-13 17:50 +0200
                  Re: [PATCH] mm: add config option to select the initial overcommit  mode Sebastian Frias <sf84@laposte.net> - 2016-05-23 15:20 +0200

Page 3 of 3 — ← Prev page 1 2 [3]

#1400844

FromMason <slash.tmp@free.fr>
Date2016-05-13 17:00 +0200
Message-ID<rymJk-5r8-13@gated-at.bofh.it>
In reply to#1400843
On 13/05/2016 16:51, Michal Hocko wrote:

> The default should cover the most use cases. If you can prove that the
> vast majority of embedded systems are different and would _benefit_ from
> a different default I wouldn't be opposed to change the default there.

It seems important to point out that Sebastian's patch does NOT change
the default behavior. It merely creates a knob allowing one to override
the default via Kconfig.

+choice
+	prompt "Overcommit Mode"
+	default OVERCOMMIT_GUESS
+	depends on EXPERT

Regards.

[toc] | [prev] | [next] | [standalone]

#1400854

FromOne Thousand Gnomes <gnomes@lxorguk.ukuu.org.uk>
Date2016-05-13 17:20 +0200
Message-ID<ryn2G-68D-7@gated-at.bofh.it>
In reply to#1400844
> It seems important to point out that Sebastian's patch does NOT change
> the default behavior. It merely creates a knob allowing one to override
> the default via Kconfig.
> 
> +choice
> +	prompt "Overcommit Mode"
> +	default OVERCOMMIT_GUESS
> +	depends on EXPERT

Which is still completely pointless given that its a single sysctl value
set at early userspace time and most distributions ship with things like
sysctl and /etc/sysctl.conf

We have a million other such knobs, putting them in kconfig just gets
silly.

Alan

[toc] | [prev] | [next] | [standalone]

#1400864

FromMichal Hocko <mhocko@kernel.org>
Date2016-05-13 17:30 +0200
Message-ID<ryncm-6eB-5@gated-at.bofh.it>
In reply to#1400854
On Fri 13-05-16 16:11:04, One Thousand Gnomes wrote:
> > It seems important to point out that Sebastian's patch does NOT change
> > the default behavior. It merely creates a knob allowing one to override
> > the default via Kconfig.
> > 
> > +choice
> > +	prompt "Overcommit Mode"
> > +	default OVERCOMMIT_GUESS
> > +	depends on EXPERT
> 
> Which is still completely pointless given that its a single sysctl value
> set at early userspace time and most distributions ship with things like
> sysctl and /etc/sysctl.conf
> 
> We have a million other such knobs, putting them in kconfig just gets
> silly.

Exactly my point from the very begining. Thanks for being so direct
here.
-- 
Michal Hocko
SUSE Labs

[toc] | [prev] | [next] | [standalone]

#1400866

FromSebastian Frias <sf84@laposte.net>
Date2016-05-13 17:40 +0200
Message-ID<rynm1-6jr-5@gated-at.bofh.it>
In reply to#1400854
Hi Alan,

On 05/13/2016 05:11 PM, One Thousand Gnomes wrote:
>> It seems important to point out that Sebastian's patch does NOT change
>> the default behavior. It merely creates a knob allowing one to override
>> the default via Kconfig.
>>
>> +choice
>> +	prompt "Overcommit Mode"
>> +	default OVERCOMMIT_GUESS
>> +	depends on EXPERT
> 
> Which is still completely pointless given that its a single sysctl value
> set at early userspace time and most distributions ship with things like
> sysctl and /etc/sysctl.conf
> 

You are right, and I said that when the thread started, but I think most people here are looking at this from a server/desktop perspective.
Also, we wanted to have more background on this setting, its history, etc. thus this discussion.
It would be interesting in know what other people working on embedded systems think about this subject, because most examples given are for much bigger systems.

Best regards,

Sebastian

[toc] | [prev] | [next] | [standalone]

#1400860

FromSebastian Frias <sf84@laposte.net>
Date2016-05-13 17:20 +0200
Message-ID<ryn2G-68D-19@gated-at.bofh.it>
In reply to#1400843
Hi Michal,

On 05/13/2016 04:51 PM, Michal Hocko wrote:
> 
> The default should cover the most use cases. If you can prove that the
> vast majority of embeded systems are different and would _benefit_ from
> a different default I wouldn't be opposed to change the default there.

I'm unsure of a way to prove that.
I mean, what was the way used to prove that "the most use cases" is ok with overcommit=guess? It seems it was an empirical thing.

Also note that this is not changing any default.
It is merely adding the option to change the initial mode without relying on the userspace.

>> :-)
>> I see, so basically it is a sort of workaround.
> 
> No it is not a workaround. It is just serving the purpose of the
> operating system. The allow using the HW as much as possible to the
> existing userspace. You cannot expect userspace will change just because
> we do not like the overcommiting the memory with all the fallouts.

I agree, but that is one of the things that is fuzzy.
My understanding is that there was a time when there was no overcommit at all.
If that's the case, understanding why overcommit was introduced would be helpful.

>> Anyway, in the embedded world the memory and system requirements are
>> usually controlled.
> 
> OK, but even when it is controlled does it suffer in any way just
> because of the default setting? Do you see OOM killer invocation
> when the overcommit would prevent from that?

I'll have to check those LTP tests again, I'll come back to this question later then.

>> Would you agree to the option if it was dependent on
>> CONFIG_EMBEDDED? Or if it was a hidden option?
>> (I understand though that it wouldn't affect the size of config space)
> 
> It could be done in the code and make the default depending on the
> existing config. But first try to think about what would be an advantage
> of such a change.

:) Well, right now I'm just trying to understand the history of this setting, because it is not obvious why it is good.

>>
>> Well, mostly the history of this setting, why it was introduced, etc.
>> more or less what we are discussing here.  Because honestly, killing
>> random processes does not seems like a straightforward idea, ie: it is
>> not obvious.  Like I was saying, without context, such behaviour looks
>> a bit crazy.
> 
> But we are not killing a random process. The semantic is quite clear. We
> are trying to kill the biggest memory hog and if it has some children
> try to sacrifice them to save as much work as possible.

Ok.
That's not the impression I have considering in my case it killed terminals and editors, but I'll try to get some examples.

>>
>> Well, a more urgent problem would be that in that case
>> overcommit=never is not really well tested.
> 
> This is a problem of the userspace and am really skeptical that a change
> in default would make any existing bugs going away. It is more likely we
> will see reports that ENOMEM has been returned even though there is
> pletny of memory available.
> 

Again, I did not propose to change the default.
The idea was just to allow setting the initial overcommit mode in the kernel without relying on the userspace.
(also beause it is still not yet clear why it is left to the userspace)

>>
>> Well, it's hard to report, since it is essentially the result of a
>> dynamic system.
> 
> Each oom killer invocation will provide a detailed report which will
> help MM developers to debug what went wrong and why.
> 

Ok.

Best regards,

Sebastian

[toc] | [prev] | [next] | [standalone]

#1400873

FromOne Thousand Gnomes <gnomes@lxorguk.ukuu.org.uk>
Date2016-05-13 17:50 +0200
Message-ID<rynvH-6ok-5@gated-at.bofh.it>
In reply to#1400860
> My understanding is that there was a time when there was no overcommit at all.
> If that's the case, understanding why overcommit was introduced would be helpful.

Linux always had overcommit.

The origin of overcommit is virtual memory for the most part. In a
classic swapping system without VM the meaning of brk() and thus malloc()
is that it allocates memory (or swap). Likewise this is true of fork()
and stack extension.

In a virtual memory system these allocate _address space_. It does not
become populated except by page faulting, copy on write and the like. It
turns out that for most use cases on a virtual memory system we get huge
amounts of page sharing or untouched space.

Historically Linux did guess based overcommit and I added no overcommit
support way back when, along with 'anything is allowed' support for
certain HPC use cases.

The beancounter patches combined with this made the entire setup
completely robust but the beancounters never hit upstream although years
later they became part of the basis of the cgroups.

You can sort of set a current Linux up for definitely no overcommit using
cgroups and no overcommit settings. It works for most stuff although last
I checked most graphics drivers were terminally broken (and not just to
no overcommit but to the point you can remote DoS Linux boxes with a
suitably constructed web page and chrome browser)

Alan

[toc] | [prev] | [next] | [standalone]

#1405357

FromSebastian Frias <sf84@laposte.net>
Date2016-05-23 15:20 +0200
Message-ID<rBXW1-6lL-9@gated-at.bofh.it>
In reply to#1400873
Hi Alan,

On 05/13/2016 05:41 PM, One Thousand Gnomes wrote:
>> My understanding is that there was a time when there was no overcommit at all.
>> If that's the case, understanding why overcommit was introduced would be helpful.
> 
> Linux always had overcommit.
> 
> The origin of overcommit is virtual memory for the most part. In a
> classic swapping system without VM the meaning of brk() and thus malloc()
> is that it allocates memory (or swap). Likewise this is true of fork()
> and stack extension.
> 
> In a virtual memory system these allocate _address space_. It does not
> become populated except by page faulting, copy on write and the like. It
> turns out that for most use cases on a virtual memory system we get huge
> amounts of page sharing or untouched space.
> 
> Historically Linux did guess based overcommit and I added no overcommit
> support way back when, along with 'anything is allowed' support for
> certain HPC use cases.
> 
> The beancounter patches combined with this made the entire setup
> completely robust but the beancounters never hit upstream although years
> later they became part of the basis of the cgroups.
> 
> You can sort of set a current Linux up for definitely no overcommit using
> cgroups and no overcommit settings. It works for most stuff although last
> I checked most graphics drivers were terminally broken (and not just to
> no overcommit but to the point you can remote DoS Linux boxes with a
> suitably constructed web page and chrome browser)
> 
> Alan
> 

Thanks for your comment, it certainly provides more clues and provided some history about the "overcommit" setting.
I will see if we can do what we want with cgroups.

Best regards,

Sebastian

[toc] | [prev] | [standalone]

Page 3 of 3 — ← Prev page 1 2 [3]

Back to top | Article view | linux.kernel

csiph-web