Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #107144 > unrolled thread

Does heap segment of a C program increases in the amount of RAM or is it fixed to a limit?

Back to article view | Back to comp.lang.c

Contents

  Does heap segment of a C program increases in the amount of RAM or is it fixed to a limit? acidlategamer@gmail.com - 2017-03-27 01:23 -0700
    Re: Does heap segment of a C program increases in the amount of RAM or is it fixed to a limit? "James R. Kuyper" <jameskuyper@verizon.net> - 2017-03-27 11:06 -0400
      Re: Does heap segment of a C program increases in the amount of RAM or is it fixed to a limit? Ken Brody <kenbrody@spamcop.net> - 2017-03-27 14:50 -0400
      Re: Does heap segment of a C program increases in the amount of RAM or is it fixed to a limit? Ben Bacarisse <ben.usenet@bsb.me.uk> - 2017-03-27 21:17 +0100
        Re: Does heap segment of a C program increases in the amount of RAM or is it fixed to a limit? Robert Wessel <robertwessel2@yahoo.com> - 2017-03-27 19:08 -0500
    Re: Does heap segment of a C program increases in the amount of RAM or is it fixed to a limit? scott@slp53.sl.home (Scott Lurndal) - 2017-03-27 15:23 +0000
      Re: Does heap segment of a C program increases in the amount of RAM or is it fixed to a limit? "Rick C. Hodgin" <rick.c.hodgin@gmail.com> - 2017-03-27 09:57 -0700
        Re: Does heap segment of a C program increases in the amount of RAM or is it fixed to a limit? BartC <bc@freeuk.com> - 2017-03-27 18:14 +0100
    Re: Does heap segment of a C program increases in the amount of RAM or is it fixed to a limit? gordonb.l1p2g@burditt.org (Gordon Burditt) - 2017-03-27 12:37 -0500
      Re: Does heap segment of a C program increases in the amount of RAM or is it fixed to a limit? "James R. Kuyper" <jameskuyper@verizon.net> - 2017-03-27 13:53 -0400
      Re: Does heap segment of a C program increases in the amount of RAM or is it fixed to a limit? scott@slp53.sl.home (Scott Lurndal) - 2017-03-27 18:06 +0000
      Re: Does heap segment of a C program increases in the amount of RAM or is it fixed to a limit? supercat@casperkitty.com - 2017-03-27 12:29 -0700
        Re: Does heap segment of a C program increases in the amount of RAM or is it fixed to a limit? Robert Wessel <robertwessel2@yahoo.com> - 2017-03-27 19:04 -0500
          Re: Does heap segment of a C program increases in the amount of RAM or is it fixed to a limit? supercat@casperkitty.com - 2017-03-27 17:31 -0700
            Re: Does heap segment of a C program increases in the amount of RAM or is it fixed to a limit? Ian Collins <ian-news@hotmail.com> - 2017-03-28 13:35 +1300
          Re: Does heap segment of a C program increases in the amount of RAM or is it fixed to a limit? Ian Collins <ian-news@hotmail.com> - 2017-03-28 13:37 +1300

#107144 — Does heap segment of a C program increases in the amount of RAM or is it fixed to a limit?

This is in context of Linux ubuntu OS,

I had this idea that heap is a portion of a c program which is used for dynamic memory allocations. If a program tries to allocate a dynamic memory it will use the memories inside heap.

Now I saw some cases where a program is trying to load memory in 1 MB chunks in a while loop. In that case I thought if the heap itself is used totally the program should crash as it do not have permission to access other application's memory segments.

But I saw a discussion that says, that memory allocation will continue, eat up all RAM, then SWAP and maybe then crash or hang.

So in case another program also uses malloc and have some portions in heap unallocated, first program will eat up that heap, then some other program's heap then swap and so on. So all the applications will hang at some point.

Is my understanding correct?
P.S
two statements about malloc and calloc,

1.    Linux doesn't actually allocate your memory until you use it.
       int *p = malloc(sizeof(int));//memory is not used
       int *s = calloc(1, p);//memory is used

2.    When you use calloc instead of malloc, it zeroes out all the memory you requested. This is equivalent to using it.

[toc] | [next] | [standalone]

#107158

On 03/27/2017 04:23 AM, acidlategamer@gmail.com wrote:
...
> 1.    Linux doesn't actually allocate your memory until you use it.
>        int *p = malloc(sizeof(int));//memory is not used
>        int *s = calloc(1, p);//memory is used

That's a configurable option, called overcommit, and that's not a 
conforming implementation of malloc() when that option is turned on. I 
have it turned off on systems under my control.

> 2.    When you use calloc instead of malloc, it zeroes out all the memory you requested. This is equivalent to using it.

It's not "equivalent" to using it; it IS using it.

[toc] | [prev] | [next] | [standalone]

#107193

On 3/27/2017 11:06 AM, James R. Kuyper wrote:
> On 03/27/2017 04:23 AM, acidlategamer@gmail.com wrote:
[...]
>> 2.    When you use calloc instead of malloc, it zeroes out all the memory
>> you requested. This is equivalent to using it.
>
> It's not "equivalent" to using it; it IS using it.

Nit:

I have seen malloc implementations which allow the user to configure which 
method of allocation is used.  Typically, this is good for debugging 
purposes.  For example, each allocation can be placed in its own MMU 
segment, allowing buffer overrun/underrun errors to be caught in hardware.

In such an implementation, it should be possible for the MMU to allocate an 
"all-zeros" segment of memory.

But, as far as C is concerned, in order to guarantee all zeros, it's "as if" 
the memory was written to.

-- 
Kenneth Brody

[toc] | [prev] | [next] | [standalone]

#107206

"James R. Kuyper" <jameskuyper@verizon.net> writes:

> On 03/27/2017 04:23 AM, acidlategamer@gmail.com wrote:
> ...
>> 1.    Linux doesn't actually allocate your memory until you use it.
>>        int *p = malloc(sizeof(int));//memory is not used
>>        int *s = calloc(1, p);//memory is used
>
> That's a configurable option, called overcommit, and that's not a
> conforming implementation of malloc() when that option is turned on. I
> have it turned off on systems under my control.
>
>> 2.  When you use calloc instead of malloc, it zeroes out all the
>> memory you requested. This is equivalent to using it.
>
> It's not "equivalent" to using it; it IS using it.

Maybe in some theoretical way, yes, but surely the system can know that
the over-committed mapped pages will be zeroed?  On my Linux system,
repeatedly calloc does not grow the memory usage.  It's only when the
pages are touched by me (i.e. with an explicit write) that the memory
usage grows.

-- 
Ben.

[toc] | [prev] | [next] | [standalone]

#107232

On Mon, 27 Mar 2017 21:17:21 +0100, Ben Bacarisse
<ben.usenet@bsb.me.uk> wrote:

>"James R. Kuyper" <jameskuyper@verizon.net> writes:
>
>> On 03/27/2017 04:23 AM, acidlategamer@gmail.com wrote:
>> ...
>>> 1.    Linux doesn't actually allocate your memory until you use it.
>>>        int *p = malloc(sizeof(int));//memory is not used
>>>        int *s = calloc(1, p);//memory is used
>>
>> That's a configurable option, called overcommit, and that's not a
>> conforming implementation of malloc() when that option is turned on. I
>> have it turned off on systems under my control.
>>
>>> 2.  When you use calloc instead of malloc, it zeroes out all the
>>> memory you requested. This is equivalent to using it.
>>
>> It's not "equivalent" to using it; it IS using it.
>
>Maybe in some theoretical way, yes, but surely the system can know that
>the over-committed mapped pages will be zeroed?  On my Linux system,
>repeatedly calloc does not grow the memory usage.  It's only when the
>pages are touched by me (i.e. with an explicit write) that the memory
>usage grows.


Of course - overcommitted virtual pages are all mapped to a read-only
page of zeros, and then (essentially) COW'd if you write to them.
Although small callocs are likely to have to do an actual zero since
they'll be pulling from an already allocated chunk of storage, that
may be getting reused.

[toc] | [prev] | [next] | [standalone]

#107159

acidlategamer@gmail.com writes:
>This is in context of Linux ubuntu OS,
>
>I had this idea that heap is a portion of a c program which is used for dynamic memory allocations. If a program tries to allocate a dynamic memory it will use the memories inside heap.
>
>Now I saw some cases where a program is trying to load memory in 1 MB chunks in a while loop. In that case I thought if the heap itself is used totally the program should crash as it do not have permission to access other application's memory segments.
>
>But I saw a discussion that says, that memory allocation will continue, eat up all RAM, then SWAP and maybe then crash or hang.
>
>So in case another program also uses malloc and have some portions in heap unallocated, first program will eat up that heap, then some other program's heap then swap and so on. So all the applications will hang at some point.
>
>Is my understanding correct?
>P.S
>two statements about malloc and calloc,
>
>1.    Linux doesn't actually allocate your memory until you use it.
>       int *p = malloc(sizeof(int));//memory is not used
>       int *s = calloc(1, p);//memory is used

malloc will call an underlying system call (sbrk or mmap) to allocate
a range of the virtual address space to the heap.    These newly allocated
pages will be initialized to "not-present" in the page tables.   When the
application subsequently references the address returned from malloc, the
operating system will take a page fault and at that time, allocate a
physical page frame to the virtual address associated with the heap page.

So yes, Linux doesn't allocate memory until you use it.   This can result
in overcommiting of memory resources.    There are mechanisms in some unix
distributions that allow the programmer to request deterministic behavior
from the memory subsystem (i.e allocate at the time of the allocation call).

Note that POSIX resource limits can be established on a per-process basis
to set limits on the size of the heap, stack and entire program.

man 2 setrlimit
man 1 ulimit

>
>2.    When you use calloc instead of malloc, it zeroes out all the memory you requested. This is equivalent to using it.

Indeed, it must touch each byte of the allocated region to zero it.

[toc] | [prev] | [next] | [standalone]

#107176

On Monday, March 27, 2017 at 11:23:32 AM UTC-4, Scott Lurndal wrote:
> acidlategamer@gmail.com writes:
> >This is in context of Linux ubuntu OS,
> >
> >I had this idea that heap is a portion of a c program which is used for dynamic memory allocations. If a program tries to allocate a dynamic memory it will use the memories inside heap.
> >
> >Now I saw some cases where a program is trying to load memory in 1 MB chunks in a while loop. In that case I thought if the heap itself is used totally the program should crash as it do not have permission to access other application's memory segments.
> >
> >But I saw a discussion that says, that memory allocation will continue, eat up all RAM, then SWAP and maybe then crash or hang.
> >
> >So in case another program also uses malloc and have some portions in heap unallocated, first program will eat up that heap, then some other program's heap then swap and so on. So all the applications will hang at some point.
> >
> >Is my understanding correct?
> >P.S
> >two statements about malloc and calloc,
> >
> >1.    Linux doesn't actually allocate your memory until you use it.
> >       int *p = malloc(sizeof(int));//memory is not used
> >       int *s = calloc(1, p);//memory is used
> 
> malloc will call an underlying system call (sbrk or mmap) to allocate
> a range of the virtual address space to the heap.    These newly allocated
> pages will be initialized to "not-present" in the page tables.   When the
> application subsequently references the address returned from malloc, the
> operating system will take a page fault and at that time, allocate a
> physical page frame to the virtual address associated with the heap page.
> 
> So yes, Linux doesn't allocate memory until you use it.   This can result
> in overcommiting of memory resources.    There are mechanisms in some unix
> distributions that allow the programmer to request deterministic behavior
> from the memory subsystem (i.e allocate at the time of the allocation call).
> 
> Note that POSIX resource limits can be established on a per-process basis
> to set limits on the size of the heap, stack and entire program.
> 
> man 2 setrlimit
> man 1 ulimit
> 
> >
> >2.    When you use calloc instead of malloc, it zeroes out all the memory you requested. This is equivalent to using it.
> 
> Indeed, it must touch each byte of the allocated region to zero it.

Scott, I observed some behavior in Linux in 2010 that I could never
resolve apart from the following explanation.  I've never gone further
and investigated it though because it wasn't that big of a deal.

I had a machine with 2 GB of RAM.  I was building textures in memory
from a data file which, through the algorithm, generated 24-bit RGB
textures.  The generated image file was about 100 MB and I used it
for textures on this OpenGL 3D project.  I was using Ubuntu 10.04
at the time with nothing special just default install and the basic
build-essential environment.

What I observed was the first time I loaded the data file it took the
amount of time I would expect (many seconds) to load.  However, on
later loads it was as though the kernel knew that I had been reading
the disk file into allocated blocks of RAM and did not re-load them
on subsequent passes because I had not modified their contents, but
only read them.

Is it possible Linux was designed to associate the disk reads into
my allocated memory, observe no changes were made, and keep that data
cached thereafter so that follow-on runs were 20x faster on load?  I
presume by mapping disk reads to pages, which were mapped to pages
mapped to my allocated block due to the read-into-allocated-block,
which then had pages flagged to be notified upon any write to then
copy to its own page and the cache invalidated... ??

Was my procedure of:

    (1)  Make changes to software
    (2)  Compile
    (3)  Run program
    (4)  Open file x.dat
    (5)  Get file size
    (6)  Allocate that much ram
    (7)  Read file into that ram
    (8)  Process through the data generating texture
    (9)  Release ram
    (10) Present 3D OpenGL view
    (11) Exit program
    (12) Goto (1)

Allowing Linux to keep that data in memory because I never changed
it?  I remember thinking at the time how I impressed I was with that
kernel if it was doing that.

My only other thought was a caching hard disk, but how would it be
able to cache about 1.5 GB worth of data?  It wouldn't I would think.
It was an inexpensive Toshiba Satellite notebook from 2007.

Thank you,
Rick C. Hodgin

[toc] | [prev] | [next] | [standalone]

#107180

On 27/03/2017 17:57, Rick C. Hodgin wrote:

> Scott, I observed some behavior in Linux in 2010 that I could never
> resolve apart from the following explanation.  I've never gone further
> and investigated it though because it wasn't that big of a deal.
>
> I had a machine with 2 GB of RAM.  I was building textures in memory
> from a data file which, through the algorithm, generated 24-bit RGB
> textures.  The generated image file was about 100 MB and I used it
> for textures on this OpenGL 3D project.  I was using Ubuntu 10.04
> at the time with nothing special just default install and the basic
> build-essential environment.
>
> What I observed was the first time I loaded the data file it took the
> amount of time I would expect (many seconds) to load.  However, on
> later loads it was as though the kernel knew that I had been reading
> the disk file into allocated blocks of RAM and did not re-load them
> on subsequent passes because I had not modified their contents, but
> only read them.
>
> Is it possible Linux was designed to associate the disk reads into
> my allocated memory, observe no changes were made, and keep that data
> cached thereafter so that follow-on runs were 20x faster on load?

Doesn't any OS do that now?

I've just tested loading a 22MB file memory. It took 1.8 seconds (more 
than I expected actually). The second time it look 0.12 seconds, and 
that was on a separate run of the program.

You would expect caching anyway by the OS's file system or even by a 
disk controller.

> My only other thought was a caching hard disk, but how would it be
> able to cache about 1.5 GB worth of data?

I thought you said it was 100MB?

-- 
bartc

[toc] | [prev] | [next] | [standalone]

#107182

> This is in context of Linux ubuntu OS,

I believe memory overcommit applies to all Linux kernels
that have memory overcommit turned on.

Have you (personally) ever had a process die because memory was
overcommitted?

> I had this idea that heap is a portion of a c program which is
> used for dynamic memory allocations. If a program tries to allocate
> a dynamic memory it will use the memories inside heap.

Each process has its own heap, drawn from the common pool.  The
memory address space allocated needs to have a physical resource
(RAM, swap space) backing it for it to actually be used.  

A fact of life is that programs often request memory they don't
ever use.  A lot of this comes from fork(), which allocates copies
of a lot of memory in the parent process for the child, (copy-on-write
means that unless it's written, one copy of a page serves for both
processes) which is often followed by exec*(), which throws out all
that memory and replaces it with another program.

A process which has allocated more than half of allowed memory
will not be able to fork() without overcommitting memory or
having the fork() fail.  Calling free() will normally not help.

> Now I saw some cases where a program is trying to load memory in
> 1 MB chunks in a while loop. In that case I thought if the heap
> itself is used totally the program should crash as it do not have
> permission to access other application's memory segments.

A refused memory allocation is *NOT* a crash.  The program should
check whether it got the memory and, as a worst case, exit gracefully
with an error message if it cannot allocate memory instead of
attempting to use it anyway.  This is not a "crash".  Perhaps a
better response is to continue with a smaller buffer.

Linux will overcommit memory (that is, let you allocate memory for
the heap without any physical resource to back it) *IF* it is set
to do that.  This is the default setting.  You can turn it off if
the sysctl vm.overcommit_memory = 2.  Typically making setting means
you also need more swap space.  You can also tune vm.overcommit_ratio
and vm.overcommit_kbytes to regulate *how much* it will overcommit.

There's also a setting vm.overcommit_memory = 1, which permits
massive overcommitting for programs which use very sparse arrays,
and it won't refuse even ridiculous overcommitting (Imagine 10TB
allocated on a system with 2GB RAM and 2GB swap.  This needs to be
a 64-bit system to have enough address space).  Some programs need
either this or LOTS more swap space that will never be used.

The response to attempting to write memory when the system has run
out of physical memory to back it is to SIGTERM the program trying
to write the memory (which is not necessarily the same program as
the "big memory hog" program - it might be some innocent program
just trying to grow its stack or use a small amount of memory it
allocated).  The program dies on a SIGTERM signal.  Not "hang".  If
memory use is high and programs are competing heavily for RAM, the
system will spend a lot of time swapping and paging and it may
appear to "hang".

> But I saw a discussion that says, that memory allocation will
> continue, eat up all RAM, then SWAP and maybe then crash or hang.

If no program has been SIGTERM'd yet, you haven't run out of physical
resources to back memory allocation yet.  The system may be spending
a lot of its time swapping and paging.  That's not a "hang".  It's
just slow.  At this point you may be praying for the out-of-memory
killer to kill something.

> So in case another program also uses malloc and have some portions
> in heap unallocated, first program will eat up that heap, then some
> other program's heap then swap and so on. So all the applications
> will hang at some point.

The memory hog program doesn't "eat up" the other program's heap,
it "eats up" the physical resources to back that heap.  That may
sound like a distinction without a difference, but it's important.
It doesn't hurt if memory is available when the other programs need
it.  Typically at any time there is some allocated memory that
hasn't been initialized yet, and the system can get away with it.
(This is similar to:  on any given airline flight, there will be a
couple of passengers who are no-shows, so airlines can often get
away with overcommitting seats.  However, every once in a while
they get caught out and end up giving passengers compensation for
getting bumped off the flight.  Passengers are not hanged (except
maybe by terrorists).)  You can tune *how much* overcommitting is
one with vm.overcommit_ratio and vm.ovecommit_kbytes if you don't
want to turn off overcommitting entirely.

> Is my understanding correct?
> P.S
> two statements about malloc and calloc,
> 
> 1.    Linux doesn't actually allocate your memory until you use it.
>        int *p = malloc(sizeof(int));//memory is not used

Memory is not used *YET*.  

>        int *s = calloc(1, p);//memory is used
> 
> 2.    When you use calloc instead of malloc, it zeroes out all
> the memory you requested. This is equivalent to using it.

If your point is "Always Use calloc()", that makes your program (or
some other program) DIE SOONER.  If the point is to make sure it
can use the memory it got before wasting its time loading huge
videos or whatever and *then* dying, ok, the policy works.

If you are regularly getting processes SIGTERM'd and are considering
turning off memory overcommit, expect that if you don't increase
swap space, you will make the problem *WORSE*, although this will
be from allocations failing, not programs getting SIGTERM'd.  So
write your code to check if memory allocations fail.

Top reasons for a program "hanging":

- Infinite loop in program (some programs like daemons are intentional infinite loops).
- Program is expecting keyboard input but user doesn't realize it.
- Overly impatient user.
- Buggy device driver (process stuck in "D" state)
- Buggy device hardware (e.g. loses interrupts) (process stuck in "D" state)
- Buggy power supply
- Overheated circuitry (dying fan)
- hard-mounted NFS disk goes offline (not really a hang, as it should come back
	when the NFS server comes back up).
- Locking deadlock
- Network goes offline (not really a "hang", but expect long delays for things to time out)
- While playing the game "hangman", the user runs out of allowed letter guesses.

[toc] | [prev] | [next] | [standalone]

#107185

As usual, I deny you (or any one else) permission to quote this message 
without properly attributing what I wrote to me.

On 03/27/2017 01:37 PM, Gordon Burditt wrote:
>> This is in context of Linux ubuntu OS,
>
> I believe memory overcommit applies to all Linux kernels
> that have memory overcommit turned on.
>
> Have you (personally) ever had a process die because memory was
> overcommitted?

Does "personally" include programs that I'm responsible for in my 
professional capacity? If so, yes. It happened so many times, in fact, 
that it was brought to my attention as a problem I needed to solve. When 
I figured out that overcommit was the cause of the failures, I had no 
trouble convincing the SAs to turn it off. The guy who turned it on in 
the first place didn't even understand know what overcommit was.

[toc] | [prev] | [next] | [standalone]

#107188

gordonb.l1p2g@burditt.org (Gordon Burditt) writes:
>> This is in context of Linux ubuntu OS,
>
>I believe memory overcommit applies to all Linux kernels
>that have memory overcommit turned on.
>
>Have you (personally) ever had a process die because memory was
>overcommitted?

yes.  the OOM killer struck.

[toc] | [prev] | [next] | [standalone]

#107198

On Monday, March 27, 2017 at 12:38:08 PM UTC-5, Gordon Burditt wrote:
> A fact of life is that programs often request memory they don't
> ever use.  A lot of this comes from fork(), which allocates copies
> of a lot of memory in the parent process for the child, (copy-on-write
> means that unless it's written, one copy of a page serves for both
> processes) which is often followed by exec*(), which throws out all
> that memory and replaces it with another program.

How often would programs request large amounts of memory that they never
use outside of situations involving fork?  How often is fork() used in
situations which wouldn't better be served by a system call that starts
a new process along with calls to create, attach, and detach mutable or
immutable shareable objects?

> A process which has allocated more than half of allowed memory
> will not be able to fork() without overcommitting memory or
> having the fork() fail.  Calling free() will normally not help.

Except in the cases where a spawned process would need a copy of everything
the parent had in memory, using a forking process that creates a logical
copy of everything seems silly to me, and I question how often copy-on-write
would be needed at the process level otherwise.

> > Now I saw some cases where a program is trying to load memory in
> > 1 MB chunks in a while loop. In that case I thought if the heap
> > itself is used totally the program should crash as it do not have
> > permission to access other application's memory segments.
> 
> A refused memory allocation is *NOT* a crash.  The program should
> check whether it got the memory and, as a worst case, exit gracefully
> with an error message if it cannot allocate memory instead of
> attempting to use it anyway.  This is not a "crash".  Perhaps a
> better response is to continue with a smaller buffer.

If a program won't be able to do anything useful unless it can get all the
memory it needs, a guarantee that a memory allocation request will either
succeed or force some kind of program termination can often eliminate the
need to have a lot of error-checking scattered throughout a program.  It
would have been helpful to have a standard means of setting up cleanup
handlers that could be used in conjunction with setjmp() to support more
systematic error-recovery behavior.

If a program will be able to adapt itself to varying amounts of available
memory, having it limit usage only when allocations fail may not yield the
most useful behavior, since such a program might grab memory which could
otherwise have been used by a program that needs it more.

> Linux will overcommit memory (that is, let you allocate memory for
> the heap without any physical resource to back it) *IF* it is set
> to do that.  This is the default setting.  You can turn it off if
> the sysctl vm.overcommit_memory = 2.  Typically making setting means
> you also need more swap space.  You can also tune vm.overcommit_ratio
> and vm.overcommit_kbytes to regulate *how much* it will overcommit.

> The response to attempting to write memory when the system has run
> out of physical memory to back it is to SIGTERM the program trying
> to write the memory (which is not necessarily the same program as
> the "big memory hog" program - it might be some innocent program
> just trying to grow its stack or use a small amount of memory it
> allocated).  The program dies on a SIGTERM signal.  Not "hang".  If
> memory use is high and programs are competing heavily for RAM, the
> system will spend a lot of time swapping and paging and it may
> appear to "hang".

That doesn't seem very helpful.  What happens if the SIGTERM handler
tries to write to a sig_atomic_t which is sitting in a shared
copy-on-write page?

[toc] | [prev] | [next] | [standalone]

#107231

On Mon, 27 Mar 2017 12:29:20 -0700 (PDT), supercat@casperkitty.com
wrote:

>On Monday, March 27, 2017 at 12:38:08 PM UTC-5, Gordon Burditt wrote:
>> A fact of life is that programs often request memory they don't
>> ever use.  A lot of this comes from fork(), which allocates copies
>> of a lot of memory in the parent process for the child, (copy-on-write
>> means that unless it's written, one copy of a page serves for both
>> processes) which is often followed by exec*(), which throws out all
>> that memory and replaces it with another program.
>
>How often would programs request large amounts of memory that they never
>use outside of situations involving fork?  How often is fork() used in
>situations which wouldn't better be served by a system call that starts
>a new process along with calls to create, attach, and detach mutable or
>immutable shareable objects?


It's not that uncommon to allocate a maximum sized area for some
process.  It's often easier to not have to deal with variable
allocations, and/or things that can grow and shrink.  Both are rather
less common in C++, where a bunch of the STL containers hide the
details from you.  I'm not saying it's a great technique, but it's far
from uncommon.

It's also very common to allocate large chunks of address space and
then suballocate them - that can lead to significant performance gains
in some situations.  Even the normal (small) heap allocation in malloc
work that way (malloc request large chunks of storage from the OS,
then hands out small chunks to the application).  Many contain objects
(including the STL ones) (can) allocate more than the minimum
increment needed when expanding a container.

It would be interesting to know how many virtual pages are actually
mapped to the read-as-zero page on typical Linux systems with
overcommit turned on.

As to fork() itself...  You certainly won't catch me defending it as a
process management primitive...  But it is part of the environment.


>If a program won't be able to do anything useful unless it can get all the
>memory it needs, a guarantee that a memory allocation request will either
>succeed or force some kind of program termination can often eliminate the
>need to have a lot of error-checking scattered throughout a program.


If that's what you want, it's easy enough to write a routine that's
small wrapper around malloc that abort if there's no storage available
(I think you could do the same thing with a mallopt).  And even touch
all the storage so obtained if you want to make sure it's committed.
Then do all of those allocations up front.

[toc] | [prev] | [next] | [standalone]

#107234

On Monday, March 27, 2017 at 7:05:13 PM UTC-5, robert...@yahoo.com wrote:
> On Mon, 27 Mar 2017 12:29:20 -0700 (PDT), supercat wrote:
> >How often would programs request large amounts of memory that they never
> >use outside of situations involving fork?  How often is fork() used in
> >situations which wouldn't better be served by a system call that starts
> >a new process along with calls to create, attach, and detach mutable or
> >immutable shareable objects?
> 
> It's not that uncommon to allocate a maximum sized area for some
> process.  It's often easier to not have to deal with variable
> allocations, and/or things that can grow and shrink.  Both are rather
> less common in C++, where a bunch of the STL containers hide the
> details from you.  I'm not saying it's a great technique, but it's far
> from uncommon.

> It's also very common to allocate large chunks of address space and
> then suballocate them - that can lead to significant performance gains
> in some situations.  Even the normal (small) heap allocation in malloc
> work that way (malloc request large chunks of storage from the OS,
> then hands out small chunks to the application).  Many contain objects
> (including the STL ones) (can) allocate more than the minimum
> increment needed when expanding a container.

It's certainly common for programs to request large chunks from malloc() and
then sub-allocate it, since an allocator that is designed around a program's
particular usage patterns can often be more efficient than a general-purpose
one like malloc().  Maybe it's common for Unix programs to allocate vastly
more than they're likely ever to need, but I'm used to approaches that try
to use an exponential-growth model to allocate a total amount that's within
a constant factor of what's required.

> It would be interesting to know how many virtual pages are actually
> mapped to the read-as-zero page on typical Linux systems with
> overcommit turned on.
> 
> As to fork() itself...  You certainly won't catch me defending it as a
> process management primitive...  But it is part of the environment.

Yeah, but if a large process which wastefully uses fork() purely for the
purpose of spawning a small process can't get enough memory to fork(), maybe
that's a sign that it should use some other means of spawning the sub-process?
I'm pretty sure Unix must have added such a means in the last few decades.

> >If a program won't be able to do anything useful unless it can get all the
> >memory it needs, a guarantee that a memory allocation request will either
> >succeed or force some kind of program termination can often eliminate the
> >need to have a lot of error-checking scattered throughout a program.
> 
> If that's what you want, it's easy enough to write a routine that's
> small wrapper around malloc that abort if there's no storage available
> (I think you could do the same thing with a mallopt).  And even touch
> all the storage so obtained if you want to make sure it's committed.
> Then do all of those allocations up front.

The problem with that approach is that in the absence of a standard, every
library that needs to request a memory allocation is apt to have its own way
of handling errors and registering cleanup handlers.  C++ implementations
can handle stack unwinding in ways that minimize cost in the no-error case,
but C can achieve similar semantics for the cost of a linked-list LIFO push
and pop per required cleanup section if everything that needs cleanup uses
the same mechanism.

[toc] | [prev] | [next] | [standalone]

#107236

On 03/28/17 01:31 PM, supercat@casperkitty.com wrote:
> On Monday, March 27, 2017 at 7:05:13 PM UTC-5, robert...@yahoo.com wrote:
>> On Mon, 27 Mar 2017 12:29:20 -0700 (PDT), supercat wrote:
>>> How often would programs request large amounts of memory that they never
>>> use outside of situations involving fork?  How often is fork() used in
>>> situations which wouldn't better be served by a system call that starts
>>> a new process along with calls to create, attach, and detach mutable or
>>> immutable shareable objects?
>>
>> It's not that uncommon to allocate a maximum sized area for some
>> process.  It's often easier to not have to deal with variable
>> allocations, and/or things that can grow and shrink.  Both are rather
>> less common in C++, where a bunch of the STL containers hide the
>> details from you.  I'm not saying it's a great technique, but it's far
>> from uncommon.
>
>> It's also very common to allocate large chunks of address space and
>> then suballocate them - that can lead to significant performance gains
>> in some situations.  Even the normal (small) heap allocation in malloc
>> work that way (malloc request large chunks of storage from the OS,
>> then hands out small chunks to the application).  Many contain objects
>> (including the STL ones) (can) allocate more than the minimum
>> increment needed when expanding a container.
>
> It's certainly common for programs to request large chunks from malloc() and
> then sub-allocate it, since an allocator that is designed around a program's
> particular usage patterns can often be more efficient than a general-purpose
> one like malloc().  Maybe it's common for Unix programs to allocate vastly
> more than they're likely ever to need, but I'm used to approaches that try
> to use an exponential-growth model to allocate a total amount that's within
> a constant factor of what's required.

Don't conflate Linux memory management with Unix: most (if not all) Unix 
systems stay well clear of over commit.

-- 
Ian

[toc] | [prev] | [next] | [standalone]

#107237

On 03/28/17 01:04 PM, Robert Wessel wrote:
> On Mon, 27 Mar 2017 12:29:20 -0700 (PDT), supercat@casperkitty.com

>> If a program won't be able to do anything useful unless it can get all the
>> memory it needs, a guarantee that a memory allocation request will either
>> succeed or force some kind of program termination can often eliminate the
>> need to have a lot of error-checking scattered throughout a program.
>
>
> If that's what you want, it's easy enough to write a routine that's
> small wrapper around malloc that abort if there's no storage available
> (I think you could do the same thing with a mallopt).  And even touch
> all the storage so obtained if you want to make sure it's committed.
> Then do all of those allocations up front.

A cleaner solution is to have a selection of allocator libraries on had 
for different uses (debugging, threading to name a couple).

-- 
Ian

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.c

csiph-web