Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #20827 > unrolled thread

Will 'free' return the memory Immediately to the OS ?

Back to article view | Back to comp.lang.c

Contents

  Will 'free' return the memory Immediately to the OS ? Raj Pashwar <raj121190@hotmail.NOSPAM.com> - 2012-05-15 21:34 +0000
    Re: Will 'free' return the memory Immediately to the OS ? Ian Collins <ian-news@hotmail.com> - 2012-05-16 09:55 +1200
    Re: Will 'free' return the memory Immediately to the OS ? James Kuyper <jameskuyper@verizon.net> - 2012-05-15 18:42 -0400
    Re: Will 'free' return the memory Immediately to the OS ? Keith Thompson <kst-u@mib.org> - 2012-05-15 16:49 -0700
    Re: Will 'free' return the memory Immediately to the OS ? Ben Pfaff <blp@cs.stanford.edu> - 2012-05-15 16:59 -0700
    Re: Will 'free' return the memory Immediately to the OS ? Eric Sosman <esosman@ieee-dot-org.invalid> - 2012-05-15 20:24 -0400
    Re: Will 'free' return the memory Immediately to the OS ? David T. Ashley <dashley@gmail.com> - 2012-05-16 13:29 -0400
      Re: Will 'free' return the memory Immediately to the OS ? Stephen Sprunk <stephen@sprunk.org> - 2012-05-16 18:36 -0500
        Re: Will 'free' return the memory Immediately to the OS ? David T. Ashley <dashley@gmail.com> - 2012-05-17 09:47 -0400
          Re: Will 'free' return the memory Immediately to the OS ? Stephen Sprunk <stephen@sprunk.org> - 2012-05-17 11:22 -0500
          Re: Will 'free' return the memory Immediately to the OS ? Kaz Kylheku <kaz@kylheku.com> - 2012-05-17 16:54 +0000
            Re: Will 'free' return the memory Immediately to the OS ? Stephen Sprunk <stephen@sprunk.org> - 2012-05-18 12:30 -0500
          Re: Will 'free' return the memory Immediately to the OS ? Keith Thompson <kst-u@mib.org> - 2012-05-17 14:09 -0700
          Re: Will 'free' return the memory Immediately to the OS ? Eric Sosman <esosman@ieee-dot-org.invalid> - 2012-05-17 21:05 -0400
            Re: Will 'free' return the memory Immediately to the OS ? Stephen Sprunk <stephen@sprunk.org> - 2012-05-18 17:41 -0500
              Re: Will 'free' return the memory Immediately to the OS ? Eric Sosman <esosman@ieee-dot-org.invalid> - 2012-05-18 20:19 -0400
                Re: Will 'free' return the memory Immediately to the OS ? Stephen Sprunk <stephen@sprunk.org> - 2012-05-22 15:08 -0500
                  Re: Will 'free' return the memory Immediately to the OS ? Ben Pfaff <blp@cs.stanford.edu> - 2012-05-22 13:45 -0700
                    Re: Will 'free' return the memory Immediately to the OS ? Robert Wessel <robertwessel2@yahoo.com> - 2012-05-23 03:02 -0500
                    Re: Will 'free' return the memory Immediately to the OS ? Robert Miles <milesrf@Usenet-News.net> - 2012-06-24 23:49 -0500
                      Re: Will 'free' return the memory Immediately to the OS ? Stephen Sprunk <stephen@sprunk.org> - 2012-06-26 10:33 -0500
          Re: Will 'free' return the memory Immediately to the OS ? jgk@panix.com (Joe keane) - 2012-05-18 19:44 +0000
            Re: Will 'free' return the memory Immediately to the OS ? Stephen Sprunk <stephen@sprunk.org> - 2012-05-18 17:27 -0500
              Re: Will 'free' return the memory Immediately to the OS ? jgk@panix.com (Joe keane) - 2012-05-22 21:52 +0000
                Re: Will 'free' return the memory Immediately to the OS ? Ian Collins <ian-news@hotmail.com> - 2012-05-23 11:12 +1200
                  Re: Will 'free' return the memory Immediately to the OS ? Robert Wessel <robertwessel2@yahoo.com> - 2012-05-23 03:15 -0500
      Re: Will 'free' return the memory Immediately to the OS ? Robert Wessel <robertwessel2@yahoo.com> - 2012-05-17 11:54 -0500

Page 2 of 2 — ← Prev page 1 [2]

#22801

On 24-Jun-12 23:49, Robert Miles wrote:
> On 5/22/2012 3:45 PM, Ben Pfaff wrote:
>> Stephen Sprunk <stephen@sprunk.org> writes:
>>> There's that problem too, but I was thinking more along the lines
>>> of 4MB pages taking ~1000 times as long as 4kB pages to evict and
>>> reload.
>>
>> I doubt that that is true, though.  I'd suspect that 4 MB pages
>> take less than 100 times as long as 4 kB pages to evict and
>> reload, maybe much less.
>>
>> I tried to find some measurements online to support this point,
>> but couldn't quickly find anything relevant.
> 
> I'd expect that to depend heavily on the rotation speed of the hard
> drive.  Eviction and reloading should each take at least an average
> of half a rotation, plus however long it takes to read or write that
> much data.
> 
> The fastest common speed for hard drives for home computers is
> 7200 RPM, although 10000 RPM is available but not common yet.
> 
> Some hard drives have memory caches to help with this if a read
> request comes soon enough after a write request for the same data.
> 
> Also, SSD drives don't use a rotating disk - they only use a slower
> interface to memory chips, and are therefore much faster than hard
> drives.

AFAIK, 10,000rpm drives, 7,200rpm drives, caching drives, and SSD drives
were not available in the consumer x86 market at the time the i386 was
being designed, which is when 4kB/4MB page sizes were chosen.

And, if you study the x86 (and even x64) page table system, there's
really no good way to add an intermediate page size without basically
revamping the entire system--which means OS developers would have to
throw away all of the paging code and experience they've built up over
the last two and a half decades of x86 development.

S

-- 
Stephen Sprunk         "God does not play dice."  --Albert Einstein
CCIE #3723         "God is an inveterate gambler, and He throws the
K5SSS        dice at every possible opportunity." --Stephen Hawking

[toc] | [prev] | [next] | [standalone]

#21003

In article <i20ar71d8m427086mra7ortps5up3tqa7v@4ax.com>,
David T. Ashley  <dashley@gmail.com> wrote:
>I figured with a modern PC having maybe 4G of memory, you might want
>to divide the memory into maybe 1,000 - 10,000 pages, so I would have
>figured page sizes in the range of 512K - 4M.  4K surprises me.

The OS can have a larger page size.  It's easy to make 16 KB pages on
hardware that is designed for 4 KB pages.  But not the other way around.

The size of a 'segment' [unit that the memory management requests more
space from the OS] would be more like 2 MB.

[toc] | [prev] | [next] | [standalone]

#21008

On 18-May-12 14:44, Joe keane wrote:
> In article <i20ar71d8m427086mra7ortps5up3tqa7v@4ax.com>,
> David T. Ashley  <dashley@gmail.com> wrote:
>> I figured with a modern PC having maybe 4G of memory, you might want
>> to divide the memory into maybe 1,000 - 10,000 pages, so I would have
>> figured page sizes in the range of 512K - 4M.  4K surprises me.
> 
> The OS can have a larger page size.  It's easy to make 16 KB pages on
> hardware that is designed for 4 KB pages.  But not the other way around.

You can _emulate_ larger pages by always handling several of them at the
same time, but the hardware does what the hardware does.

I question whether this is a good idea.  Any reduction in page faults
due to loading multiple pages at a time is likely to be offset by an
increase in page faults due to evicting multiple pages at a time.  So,
overall, you spend more time swapping pages to/from disk for no gain.

> The size of a 'segment' [unit that the memory management requests more
> space from the OS] would be more like 2 MB.

"Segment" has a very different meaning on many architectures.

Also, I would expect malloc() et al to request/release memory from the
OS in whatever multiple of the page size it found convenient, not some
arbitrarily large (or small, for some workloads) amount like 2MB.

S

-- 
Stephen Sprunk         "God does not play dice."  --Albert Einstein
CCIE #3723         "God is an inveterate gambler, and He throws the
K5SSS        dice at every possible opportunity." --Stephen Hawking

[toc] | [prev] | [next] | [standalone]

#21214

In article <jp6ic1$lp6$1@dont-email.me>,
Stephen Sprunk  <stephen@sprunk.org> wrote:
>I question whether this is a good idea.

It is a good idea.  These are -disk drives-.

If your drive has a decent buffer, the time to get additional pages
could be no more than to transfer data over the bus.

[toc] | [prev] | [next] | [standalone]

#21218

On 05/23/12 09:52 AM, Joe keane wrote:
> In article<jp6ic1$lp6$1@dont-email.me>,
> Stephen Sprunk<stephen@sprunk.org>  wrote:
>> I question whether this is a good idea.
>
> It is a good idea.  These are -disk drives-.
>
> If your drive has a decent buffer, the time to get additional pages
> could be no more than to transfer data over the bus.

If only life were that simple!  Consider a diskless client.

The point you snipped:

"The OS can have a larger page size.  It's easy to make 16 KB pages on
hardware that is designed for 4 KB pages.  But not the other way around."

A reasonable OS will allow multiple page sizes, in part to minimise the 
use of MMU slots when applications use large amounts of memory.  One 
negative side effect is the bigger the page size an application uses, 
the more likely it is to hit an out of memory condition due to 
fragmentation.

-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

#21226

On Wed, 23 May 2012 11:12:43 +1200, Ian Collins <ian-news@hotmail.com>
wrote:

>On 05/23/12 09:52 AM, Joe keane wrote:
>> In article<jp6ic1$lp6$1@dont-email.me>,
>> Stephen Sprunk<stephen@sprunk.org>  wrote:
>>> I question whether this is a good idea.
>>
>> It is a good idea.  These are -disk drives-.
>>
>> If your drive has a decent buffer, the time to get additional pages
>> could be no more than to transfer data over the bus.
>
>If only life were that simple!  Consider a diskless client.
>
>The point you snipped:
>
>"The OS can have a larger page size.  It's easy to make 16 KB pages on
>hardware that is designed for 4 KB pages.  But not the other way around."
>
>A reasonable OS will allow multiple page sizes, in part to minimise the 
>use of MMU slots when applications use large amounts of memory.  One 
>negative side effect is the bigger the page size an application uses, 
>the more likely it is to hit an out of memory condition due to 
>fragmentation.


I think his point is that most MMUs (and their associated TLBs) are
very restrictive in the sizes of pages they support, and while you can
certainly emulate 16KB pages on an MMU supporting only 4KB pages, each
"16KB" page will take four TLB entries to map (and four TLB misses to
fill).  So emulating larger pages does not help with TLB pressures.

Even when different page sizes are supported, TLB usage is not
necessarily simply split between the sizes.  For example, there have
been x86 implementations (which support 4KB and 2/4MB pages) where the
large pages had a separate, and quite small, TLB, and it was easy to
thrash the large-page TLB.  Some ISAs that supported several (small)
pages sizes have had implementations that did not directly support all
the sizes in the TLB, and would use several TLB slots to map the
bigger pages (leading to similar issues as OS emulation of larger
pages).

[toc] | [prev] | [next] | [standalone]

#20916

On Wed, 16 May 2012 13:29:48 -0400, David T. Ashley
<dashley@gmail.com> wrote:

>On Tue, 15 May 2012 21:34:57 +0000 (UTC), Raj Pashwar
><raj121190@hotmail.NOSPAM.com> wrote:
>
>>As per subject-line, thank-you
>
>Generally, it depends on the "division of labor" between the run-time
>library and the operating system.
>
>Some operating systems (in the old days before PCs used hardware
>memory management and all that) had OS function calls that would
>handle small blocks quite efficiently.  malloc() and free() were just
>empty wrappers, and dynamically-allocated memory belonging to the OS
>and the application were freely mixed.  Some embedded systems may
>still have this kind of division of labor.
>
>Nowadays, on a modern PC, the landscape is different.
>
>No OS is likely to have the capability to hand the run-time library a
>chunk of memory smaller than the page size supported by the memory
>management hardware.  I don't know what this figure is nowadays, but
>I'd guess maybe between 256K and 1M.  You typically wouldn't be able
>to allocate from the operating system, for example, a 30-byte chunk of
>memory.


The vast majority of systems use 4-64KB pages (and 4KB pages are most
common of all).  ARM allowed (in the past) smaller pages.  Many
systems also allow large pages, usually by omitting the lowest level
of the page table hierarchy (x86, for example, allows 2MB or 4MB
pages, depending on which format page tables you're using), but these
tend to be used mainly to replace large contiguous allocations of
"normal" sized pages, and particularly for areas some subject to
paging or allocation (for example, the OS areas are fairly commonly
mapped with large pages).

As for OS's allowing small allocations...  Obviously you need to
define the OS's boundaries rather carefully.  For example Windows
Heap*() functions are defined as part of the OS API, but act more like
what you'd expect malloc() to do (and are implemented largely in user
space - in fact some Windows malloc-like implementations map simply to
HeapAlloc).  As another example, zOS has the "GETMAIN" macro as its
primary memory allocation API, which also handles small chunks.


>The run-time library is likely to take these large chunks allocated by
>the operating system and subdivide them so it can hand out smaller
>chunks via malloc().
>
>There is no guarantee that the algorithm used by malloc() and free()
>would allow any memory to be returned to the OS until every block had
>been free()'d.


True.


>There is no reason usually to return memory to the OS until the
>process terminates.  Most programs that use dynamic memory use a bunch
>and then terminate ... it wouldn't benefit the average program to
>return anything to the OS until the program terminates ... just adds
>complexity and decreases speed.


That's sometimes true, sometimes not.  Consider an application that
lets you load a large file to work on for a while, and then lets you
work on something else.  Returning that memory to the OS would only be
polite.

In any event, many implementations do things like handle small
allocations internally (by subdividing larger chunks allocated from
the OS), and large allocations as OS calls. So often freeing a large
allocated chunk actually does return the storage to the OS.

In short, it's impossible to generalize.

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | comp.lang.c

csiph-web