Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #78673 > unrolled thread

Re: "High water" Memory fragmentation still a thing?

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: "High water" Memory fragmentation still a thing? Chris Angelico <rosuav@gmail.com> - 2014-10-04 09:26 +1000
    Re: "High water" Memory fragmentation still a thing? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-10-04 11:02 +1000
      Re: "High water" Memory fragmentation still a thing? Chris Angelico <rosuav@gmail.com> - 2014-10-04 12:22 +1000
        Re: "High water" Memory fragmentation still a thing? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-10-04 15:48 +1000
          Re: "High water" Memory fragmentation still a thing? Chris Angelico <rosuav@gmail.com> - 2014-10-04 15:57 +1000
          Re: "High water" Memory fragmentation still a thing? Sturla Molden <sturla.molden@gmail.com> - 2014-10-04 17:51 +0000
      Re: "High water" Memory fragmentation still a thing? Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2014-10-04 16:20 -0400
      Re: "High water" Memory fragmentation still a thing? mm0fmf <none@mailinator.com> - 2014-10-05 20:23 +0100
        Re: "High water" Memory fragmentation still a thing? Steven D'Aprano <steve@pearwood.info> - 2014-10-06 07:20 +0000
    Re: "High water" Memory fragmentation still a thing? bryanjugglercryptographer@yahoo.com - 2014-10-06 09:21 -0700
      Re: "High water" Memory fragmentation still a thing? Chris Angelico <rosuav@gmail.com> - 2014-10-07 03:43 +1100
        Re: "High water" Memory fragmentation still a thing? bryanjugglercryptographer@yahoo.com - 2014-10-08 10:28 -0700
          Re: "High water" Memory fragmentation still a thing? Emile van Sebille <emile@fenx.com> - 2014-10-09 14:39 -0700
          Re: "High water" Memory fragmentation still a thing? Chris Angelico <rosuav@gmail.com> - 2014-10-10 08:42 +1100
          Re: "High water" Memory fragmentation still a thing? dieter <dieter@handshake.de> - 2014-10-10 08:09 +0200
          Re: "High water" Memory fragmentation still a thing? Chris Angelico <rosuav@gmail.com> - 2014-10-10 17:52 +1100

#78673 — Re: "High water" Memory fragmentation still a thing?

On Sat, Oct 4, 2014 at 4:36 AM, Croepha <croepha@gmail.com> wrote:
> What this means is that processes that do need to use a lot of memory will
> exhibit a "high water" behavior, where they remain forever at the level of
> memory usage that they required at their peak.

This is almost never true. What you will see, though, is something
like Christian described; pages get allocated, and then partially
used, and you don't always get all that memory back. In theory, a high
level language like Python would be allowed to move objects around to
compact memory, but CPython doesn't do this, and there's no proof that
it'd really help anything anyway. (Look at Christian's comments about
"Should you return it or not?" and the cost of system calls... now
consider the orders-of-magnitude worse cost of actually moving memory
around.)

This is why a lot of long-duration processes are built to be restarted
periodically. It's not strictly necessary, but it can be the most
effective way of solving a problem. I tend to ignore that, though, and
let my processes just keep on running... for 88 wk 4d 23:56:27 so far,
on one of those processes. It's consuming less than half a gig of
virtual memory, quarter gig resident, and it's been doing a fair bit
(it keeps all sorts of things in memory to avoid re-decoding from
disk). So don't worry too much about memory usage until you see that
there's actually a problem; with most Python processes, you'll restart
them to deploy new code sooner than you'll restart them to fix memory
problems. (The above example isn't a Python process, and code updates
happen live.) In fact, I'd advise that as a general policy: Don't
panic about any Python limitation until you've proven that it's
actually a problem. :)

ChrisA

[toc] | [next] | [standalone]

#78674

Chris Angelico wrote:

> In theory, a high
> level language like Python would be allowed to move objects around to
> compact memory, but CPython doesn't do this, and there's no proof that
> it'd really help anything anyway.

I welcome correction, but I understand that both the JVM and the CLR memory
managers move memory around. That's why Jython and IronPython use
sequential integers as object IDs, since memory locations are not fixed.

Way back in the mid 1980s, Apple Macintoshes used a memory manager which
could move memory around. Given that the Macs of the day had 128K of RAM,
of which something like a third or a half was used for the screen, being
able to move blocks of memory around to avoid fragmentation was critical,
so I guess that proves that it would help at least one thing.


> (Look at Christian's comments about 
> "Should you return it or not?" and the cost of system calls... now
> consider the orders-of-magnitude worse cost of actually moving memory
> around.)
> 
> This is why a lot of long-duration processes are built to be restarted
> periodically. 

Ironically, the cost of restarting the process periodically is likely to be
orders of magnitude more expensive than that of moving a few blocks of
memory around from time to time. Especially on Windows, where starting
processes is expensive, but even on Linux you have to shut the running
application down, then start it up again and rebuild all the internal data
structures that you just tore down...


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#78677

On Sat, Oct 4, 2014 at 11:02 AM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> Chris Angelico wrote:
>
>> In theory, a high
>> level language like Python would be allowed to move objects around to
>> compact memory, but CPython doesn't do this, and there's no proof that
>> it'd really help anything anyway.
>
> I welcome correction, but I understand that both the JVM and the CLR memory
> managers move memory around. That's why Jython and IronPython use
> sequential integers as object IDs, since memory locations are not fixed.

Right; I should have made it clearer that there's no proof that it'd
help anything *in CPython*. Removing the GIL is periodically proposed,
too, but there's no proof that its removal would benefit CPython; it's
not just that nobody's gotten around to writing a memory compactor for
CPython.

> Ironically, the cost of restarting the process periodically is likely to be
> orders of magnitude more expensive than that of moving a few blocks of
> memory around from time to time. Especially on Windows, where starting
> processes is expensive, but even on Linux you have to shut the running
> application down, then start it up again and rebuild all the internal data
> structures that you just tore down...

Maybe. But you deal with a number of things all at once:

1) Code updates (including interpreter updates)
2) Compaction of Python objects
3) Disposal of anything that got "high level leaked" - unintended
longevity caused by a global reference of some sort
4) Cleanup of low-level allocations that don't go through the Python
memory manager
etc etc etc.

So, yes, it's expensive. And sometimes it's not even possible (there's
no way to retain socket connections across a restart, AFAIK). But it's
there if you want it.

Personally, I like to keep processes running, but that's me. :)

ChrisA

[toc] | [prev] | [next] | [standalone]

#78679

Chris Angelico wrote:

> On Sat, Oct 4, 2014 at 11:02 AM, Steven D'Aprano
> <steve+comp.lang.python@pearwood.info> wrote:
>> Chris Angelico wrote:
>>
>>> In theory, a high
>>> level language like Python would be allowed to move objects around to
>>> compact memory, but CPython doesn't do this, and there's no proof that
>>> it'd really help anything anyway.
>>
>> I welcome correction, but I understand that both the JVM and the CLR
>> memory managers move memory around. That's why Jython and IronPython use
>> sequential integers as object IDs, since memory locations are not fixed.
> 
> Right; I should have made it clearer that there's no proof that it'd
> help anything *in CPython*. Removing the GIL is periodically proposed,
> too, but there's no proof that its removal would benefit CPython; it's
> not just that nobody's gotten around to writing a memory compactor for
> CPython.

I think that you're conflating a couple of different issues, although I
welcome correction.

I don't think that removing the GIL is a requirement for a memory compactor,
or vice versa. I think that PyPy is capable of plugging in various
different garbage collectors, including some without the GIL, which may or
may not include memory compactors. So as far as I can tell, the two are
independent.

As far as the GIL in CPython goes, there have been at least two attempts to
remove it, and they do show strong improvements for multi-threaded code
running on multi-core machines. Alas, they also show significant *slowdown*
for single-core machines, and very little improvement on dual-core
machines.

[Aside: The thing that people fail to understand is that the GIL is not in
fact something which *prevents* multi-tasking, but it *enables* cooperative
multi-tasking:

http://www.dabeaz.com/python/GIL.pdf

although that's not to say that there aren't some horrible performance
characteristics of the GIL. David Beazley has identified issues with the
GIL which suggest room for improving the GIL and avoiding "GIL battles"
which are responsible for much of the overhead of CPU-bound threads. Any C
programmers who want to hack on the interpreter core?]


Nevertheless, you're right that since nobody has actually built a version of
CPython with memory compactor, there's no *proof* that it would help
anything.


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#78681

On Sat, Oct 4, 2014 at 3:48 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> I think that you're conflating a couple of different issues, although I
> welcome correction.
>
> I don't think that removing the GIL is a requirement for a memory compactor,
> or vice versa. I think that PyPy is capable of plugging in various
> different garbage collectors, including some without the GIL, which may or
> may not include memory compactors. So as far as I can tell, the two are
> independent.
>
> As far as the GIL in CPython goes, there have been at least two attempts to
> remove it, and they do show strong improvements for multi-threaded code
> running on multi-core machines. Alas, they also show significant *slowdown*
> for single-core machines, and very little improvement on dual-core
> machines.

Not conflating, comparing. In both cases, it's perfectly possible *in
theory* to build a CPython with this change. (GIL removal is clearly
possible in theory, as it's been done in practice.) And *in theory*,
there's some benefit to be gained by doing so. But it's not a case of
"why doesn't python-dev just knuckle down and do it already", as
there's no evidence that it's a real improvement. (A "compile to
machine code" feature might well be purely beneficial, and that's
simply a matter of work - how much work, for how many CPU
architectures, to get how much benefit. But at least that's going to
have a fairly good expectation of performance improvement.) A memory
compactor might well help a narrow set of Python programs (namely,
those which allocate heaps and heaps of memory, then throw away most
of it, and keep running for a long time), but the complexity cost will
make it unlikely to be of benefit.

ChrisA

[toc] | [prev] | [next] | [standalone]

#78701

Steven D'Aprano <steve+comp.lang.python@pearwood.info> wrote:

> [Aside: The thing that people fail to understand is that the GIL is not in
> fact something which *prevents* multi-tasking, but it *enables* cooperative
> multi-tasking:
> 
> http://www.dabeaz.com/python/GIL.pdf
> 
> although that's not to say that there aren't some horrible performance
> characteristics of the GIL. David Beazley has identified issues with the
> GIL which suggest room for improving the GIL and avoiding "GIL battles"
> which are responsible for much of the overhead of CPU-bound threads. Any C
> programmers who want to hack on the interpreter core?]

Didn't the "new GIL" fix some of these problems?


Sturla

[toc] | [prev] | [next] | [standalone]

#78706

On Sat, 04 Oct 2014 11:02:02 +1000, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> declaimed the following:

>
>Way back in the mid 1980s, Apple Macintoshes used a memory manager which
>could move memory around. Given that the Macs of the day had 128K of RAM,
>of which something like a third or a half was used for the screen, being
>able to move blocks of memory around to avoid fragmentation was critical,
>so I guess that proves that it would help at least one thing.
>
	It also relied upon everything being double-dereferenced... Allocating
memory did not return the address of the allocated memory, but a handle
(pointer) to a structure that itself held the address of the target memory.

	Handles could not be moved, but the memory compaction algorithm was
free to move the allocated memory and update the handle to point to the new
location. Of course, this means one better not run out of space in the
"handle table" <G>

-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
    wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/

[toc] | [prev] | [next] | [standalone]

#78720

On 04/10/2014 02:02, Steven D'Aprano wrote:
> Way back in the mid 1980s, Apple Macintoshes used a memory manager which
> could move memory around.

But the memory manager didn't return a pointer to memory the way malloc 
does. It returned a pointer to the pointer and you had to double 
dereference it to get the heap address (ISTR, 30 years ago now). The 
advantage being the memory manager could shuffle the memory about and 
update the pointers. Your pointer to a pointer would still point to the 
same block after a shuffle. Of course you couldn't hold on to a partial 
dereference across system calls... can you guess why? :-)

Linux has (had) a allocation scheme where blocks came from different 
sized areas depending on the size requested. So all requests below 4k 
came from one heap area, and so on for 16k, 64k 256k, 1M etc. Meaning 
that code requesting a freeing up small amounts fragged the small 
allocation zone and so a big allocation would die due to fragmentation 
of small amounts. That was in kernel 2.4 days, sorry I'm off the 
bleeding edge now with how the allocator works in modern kernels.

Andy

[toc] | [prev] | [next] | [standalone]

#78740

On Sun, 05 Oct 2014 20:23:42 +0100, mm0fmf wrote:

> On 04/10/2014 02:02, Steven D'Aprano wrote:
>> Way back in the mid 1980s, Apple Macintoshes used a memory manager
>> which could move memory around.
> 
> But the memory manager didn't return a pointer to memory the way malloc
> does. It returned a pointer to the pointer and you had to double
> dereference it to get the heap address (ISTR, 30 years ago now).

Correct.


> The
> advantage being the memory manager could shuffle the memory about and
> update the pointers. Your pointer to a pointer would still point to the
> same block after a shuffle. Of course you couldn't hold on to a partial
> dereference across system calls... can you guess why? :-)

Because system calls might trigger a memory compaction or move.


Before the move, you have a managed "pointer to pointer" (handle) 
pointing to a managed point which points to a block of memory:


handle -----> pointer -----> "Some stuff here"

Grab a copy of the pointer with a partial deref:

myPtr := handle^;  (*I'm an old Pascal guy.*)


So we have this:

handle -----> pointer -----> "Some stuff here"
myPtr -----------------------^


Then you call a system routine that moves memory, and the memory manager 
moves the block, updating the pointer, but leaving myPtr pointing at 
garbage:


handle -----> pointer ----------------------------> "Some stuff here"
myPtr -----------------------^


and as soon as you try using myPtr, you likely get the dreaded Bomb 
dialog box:

http://www.macobserver.com/tmo/article/happy-birthday-mac-how-to-recover-from-the-dreaded-bomb-box-error-message

I'm not suggesting that a 1984 memory manager is the solution to all our 
problems. I'm just pointing it out as proof that the concept works. If I 
knew more about Java and .Net, I could use them as examples instead :-)



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#78768

Chris Angelico wrote:
> This is why a lot of long-duration processes are built to be restarted
> periodically. It's not strictly necessary, but it can be the most
> effective way of solving a problem. I tend to ignore that, though, and
> let my processes just keep on running... for 88 wk 4d 23:56:27 so far,
> on one of those processes. It's consuming less than half a gig of
> virtual memory, quarter gig resident, and it's been doing a fair bit [...]

A shipping product has to meet a higher standard. Planned process mortality is a reasonably simple strategy for building robust services from tools that have flaws in resource management. It assumes only that the operating system reliably reclaims resources from dead processes. 

The usual pattern is to have one or two parent processes that keep several worker processes running but do not themselves directly serve clients. The workers do the heavy lifting and are programmed to eventually die, letting younger workers take over.

For an example see the Apache HTTP daemon, particularly the classic pre-forking server. There's a configuration parameter, "MaxRequestsPerChild", that sets how many requests a process should answer before terminating.

[toc] | [prev] | [next] | [standalone]

#78770

On Tue, Oct 7, 2014 at 3:21 AM,
<bryanjugglercryptographer@yahoo.com.dmarc.invalid> wrote:
> Chris Angelico wrote:
>> This is why a lot of long-duration processes are built to be restarted
>> periodically. It's not strictly necessary, but it can be the most
>> effective way of solving a problem. I tend to ignore that, though, and
>> let my processes just keep on running... for 88 wk 4d 23:56:27 so far,
>> on one of those processes. It's consuming less than half a gig of
>> virtual memory, quarter gig resident, and it's been doing a fair bit [...]
>
> A shipping product has to meet a higher standard. Planned process mortality is a reasonably simple strategy for building robust services from tools that have flaws in resource management. It assumes only that the operating system reliably reclaims resources from dead processes.
>

Sure, and that's all well and good. But what I just cited there *is* a
shipping product. That's a live server that runs a game that I'm admin
of. So it's possible to do without the resource safety net of periodic
restarts.

> For an example see the Apache HTTP daemon, particularly the classic pre-forking server. There's a configuration parameter, "MaxRequestsPerChild", that sets how many requests a process should answer before terminating.
>

That assumes that requests can be handled equally by any server
process - and more so, that there are such things as discrete
requests. That's true of HTTP, but not of everything. And even with
HTTP, if you do "long polls" [1] then clients might remain connected
for arbitrary lengths of time; either you have to cut them off when
you terminate the server process (hopefully that isn't too often, or
you lose the benefit of long polling), or you retain processes for
much longer.

Restarting isn't necessary. It's like rebooting a computer: people get
into the habit of doing it, because it "fixes problems", but all that
means is that it allows you to get sloppy with resource management.
Working under the constraint that your one process will remain running
for at least a year forces you to be careful, and IMO results in
better code overall.

ChrisA

[1] https://en.wikipedia.org/wiki/Push_technology#Long_polling

[toc] | [prev] | [next] | [standalone]

#78844

Chris Angelico wrote:
> Sure, and that's all well and good. But what I just cited there *is* a
> shipping product. That's a live server that runs a game that I'm admin
> of. So it's possible to do without the resource safety net of periodic
> restarts.

Nice that the non-Python server you administer stayed up for 88 weeks, but that doesn't really have anything to do with the issue here. The answer to the OP's title question is "yes", high-water memory fragmentation is a real thing, in most platforms including CPython.

The cited article tells of Celery hitting the problem, and the working solution was to "roll the celery worker processes". That doesn't mean to tell a human administrator to regularly restart the server. It's programmatic and it's a reasonably simple and well-established design pattern.

> > For an example see the Apache HTTP daemon, particularly the classic pre-forking server. There's a configuration parameter, "MaxRequestsPerChild", that sets how many requests a process should answer before terminating.
> 
> That assumes that requests can be handled equally by any server
> process - and more so, that there are such things as discrete
> requests. That's true of HTTP, but not of everything. 

It's true of HTTP and many other protocols because they were designed to support robust operation even as individual components may fail.

> And even with
> HTTP, if you do "long polls" [1] then clients might remain connected
> for arbitrary lengths of time; either you have to cut them off when
> you terminate the server process (hopefully that isn't too often, or
> you lose the benefit of long polling), or you retain processes for
> much longer.

If you look at actual long-polling protocols, you'll see that the server occasionally closing connections is no problem at all. They're actually designed to be robust even against connections that drop without proper shutdown.

> Restarting isn't necessary. It's like rebooting a computer: people get
> into the habit of doing it, because it "fixes problems", but all that
> means is that it allows you to get sloppy with resource management.

CPython, and for that matter malloc/free, have known problems in resource management, such as the fragmentation issue noted here. There are more. Try a Google site search for "memory leak" on http://bugs.python.org/. Do you think the last memory leak is fixed now?

From what I've seen, planned process replacement is the primary techniques to support long-lived mission-critical services in face of resource management flaws. Upon process termination the OS recovers the resources. I love CPython, but on this point I trust the Linux kernel much more.

-- 
--Bryan

[toc] | [prev] | [next] | [standalone]

#78914

On 10/8/2014 10:28 AM, bryanjugglercryptographer@yahoo.com.dmarc.invalid 
wrote:

> That doesn't mean to tell a human administrator to regularly restart the server. It's programmatic and it's a reasonably simple and well-established design pattern.

I'd call it more a compensation technique than a design pattern*.  You 
know, like rebooting windows routinely.  :)

Emile


*) Alternately known as a workaround or kludge.

[toc] | [prev] | [next] | [standalone]

#78915

On Fri, Oct 10, 2014 at 8:39 AM, Emile van Sebille <emile@fenx.com> wrote:
> On 10/8/2014 10:28 AM, bryanjugglercryptographer@yahoo.com.dmarc.invalid
> wrote:
>
>> That doesn't mean to tell a human administrator to regularly restart the
>> server. It's programmatic and it's a reasonably simple and well-established
>> design pattern.
>
> I'd call it more a compensation technique than a design pattern*.  You know,
> like rebooting windows routinely.  :)
>
> *) Alternately known as a workaround or kludge.

That sounds about right to me.

ChrisA

[toc] | [prev] | [next] | [standalone]

#78924

Emile van Sebille <emile@fenx.com> writes:

> On 10/8/2014 10:28 AM,
> bryanjugglercryptographer@yahoo.com.dmarc.invalid wrote:
>
>> That doesn't mean to tell a human administrator to regularly restart the server. It's programmatic and it's a reasonably simple and well-established design pattern.
>
> I'd call it more a compensation technique than a design pattern*.  You
> know, like rebooting windows routinely.  :)
>
> Emile
>
>
> *) Alternately known as a workaround or kludge.

Well, everything has a price.

Python does not use memory compaction but places many objects on the
heap and therefore, long running processes
usually are subject to memory fragmentation. As a consequence, those
processes need to be restarted from time to time.

Would Python use memory compaction, implementing C extensions would
be much more tedious and error prone and there would be less such
extensions - limiting the domain where Python is used.

One has to choose.


People who do not like the choice of the
"CPython" implementation (no memory compaction) may look at
"Jython" (based on Java, with memory compaction). If they are lucky,
all the extensions they need are available there.

[toc] | [prev] | [next] | [standalone]

#78927

On Fri, Oct 10, 2014 at 5:09 PM, dieter <dieter@handshake.de> wrote:
> Python does not use memory compaction but places many objects on the
> heap and therefore, long running processes
> usually are subject to memory fragmentation. As a consequence, those
> processes need to be restarted from time to time.

Pike doesn't use memory compaction either, and also places pretty much
everything on the heap. Yet its processes don't need to be restarted;
in fact, everything's designed around keeping stuff running
permanently. As Emile said, it's not a requirement, it's a
compensation technique. I've yet to build a Python process that runs
for an entire year (as I'm not confident that I can get it so right
that I don't need to update any code), so I can't say for sure that it
would work, but I do have Python processes running for multiple months
without suffering serious fragmentation.

ChrisA

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web