Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #78682 > unrolled thread

Re: "High water" Memory fragmentation still a thing?

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: "High water" Memory fragmentation still a thing? dieter <dieter@handshake.de> - 2014-10-04 08:33 +0200
    Re: "High water" Memory fragmentation still a thing? Marko Rauhamaa <marko@pacujo.net> - 2014-10-04 11:27 +0300
    Re: "High water" Memory fragmentation still a thing? bryanjugglercryptographer@yahoo.com - 2014-10-06 10:10 -0700

#78682 — Re: "High water" Memory fragmentation still a thing?

Croepha <croepha@gmail.com> writes:
> Does python in general terms (apart from extensions or gc manipulation),
> exhibit a "high water" type leak of allocated memory in recent python
> versions (2.7+)?

Likely because it is very difficult to avoid. Would you need to
definitely prevent it is "memory compaction": not only is garbage
collected and freed but in addition, the used memory is also
relocated to form contigous blocks of used and free memory.

Without memory compaction, long running processes tend to suffer
from "memory fragmentation": while sufficient free memory is available,
it is available only in small blocks, not large enough for some memory
requests and those requests then call for more memory from the operating
system. When this memory is later released, it may become split up in
smaller blocks and when another large memory request arrives, new memory
may be requested from the operating system (even though the original
block would have been large enough).

Python tries hard to limit the effect of fragmentation (maintaining the
free blocks in bins of given size) but cannot eliminate it completely.


In order to support memory compaction, all C extensions must adhere
to a strict memory access protocol: they cannot freely use C pointers
to access the memory (as the compaction may invalidate those pointers
at any time) but must beforehand announce "I will now be using this pointer"
(such that the compaction does not move the memory) and afterwards
announce "I am no longer using this pointer" (such that memory
compaction becomes again possible for this memory).

As you see from the description, memory compaction presents a heavy burden
for all extension writers.

[toc] | [next] | [standalone]

#78683

dieter <dieter@handshake.de>:

> Without memory compaction, long running processes tend to suffer from
> "memory fragmentation": while sufficient free memory is available, it
> is available only in small blocks, not large enough for some memory
> requests

Now this is a question for the computer scientists. The problem is quite
amenable to purely mathematical/statistical treatment. No doubt they've
been at it for decades.

My personal hunch is that GC in general works best with an ample amount
of RAM, where "ample" means, say, ten times the minimum amount needed.
As a bonus, I'm guessing the ample room would also all but remove the
memory fragmentation issue.


Marko

[toc] | [prev] | [next] | [standalone]

#78774

dieter wrote:
> As you see from the description, memory compaction presents a heavy burden
> for all extension writers.

Particularly because many CPython extensions are actually interfaces to pre-existing libraries. To leverage the system's facilities CPython has to follow the system's conventions, which memory compaction would break.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web