Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!feeder.erje.net!eu.feeder.erje.net!weretis.net!feeder4.news.weretis.net!ecngs!feeder2.ecngs.de!novso.com!newsfeed.xs4all.nl!newsfeed2.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
MIME-Version: 1.0
Date: Fri, 3 Oct 2014 13:36:20 -0500
Subject: "High water" Memory fragmentation still a thing?
From: Croepha <croepha@gmail.com>
To: python-list@python.org
Content-Type: multipart/alternative; boundary=089e014944a461228a050489023a
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.14580.1412361383.18130.python-list@python.org>
Lines: 71
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:78660

--089e014944a461228a050489023a
Content-Type: text/plain; charset=UTF-8

Question:

Does python in general terms (apart from extensions or gc manipulation),
exhibit a "high water" type leak of allocated memory in recent python
versions (2.7+)?

Background:


>From the post:
http://chase-seibert.github.io/blog/2013/08/03/diagnosing-memory-leaks-python.html

Begin quote:

Long running Python jobs that consume a lot of memory while running may not
return that memory to the operating system until the process actually
terminates, even if everything is garbage collected properly. That was news
to me, but it's true. What this means is that processes that do need to use
a lot of memory will exhibit a "high water" behavior, where they remain
forever at the level of memory usage that they required at their peak.

Note: this behavior may be Linux specific; there are anecdotal reports that
Python on Windows does not have this problem.

This problem arises from the fact that the Python VM does its own internal
memory management. It's commonly know as memory fragmentation.
Unfortunately, there doesn't seem to be any fool-proof method of avoiding
it.

End Quote

However this paper seems to indicate that that is not a modern problem:
http://www.evanjones.ca/memoryallocator/


--Thanks
Dave Butler

--089e014944a461228a050489023a
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Question:</div><div><br></div><div>Does python in gen=
eral terms (apart from extensions or gc manipulation), exhibit a &quot;high=
 water&quot; type leak of allocated memory in recent python versions (2.7+)=
?</div><div><br></div><div>Background:</div><div><br></div><div><br></div>F=
rom the post: <a href=3D"http://chase-seibert.github.io/blog/2013/08/03/dia=
gnosing-memory-leaks-python.html">http://chase-seibert.github.io/blog/2013/=
08/03/diagnosing-memory-leaks-python.html</a><br><div><br></div><div>Begin =
quote:</div><div><br></div><div><div>Long running Python jobs that consume =
a lot of memory while running may not return that memory to the operating s=
ystem until the process actually terminates, even if everything is garbage =
collected properly. That was news to me, but it&#39;s true. What this means=
 is that processes that do need to use a lot of memory will exhibit a &quot=
;high water&quot; behavior, where they remain forever at the level of memor=
y usage that they required at their peak.</div><div><br></div><div>Note: th=
is behavior may be Linux specific; there are anecdotal reports that Python =
on Windows does not have this problem.</div><div><br></div><div>This proble=
m arises from the fact that the Python VM does its own internal memory mana=
gement. It&#39;s commonly know as memory fragmentation. Unfortunately, ther=
e doesn&#39;t seem to be any fool-proof method of avoiding it.</div></div><=
div><br></div><div>End Quote</div><div><br></div><div>However this paper se=
ems to indicate that that is not a modern problem:=C2=A0<a href=3D"http://w=
ww.evanjones.ca/memoryallocator/">http://www.evanjones.ca/memoryallocator/<=
/a></div><div><br></div><div><br></div><div>--Thanks</div><div>Dave Butler<=
/div></div>

--089e014944a461228a050489023a--