Path: csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!news.albasani.net!newsfeed.freenet.ag!news2.euro.net!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
From: Juan Declet-Barreto <Juan.Declet-Barreto@MesaAZ.gov>
To: "python-list@python.org" <python-list@python.org>
Date: Mon, 7 Nov 2011 13:33:09 -0700
Subject: RE: memory management
Thread-Topic: memory management
Thread-Index: Acydiq+B6JBKcGiXTgKknLONHr0dvwAAGZkA
References: <3CB1190388197146B35D522652EAB501014C71D0C4@MESAMAIL01.acctcom.mesa> <4EB83D69.2000409@dejaviewphoto.com>
In-Reply-To: <4EB83D69.2000409@dejaviewphoto.com>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.2513.1320697994.27778.python-list@python.org>
Lines: 65
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: x330-a1.tempe.blueboxinc.net comp.lang.python:15435

Well, I am using Python 2.5 (and the IDLE shell) in Windows XP, which ships=
 with ESRI's ArcGIS. In addition, I am using some functions in the arcgissc=
ripting Python geoprocessing module for geographic information systems (GIS=
) applications, which can complicate things. I am currently isolating stand=
ard library Python code (e.g., os.walk()) from the arcgisscripting module t=
o evaluate in which module the environment crash is occurring.=20

-----Original Message-----
From: Dave Angel [mailto:davea@dejaviewphoto.com]=20
Sent: Monday, November 07, 2011 1:20 PM
To: Juan Declet-Barreto
Cc: python-list@python.org
Subject: Re: memory management


On 11/07/2011 02:43 PM, Juan Declet-Barreto wrote:
> Hi,
>
> Can anyone provide links or basic info on memory management, variable der=
eferencing, or the like? I have a script that traverses a file structure us=
ing os.walk and adds directory names to a list. It works for a small number=
 of directories, but when I set it loose on a directory with thousands of d=
irs/subdirs, it crashes the DOS session and also the Python shell (when I r=
un it from the shell).  This makes it difficult to figure out if the alloca=
ted memory or heap space for the DOS/shell session have overflown, or why i=
t is crashing.
>
> Juan Declet-Barreto [ciId:image001.png@01CC9D4A.CB6B9D70]
I don't have any reference to point you to, but CPython's memory management=
 is really pretty simple.  However, it's important to tell us the build of =
Python, as there are several, with very different memory rules.  For exampl=
e Jython, which is Python running in a Java VM, lets the java garbage colle=
ctor handle things, and it's entirely different.

Likewise, the OS may be relevant.  You're using Windows-kind of terminology=
, but that doesn't prove you're on Windows, nor does it say what version.

Assuming 32 bit CPython 2.7 on XP, the principles are simple.  When an=20
object is no longer accessible, it gets garbage collected*.   So if you=20
build a list inside a function, and the only reference is from a function's=
 local var, then the whole list will be freed when the function exits.  The=
 mistakes many people make are unnecessarily using globals, and using lists=
 when iterables would work just as well.

The tool on XP to tell how much memory is in use is the task manager. =20
As you point out, its hard to catch a short-running app in the act.  So you=
 want to add a counter to your code (global), and see how high it gets when=
 it crashes.  Then put a test in your code for the timer value, and do an "=
input" somewhat earlier.

At that point, see how much memory the program is actually using.

Now, when an object is freed, a new one of the same size is likely to immed=
iately re-use the space.  But if they're all different sizes, it's somewhat=
 statistical.  You might get fragmentation, for example.  When Python's poo=
l is full, it asks the OS for more (perhaps using swap space), but I don't =
think it ever gives it back.  So your memory use is a kind of ceiling case.=
  That's why it's problematic to build a huge data structure, and then walk=
 through it, then delete it.  The script will probably continue to show the=
 peak memory use, indefinitely.

* (technically, this is ref counted.  When the ref reaches zero the object =
is freed.  Real gc is more lazy scanning)