Path: csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!gegeweb.org!de-l.enfer-du-nord.net!feeder2.enfer-du-nord.net!feeds.phibee-telecom.net!newsfeed.xs4all.nl!newsfeed5.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Return-Path: <d@davea.name>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.029
X-Spam-Evidence: '*H*': 0.94; '*S*': 0.00; 'python': 0.08; 'complicate': 0.09; 'filename': 0.09; 'files:': 0.09; 'iterate': 0.09; 'things.': 0.13; 'def': 0.14; 'library': 0.14; 'cc:addr :python-list': 0.15; 'received:192.168.1.104': 0.16; 'subject:memory': 0.16; 'wrote:': 0.18; '(e.g.,': 0.19; 'memory': 0.20; 'file,': 0.21; 'header:In-Reply-To:1': 0.23; 'cc:2**0': 0.25; 'code': 0.25; 'windows': 0.26; '(and': 0.28; 'idle': 0.28; 'order.': 0.28; 'yield': 0.28; 'cc:addr:python.org': 0.29; 'environment': 0.30; 'module': 0.30; 'logic': 0.30; 'pm,': 0.31; 'list': 0.32; 'app': 0.32; 'it.': 0.33; 'instead': 0.33; 'header :User-Agent:1': 0.33; 'apply': 0.33; 'usual': 0.34; 'file': 0.36; 'addition,': 0.36; 'before.': 0.36; 'doing': 0.37; 'list,': 0.37; 'using': 0.37; 'some': 0.38; 'perhaps': 0.38; 'received:192': 0.38; 'mailing': 0.38; 'should': 0.39; 'subject:: ': 0.39; 'getting': 0.40; 'your': 0.61; 'information,': 0.66; 'crash': 0.67; 'header:Reply-To:1': 0.70; 'evaluate': 0.71; 'reply-to:no real name:2**0': 0.71; 'generator:': 0.84; 'subject:management': 0.84; 'xp,': 0.84; 'ships': 0.91
Date: Mon, 07 Nov 2011 15:50:19 -0500
From: Dave Angel <d@davea.name>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.23) Gecko/20110922 Thunderbird/3.1.15
MIME-Version: 1.0
To: Juan Declet-Barreto <Juan.Declet-Barreto@MesaAZ.gov>
Subject: Re: memory management
References: <3CB1190388197146B35D522652EAB501014C71D0C4@MESAMAIL01.acctcom.mesa>	<4EB83D69.2000409@dejaviewphoto.com> <3CB1190388197146B35D522652EAB501014C71D167@MESAMAIL01.acctcom.mesa>
In-Reply-To: <3CB1190388197146B35D522652EAB501014C71D167@MESAMAIL01.acctcom.mesa>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Provags-ID: V02:K0:SfsAZt6+dhEpSl2Ny/fKKKszT/kzjDoEOPSn24/Ripn HRBThwEE4ZkiH7R2uLLWE56+zS9SWzozgoKWrYbCzyNwOTEb+1 7nNWbSeLTAr0gxdV8kOfG5+bd0fwneMOcXtzYmEX0b/ONF2IOh UUXWXBeX+qpa5xi3tRnbOvom3aA/ciVC/jFTRuQAEB3yVwSR+E eq54EljvoxG17p+UhLFOsbiUfN4SyL3n9s8wjEXF/7QqpBXqGK s1k4On26pXAjp1DQORXV12J69X42R96Jc6xfHf/k8fbI13o4O7 D9Jzwb2E77Civ5UULclCwR8eZ6LRDAVFYZHDdouyK6c0vp5sQ= =
Cc: "python-list@python.org" <python-list@python.org>
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.12
Precedence: list
Reply-To: d@davea.name
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups: comp.lang.python
Message-ID: <mailman.2515.1320699053.27778.python-list@python.org>
Lines: 34
NNTP-Posting-Host: 2001:888:2000:d::a6
X-Trace: 1320699053 news.xs4all.nl 6962 [2001:888:2000:d::a6]:55855
X-Complaints-To: abuse@xs4all.nl
Xref: x330-a1.tempe.blueboxinc.net comp.lang.python:15437

On 11/07/2011 03:33 PM, Juan Declet-Barreto wrote:
> Well, I am using Python 2.5 (and the IDLE shell) in Windows XP, which ships with ESRI's ArcGIS. In addition, I am using some functions in the arcgisscripting Python geoprocessing module for geographic information systems (GIS) applications, which can complicate things. I am currently isolating standard library Python code (e.g., os.walk()) from the arcgisscripting module to evaluate in which module the environment crash is occurring.
You top-posted.  In this mailing list, one should type new information 
after the quoted information, not before.

Perhaps a pre-emptive strike is in order.  On the assumption that it may 
be a memory problem, how about you turn the app inside out.  Instead of 
walking the entire tree, getting a list with all the paths, and then 
working on the list, how about doing the work on each file as you get 
it.  Or even make your own generator from os.walk, so the app can call 
your logic on each file, and never have all the file (name)s in memory 
at the same time.


Generator:

def  filelist(top, criteria):
       for  a, b, c in os.walk():
              for fiile in files:
                    apply some criteria
                    yield file


Now the main app can iterate through this "list" in the usual way

for filename in filelist(top, "*.txt"):
        dosomething...



-- 

DaveA