Path: csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!gegeweb.org!de-l.enfer-du-nord.net!feeder2.enfer-du-nord.net!feeder.news-service.com!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Date: Tue, 02 Aug 2011 13:09:57 +0200
From: Thomas Jollans <t@jollybox.de>
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:5.0) Gecko/20110628 Thunderbird/5.0
MIME-Version: 1.0
To: python-list@python.org
Subject: Re: python reading file memory cost
References: <000f01cc505c$74e01e80$5ea05b80$@com>	<CAGGBd_quXLPkr0kNDPJhOwcdrf2BxELQmXcq7_RKW-e9Ox=yFg@mail.gmail.com>	<1312255370.31275.20.camel@ZTFEL>	<CAMZYqRSwsPyz+GK76z6Hvbi8smAj_PWVm3C4W8nahaO1XsF6fQ@mail.gmail.com> <j18car$i0u$1@dough.gmane.org> <000a01cc5103$6be8ea80$43babf80$@sinap.ac.cn>
In-Reply-To: <000a01cc5103$6be8ea80$43babf80$@sinap.ac.cn>
OpenPGP: id=5C8691ED
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.1762.1312283392.1164.python-list@python.org>
Lines: 55
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: x330-a1.tempe.blueboxinc.net comp.lang.python:10717

On 02/08/11 13:00, 张彤 wrote:
> Thanks Peter! Your explanation is great!
> And one more question:
> Why it is still keeping the memory even when I del the large array in
> interactive python mode?

This is an optimisation of the way the Python interpreter allocates
memory: it holds on to memory it's not using any more for a while so it
can be easily re-used for new objects --- this is more efficient than
giving the memory back to the operating system only to request it again
shortly afterwards.

> 
> -----Original Message-----
> From: Peter Otten [mailto:__peter__@web.de] 
> Sent: Tuesday, August 02, 2011 4:26 PM
> To: python-list@python.org
> Subject: Re: python reading file memory cost
> 
> Chris Rebert wrote:
> 
>>> The running result was that read a 500M file consume almost 2GB RAM, 
>>> I cannot figure it out, somebody help!
>>
>> If you could store the floats themselves, rather than their string 
>> representations, that would be more space-efficient. You could then 
>> also use the `array` module, which is more space-efficient than lists 
>> (http://docs.python.org/library/array.html ). Numpy would also be 
>> worth investigating since multidimensional arrays are involved.
>>
>> The next obvious question would then be: do you /really/ need /all/ of 
>> the data in memory at once?
> 
> This is what you (OP) should think about really hard before resorting to the
> optimizations mentioned above. Perhaps you can explain what you are doing
> with the data once you've loaded it into memory?
> 
>> Also, just so you're aware:
>> http://docs.python.org/library/sys.html#sys.getsizeof
> 
> To give you an idea how memory usage explodes:
> 
>>>> line = "1.23 4.56 7.89 0.12\n"
>>>> len(line) # size in the file
> 20
>>>> sys.getsizeof(line)
> 60
>>>> formatted = ["%2.6E" % float(x) for x in line.split()]
>>>> sys.getsizeof(formatted) + sum(sys.getsizeof(s) for s in formatted)
> 312
> 
> 
> 
>