Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #31796

Re: Python does not take up available physical memory

Show all headers | View raw

Thomas Rachel
<nutznetz-0c1b6768-bfa9-48d5-a470-7603bd3aa915@spamschutz.glglgl.de>
writes:

> Am 19.10.2012 21:03 schrieb Pradipto Banerjee:

[...]
>> Still got MemoryError, but at least this time python tried to use the
>> physical memory. What I noticed is that before it gave me the error
>> it used up to 1.5GB (of the 2.23 GB originally showed as available) -
>> so in general, python takes up more memory than the size of the file
>> itself.
>
> Of course - the file is not the only thing to be held by the process.
>
> I see several approaches here:
>
> * Process the file part by part - as the others already suggested,
> line-wise, but if you have e.g. a binary file format, other partings
> may be suitable as well - e.g. fixed block size, or parts given by the
> file format.
>
> * If you absolutely have to keep the whole file data in memory, split
> it up in several strings. Why? Well, the free space in virtual memory
> is not necessarily contiguous. So even if you have 1.5G free, you
> might not be able to read 1.5G at once, but you might succeed in
> reading 3*0.5G.

* try mmap, if you're lucky it will give you access to your data.

(Note that it is completely unreasonable to load several Gs of data in a
32-bit address space, especially if this is text. So my real advice
would be:

* read the file line per line and pack the contents of every line into
  a list of objects; once you have all your stuff, process it

-- Alain.

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

RE: Python does not take up available physical memory Pradipto Banerjee <pradipto.banerjee@adainvestments.com> - 2012-10-19 14:03 -0500
  Re: Python does not take up available physical memory Thomas Rachel <nutznetz-0c1b6768-bfa9-48d5-a470-7603bd3aa915@spamschutz.glglgl.de> - 2012-10-19 23:34 +0200
    Re: Python does not take up available physical memory Alain Ketterlin <alain@dpt-info.u-strasbg.fr> - 2012-10-20 07:09 +0200
  Re: Python does not take up available physical memory Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-10-19 22:12 +0000
    RE: Python does not take up available physical memory Pradipto Banerjee <pradipto.banerjee@adainvestments.com> - 2012-10-19 17:20 -0500
    Re: Python does not take up available physical memory Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2012-10-19 20:40 -0400
    RE: Python does not take up available physical memory Pradipto Banerjee <pradipto.banerjee@adainvestments.com> - 2012-10-21 09:14 -0500

csiph-web