Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder1.news.weretis.net!feeder.erje.net!newsfeed.xs4all.nl!newsfeed5.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
From: Pradipto Banerjee <pradipto.banerjee@adainvestments.com>
To: Dennis Lee Bieber <wlfraed@ix.netcom.com>, "python-list@python.org" <python-list@python.org>
Date: Fri, 19 Oct 2012 17:22:15 -0500
Subject: RE: Python does not take up available physical memory
Thread-Topic: Python does not take up available physical memory
Thread-Index: Ac2uRgWyYMXLoAsNSLqUqXak32HXSQAAe2HQ
References: <76D03718A3233B4C8CC236C169B535B5A23E063E45@AUSP01VMBX08.collaborationhost.net> <9oi388heolcjt1es6em5f9c93q73cranbh@invalid.netcom.com>
In-Reply-To: <9oi388heolcjt1es6em5f9c93q73cranbh@invalid.netcom.com>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.2541.1350685389.27098.python-list@python.org>
Lines: 93
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:31781

Dennis,

1. Yes, .readlines() work where .read() fails. Thanks for the suggestion - =
this has really given a big boost to the size of the data I can read.
2. Yes, I am running the python interpretation within the IDE (Spyder).

Thanks


-----Original Message-----
From: Python-list [mailto:python-list-bounces+pradipto.banerjee=3Dadainvest=
ments.com@python.org] On Behalf Of Dennis Lee Bieber
Sent: Friday, October 19, 2012 6:01 PM
To: python-list@python.org
Subject: Re: Python does not take up available physical memory

On Fri, 19 Oct 2012 12:08:53 -0500, Pradipto Banerjee
<pradipto.banerjee@adainvestments.com> declaimed the following in
gmane.comp.python.general:

> Hi,
>
> I am trying to read a file into memory. The size of the file is around 1 =
GB. I have a 3GB memory PC and the Windows Task Manager shows  2.3 GB avail=
able physical memory when I was trying to read the file. I tried to read th=
e file as follows:

1)      32-bit Windows OS can only physically address 4GB BUT
2)      Unless you have a SERVER version of windows, the most allocated to =
a
user process is 2GB (there is a boot-time parameter which can configure
WinXP [for example] to allow 3GB to user processes -- the last GB is
always reserved for the OS)

> Is there any reason why python can't read a 1GB file in memory even when =
a 2.3 GB physical memory is available? Do I need to make a change in some s=
etting or preferences?
>
> I am using python(x,y) distribution (python 2.7) and use Spyder as the ID=
E.
>
        Are you /running/ from within the IDE -- such that the Python
interpreter is considered part of the IDE address space?

        <snipped>

        My first thought would be that you should NOT be loading an entire
GB file as one chunk (for all I know, Windows low-level I/O read the
file into a OS buffer [1GB] and then tries to pass it on to the Python
memory buffer [another 1GB]... Or Python is reading in chunks -- say 1MB
at a time, and appending chunks... That means it goes "whoops, I've got
a 512MB data buffer and just read another 1MB -- better allocate a 768MB
buffer [total now is 512 + 768 + 1 =3D> 1281MB], copy the 512MB into the
new buffer, append the 1MB, NOW free the older 512MB... You might have
the raw memory, but it may be fragmented such that the system can not
allocate a contiguous 1MB chunk).

        If .readlines() works when .read() fails, it is memory fragmentatio=
n
(each parsed line is an individual chunk of memory since the list of
lines is just a list of addresses to the lines)
--
        Wulfraed                 Dennis Lee Bieber         AF6VN
        wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/

--
http://mail.python.org/mailman/listinfo/python-list

 This communication is for informational purposes only. It is not intended =
to be, nor should it be construed or used as, financial, legal, tax or inve=
stment advice or an offer to sell, or a solicitation of any offer to buy, a=
n interest in any fund advised by Ada Investment Management LP, the Investm=
ent advisor.  Any offer or solicitation of an investment in any of the Fund=
s may be made only by delivery of such Funds confidential offering material=
s to authorized prospective investors.  An investment in any of the Funds i=
s not suitable for all investors.  No representation is made that the Funds=
 will or are likely to achieve their objectives, or that any investor will =
or is likely to achieve results comparable to those shown, or will make any=
 profit at all or will be able to avoid incurring substantial losses.  Perf=
ormance results are net of applicable fees, are unaudited and reflect reinv=
estment of income and profits.  Past performance is no guarantee of future =
results. All financial data and other information are not warranted as to c=
ompleteness or accuracy and are subject to change without notice.

Any comments or statements made herein do not necessarily reflect those of =
Ada Investment Management LP and its affiliates. This transmission may cont=
ain information that is confidential, legally privileged, and/or exempt fro=
m disclosure under applicable law. If you are not the intended recipient, y=
ou are hereby notified that any disclosure, copying, distribution, or use o=
f the information contained herein (including any reliance thereon) is stri=
ctly prohibited. If you received this transmission in error, please immedia=
tely contact the sender and destroy the material in its entirety, whether i=
n electronic or hard copy format.