Re: Python does not take up available physical memory

Path	csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!news.mixmin.net!feed.xsnews.nl!border-3.ams.xsnews.nl!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Return-Path	<rosuav@gmail.com>
X-Original-To	python-list@python.org
Delivered-To	python-list@mail.python.org
X-Spam-Status	OK 0.008
X-Spam-Evidence	'H': 0.98; 'S': 0.00; 'subject:Python': 0.05; 'fails.': 0.09; 'retained': 0.09; 'subject:not': 0.11; 'read.': 0.13; 'sat,': 0.15; 'able,': 0.16; 'from:addr:rosuav': 0.16; 'from:name:chris angelico': 0.16; 'iteratively': 0.16; 'limit,': 0.16; 'oct': 0.16; 'wrote:': 0.17; 'instance,': 0.17; 'sort': 0.21; 'header:In-Reply-To:1': 0.25; 'am,': 0.27; 'possible,': 0.27; 'message-id:@mail.gmail.com': 0.27; 'file': 0.32; 'suggestion': 0.32; 'running': 0.32; 'utility': 0.33; 'handle': 0.33; 'to:addr:python-list': 0.33; 'received:google.com': 0.34; 'thanks': 0.34; 'too.': 0.35; 'received:209.85': 0.35; 'really': 0.36; 'compare': 0.36; 'received:209': 0.37; 'data': 0.37; 'subject:: ': 0.38; 'files': 0.38; 'to:addr:python.org': 0.39; 'where': 0.40; 'header:Received:5': 0.40; 'think': 0.40; 'days': 0.60; '20,': 0.65; 'subject:available': 0.96
DKIM-Signature	v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=BU0tfB0jEqkgfq7p1uQvUTu4HHj+xXGkyqOdwW8EFms=; b=Cmk793KDMTmenIcDqpVJhp+DZ4iD9FyLSnTUOOvBTwc+Sbsudx2XQRygpP/mYpt3xd caO5DlN9sRneztDOLcX3ANjOTx6v6Gy1/jbpDYJcq65KxYts7spFLhvsoy/H+ME3SDdg pecMG9ISymNUQav5Y41G+E7Kz94cBsYSZdpPADkVWwFttbdFH44TMxFbndZG/yaEdTBl 8fJcAP5g87fe++Y6Clrr4JRgRo87EENYb/kPQuqwSohtk9okXp/YLHsKG70mwwFqBX9T xcThwjdi1afwqJYaJmDdH3aIbnjj9CjeZwk75640SsgWR5fS8d+Ml5U78AGnYvrZjL42 yZlw==
MIME-Version	1.0
In-Reply-To	<76D03718A3233B4C8CC236C169B535B5A23E063F6D@AUSP01VMBX08.collaborationhost.net>
References	<76D03718A3233B4C8CC236C169B535B5A23E063E45@AUSP01VMBX08.collaborationhost.net> <9oi388heolcjt1es6em5f9c93q73cranbh@invalid.netcom.com> <76D03718A3233B4C8CC236C169B535B5A23E063F6D@AUSP01VMBX08.collaborationhost.net>
Date	Sat, 20 Oct 2012 13:15:10 +1100
Subject	Re: Python does not take up available physical memory
From	Chris Angelico <rosuav@gmail.com>
To	python-list@python.org
Content-Type	text/plain; charset=ISO-8859-1
X-BeenThere	python-list@python.org
X-Mailman-Version	2.1.15
Precedence	list
List-Id	General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe	<http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive	<http://mail.python.org/pipermail/python-list/>
List-Post	<mailto:python-list@python.org>
List-Help	<mailto:python-list-request@python.org?subject=help>
List-Subscribe	<http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups	comp.lang.python
Message-ID	<mailman.2548.1350699319.27098.python-list@python.org> (permalink)
Lines	16
NNTP-Posting-Host	2001:888:2000:d::a6
X-Trace	1350699319 news.xs4all.nl 6865 [2001:888:2000:d::a6]:44434
X-Complaints-To	abuse@xs4all.nl
Xref	csiph.com comp.lang.python:31790

Show key headers only | View raw

On Sat, Oct 20, 2012 at 9:22 AM, Pradipto Banerjee
<pradipto.banerjee@adainvestments.com> wrote:
> Dennis,
>
> 1. Yes, .readlines() work where .read() fails. Thanks for the suggestion - this has really given a big boost to the size of the data I can read.

If at all possible, consider reading the file iteratively and
retaining only the current line. You say you need to "compare one line
versus another" - are you able, for instance, to sort the file
(externally; I think every OS these days will have a sort utility that
can handle files >RAM) and then compare the current line against a
small amount of retained state? This sort (pun intended) of trick can
not only get you past the limit, it will often improve running time
enormously too.

ChrisA

Back to comp.lang.python | Previous | Next | Find similar | Unroll thread

Thread

Re: Python does not take up available physical memory Chris Angelico <rosuav@gmail.com> - 2012-10-20 13:15 +1100

csiph-web