Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #31790 > unrolled thread

Re: Python does not take up available physical memory

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Python does not take up available physical memory Chris Angelico <rosuav@gmail.com> - 2012-10-20 13:15 +1100

#31790 — Re: Python does not take up available physical memory

On Sat, Oct 20, 2012 at 9:22 AM, Pradipto Banerjee
<pradipto.banerjee@adainvestments.com> wrote:
> Dennis,
>
> 1. Yes, .readlines() work where .read() fails. Thanks for the suggestion - this has really given a big boost to the size of the data I can read.

If at all possible, consider reading the file iteratively and
retaining only the current line. You say you need to "compare one line
versus another" - are you able, for instance, to sort the file
(externally; I think every OS these days will have a sort utility that
can handle files >RAM) and then compare the current line against a
small amount of retained state? This sort (pun intended) of trick can
not only get you past the limit, it will often improve running time
enormously too.

ChrisA

[toc] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web