Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #31790

Re: Python does not take up available physical memory

Show all headers | View raw

On Sat, Oct 20, 2012 at 9:22 AM, Pradipto Banerjee
<pradipto.banerjee@adainvestments.com> wrote:
> Dennis,
>
> 1. Yes, .readlines() work where .read() fails. Thanks for the suggestion - this has really given a big boost to the size of the data I can read.

If at all possible, consider reading the file iteratively and
retaining only the current line. You say you need to "compare one line
versus another" - are you able, for instance, to sort the file
(externally; I think every OS these days will have a sort utility that
can handle files >RAM) and then compare the current line against a
small amount of retained state? This sort (pun intended) of trick can
not only get you past the limit, it will often improve running time
enormously too.

ChrisA

Back to comp.lang.python | Previous | Next | Find similar | Unroll thread

Thread

Re: Python does not take up available physical memory Chris Angelico <rosuav@gmail.com> - 2012-10-20 13:15 +1100

csiph-web