Groups > comp.lang.python > #31973

Re: Fast forward-backward (write-read)

From	emile <emile@fenx.com>
Subject	Re: Fast forward-backward (write-read)
Date	2012-10-23 16:35 -0700
References	<5086AA35.4000509@it.uu.se> <CA+vVgJV6feUL0gTPC==3fp3Wq8zvRXgoyhVaUYnZNtMfF8qpLw@mail.gmail.com> <CA+vVgJWnAThHhD4cUJzXLGdVojCNA1oV_qKYwa+7UsEqS=x7XQ@mail.gmail.com> <CA+vVgJUNOZD3vBVFDXvXzdXb=c5THQG+B5dSOp_uF1nqDHhPug@mail.gmail.com> <CA+vVgJV2A7oJQCRGFypqsfYXHNVxSuhWJUhCF+FM4NWnXanGcA@mail.gmail.com>
Newsgroups	comp.lang.python
Message-ID	<mailman.2702.1351035353.27098.python-list@python.org> (permalink)

Show all headers | View raw

On 10/23/2012 04:19 PM, David Hutto wrote:
> Whether this is fast enough, or not, I don't know:

well, the OP's original post started with
   "I am working with some rather large data files (>100GB)..."

> filename = "data_file.txt"
> f = open(filename, 'r')
> forward =  [line.rstrip('\n') for line in f.readlines()]

f.readlines() will be big(!) and have overhead... and forward results in 
something again as big.

> backward =  [line.rstrip('\n') for line in reversed(forward)]

and defining backward looks to me to require space to build backward and 
hold reversed(forward)

So, let's see, at that point in time (building backward) you've got
probably somewhere close to 400-500Gb in memory.

My guess -- probably not so fast.  Thrashing is sure to be a factor on 
all but machines I'll never have a chance to work on.

> f.close()
> print forward, "\n\n", "********************\n\n", backward, "\n"

It's good to retain context.

Emile

Back to comp.lang.python | Previous | Next | Find similar | Unroll thread

Thread

Re: Fast forward-backward (write-read) emile <emile@fenx.com> - 2012-10-23 16:35 -0700

csiph-web