Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #31946

Re: Fast forward-backward (write-read)

Show key headers only | View raw

On 23-Oct-2012 18:17, Paul Rubin wrote:
> Virgil Stokes <vs@it.uu.se> writes:
>> Finally, to my question --- What is a fast way to write these
>> variables to an external file and then read them in backwards?
> Seeking backwards in files works, but the performance hit is
> significant.  There is also a performance hit to scanning pointers
> backwards in memory, due to cache misprediction.  If it's something
> you're just running a few times, seeking backwards the simplest
> approach.  If you're really trying to optimize the thing, you might
> buffer up large chunks (like 1 MB) before writing.  If you're writing
> once and reading multiple times, you might reverse the order of records
> within the chunks during the writing phase.
I am writing (forward) once and reading (backward) once.
>
> You're of course taking a performance bath from writing the program in
> Python to begin with (unless using scipy/numpy or the like), enough that
> it might dominate any effects of how the files are written.
I am currently using SciPy/NumPy
>
> Of course (it should go without saying) that you want to dump in a
> binary format rather than converting to decimal.
Yes, I am doing this (but thanks for "underlining" it!)

Thanks Paul :-)

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Fast forward-backward (write-read) Virgil Stokes <vs@it.uu.se> - 2012-10-23 16:31 +0200
  Re: Fast forward-backward (write-read) Paul Rubin <no.email@nospam.invalid> - 2012-10-23 09:17 -0700
    Re: Fast forward-backward (write-read) Paul Rubin <no.email@nospam.invalid> - 2012-10-23 09:22 -0700
    Re: Fast forward-backward (write-read) Tim Chase <python.list@tim.thechases.com> - 2012-10-23 11:53 -0500
      Re: Fast forward-backward (write-read) Paul Rubin <no.email@nospam.invalid> - 2012-10-23 09:58 -0700
    Re: Fast forward-backward (write-read) Virgil Stokes <vs@it.uu.se> - 2012-10-23 19:06 +0200
  Re: Fast forward-backward (write-read) rusi <rustompmody@gmail.com> - 2012-10-24 08:11 -0700
    Re: Fast forward-backward (write-read) Virgil Stokes <vs@it.uu.se> - 2012-10-25 11:00 +0200

csiph-web