Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #31933

Fast forward-backward (write-read)

Show all headers | View raw

I am working with some rather large data files (>100GB) that contain time series 
data. The data (t_k,y(t_k)), k = 0,1,...,N are stored in ASCII format. I perform 
various types of processing on these data (e.g. moving median, moving average, 
and Kalman-filter, Kalman-smoother) in a sequential manner and only a small 
number of these data need be stored in RAM when being processed. When performing 
Kalman-filtering (forward in time pass, k = 0,1,...,N) I need to save to an 
external file several variables (e.g. 11*32 bytes) for each (t_k, y(t_k)). These 
are inputs to the Kalman-smoother (backward in time pass, k = N,N-1,...,0). 
Thus, I will need to input these variables saved to an external file from the 
forward pass, in reverse order --- from last written to first written.

Finally, to my question --- What is a fast way to write these variables to an 
external file and then read them in backwards?

Back to comp.lang.python | Previous | Next — Next in thread | Find similar | Unroll thread

Thread

Fast forward-backward (write-read) Virgil Stokes <vs@it.uu.se> - 2012-10-23 16:31 +0200
  Re: Fast forward-backward (write-read) Paul Rubin <no.email@nospam.invalid> - 2012-10-23 09:17 -0700
    Re: Fast forward-backward (write-read) Paul Rubin <no.email@nospam.invalid> - 2012-10-23 09:22 -0700
    Re: Fast forward-backward (write-read) Tim Chase <python.list@tim.thechases.com> - 2012-10-23 11:53 -0500
      Re: Fast forward-backward (write-read) Paul Rubin <no.email@nospam.invalid> - 2012-10-23 09:58 -0700
    Re: Fast forward-backward (write-read) Virgil Stokes <vs@it.uu.se> - 2012-10-23 19:06 +0200
  Re: Fast forward-backward (write-read) rusi <rustompmody@gmail.com> - 2012-10-24 08:11 -0700
    Re: Fast forward-backward (write-read) Virgil Stokes <vs@it.uu.se> - 2012-10-25 11:00 +0200

csiph-web