Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder1.news.weretis.net!feeder.erje.net!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
MIME-Version: 1.0
In-Reply-To: <5086AA35.4000509@it.uu.se>
References: <5086AA35.4000509@it.uu.se>
Date: Wed, 24 Oct 2012 01:06:13 +0100
Subject: Re: Fast forward-backward (write-read)
From: Oscar Benjamin <oscar.j.benjamin@gmail.com>
To: Virgil Stokes <vs@it.uu.se>
Content-Type: text/plain; charset=ISO-8859-1
Cc: "python-list@python.org" <python-list@python.org>
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.2706.1351037177.27098.python-list@python.org>
Lines: 61
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:31980

On 23 October 2012 15:31, Virgil Stokes <vs@it.uu.se> wrote:
> I am working with some rather large data files (>100GB) that contain time
> series data. The data (t_k,y(t_k)), k = 0,1,...,N are stored in ASCII
> format. I perform various types of processing on these data (e.g. moving
> median, moving average, and Kalman-filter, Kalman-smoother) in a sequential
> manner and only a small number of these data need be stored in RAM when
> being processed. When performing Kalman-filtering (forward in time pass, k =
> 0,1,...,N) I need to save to an external file several variables (e.g. 11*32
> bytes) for each (t_k, y(t_k)). These are inputs to the Kalman-smoother
> (backward in time pass, k = N,N-1,...,0). Thus, I will need to input these
> variables saved to an external file from the forward pass, in reverse order
> --- from last written to first written.
>
> Finally, to my question --- What is a fast way to write these variables to
> an external file and then read them in backwards?

You mentioned elsewhere that you are using numpy. I'll assume that the
data you want to read/write are numpy arrays.

Numpy arrays can be written very efficiently in binary form using
tofile/fromfile:

>>> import numpy
>>> a = numpy.array([1, 2, 5], numpy.int64)
>>> a
array([1, 2, 5])
>>> with open('data.bin', 'wb') as f:
...   a.tofile(f)
...

You can then reload the array with:

>>> with open('data.bin', 'rb') as f:
...   a2 = numpy.fromfile(f, numpy.int64)
...
>>> a2
array([1, 2, 5])

Numpy arrays can be reversed before writing or after reading using;

>>> a2
array([1, 2, 5])
>>> a2[::-1]
array([5, 2, 1])

Assuming you wrote the file forwards you can make an iterator to yield
the file in chunks backwards like so (untested):

def read_backwards(f, dtype, chunksize=1024 ** 2):
    dtype = numpy.dtype(dtype)
    nbytes = chunksize * dtype.itemsize
    f.seek(0, 2)
    fpos = f.tell()
    while fpos > nbytes:
        f.seek(fpos, 0)
        yield numpy.fromfile(f, dtype, chunksize)[::-1]
        fpos -= nbytes
    yield numpy.fromfile(f, dtype)[::-1]


Oscar