Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder4.news.weretis.net!feeds.phibee-telecom.net!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Date: Wed, 24 Oct 2012 09:12:29 +0200
From: Virgil Stokes <vs@it.uu.se>
User-Agent: Mozilla/5.0 (Windows NT 6.0; rv:17.0) Gecko/17.0 Thunderbird/17.0
MIME-Version: 1.0
To: "python-list@python.org" <python-list@python.org>
Subject: Re: Fast forward-backward (write-read)
References: <5086AA35.4000509@it.uu.se> <CAHVvXxRRSwV=A9Sx6odkpG2wOYwito7=OD6YPatcAutoBuAcgw@mail.gmail.com>
In-Reply-To: <CAHVvXxRRSwV=A9Sx6odkpG2wOYwito7=OD6YPatcAutoBuAcgw@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-15; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.2733.1351062745.27098.python-list@python.org>
Lines: 63
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:32010

On 24-Oct-2012 02:06, Oscar Benjamin wrote:
> On 23 October 2012 15:31, Virgil Stokes <vs@it.uu.se> wrote:
>> I am working with some rather large data files (>100GB) that contain time
>> series data. The data (t_k,y(t_k)), k = 0,1,...,N are stored in ASCII
>> format. I perform various types of processing on these data (e.g. moving
>> median, moving average, and Kalman-filter, Kalman-smoother) in a sequential
>> manner and only a small number of these data need be stored in RAM when
>> being processed. When performing Kalman-filtering (forward in time pass, k =
>> 0,1,...,N) I need to save to an external file several variables (e.g. 11*32
>> bytes) for each (t_k, y(t_k)). These are inputs to the Kalman-smoother
>> (backward in time pass, k = N,N-1,...,0). Thus, I will need to input these
>> variables saved to an external file from the forward pass, in reverse order
>> --- from last written to first written.
>>
>> Finally, to my question --- What is a fast way to write these variables to
>> an external file and then read them in backwards?
> You mentioned elsewhere that you are using numpy. I'll assume that the
> data you want to read/write are numpy arrays.
>
> Numpy arrays can be written very efficiently in binary form using
> tofile/fromfile:
>
>>>> import numpy
>>>> a = numpy.array([1, 2, 5], numpy.int64)
>>>> a
> array([1, 2, 5])
>>>> with open('data.bin', 'wb') as f:
> ...   a.tofile(f)
> ...
>
> You can then reload the array with:
>
>>>> with open('data.bin', 'rb') as f:
> ...   a2 = numpy.fromfile(f, numpy.int64)
> ...
>>>> a2
> array([1, 2, 5])
>
> Numpy arrays can be reversed before writing or after reading using;
>
>>>> a2
> array([1, 2, 5])
>>>> a2[::-1]
> array([5, 2, 1])
>
> Assuming you wrote the file forwards you can make an iterator to yield
> the file in chunks backwards like so (untested):
>
> def read_backwards(f, dtype, chunksize=1024 ** 2):
>      dtype = numpy.dtype(dtype)
>      nbytes = chunksize * dtype.itemsize
>      f.seek(0, 2)
>      fpos = f.tell()
>      while fpos > nbytes:
>          f.seek(fpos, 0)
>          yield numpy.fromfile(f, dtype, chunksize)[::-1]
>          fpos -= nbytes
>      yield numpy.fromfile(f, dtype)[::-1]
>
>
> Oscar
Ok Oscar,
Thanks for the tip and I will look into this more.