Path: csiph.com!newsfeed.hal-mli.net!feeder3.hal-mli.net!newsfeed.hal-mli.net!feeder1.hal-mli.net!newsfeed.xs4all.nl!newsfeed5.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
MIME-Version: 1.0
In-Reply-To: <CAHVvXxRRSwV=A9Sx6odkpG2wOYwito7=OD6YPatcAutoBuAcgw@mail.gmail.com>
References: <5086AA35.4000509@it.uu.se> <CAHVvXxRRSwV=A9Sx6odkpG2wOYwito7=OD6YPatcAutoBuAcgw@mail.gmail.com>
Date: Tue, 23 Oct 2012 22:29:09 -0400
Subject: Re: Fast forward-backward (write-read)
From: David Hutto <dwightdhutto@gmail.com>
To: Oscar Benjamin <oscar.j.benjamin@gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
Cc: "python-list@python.org" <python-list@python.org>
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.2708.1351045752.27098.python-list@python.org>
Lines: 75
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:31983

On Tue, Oct 23, 2012 at 8:06 PM, Oscar Benjamin
<oscar.j.benjamin@gmail.com> wrote:
> On 23 October 2012 15:31, Virgil Stokes <vs@it.uu.se> wrote:
>> I am working with some rather large data files (>100GB) that contain time
>> series data. The data (t_k,y(t_k)), k = 0,1,...,N are stored in ASCII
>> format. I perform various types of processing on these data (e.g. moving
>> median, moving average, and Kalman-filter, Kalman-smoother) in a sequential
>> manner and only a small number of these data need be stored in RAM when
>> being processed. When performing Kalman-filtering (forward in time pass, k =
>> 0,1,...,N) I need to save to an external file several variables (e.g. 11*32
>> bytes) for each (t_k, y(t_k)). These are inputs to the Kalman-smoother
>> (backward in time pass, k = N,N-1,...,0). Thus, I will need to input these
>> variables saved to an external file from the forward pass, in reverse order
>> --- from last written to first written.
>>
>> Finally, to my question --- What is a fast way to write these variables to
>> an external file and then read them in backwards?
>
> You mentioned elsewhere that you are using numpy. I'll assume that the
> data you want to read/write are numpy arrays.

If that is the case always timeit. The following is an example of 3
functions, with repetitions of time that give an average:

import timeit
#3 dimensional matrix
x_dim = -1
y_dim = -1
z_dim = -1
s = """\

x_dim = -1
y_dim = -1
z_dim = -1
dict_1 = {}

for i in xrange(0,6):
	x_dim = 1
	y_dim = 1
	z_dim = 1
	dict_1['%s' % (i) ] = ['x = %i' % (x_dim), 'y = %i' % (y_dim),  'z =
%i' % (z_dim)]

"""

t = """\
import numpy
numpy.array([[ 1.,  0.,  0.],
       [ 0.,  1.,  2.]])
"""

u = """\
list_count = 0
an_array = []
for i in range(0,10):

	if list_count > 3:
		break
	
	if i % 3 != 0:
		an_array.append(i)

	if i % 3 == 0:
		list_count += 1

"""
print timeit.timeit(stmt=s, number=100000)
print timeit.timeit(stmt=t, number=100000)
print timeit.timeit(stmt=u, number=100000)


-- 
Best Regards,
David Hutto
CEO: http://www.hitwebdevelopment.com