Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #31945 > unrolled thread

Re: Fast forward-backward (write-read)

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Fast forward-backward (write-read) Virgil Stokes <vs@it.uu.se> - 2012-10-23 19:17 +0200
    Re: Fast forward-backward (write-read) Cousin Stanley <cousinstanley@gmail.com> - 2012-10-23 20:03 +0000
      Re: Fast forward-backward (write-read) Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2012-10-24 01:20 -0400
      Re: Fast forward-backward (write-read) Virgil Stokes <vs@it.uu.se> - 2012-10-24 09:07 +0200
      Re: Fast forward-backward (write-read) Tim Golden <mail@timgolden.me.uk> - 2012-10-24 09:12 +0100

#31945 — Re: Fast forward-backward (write-read)

On 23-Oct-2012 18:09, Tim Chase wrote:
> On 10/23/12 09:31, Virgil Stokes wrote:
>> I am working with some rather large data files (>100GB) that contain time series
>> data. The data (t_k,y(t_k)), k = 0,1,...,N are stored in ASCII format. I perform
>> various types of processing on these data (e.g. moving median, moving average,
>> and Kalman-filter, Kalman-smoother) in a sequential manner and only a small
>> number of these data need be stored in RAM when being processed. When performing
>> Kalman-filtering (forward in time pass, k = 0,1,...,N) I need to save to an
>> external file several variables (e.g. 11*32 bytes) for each (t_k, y(t_k)). These
>> are inputs to the Kalman-smoother (backward in time pass, k = N,N-1,...,0).
>> Thus, I will need to input these variables saved to an external file from the
>> forward pass, in reverse order --- from last written to first written.
>>
>> Finally, to my question --- What is a fast way to write these variables to an
>> external file and then read them in backwards?
> Am I missing something, or would the fairly-standard "tac" utility
> do the reversal you want?  It should[*] be optimized to handle
> on-disk files in a smart manner.
Not sure about "tac" --- could you provide more details on this and/or a simple 
example of how it could be used for fast reversed "reading" of a data file?
>
> Otherwise, if you can pad the record-lengths so they're all the
> same, and you know the total number of records, you can seek to
> Total-(RecSize*OneBasedOffset) and write the record,optionally
> padding if you need/can.  At least on *nix-like OSes, you can seek
> into a sparse-file with no problems (untested on Win32).
The records lengths will all be the same and yes seek could be used; but, I was 
hoping for a faster method.

Thanks Tim! :-)
>
> -tkc
>
>
>
> [*]
> Just guessing here. Would be disappointed if it *wasn't*.

[toc] | [next] | [standalone]

#31954

Virgil Stokes wrote:

> Not sure about "tac" --- could you provide more details on this 
> and/or a simple example of how it could be used for fast reversed 
> "reading" of a data file ?

  tac is available as a command under linux .... 

  $ whatis tac
  tac (1) - concatenate and print files in reverse

  $ whereis tac
  tac: /usr/bin/tac /usr/bin/X11/tac /usr/share/man/man1/tac.1.gz

  $ man tac

  SYNOPSIS
    tac [OPTION]... [FILE]...

  DESCRIPTION

    Write each FILE to standard output, last line first.  

    With no FILE, or when FILE is -, read standard input.


  I only know that the  tac  command exists
  but have never used it myself ....


-- 
Stanley C. Kitching
Human Being
Phoenix, Arizona

[toc] | [prev] | [next] | [standalone]

#31990

On Tue, 23 Oct 2012 20:03:39 +0000 (UTC), Cousin Stanley
<cousinstanley@gmail.com> declaimed the following in
gmane.comp.python.general:

> 
>   $ whatis tac
>   tac (1) - concatenate and print files in reverse
>

	Oh... a backwards "cat" 
-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
        wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/

[toc] | [prev] | [next] | [standalone]

#32015

On 23-Oct-2012 22:03, Cousin Stanley wrote:
> Virgil Stokes wrote:
>
>> Not sure about "tac" --- could you provide more details on this
>> and/or a simple example of how it could be used for fast reversed
>> "reading" of a data file ?
>    tac is available as a command under linux ....
>
>    $ whatis tac
>    tac (1) - concatenate and print files in reverse
>
>    $ whereis tac
>    tac: /usr/bin/tac /usr/bin/X11/tac /usr/share/man/man1/tac.1.gz
>
>    $ man tac
>
>    SYNOPSIS
>      tac [OPTION]... [FILE]...
>
>    DESCRIPTION
>
>      Write each FILE to standard output, last line first.
>
>      With no FILE, or when FILE is -, read standard input.
>
>
>    I only know that the  tac  command exists
>    but have never used it myself ....
>
>
Unfortunately, I may be forced to process the data on a Windows platform; but, 
thanks Cousin for the Linux tip.

[toc] | [prev] | [next] | [standalone]

#32020

On 24/10/2012 08:07, Virgil Stokes wrote:
> On 23-Oct-2012 22:03, Cousin Stanley wrote:
>> Virgil Stokes wrote:
>>
>>> Not sure about "tac" --- could you provide more details on this
>>> and/or a simple example of how it could be used for fast reversed
>>> "reading" of a data file ?
>>    tac is available as a command under linux ....
>>
>>    $ whatis tac
>>    tac (1) - concatenate and print files in reverse
>>
>>    $ whereis tac
>>    tac: /usr/bin/tac /usr/bin/X11/tac /usr/share/man/man1/tac.1.gz
>>
>>    $ man tac
>>
>>    SYNOPSIS
>>      tac [OPTION]... [FILE]...
>>
>>    DESCRIPTION
>>
>>      Write each FILE to standard output, last line first.
>>
>>      With no FILE, or when FILE is -, read standard input.
>>
>>
>>    I only know that the  tac  command exists
>>    but have never used it myself ....
>>
>>
> Unfortunately, I may be forced to process the data on a Windows
> platform; but, thanks Cousin for the Linux tip.

Well, addressing that specific point, tac is available for Windows:

  http://unxutils.sourceforge.net/

No idea how efficient it is...

TJG

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web