Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #21545 > unrolled thread

Re: Fast file data retrieval?

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Fast file data retrieval? Arnaud Delobelle <arnodel@gmail.com> - 2012-03-12 20:31 +0000

#21545 — Re: Fast file data retrieval?

On 12 March 2012 19:39, Virgil Stokes <vs@it.uu.se> wrote:
> I have a rather large ASCII file that is structured as follows
>
> header line
> 9 nonblank lines with alphanumeric data
> header line
> 9 nonblank lines with alphanumeric data
> ...
> ...
> ...
> header line
> 9 nonblank lines with alphanumeric data
> EOF
>
> where, a data set contains 10 lines (header + 9 nonblank) and there can be
> several thousand
> data sets in a single file. In addition, each header has a unique ID code.
>
> Is there a fast method for the retrieval of a data set from this large file
> given its ID code?

It depends.  I guess if it's a long running application, you could
load up all the data into a dictionary at startup time (several
thousand data sets doesn't sound like that much).  Another option
would be to put all this into a dbm database file
(http://docs.python.org/library/dbm.html) - it would be very easy to
do.

Or you could have your own custom solution where you scan the file and
build a dictionary mapping keys to file offsets, then when requesting
a dataset you can seek directly to the correct position.
-- 
Arnaud

[toc] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web