Path: csiph.com!usenet.pasdenom.info!gegeweb.42!gegeweb.eu!nntpfeed.proxad.net!proxad.net!feeder1-2.proxad.net!usenet-fr.net!de-l.enfer-du-nord.net!feeder1.enfer-du-nord.net!newsfeed.eweka.nl!eweka.nl!feeder3.eweka.nl!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Return-Path: <joncle@googlemail.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.011
X-Spam-Evidence: '*H*': 0.98; '*S*': 0.00; 'mrab': 0.04; 'ascii': 0.07; 'dict': 0.09; 'eof': 0.09; 'to:addr:comp.lang.python': 0.09; 'server,': 0.12; 'subject:file': 0.13; 'code?': 0.16; 'sqlite3': 0.16; 'stokes': 0.16; 'tokyo': 0.16; 'utc,': 0.16; 'cc:addr :python-list': 0.16; 'wrote:': 0.18; 'cc:no real name:2**0': 0.21; 'file,': 0.21; 'maybe': 0.21; 'header:In-Reply-To:1': 0.22; '(or': 0.22; 'subject:data': 0.25; 'cc:2**0': 0.26; 'position.': 0.28; 'module.': 0.29; 'retrieval': 0.29; 'cc:addr:python.org': 0.29; 'lines': 0.30; 'from:addr:googlemail.com': 0.31; 'subject:?': 0.31; 'file.': 0.31; 'that,': 0.32; 'there': 0.33; 'header:User- Agent:1': 0.33; 'file': 0.34; 'rather': 0.34; 'structured': 0.34; 'probably': 0.35; '...': 0.35; 'sets': 0.35; 'offset': 0.37; 'received:google.com': 0.37; 'either': 0.37; 'received:209.85': 0.38; 'monday,': 0.38; 'could': 0.38; 'several': 0.38; 'data': 0.38; 'header': 0.39; 'received:209': 0.39; 'put': 0.40; 'march': 0.61; 'full': 0.62; 'alphanumeric': 0.67; 'database.': 0.74; 'thousand': 0.74; 'blown': 0.84; 'cabinet': 0.84; 'subject:Fast': 0.84
Newsgroups: comp.lang.python
Date: Mon, 12 Mar 2012 20:38:25 -0700 (PDT)
In-Reply-To: <mailman.592.1331584145.3037.python-list@python.org>
Complaints-To: groups-abuse@google.com
Injection-Info: glegroupsg2000goo.googlegroups.com; posting-host=86.156.91.130; posting-account=HLD_OAoAAAD-0RilNRZUjdKEwXt97Q9q
References: <4F5E50F6.9070309@it.uu.se> <mailman.592.1331584145.3037.python-list@python.org>
User-Agent: G2/1.0
X-Google-Web-Client: true
MIME-Version: 1.0
Subject: Re: Fast file data retrieval?
From: Jon Clements <joncle@googlemail.com>
To: comp.lang.python@googlegroups.com
Content-Type: text/plain; charset=ISO-8859-1
Cc: python-list@python.org
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Message-ID: <mailman.599.1331609909.3037.python-list@python.org>
Lines: 31
NNTP-Posting-Host: 2001:888:2000:d::a6
X-Trace: 1331609909 news.xs4all.nl 6904 [2001:888:2000:d::a6]:52680
X-Complaints-To: abuse@xs4all.nl
Xref: csiph.com comp.lang.python:21550

On Monday, 12 March 2012 20:31:35 UTC, MRAB  wrote:
> On 12/03/2012 19:39, Virgil Stokes wrote:
> > I have a rather large ASCII file that is structured as follows
> >
> > header line
> > 9 nonblank lines with alphanumeric data
> > header line
> > 9 nonblank lines with alphanumeric data
> > ...
> > ...
> > ...
> > header line
> > 9 nonblank lines with alphanumeric data
> > EOF
> >
> > where, a data set contains 10 lines (header + 9 nonblank) and there can
> > be several thousand
> > data sets in a single file. In addition,*each header has a* *unique ID
> > code*.
> >
> > Is there a fast method for the retrieval of a data set from this large
> > file given its ID code?
> >
> Probably the best solution is to put it into a database. Have a look at
> the sqlite3 module.
> 
> Alternatively, you could scan the file, recording the ID and the file
> offset in a dict so that, given an ID, you can seek directly to that
> file position.

I would have a look at either bsddb, Tokyo (or Kyoto) Cabinet or hamsterdb. If it's really going to get large and needs a full blown server, maybe MongoDB/redis/hadoop...