Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder4.news.weretis.net!newsreader4.netcologne.de!news.netcologne.de!feeder2.cambriumusenet.nl!feed.tweaknews.nl!194.109.133.85.MISMATCH!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.027 X-Spam-Evidence: '*H*': 0.95; '*S*': 0.00; 'ascii': 0.07; 'eof': 0.09; 'subject:file': 0.13; 'startup': 0.15; 'code?': 0.16; 'depends.': 0.16; 'stokes': 0.16; 'url:html)': 0.16; 'cc:addr:python-list': 0.16; 'wrote:': 0.18; 'cc:no real name:2**0': 0.21; "doesn't": 0.22; 'header:In-Reply-To:1': 0.22; 'dictionary': 0.23; 'received:209.85.213.46': 0.23; 'received:mail- yw0-f46.google.com': 0.23; 'subject:data': 0.25; 'guess': 0.26; 'cc:2**0': 0.26; 'code.': 0.26; 'load': 0.26; 'position.': 0.28; 'sound': 0.28; 'message-id:@mail.gmail.com': 0.29; 'mapping': 0.29; 'retrieval': 0.29; 'cc:addr:python.org': 0.29; 'correct': 0.29; 'lines': 0.30; 'url:library': 0.31; 'subject:?': 0.31; 'do.': 0.31; 'file.': 0.31; 'there': 0.33; 'addition,': 0.34; 'file': 0.34; 'rather': 0.34; 'running': 0.34; 'keys': 0.34; 'structured': 0.34; '...': 0.35; 'sets': 0.35; 'url:python': 0.35; 'received:google.com': 0.37; 'another': 0.37; 'received:209.85': 0.38; 'could': 0.38; 'several': 0.38; 'data': 0.38; 'url:org': 0.39; 'option': 0.39; 'header': 0.39; 'received:209': 0.39; 'put': 0.40; 'easy': 0.60; 'unique': 0.61; 'custom': 0.61; 'your': 0.61; 'march': 0.61; 'alphanumeric': 0.67; 'thousand': 0.74; 'subject:Fast': 0.84 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=Ni6neRw/3sjlHrBOsbl0RA/B3hfODnYRpVXE0wzJ12k=; b=ejRVBL1l7KU3ItaWM1mifIk2Lt5GoZPXl1286F6VvHLpwHfhUpdP7hMs2ldJh4qbqu TyScZqxmwxTHWhqCvi71eQFW+9uZJ4DaiCFXx0fj+IJEjVvDdH7YbLdgtQqY++ZYmQrw VlbLC1sUNnYgkEIJvPDC2kzkpxaDsk8u7Yz92P020EesnjBOvkmL6ufdLbXMo4pr28AU KAsY354y+v0OzPt6bjZ3GTW7z6PEcsLgD8rMH/e45Vgn05KRuHtbWb1M5J205ANmWvze XIAeZ8Vf21CX3jLe4Egwb4vvofb42koOgPkdBNcsuN+kBaUz/OSXbhNq8SCJ/YOmY834 KLbg== MIME-Version: 1.0 In-Reply-To: <4F5E50F6.9070309@it.uu.se> References: <4F5E50F6.9070309@it.uu.se> Date: Mon, 12 Mar 2012 20:31:26 +0000 Subject: Re: Fast file data retrieval? From: Arnaud Delobelle To: Virgil Stokes Content-Type: text/plain; charset=UTF-8 Cc: python-list@python.org X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 33 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1331584290 news.xs4all.nl 6988 [2001:888:2000:d::a6]:39905 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:21545 On 12 March 2012 19:39, Virgil Stokes wrote: > I have a rather large ASCII file that is structured as follows > > header line > 9 nonblank lines with alphanumeric data > header line > 9 nonblank lines with alphanumeric data > ... > ... > ... > header line > 9 nonblank lines with alphanumeric data > EOF > > where, a data set contains 10 lines (header + 9 nonblank) and there can be > several thousand > data sets in a single file. In addition, each header has a unique ID code. > > Is there a fast method for the retrieval of a data set from this large file > given its ID code? It depends. I guess if it's a long running application, you could load up all the data into a dictionary at startup time (several thousand data sets doesn't sound like that much). Another option would be to put all this into a dbm database file (http://docs.python.org/library/dbm.html) - it would be very easy to do. Or you could have your own custom solution where you scan the file and build a dictionary mapping keys to file offsets, then when requesting a dataset you can seek directly to the correct position. -- Arnaud