Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.lang.python > #21550
| Path | csiph.com!usenet.pasdenom.info!gegeweb.42!gegeweb.eu!nntpfeed.proxad.net!proxad.net!feeder1-2.proxad.net!usenet-fr.net!de-l.enfer-du-nord.net!feeder1.enfer-du-nord.net!newsfeed.eweka.nl!eweka.nl!feeder3.eweka.nl!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail |
|---|---|
| Return-Path | <joncle@googlemail.com> |
| X-Original-To | python-list@python.org |
| Delivered-To | python-list@mail.python.org |
| X-Spam-Status | OK 0.011 |
| X-Spam-Evidence | '*H*': 0.98; '*S*': 0.00; 'mrab': 0.04; 'ascii': 0.07; 'dict': 0.09; 'eof': 0.09; 'to:addr:comp.lang.python': 0.09; 'server,': 0.12; 'subject:file': 0.13; 'code?': 0.16; 'sqlite3': 0.16; 'stokes': 0.16; 'tokyo': 0.16; 'utc,': 0.16; 'cc:addr :python-list': 0.16; 'wrote:': 0.18; 'cc:no real name:2**0': 0.21; 'file,': 0.21; 'maybe': 0.21; 'header:In-Reply-To:1': 0.22; '(or': 0.22; 'subject:data': 0.25; 'cc:2**0': 0.26; 'position.': 0.28; 'module.': 0.29; 'retrieval': 0.29; 'cc:addr:python.org': 0.29; 'lines': 0.30; 'from:addr:googlemail.com': 0.31; 'subject:?': 0.31; 'file.': 0.31; 'that,': 0.32; 'there': 0.33; 'header:User- Agent:1': 0.33; 'file': 0.34; 'rather': 0.34; 'structured': 0.34; 'probably': 0.35; '...': 0.35; 'sets': 0.35; 'offset': 0.37; 'received:google.com': 0.37; 'either': 0.37; 'received:209.85': 0.38; 'monday,': 0.38; 'could': 0.38; 'several': 0.38; 'data': 0.38; 'header': 0.39; 'received:209': 0.39; 'put': 0.40; 'march': 0.61; 'full': 0.62; 'alphanumeric': 0.67; 'database.': 0.74; 'thousand': 0.74; 'blown': 0.84; 'cabinet': 0.84; 'subject:Fast': 0.84 |
| Newsgroups | comp.lang.python |
| Date | Mon, 12 Mar 2012 20:38:25 -0700 (PDT) |
| In-Reply-To | <mailman.592.1331584145.3037.python-list@python.org> |
| Complaints-To | groups-abuse@google.com |
| Injection-Info | glegroupsg2000goo.googlegroups.com; posting-host=86.156.91.130; posting-account=HLD_OAoAAAD-0RilNRZUjdKEwXt97Q9q |
| References | <4F5E50F6.9070309@it.uu.se> <mailman.592.1331584145.3037.python-list@python.org> |
| User-Agent | G2/1.0 |
| X-Google-Web-Client | true |
| MIME-Version | 1.0 |
| Subject | Re: Fast file data retrieval? |
| From | Jon Clements <joncle@googlemail.com> |
| To | comp.lang.python@googlegroups.com |
| Content-Type | text/plain; charset=ISO-8859-1 |
| Cc | python-list@python.org |
| X-BeenThere | python-list@python.org |
| X-Mailman-Version | 2.1.12 |
| Precedence | list |
| List-Id | General discussion list for the Python programming language <python-list.python.org> |
| List-Unsubscribe | <http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe> |
| List-Archive | <http://mail.python.org/pipermail/python-list> |
| List-Post | <mailto:python-list@python.org> |
| List-Help | <mailto:python-list-request@python.org?subject=help> |
| List-Subscribe | <http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe> |
| Message-ID | <mailman.599.1331609909.3037.python-list@python.org> (permalink) |
| Lines | 31 |
| NNTP-Posting-Host | 2001:888:2000:d::a6 |
| X-Trace | 1331609909 news.xs4all.nl 6904 [2001:888:2000:d::a6]:52680 |
| X-Complaints-To | abuse@xs4all.nl |
| Xref | csiph.com comp.lang.python:21550 |
Show key headers only | View raw
On Monday, 12 March 2012 20:31:35 UTC, MRAB wrote: > On 12/03/2012 19:39, Virgil Stokes wrote: > > I have a rather large ASCII file that is structured as follows > > > > header line > > 9 nonblank lines with alphanumeric data > > header line > > 9 nonblank lines with alphanumeric data > > ... > > ... > > ... > > header line > > 9 nonblank lines with alphanumeric data > > EOF > > > > where, a data set contains 10 lines (header + 9 nonblank) and there can > > be several thousand > > data sets in a single file. In addition,*each header has a* *unique ID > > code*. > > > > Is there a fast method for the retrieval of a data set from this large > > file given its ID code? > > > Probably the best solution is to put it into a database. Have a look at > the sqlite3 module. > > Alternatively, you could scan the file, recording the ID and the file > offset in a dict so that, given an ID, you can seek directly to that > file position. I would have a look at either bsddb, Tokyo (or Kyoto) Cabinet or hamsterdb. If it's really going to get large and needs a full blown server, maybe MongoDB/redis/hadoop...
Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread
Re: Fast file data retrieval? MRAB <python@mrabarnett.plus.com> - 2012-03-12 20:31 +0000
Re: Fast file data retrieval? Jon Clements <joncle@googlemail.com> - 2012-03-12 20:38 -0700
Re: Fast file data retrieval? Jon Clements <joncle@googlemail.com> - 2012-03-12 20:38 -0700
Re: Fast file data retrieval? Jorgen Grahn <grahn+nntp@snipabacken.se> - 2012-03-13 20:44 +0000
Re: Fast file data retrieval? Stefan Behnel <stefan_ml@behnel.de> - 2012-03-21 17:32 +0100
csiph-web