Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.lang.python > #56887
| From | Dennis Lee Bieber <wlfraed@ix.netcom.com> |
|---|---|
| Subject | Re: How pickle helps in reading huge files? |
| Date | 2013-10-16 13:32 -0400 |
| Organization | IISS Elusive Unicorn |
| References | <0044bfd0-f07f-4f7b-b976-5df034b6fec6@googlegroups.com> |
| Newsgroups | comp.lang.python |
| Message-ID | <mailman.1115.1381944762.18130.python-list@python.org> (permalink) |
On Tue, 15 Oct 2013 23:55:26 -0700 (PDT), Harsh Jha
<harshjha2006@gmail.com> declaimed the following:
>I've a huge csv file and I want to read stuff from it again and again. Is it useful to pickle it and keep and then unpickle it whenever I need to use that data? Is it faster that accessing that file simply by opening it again and again? Please explain, why?
>
As others mention, what is "huge"?
Does it get updated often? How extensive are updates?
I suspect I'd use the CSV module to parse it into an SQLite3 database,
then use the database for the repetitive access. NOTE: I've never used
pickle -- but for stuff that is coming in as simple CSV I'd suspect the
parsing (even including the various int()/float() wrapping of numeric
fields) can't be much slower than the object creation/unwrapping used by
pickle; SQLite3 should let you leave the data in numeric formats without
the translation penalty on each use.
--
Wulfraed Dennis Lee Bieber AF6VN
wlfraed@ix.netcom.com HTTP://wlfraed.home.netcom.com/
Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread
How pickle helps in reading huge files? Harsh Jha <harshjha2006@gmail.com> - 2013-10-15 23:55 -0700
Re: How pickle helps in reading huge files? Stephane Wirtel <stephane@wirtel.be> - 2013-10-16 09:05 +0200
Re: How pickle helps in reading huge files? rusi <rustompmody@gmail.com> - 2013-10-16 01:51 -0700
Re: How pickle helps in reading huge files? Chris Angelico <rosuav@gmail.com> - 2013-10-16 20:09 +1100
Re: How pickle helps in reading huge files? Mark Lawrence <breamoreboy@yahoo.co.uk> - 2013-10-16 08:39 +0100
Re: How pickle helps in reading huge files? Roy Smith <roy@panix.com> - 2013-10-16 08:29 -0400
Re: How pickle helps in reading huge files? Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2013-10-16 13:32 -0400
Re: How pickle helps in reading huge files? Peter Cacioppi <peter.cacioppi@gmail.com> - 2013-10-16 14:04 -0700
Re: How pickle helps in reading huge files? Irmen de Jong <irmen.NOSPAM@xs4all.nl> - 2013-10-16 23:09 +0200
csiph-web