Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.lang.python > #15137
| Path | csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!aioe.org!news.stack.nl!newsfeed.xs4all.nl!newsfeed5.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail |
|---|---|
| Return-Path | <python-python-list@m.gmane.org> |
| X-Original-To | python-list@python.org |
| Delivered-To | python-list@mail.python.org |
| X-Spam-Status | OK 0.021 |
| X-Spam-Evidence | '*H*': 0.96; '*S*': 0.00; 'received:80.91': 0.09; 'received:80.91.229': 0.09; 'received:80.91.229.12': 0.09; 'received:gmane.org': 0.09; 'received:list': 0.09; 'received:lo.gmane.org': 0.09; 'am,': 0.12; 'compression': 0.16; 'extracting': 0.16; 'first),': 0.16; 'roy': 0.16; 'scenarios': 0.16; 'sequential': 0.16; 'subject:memory': 0.16; 'subject:print': 0.16; 'wrote:': 0.18; 'figure': 0.20; 'memory': 0.20; 'subject:data': 0.21; "doesn't": 0.23; 'works.': 0.23; 'header:In- Reply-To:1': 0.23; 'tests': 0.25; 'fixed': 0.28; 'seem': 0.30; 'tuples': 0.30; "didn't": 0.30; 'least': 0.31; 'to:addr:python- list': 0.32; 'header:User-Agent:1': 0.33; 'header:X-Complaints- To:1': 0.33; 'algorithms': 0.34; 'keys': 0.34; 'surprised': 0.34; 'file': 0.36; 'convenient': 0.36; 'but': 0.37; 'two': 0.37; 'received:org': 0.37; 'some': 0.38; "there's": 0.39; "it's": 0.39; 'subject: (': 0.39; 'subject:: ': 0.39; 'to:addr:python.org': 0.39; 'one,': 0.40; 'data': 0.40; 'more': 0.60; 'your': 0.61; 'efficient': 0.61; 'cost': 0.63; 'manner': 0.65; 'here.': 0.66; 'carefully': 0.68; 'article': 0.76; 'experiment': 0.84; 'subject:foot': 0.84; 'subject:types': 0.84; 'required)': 0.91 |
| X-Injected-Via-Gmane | http://gmane.org/ |
| To | python-list@python.org |
| From | Gelonida N <gelonida@gmail.com> |
| Subject | Re: save tuple of simple data types to disk (low memory foot print) |
| Date | Sat, 29 Oct 2011 18:47:57 +0200 |
| References | <mailman.2293.1319834877.27778.python-list@python.org> <roy-88E0AF.19084228102011@news.panix.com> |
| Mime-Version | 1.0 |
| Content-Type | text/plain; charset=ISO-8859-1 |
| Content-Transfer-Encoding | 7bit |
| X-Gmane-NNTP-Posting-Host | unicorn.dungeon.de |
| User-Agent | Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.23) Gecko/20110921 Lightning/1.0b2 "" |
| In-Reply-To | <roy-88E0AF.19084228102011@news.panix.com> |
| X-BeenThere | python-list@python.org |
| X-Mailman-Version | 2.1.12 |
| Precedence | list |
| List-Id | General discussion list for the Python programming language <python-list.python.org> |
| List-Unsubscribe | <http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe> |
| List-Archive | <http://mail.python.org/pipermail/python-list> |
| List-Post | <mailto:python-list@python.org> |
| List-Help | <mailto:python-list-request@python.org?subject=help> |
| List-Subscribe | <http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe> |
| Newsgroups | comp.lang.python |
| Message-ID | <mailman.2309.1319907010.27778.python-list@python.org> (permalink) |
| Lines | 26 |
| NNTP-Posting-Host | 2001:888:2000:d::a6 |
| X-Trace | 1319907010 news.xs4all.nl 6853 [2001:888:2000:d::a6]:51186 |
| X-Complaints-To | abuse@xs4all.nl |
| Xref | x330-a1.tempe.blueboxinc.net comp.lang.python:15137 |
Show key headers only | View raw
On 10/29/2011 01:08 AM, Roy Smith wrote: > In article <mailman.2293.1319834877.27778.python-list@python.org>, > Gelonida N <gelonida@gmail.com> wrote: > >> I would like to save many dicts with a fixed amount of keys >> tuples to a file in a memory efficient manner (no random, but only >> sequential access is required) > > There's two possible scenarios here. One, which you seem to be > exploring, is to carefully study your data and figure out the best way > to externalize it which reduces volume. > > The other is to just write it out in whatever form is most convenient > (JSON is a reasonable thing to try first), and compress the output. Let > the compression algorithms worry about extracting the entropy. You may > be surprised at how well it works. It's also an easy experiment to try, > so if it doesn't work well, at least it didn't cost you much to find out. Yes I have to make some more tests to see the defference between just compressing aplain format (JSON / pickle) and compressing the 'optimized' representation.
Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread
save tuple of simple data types to disk (low memory foot print) Gelonida N <gelonida@gmail.com> - 2011-10-28 22:47 +0200
Re: save tuple of simple data types to disk (low memory foot print) Roy Smith <roy@panix.com> - 2011-10-28 19:08 -0400
Re: save tuple of simple data types to disk (low memory foot print) Gelonida N <gelonida@gmail.com> - 2011-10-29 18:47 +0200
Re: save tuple of simple data types to disk (low memory foot print) Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-10-29 01:00 +0000
Re: save tuple of simple data types to disk (low memory foot print) Gelonida N <gelonida@gmail.com> - 2011-10-29 18:44 +0200
Re: save tuple of simple data types to disk (low memory foot print) Tim Chase <python.list@tim.thechases.com> - 2011-10-29 12:47 -0500
csiph-web