Path: csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!aioe.org!news.stack.nl!newsfeed.xs4all.nl!newsfeed5.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Return-Path: <python-python-list@m.gmane.org>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.021
X-Spam-Evidence: '*H*': 0.96; '*S*': 0.00; 'received:80.91': 0.09; 'received:80.91.229': 0.09; 'received:80.91.229.12': 0.09; 'received:gmane.org': 0.09; 'received:list': 0.09; 'received:lo.gmane.org': 0.09; 'am,': 0.12; 'compression': 0.16; 'extracting': 0.16; 'first),': 0.16; 'roy': 0.16; 'scenarios': 0.16; 'sequential': 0.16; 'subject:memory': 0.16; 'subject:print': 0.16; 'wrote:': 0.18; 'figure': 0.20; 'memory': 0.20; 'subject:data': 0.21; "doesn't": 0.23; 'works.': 0.23; 'header:In- Reply-To:1': 0.23; 'tests': 0.25; 'fixed': 0.28; 'seem': 0.30; 'tuples': 0.30; "didn't": 0.30; 'least': 0.31; 'to:addr:python- list': 0.32; 'header:User-Agent:1': 0.33; 'header:X-Complaints- To:1': 0.33; 'algorithms': 0.34; 'keys': 0.34; 'surprised': 0.34; 'file': 0.36; 'convenient': 0.36; 'but': 0.37; 'two': 0.37; 'received:org': 0.37; 'some': 0.38; "there's": 0.39; "it's": 0.39; 'subject: (': 0.39; 'subject:: ': 0.39; 'to:addr:python.org': 0.39; 'one,': 0.40; 'data': 0.40; 'more': 0.60; 'your': 0.61; 'efficient': 0.61; 'cost': 0.63; 'manner': 0.65; 'here.': 0.66; 'carefully': 0.68; 'article': 0.76; 'experiment': 0.84; 'subject:foot': 0.84; 'subject:types': 0.84; 'required)': 0.91
X-Injected-Via-Gmane: http://gmane.org/
To: python-list@python.org
From: Gelonida N <gelonida@gmail.com>
Subject: Re: save tuple of simple data types to disk (low memory foot print)
Date: Sat, 29 Oct 2011 18:47:57 +0200
References: <mailman.2293.1319834877.27778.python-list@python.org> <roy-88E0AF.19084228102011@news.panix.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Gmane-NNTP-Posting-Host: unicorn.dungeon.de
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.23) Gecko/20110921 Lightning/1.0b2 ""
In-Reply-To: <roy-88E0AF.19084228102011@news.panix.com>
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups: comp.lang.python
Message-ID: <mailman.2309.1319907010.27778.python-list@python.org>
Lines: 26
NNTP-Posting-Host: 2001:888:2000:d::a6
X-Trace: 1319907010 news.xs4all.nl 6853 [2001:888:2000:d::a6]:51186
X-Complaints-To: abuse@xs4all.nl
Xref: x330-a1.tempe.blueboxinc.net comp.lang.python:15137

On 10/29/2011 01:08 AM, Roy Smith wrote:
> In article <mailman.2293.1319834877.27778.python-list@python.org>,
>  Gelonida N <gelonida@gmail.com> wrote:
> 
>> I would like to save many dicts with a fixed amount of keys
>> tuples to a file  in a memory efficient manner (no random, but only
>> sequential access is required)
> 
> There's two possible scenarios here.  One, which you seem to be 
> exploring, is to carefully study your data and figure out the best way 
> to externalize it which reduces volume.
> 
> The other is to just write it out in whatever form is most convenient 
> (JSON is a reasonable thing to try first), and compress the output.  Let 
> the compression algorithms worry about extracting the entropy.  You may 
> be surprised at how well it works.  It's also an easy experiment to try, 
> so if it doesn't work well, at least it didn't cost you much to find out.


Yes I have to make some more tests to see the defference between
just compressing aplain format (JSON / pickle) and compressing the
'optimized' representation.