Path: csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!aioe.org!feeder.news-service.com!news2.euro.net!newsgate.cistron.nl!newsgate.news.xs4all.nl!194.109.133.85.MISMATCH!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Return-Path: <python-python-list@m.gmane.org>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.001
X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'string.': 0.04; '(except': 0.05; 'received:80.91': 0.09; 'received:80.91.229': 0.09; 'received:80.91.229.12': 0.09; 'received:gmane.org': 0.09; 'received:list': 0.09; 'received:lo.gmane.org': 0.09; 'suggestion.': 0.09; 'tuple': 0.09; 'library': 0.15; 'converting': 0.15; 'fixed.': 0.15; '(double)': 0.16; '(long': 0.16; 'pack.': 0.16; 'sequential': 0.16; 'simplest': 0.16; 'subject:memory': 0.16; 'subject:print': 0.16; 'advance': 0.18; 'memory': 0.21; 'subject:data': 0.21; 'variable': 0.24; "i'm": 0.27; 'fixed': 0.29; 'efficiently': 0.30; 'strings,': 0.30; 'tuples': 0.30; 'thanks': 0.30; 'hi,': 0.32; 'there': 0.33; 'to:addr:python-list': 0.33; 'header:User-Agent:1': 0.34; 'keys': 0.34; 'header:X -Complaints-To:1': 0.35; 'entry': 0.35; 'file': 0.36; 'fastest': 0.37; 'formats': 0.37; 'but': 0.37; 'could': 0.38; 'received:org': 0.38; 'some': 0.38; 'header:Mime-Version:1': 0.39; 'data': 0.39; 'to:addr:python.org': 0.39; 'subject: (': 0.39; 'manner': 0.64; 'subject:foot': 0.84; 'subject:types': 0.84; 'required)': 0.91
X-Injected-Via-Gmane: http://gmane.org/
To: python-list@python.org
From: Gelonida N <gelonida@gmail.com>
Subject: save tuple of simple data types to disk (low memory foot print)
Date: Fri, 28 Oct 2011 22:47:42 +0200
Mime-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Gmane-NNTP-Posting-Host: unicorn.dungeon.de
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.23) Gecko/20110921 Lightning/1.0b2 ""
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups: comp.lang.python
Message-ID: <mailman.2293.1319834877.27778.python-list@python.org>
Lines: 25
NNTP-Posting-Host: 2001:888:2000:d::a6
X-Trace: 1319834877 news.xs4all.nl 6939 [2001:888:2000:d::a6]:45823
X-Complaints-To: abuse@xs4all.nl
Xref: x330-a1.tempe.blueboxinc.net comp.lang.python:15115

Hi,

I would like to save many dicts with a fixed amount of keys
tuples to a file  in a memory efficient manner (no random, but only
sequential access is required)

As the keys are the same for each entry  I considered converting them to
tuples.

The tuples contain only strings, ints (long ints) and floats (double)
and the data types for each position within the tuple are fixed.

The fastest and simplest way is to pickle the data or to use json.
Both formats however are not that optimal.


I could store ints and floats with pack. As strings have variable length
I'm not sure how to save them efficiently
(except adding a length first and then the string.

Is there already some 'standard' way or standard library to store
such data efficiently?

Thanks in advance for any suggestion.