Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #15115

save tuple of simple data types to disk (low memory foot print)

Path csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!aioe.org!feeder.news-service.com!news2.euro.net!newsgate.cistron.nl!newsgate.news.xs4all.nl!194.109.133.85.MISMATCH!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Return-Path <python-python-list@m.gmane.org>
X-Original-To python-list@python.org
Delivered-To python-list@mail.python.org
X-Spam-Status OK 0.001
X-Spam-Evidence '*H*': 1.00; '*S*': 0.00; 'string.': 0.04; '(except': 0.05; 'received:80.91': 0.09; 'received:80.91.229': 0.09; 'received:80.91.229.12': 0.09; 'received:gmane.org': 0.09; 'received:list': 0.09; 'received:lo.gmane.org': 0.09; 'suggestion.': 0.09; 'tuple': 0.09; 'library': 0.15; 'converting': 0.15; 'fixed.': 0.15; '(double)': 0.16; '(long': 0.16; 'pack.': 0.16; 'sequential': 0.16; 'simplest': 0.16; 'subject:memory': 0.16; 'subject:print': 0.16; 'advance': 0.18; 'memory': 0.21; 'subject:data': 0.21; 'variable': 0.24; "i'm": 0.27; 'fixed': 0.29; 'efficiently': 0.30; 'strings,': 0.30; 'tuples': 0.30; 'thanks': 0.30; 'hi,': 0.32; 'there': 0.33; 'to:addr:python-list': 0.33; 'header:User-Agent:1': 0.34; 'keys': 0.34; 'header:X -Complaints-To:1': 0.35; 'entry': 0.35; 'file': 0.36; 'fastest': 0.37; 'formats': 0.37; 'but': 0.37; 'could': 0.38; 'received:org': 0.38; 'some': 0.38; 'header:Mime-Version:1': 0.39; 'data': 0.39; 'to:addr:python.org': 0.39; 'subject: (': 0.39; 'manner': 0.64; 'subject:foot': 0.84; 'subject:types': 0.84; 'required)': 0.91
X-Injected-Via-Gmane http://gmane.org/
To python-list@python.org
From Gelonida N <gelonida@gmail.com>
Subject save tuple of simple data types to disk (low memory foot print)
Date Fri, 28 Oct 2011 22:47:42 +0200
Mime-Version 1.0
Content-Type text/plain; charset=ISO-8859-1
Content-Transfer-Encoding 7bit
X-Gmane-NNTP-Posting-Host unicorn.dungeon.de
User-Agent Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.23) Gecko/20110921 Lightning/1.0b2 ""
X-BeenThere python-list@python.org
X-Mailman-Version 2.1.12
Precedence list
List-Id General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe <http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive <http://mail.python.org/pipermail/python-list>
List-Post <mailto:python-list@python.org>
List-Help <mailto:python-list-request@python.org?subject=help>
List-Subscribe <http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups comp.lang.python
Message-ID <mailman.2293.1319834877.27778.python-list@python.org> (permalink)
Lines 25
NNTP-Posting-Host 2001:888:2000:d::a6
X-Trace 1319834877 news.xs4all.nl 6939 [2001:888:2000:d::a6]:45823
X-Complaints-To abuse@xs4all.nl
Xref x330-a1.tempe.blueboxinc.net comp.lang.python:15115

Show key headers only | View raw

Hi,

I would like to save many dicts with a fixed amount of keys
tuples to a file  in a memory efficient manner (no random, but only
sequential access is required)

As the keys are the same for each entry  I considered converting them to
tuples.

The tuples contain only strings, ints (long ints) and floats (double)
and the data types for each position within the tuple are fixed.

The fastest and simplest way is to pickle the data or to use json.
Both formats however are not that optimal.


I could store ints and floats with pack. As strings have variable length
I'm not sure how to save them efficiently
(except adding a length first and then the string.

Is there already some 'standard' way or standard library to store
such data efficiently?

Thanks in advance for any suggestion.

Back to comp.lang.python | Previous | Next — Next in thread | Find similar | Unroll thread

Thread

save tuple of simple data types to disk (low memory foot print) Gelonida N <gelonida@gmail.com> - 2011-10-28 22:47 +0200
  Re: save tuple of simple data types to disk (low memory foot print) Roy Smith <roy@panix.com> - 2011-10-28 19:08 -0400
    Re: save tuple of simple data types to disk (low memory foot print) Gelonida N <gelonida@gmail.com> - 2011-10-29 18:47 +0200
  Re: save tuple of simple data types to disk (low memory foot print) Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-10-29 01:00 +0000
    Re: save tuple of simple data types to disk (low memory foot print) Gelonida N <gelonida@gmail.com> - 2011-10-29 18:44 +0200
    Re: save tuple of simple data types to disk (low memory foot print) Tim Chase <python.list@tim.thechases.com> - 2011-10-29 12:47 -0500

csiph-web