Path: csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!aioe.org!feeder.news-service.com!news2.euro.net!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Date: Sat, 29 Oct 2011 12:47:42 -0500
From: Tim Chase <python.list@tim.thechases.com>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.20) Gecko/20110826 Icedove/3.1.12
MIME-Version: 1.0
To: Gelonida N <gelonida@gmail.com>
Subject: Re: save tuple of simple data types to disk (low memory foot print)
References: <mailman.2293.1319834877.27778.python-list@python.org>	<4eab5021$0$29968$c3e8da3$5496439d@news.astraweb.com> <j8hagu$1o3$1@dough.gmane.org>
In-Reply-To: <j8hagu$1o3$1@dough.gmane.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Cc: python-list@python.org
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.2310.1319912594.27778.python-list@python.org>
Lines: 104
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: x330-a1.tempe.blueboxinc.net comp.lang.python:15138

On 10/29/11 11:44, Gelonida N wrote:
> I would like to save many dicts with a fixed (and known) amount of keys
> in a memory efficient manner (no random, but only sequential access is
> required) to a file (which can later be sent over a slow expensive
> network to other machines)
>
> Example:
> Every dict will have the keys 'timestamp', 'floatvalue', 'intvalue',
> 'message1', 'message2'
> 'timestamp' is an integer
> 'floatvalue' is a float
> 'intvalue' an int
> 'message1' is a string with a length of max 2000 characters, but can
> often be very short
> 'message2' the same as message1
>
> so a typical dict will look like
> { 'timetamp' : 12, 'floatvalue': 3.14159, 'intvalue': 42,
>   'message1' : '', 'message2' : '=' * 1999 }
>
>
>>
>> What do you call "many"? Fifty? A thousand? A thousand million? How many
>> items in each dict? Ten? A million?
>
> File size can be between 100kb and over 100Mb per file. Files will be
> accumulated over months.

If Steven's pickle-protocol2 solution doesn't quite do what you 
need, you can do something like the code below.  Gzip is pretty 
good at addressing...

>> Or have you considered simply compressing the files?
> Compression makes sense but the inital file format should be
> already rather 'compact'

...by compressing out a lot of the duplicate aspects.  Which also 
mitigates some of the verbosity of CSV.

It serializes the data to a gzipped CSV file then unserializes 
it.  Just point it at the appropriate data-source, adjust the 
column-names and data-types

-tkc

from gzip import GzipFile
from csv import writer, reader

data = [ # use your real data here
     {
     'timestamp': 12,
     'floatvalue': 3.14159,
     'intvalue': 42,
     'message1': 'hello world',
     'message2': '=' * 1999,
     },
     ] * 10000


f = GzipFile('data.gz', 'wb')
try:
     w = writer(f)
     for row in data:
         w.writerow([
             row[name] for name in (
             # use your real col-names here
             'timestamp',
             'floatvalue',
             'intvalue',
             'message1',
             'message2',
             )])
finally:
     f.close()

output = []
for row in reader(GzipFile('data.gz')):
     d = dict((
         (name, f(row[i]))
         for i, (f,name) in enumerate((
             # adjust for your column-names/data-types
             (int, 'timestamp'),
             (float, 'floatvalue'),
             (int, 'intvalue'),
             (str, 'message1'),
             (str, 'message2'),
             ))))
     output.append(d)

# or

output = [
     dict((
         (name, f(row[i]))
         for i, (f,name) in enumerate((
             # adjust for your column-names/data-types
             (int, 'timestamp'),
             (float, 'floatvalue'),
             (int, 'intvalue'),
             (str, 'message1'),
             (str, 'message2'),
             ))))
     for row in reader(GzipFile('data.gz'))
     ]