Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #15137

Re: save tuple of simple data types to disk (low memory foot print)

Show all headers | View raw

On 10/29/2011 01:08 AM, Roy Smith wrote:
> In article <mailman.2293.1319834877.27778.python-list@python.org>,
>  Gelonida N <gelonida@gmail.com> wrote:
> 
>> I would like to save many dicts with a fixed amount of keys
>> tuples to a file  in a memory efficient manner (no random, but only
>> sequential access is required)
> 
> There's two possible scenarios here.  One, which you seem to be 
> exploring, is to carefully study your data and figure out the best way 
> to externalize it which reduces volume.
> 
> The other is to just write it out in whatever form is most convenient 
> (JSON is a reasonable thing to try first), and compress the output.  Let 
> the compression algorithms worry about extracting the entropy.  You may 
> be surprised at how well it works.  It's also an easy experiment to try, 
> so if it doesn't work well, at least it didn't cost you much to find out.


Yes I have to make some more tests to see the defference between
just compressing aplain format (JSON / pickle) and compressing the
'optimized' representation.

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

save tuple of simple data types to disk (low memory foot print) Gelonida N <gelonida@gmail.com> - 2011-10-28 22:47 +0200
  Re: save tuple of simple data types to disk (low memory foot print) Roy Smith <roy@panix.com> - 2011-10-28 19:08 -0400
    Re: save tuple of simple data types to disk (low memory foot print) Gelonida N <gelonida@gmail.com> - 2011-10-29 18:47 +0200
  Re: save tuple of simple data types to disk (low memory foot print) Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-10-29 01:00 +0000
    Re: save tuple of simple data types to disk (low memory foot print) Gelonida N <gelonida@gmail.com> - 2011-10-29 18:44 +0200
    Re: save tuple of simple data types to disk (low memory foot print) Tim Chase <python.list@tim.thechases.com> - 2011-10-29 12:47 -0500

csiph-web