Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #3208

Re: memory usage multi value hash

Show all headers | View raw

On 4/14/2011 12:55 PM, Peter Otten wrote:

> I don't expect that it matters much, but you don't need to sort your data if
> you use a dictionary anyway:

Which means that one can build the dict line by line, as each is read, 
instead of reading the entire file into memory. So it does matter for 
intermediate memory use.

> import csv
> import sys
>
> infile, outfile = sys.argv[1:]
>
> d = {}
> with open(infile, "rb") as instream:
>      for key, value in csv.reader(instream, delimiter=';'):
>          d.setdefault(key, [key]).append(value)
>
> with open(outfile, "wb") as outstream:
>      csv.writer(outstream).writerows(d.itervalues())


-- 
Terry Jan Reedy

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

memory usage multi value hash christian <ozric@web.de> - 2011-04-14 09:13 -0700
  Re: memory usage multi value hash Peter Otten <__peter__@web.de> - 2011-04-14 18:55 +0200
  Re: memory usage multi value hash Terry Reedy <tjreedy@udel.edu> - 2011-04-14 13:28 -0400
    Re: memory usage multi value hash Peter Otten <__peter__@web.de> - 2011-04-15 10:15 +0200
  Re: memory usage multi value hash Algis Kabaila <akabaila@pcug.org.au> - 2011-04-15 18:01 +1000

csiph-web