Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #76590

Re: Functions on list items

Show key headers only | View raw

On 2014-08-19 10:34, Kurt wrote:
> I am trying to process the following calendar and data attributes
> in a file: Da Mo Yr AttrA AttrB AttrC...
> I need to average AttrA for each of 365 Da days across Yr years.
> Then do the same for 27K files. Repeat for AttrB, AttrC etc. Can I
> do the averaging with lists or do I need some Python Db module (if
> it exists)? 

Unless these files are huge *AND* you plan to perform this operation
repeatedly (rather than once), just do it in plain Python.  (though
Python does offer both the "anydbm" and "sqlite3" modules for
database'ish stuff)

You omit some key details, so the example below would need to be
tweaked depending on things like:

 - does it have headers
 - how are the columns separated? (tabs, commas, spaces?)
 - can a single date appear multiple times in a single file? (and if
   so, what would you intend to do with it?)
 - do you need aggregate averages across all the files, or just
   per-file?
 - are those values you're averaging integers or floats?

It would look something like (untested)

  import csv
  from collections import defaultdict

  def report(fname, names_avgs):
    "Do something with the averages"
    print(fname)
    for name, avg in names_avgs:
      print(" %s: %f" % (name, avg))

  def no_data(fname):
    "Do something if a file has no data"
    pass

  for fname in fname_iter():
    sums = defaultdict(float)
    row_count = 0
    with open(fname) as f:
      r = csv.DictReader(f):
      for row in r:
        row_count += 1
        for data in ("AttrA", "AttrB", "AttrC"):
          sums[data] += float(row[data])
    if row_count:
      report(fname, [
        (fieldname, total/row_count)
        for fieldname, total
        in sums
        ])
    else:
      no_data(fname)

I leave the details of fname_iter() and tweaking it based on the
omitted details as an exercise to the reader. :-)

-tkc

Back to comp.lang.python | Previous | Next — Previous in thread | Find similar | Unroll thread

Thread

Functions on list items Kurt <kplunt@gmail.com> - 2014-08-19 10:34 -0700
  Re: Functions on list items Mark Lawrence <breamoreboy@yahoo.co.uk> - 2014-08-19 18:43 +0100
    Re: Functions on list items Kurt <kplunt@gmail.com> - 2014-08-19 10:56 -0700
  Re: Functions on list items Tim Chase <python.list@tim.thechases.com> - 2014-08-19 12:58 -0500

csiph-web