Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #95953

Re: continue vs. pass in this IO reading and writing

Show all headers | View raw

On Fri, Sep 4, 2015 at 1:57 AM, kbtyo <ahlusar.ahluwalia@gmail.com> wrote:
> I have used CSV and collections. For some reason when I apply this algorithm, all of my files are not added (the output is ridiculously small considering how much goes in - think KB output vs MB input):
>
> from glob import iglob
> import csv
> from collections import OrderedDict
>
> files = sorted(iglob('*.csv'))
> header = OrderedDict()
> data = []
>
> for filename in files:
>     with open(filename, 'r') as fin:
>         csvin = csv.DictReader(fin)
>         header.update(OrderedDict.fromkeys(csvin.fieldnames))
>         data.append(next(csvin))
>
> with open('output_filename_version2.csv', 'w') as fout:
>     csvout = csv.DictWriter(fout, fieldnames=list(header))
>     csvout.writeheader()
>     csvout.writerows(data)

You're collecting up just one row from each file. Since you say your
input is measured in MB (not GB or anything bigger), the simplest
approach is probably fine: instead of "data.append(next(csvin))", just
use "data.extend(csvin)", which should grab them all. That'll store
all your input data in memory, which should be fine if it's only a few
meg, and probably not a problem for anything under a few hundred meg.

ChrisA

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

continue vs. pass in this IO reading and writing kbtyo <ahlusar.ahluwalia@gmail.com> - 2015-09-03 08:05 -0700
  Re: continue vs. pass in this IO reading and writing Chris Angelico <rosuav@gmail.com> - 2015-09-04 01:27 +1000
    Re: continue vs. pass in this IO reading and writing kbtyo <ahlusar.ahluwalia@gmail.com> - 2015-09-03 08:38 -0700
      Re: continue vs. pass in this IO reading and writing Chris Angelico <rosuav@gmail.com> - 2015-09-04 01:51 +1000
        Re: continue vs. pass in this IO reading and writing kbtyo <ahlusar.ahluwalia@gmail.com> - 2015-09-03 08:57 -0700
          Re: continue vs. pass in this IO reading and writing Chris Angelico <rosuav@gmail.com> - 2015-09-04 02:11 +1000
            Re: continue vs. pass in this IO reading and writing kbtyo <ahlusar.ahluwalia@gmail.com> - 2015-09-03 09:35 -0700
    Re: continue vs. pass in this IO reading and writing kbtyo <ahlusar.ahluwalia@gmail.com> - 2015-09-03 08:49 -0700
  Re: continue vs. pass in this IO reading and writing Terry Reedy <tjreedy@udel.edu> - 2015-09-03 12:05 -0400
  Re: continue vs. pass in this IO reading and writing Luca Menegotto <otlucaDELETE@DELETEyahoo.it> - 2015-09-03 18:37 +0200

csiph-web