Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #68898 > unrolled thread

Re: Merge/append CSV files with different headers

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Merge/append CSV files with different headers Chris Angelico <rosuav@gmail.com> - 2014-03-25 06:44 +1100

#68898 — Re: Merge/append CSV files with different headers

On Tue, Mar 25, 2014 at 4:50 AM, Vincent Davis <vincent@vincentdavis.net> wrote:
> I have several csv file I need to append (vertically). They have different
> but overlapping headers. For example;
> file1 headers ['a', 'b', 'c']
> file2 headers ['d', 'e']
> file3 headers ['c', 'd']
>
> Is there a better way than this

Summary of your code:

1) Build up a set of all headers used, by opening each file and
reading the headers.
2) Go through each file a second time and write them out.

That seems like the best approach, broadly. You might be able to
improve it a bit (it might be tidier to open each file once, but since
you're using two different CSV readers, it'd probably not be), but by
and large, I'd say you have the right technique. Your processing time
here is going to be dominated by the actual work of copying.

The only thing you might want to consider is order. The headers all
have a set order to them, and it'd make sense to have the output come
out as ['a', 'b', 'c', 'd', 'e'] - the first three from the first
file, then adding in everything from subsequent files in the order
they were found. Could be done easily enough by using 'in' and
.append() on a list, rather than using a set. But if that doesn't
matter to you, or if something simple like "sort the headers
alphabetically" will do, then I think you basically have what you
want.

ChrisA

[toc] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web