Path: csiph.com!newsfeed.hal-mli.net!feeder3.hal-mli.net!newsfeed.hal-mli.net!feeder1.hal-mli.net!newsfeed.xs4all.nl!newsfeed3.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.000 X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'from:addr:yahoo.co.uk': 0.04; 'append': 0.09; 'lawrence': 0.09; 'parameter': 0.09; 'received:80.91': 0.09; 'received:80.91.229': 0.09; 'received:gmane.org': 0.09; 'received:list': 0.09; 'rows': 0.09; 'subject:files': 0.09; 'def': 0.12; 'language.': 0.14; "'b',": 0.16; "'e']": 0.16; "'rb')": 0.16; "['a',": 0.16; 'csv': 0.16; 'file1': 0.16; 'headers.': 0.16; 'received:80.91.229.3': 0.16; 'received:plane.gmane.org': 0.16; 'subject:CSV': 0.16; 'subject:Merge': 0.16; 'subject:headers': 0.16; 'language': 0.16; 'wrote:': 0.18; 'looked': 0.18; 'import': 0.22; 'header:User- Agent:1': 0.23; 'headers': 0.24; "haven't": 0.24; '---': 0.24; 'class.': 0.26; 'subject:/': 0.26; 'header:X-Complaints-To:1': 0.27; 'header:In-Reply-To:1': 0.27; 'skip:p 30': 0.29; 'code': 0.31; 'file': 0.32; 'class': 0.32; 'url:python': 0.33; 'guess': 0.33; 'skip:d 20': 0.34; 'could': 0.34; 'subject:with': 0.35; 'but': 0.35; 'there': 0.35; 'url:org': 0.36; 'too': 0.37; 'skip:o 20': 0.38; 'url:library': 0.38; 'to:addr:python-list': 0.38; 'to:addr:python.org': 0.39; 'received:org': 0.40; 'free': 0.61; 'url:3': 0.61; 'viruses': 0.61; 'making': 0.63; 'protection': 0.63; 'skip:n 10': 0.64; 'our': 0.64; 'different': 0.65; 'antivirus': 0.68; 'received:2': 0.84 X-Injected-Via-Gmane: http://gmane.org/ To: python-list@python.org From: Mark Lawrence Subject: Re: Merge/append CSV files with different headers Date: Mon, 24 Mar 2014 18:24:10 +0000 References: Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Gmane-NNTP-Posting-Host: host-2-98-198-214.as13285.net User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:24.0) Gecko/20100101 Thunderbird/24.4.0 In-Reply-To: X-Antivirus: avast! (VPS 140324-1, 24/03/2014), Outbound message X-Antivirus-Status: Clean X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 47 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1395685473 news.xs4all.nl 2977 [2001:888:2000:d::a6]:58169 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:68890 On 24/03/2014 17:50, Vincent Davis wrote: > I have several csv file I need to append (vertically). They have > different but overlapping headers. For example; > file1 headers ['a', 'b', 'c'] > file2 headers ['d', 'e'] > file3 headers ['c', 'd'] > > Is there a better way than this > import csv > def merge_csv(fileList, newFileName): > allHeaders = set([]) > for afile in fileList: > with open(afile, 'rb') as csvfilesin: > eachheader = csv.reader(csvfilesin, delimiter=',').next() > allHeaders.update(eachheader) > print(allHeaders) > with open(newFileName, 'wb') as csvfileout: > outfile = csv.DictWriter(csvfileout, allHeaders) > outfile.writeheader() > for afile in fileList: > print('***'+afile) > with open(afile, 'rb') as csvfilesin: > rows = csv.DictReader(csvfilesin, delimiter=',') > for r in rows: > print(allHeaders.issuperset(r.keys())) > outfile.writerow(r) > > Vincent Davis > I haven't looked too hard at your code but guess you could simplify it by using the fieldnames parameter to the DictReader class or making use of the Sniffer class. See http://docs.python.org/3/library/csv.html#module-csv -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence --- This email is free from viruses and malware because avast! Antivirus protection is active. http://www.avast.com