Path: csiph.com!news.mixmin.net!feeds.phibee-telecom.net!newsfeed.xs4all.nl!newsfeed7.news.xs4all.nl!nzpost1.xs4all.net!not-for-mail
To: python-list@python.org
From: Peter Otten <__peter__@web.de>
Subject: Re: Finding Blank Columns in CSV
Date: Tue, 06 Oct 2015 14:33:51 +0200
Organization: None
References: <muttuu$pc5$1@ger.gmane.org> <CAPTjJmrdE-n=VazddU7PZS6E=rbBf1XVoz_g2ArxK6454D5qxw@mail.gmail.com> <CAPTjJmqYwD0=KNLaHu_w_sOjUoacQRXFhb4Zxdv6n++5A-Y8iQ@mail.gmail.com> <20151005090652.1c9faed7@bigbox.christie.dr> <CAPTjJmpv1OOEomjjaqQ5qiGQDHpxAR4RpxLxn1Rt+j=TQBz4Yg@mail.gmail.com> <mv0b1h$5rd$1@ger.gmane.org>
Mime-Version: 1.0
Content-Type: text/plain; charset="ISO-8859-1"
Content-Transfer-Encoding: 7Bit
User-Agent: KNode/4.13.3
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.421.1444134844.28679.python-list@python.org>
Lines: 58
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:97441

Jaydip Chakrabarty wrote:

> On Tue, 06 Oct 2015 01:34:17 +1100, Chris Angelico wrote:
> 
>> On Tue, Oct 6, 2015 at 1:06 AM, Tim Chase
>> <python.list@tim.thechases.com> wrote:
>>> That way, if you determine by line 3 that your million-row CSV file has
>>> no blank columns, you can get away with not processing all million
>>> rows.
>> 
>> Sure, although that effectively means the entire job is moot. I kinda
>> assume that the OP knows that there are some blank columns (maybe lots
>> of them). The extra check is unnecessary unless it's actually plausible
>> that there'll be no blanks whatsoever.
>> 
>> Incidentally, you have an ordered_headers list which is the blank
>> columns in order; I think the OP was looking for a list of the
>> _non_blank columns. But that's a trivial difference, easy to tweak.
>> 
>> ChrisA
> 
> Thanks to you all. I got it this far. But while writing back to another
> csv file, I got this error - "ValueError: dict contains fields not in
> fieldnames: None". Here is my code.
> 
> rdr = csv.DictReader(fin, delimiter=',')
> header_set = set(rdr.fieldnames)
> for r in rdr:
>     header_set = set(h for h in header_set if not r[h])
>     if not header_set:
>         break
> 
> for r in rdr:
>     data = list(r[i] for i in header_set)
> 
> dw = csv.DictWriter(fout, header_set)
> dw.writeheader()
> dw.writerows(data)

Sorry, this is not the code you ran. I could guess what the missing parts 
might be, but it is easier for both sides if you provide a small script that 
actually can be executed and a small dataset that shows the behaviour you 
describe. Then post the session and especially the traceback. Example:

$ cat my_data.csv
0
$ cat my_code.py 
print 1/int(open("my_data.csv").read())
$ python my_code.py 
Traceback (most recent call last):
  File "my_code.py", line 1, in <module>
    print 1/int(open("my_data.csv").read())
ZeroDivisionError: integer division or modulo by zero

Don't retype, use cut and paste. Thank you.