Path: csiph.com!news.mixmin.net!feeds.phibee-telecom.net!newsfeed.xs4all.nl!newsfeed7.news.xs4all.nl!nzpost1.xs4all.net!not-for-mail
To: python-list@python.org
From: Peter Otten <__peter__@web.de>
Subject: Re: Finding Blank Columns in CSV
Date: Tue, 06 Oct 2015 20:20:40 +0200
Organization: None
References: <muttuu$pc5$1@ger.gmane.org> <CAPTjJmrdE-n=VazddU7PZS6E=rbBf1XVoz_g2ArxK6454D5qxw@mail.gmail.com> <CAPTjJmqYwD0=KNLaHu_w_sOjUoacQRXFhb4Zxdv6n++5A-Y8iQ@mail.gmail.com> <20151005090652.1c9faed7@bigbox.christie.dr> <CAPTjJmpv1OOEomjjaqQ5qiGQDHpxAR4RpxLxn1Rt+j=TQBz4Yg@mail.gmail.com> <mv0b1h$5rd$1@ger.gmane.org> <mv0f3g$8pd$1@ger.gmane.org> <mv102j$oil$1@ger.gmane.org>
Mime-Version: 1.0
Content-Type: text/plain; charset="ISO-8859-1"
Content-Transfer-Encoding: 7Bit
User-Agent: KNode/4.13.3
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.432.1444155660.28679.python-list@python.org>
Lines: 169
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:97454

Jaydip Chakrabarty wrote:

> On Tue, 06 Oct 2015 14:33:51 +0200, Peter Otten wrote:
> 
>> Jaydip Chakrabarty wrote:
>> 
>>> On Tue, 06 Oct 2015 01:34:17 +1100, Chris Angelico wrote:
>>> 
>>>> On Tue, Oct 6, 2015 at 1:06 AM, Tim Chase
>>>> <python.list@tim.thechases.com> wrote:
>>>>> That way, if you determine by line 3 that your million-row CSV file
>>>>> has no blank columns, you can get away with not processing all
>>>>> million rows.
>>>> 
>>>> Sure, although that effectively means the entire job is moot. I kinda
>>>> assume that the OP knows that there are some blank columns (maybe lots
>>>> of them). The extra check is unnecessary unless it's actually
>>>> plausible that there'll be no blanks whatsoever.
>>>> 
>>>> Incidentally, you have an ordered_headers list which is the blank
>>>> columns in order; I think the OP was looking for a list of the
>>>> _non_blank columns. But that's a trivial difference, easy to tweak.
>>>> 
>>>> ChrisA
>>> 
>>> Thanks to you all. I got it this far. But while writing back to another
>>> csv file, I got this error - "ValueError: dict contains fields not in
>>> fieldnames: None". Here is my code.
>>> 
>>> rdr = csv.DictReader(fin, delimiter=',')
>>> header_set = set(rdr.fieldnames)
>>> for r in rdr:
>>>     header_set = set(h for h in header_set if not r[h])
>>>     if not header_set:
>>>         break
>>> 
>>> for r in rdr:
>>>     data = list(r[i] for i in header_set)
>>> 
>>> dw = csv.DictWriter(fout, header_set)
>>> dw.writeheader()
>>> dw.writerows(data)
>> 
>> Sorry, this is not the code you ran. I could guess what the missing
>> parts might be, but it is easier for both sides if you provide a small
>> script that actually can be executed and a small dataset that shows the
>> behaviour you describe. Then post the session and especially the
>> traceback. Example:
>> 
>> $ cat my_data.csv 0
>> $ cat my_code.py print 1/int(open("my_data.csv").read())
>> $ python my_code.py Traceback (most recent call last):
>>   File "my_code.py", line 1, in <module>
>>     print 1/int(open("my_data.csv").read())
>> ZeroDivisionError: integer division or modulo by zero
>> 
>> Don't retype, use cut and paste. Thank you.
> 
> I downloaded gmail contacts in google csv format. There are so many
> columns. So I was trying to create another csv with the required columns.
> Now when I tried to open the gmail csv file with csv DictReader, it said
> the file contained NULL characters.
> So first I did -
> 
> data = open(fn, 'rb').read()
> fout = open(ofn, 'wb')
> fout.write(data.replace('\x00', ''))
> fout.close()
> shutil.move(ofn, fn)
> 
> Then I found, there were some special characters in the file. So, once
> again I opened the file and did -
> 
> data = open(fn, 'rb').read()
> fout = open(ofn, 'wb')
> fout.write(data.replace('\xff\xfe', ''))
> fout.close()
> shutil.move(ofn, fn)

Uh this looks like the file is in UTF-16. Use

import codecs
fn = ...
ofn = ...
with codecs.open(fn, encoding="utf-16") as f:
    with codecs.open(ofn, "w", encoding="utf-8") as g:
        g.writelines(f)
...

to convert it to UTF-8 which is compatible with the csv module of Python 2.

> Now it seemed right. 

Only if all characters are encodable as iso-8859-1.

> So I started to remove empty columns.
> 
> fin = open(fn, 'rb')
> fout = open(ofn, 'wb')
> 
> rdr = csv.DictReader(fin, delimiter=',')
> flds = rdr.fieldnames
> header_set = set(rdr.fieldnames)
> for r in rdr:
>     header_set = set(h for h in header_set if not r[h])
>     if not header_set:
>         break
> for r in rdr:
>     data = list(r[i] for i in header_set)
> 
> dw = csv.DictWriter(fout, data[0].keys())
> dw.writeheader()
> dw.writerows(data)
> 
> fin.close()
> fout.close()
> 
> But, I am getting error at dw.writerows(data). I put the whole code here.
> Please help.

I really meant it when I asked you to post the code you actually ran, and 
the traceback it produces. 

When I fill in the blanks by guessing

$ cat in.csv
one,two,three
foo,,
bar,,baz
$ cat remove_empty_colums.py 
import csv
fn = "in.csv"
ofn = "out.csv"

fin = open(fn, 'rb')
fout = open(ofn, 'wb')

rdr = csv.DictReader(fin, delimiter=',')
flds = rdr.fieldnames
header_set = set(rdr.fieldnames)
for r in rdr:
    header_set = set(h for h in header_set if not r[h])
    if not header_set:
        break
for r in rdr:
    data = list(r[i] for i in header_set)

dw = csv.DictWriter(fout, data[0].keys())
dw.writeheader()
dw.writerows(data)

fin.close()
fout.close()

and then run the resulting script I get

$ python remove_empty_colums.py 
Traceback (most recent call last):
  File "remove_empty_colums.py", line 18, in <module>
    dw = csv.DictWriter(fout, data[0].keys())
NameError: name 'data' is not defined

So this is my traceback, and while the NameError is trivial to fix (reopen 
the file or do a seek) but not sufficient to make the script do what you 
want it doesn't seem to be the problem you ran into. 
So you have a different script. I'd really like to see it, and the traceback 
it produces.