Path: csiph.com!eternal-september.org!feeder.eternal-september.org!newsfeed1.swip.net!uio.no!news.tele.dk!news.tele.dk!small.news.tele.dk!newsgate.cistron.nl!newsgate.news.xs4all.nl!nzpost1.xs4all.net!not-for-mail
Date: Mon, 5 Oct 2015 09:06:52 -0500
From: Tim Chase <python.list@tim.thechases.com>
To: python-list@python.org
Subject: Re: Finding Blank Columns in CSV
In-Reply-To: <CAPTjJmqYwD0=KNLaHu_w_sOjUoacQRXFhb4Zxdv6n++5A-Y8iQ@mail.gmail.com>
References: <muttuu$pc5$1@ger.gmane.org> <CAPTjJmrdE-n=VazddU7PZS6E=rbBf1XVoz_g2ArxK6454D5qxw@mail.gmail.com> <CAPTjJmqYwD0=KNLaHu_w_sOjUoacQRXFhb4Zxdv6n++5A-Y8iQ@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.393.1444054086.28679.python-list@python.org>
Lines: 43
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:97415

On 2015-10-06 00:51, Chris Angelico wrote:
> fn = "tmp1.csv"
> fin = open(fn, 'rb')
> rdr = csv.DictReader(fin, delimiter=',')
> # all the same down to here
> blanks = set(rdr.fieldnames)
> for row in rdr:
>     blanks = {col for col in blanks if not row[col]}
> mt = [col for col in rdr.fieldnames if col not in blanks]
> print mt

My only other modification would be to add a check that, if you no
longer have any blank columns, bail early from the loop:

  from cStringIO import StringIO
  import csv

  s = StringIO("""Name,Surname,Age,Sex
  abc,def,,M
  ,ghi,,F
  jkl,mno,,
  pqr,,,F
  """)

  dr = csv.DictReader(s)
  header_set = set(dr.fieldnames)
  for row in dr:
    header_set = set(h for h in header_set if not row[h])
    if not header_set:
      # we no longer have any headers, bail early
      break
  ordered_headers = [h for h in dr.fieldnames if h in header_set]
  print(header_set)
  print(ordered_headers)

That way, if you determine by line 3 that your million-row CSV file
has no blank columns, you can get away with not processing all
million rows.

-tkc