Path: csiph.com!usenet.pasdenom.info!news.albasani.net!news.stack.nl!newsfeed.xs4all.nl!newsfeed4.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
To: python-list@python.org
From: Peter Otten <__peter__@web.de>
Subject: Re: Reading csv file
Date: Tue, 17 Dec 2013 11:51:04 +0100
Organization: None
References: <CA+FnnTwpiq=Z6tswub9vab2wxzzQycwyL2jNCgK2RTQuhGrrMQ@mail.gmail.com> <l8p1f5$gk0$1@ger.gmane.org> <l8p3gj$3gb$1@ger.gmane.org> <CA+FnnTw+Qt-g8+aDeMi2nDENsWDDVzW6AZBLJzcCJtvEKjrRZg@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="ISO-8859-1"
Content-Transfer-Encoding: 7Bit
User-Agent: KNode/4.7.3
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.4275.1387277432.18130.python-list@python.org>
Lines: 99
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:62176

Igor Korot wrote:

> Hi, guys,
> 
> On Tue, Dec 17, 2013 at 12:55 AM, Peter Otten <__peter__@web.de> wrote:
>> Peter Otten wrote:
>>
>>> You are still reading the complete csv file. Assuming
>>>
>>> (1) the first row of the csv contains the column names
>>> (2) you want to skip the first five rows of data
> 
> Looking at the Peter's reply I realized I missed very important piece:
> 
> The first row may or may not contain column names.
> If it does not, the first row will just contain some text, i.e. "abc"
> and the column names will be located on the row 6.
> 
> I know if does complicate things but I am deeply sorry.
> The csv file is generated by some program run and I guess depending on
> the switches passed to
> that program it either creates the header in the csv (report name,
> time slice it ran at, user it ran under
> and some other info.
> Or it can be run without such switch and then it generates a normal csv.
> 
> The report it generates is huge: it has about 30+ fields and I need to
> read this report, parse it and
> push accordingly to the database of mySQL.
> 
> Thank you for any suggestions and sorry for not posting complete task.

Try the following (without the mock-ups of course):


$ cat csv_skip_header.py
import csv
import sys
from contextlib import contextmanager

filename = "ignored"

@contextmanager
def open(*args):
    "mock-up, replace with open() built-in"
    from StringIO import StringIO
    lines = range(10)
    if len(sys.argv) > 1 and sys.argv[1] == "--skip":
        lines[0] = "skipped"
        lines[6] = "field1-from-line6,field2-from-line6"
    else:
        lines[0] = "field1-from-line1,field2-from-line1"
    yield StringIO("\r\n".join(map(str, lines)))

def is_arbitrary_text(fieldnames):
    "mock-up, replace with the actual check"
    return "skipped" in fieldnames

with open(filename, "rb") as f:
    reader = csv.DictReader(f)
    if is_arbitrary_text(reader.fieldnames):
        for _ in range(5):
            next(reader, None)
        reader._fieldnames = None # underscore necessary,
                                  # fieldnames setter doesn't work
        reader.fieldnames # used for its side-effect
    for row in reader:
        print row
$ python csv_skip_header.py 
{'field2-from-line1': None, 'field1-from-line1': '1'}
{'field2-from-line1': None, 'field1-from-line1': '2'}
{'field2-from-line1': None, 'field1-from-line1': '3'}
{'field2-from-line1': None, 'field1-from-line1': '4'}
{'field2-from-line1': None, 'field1-from-line1': '5'}
{'field2-from-line1': None, 'field1-from-line1': '6'}
{'field2-from-line1': None, 'field1-from-line1': '7'}
{'field2-from-line1': None, 'field1-from-line1': '8'}
{'field2-from-line1': None, 'field1-from-line1': '9'}
$ python csv_skip_header.py --skip
{'field1-from-line6': '7', 'field2-from-line6': None}
{'field1-from-line6': '8', 'field2-from-line6': None}
{'field1-from-line6': '9', 'field2-from-line6': None}

You may find the following a bit cleaner:

with open(filename, "rb") as f:
    reader = csv.reader(f)
    fieldnames = next(reader)
    if is_arbitrary_text(fieldnames):
        for _ in range(5):
            next(reader, None)
        fieldnames = None
    reader = csv.DictReader(f, fieldnames=fieldnames)
    for row in reader:
        print row

Or you do the skipping on the file (only if the rows don't have embedded 
newlines).