Path: csiph.com!usenet.pasdenom.info!goblin2!goblin.stu.neva.ru!newsfeed.xs4all.nl!newsfeed1.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
MIME-Version: 1.0
In-Reply-To: <l8pa8t$mmi$1@ger.gmane.org>
References: <CA+FnnTwpiq=Z6tswub9vab2wxzzQycwyL2jNCgK2RTQuhGrrMQ@mail.gmail.com> <l8p1f5$gk0$1@ger.gmane.org> <l8p3gj$3gb$1@ger.gmane.org> <CA+FnnTw+Qt-g8+aDeMi2nDENsWDDVzW6AZBLJzcCJtvEKjrRZg@mail.gmail.com> <l8pa8t$mmi$1@ger.gmane.org>
Date: Thu, 19 Dec 2013 01:22:26 -0800
Subject: Re: Reading csv file
From: Igor Korot <ikorot01@gmail.com>
To: Peter Otten <__peter__@web.de>, python-list@python.org
Content-Type: text/plain; charset=ISO-8859-1
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.4407.1387444950.18130.python-list@python.org>
Lines: 147
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:62390

Hi, Peter,
Thank you for the great suggestion.

I tried to implement you code but failed.

Here's what I have:

class FileReader:
        def __init__(self, filename, isSkip):
                self.path = filename
                self.isSkip = isSkip

        @contextmanager
        def open(*args):
                from StringIO import StringIO
                lines = range(10)
                if self.isSkip:
                        lines[0] = "skipped"
                        lines[6] = "field1-from-line6,field2-from-line6"
                else:
                        lines[0] = "field1-from-line1,field2-from-line1"
                yield StringIO("\r\n".join(map(str, lines)))

        def is_arbitrary_text(self,fieldnames):
                return "skipped" in fieldnames

        def readData(self):
                with self.open(self.path, "r") as f:
                        reader = csv.DictReader(f)
                        if self.is_arbitrary_text(reader.fieldnames):
                                for _ in range(5):
                                        next(reader, None)
                                reader._fieldnames = None
                        for row in reader:
                                print row

Unfortunately this does not work as "def open()" does not belong to my
class and if I comment the "@contextmanager" line
I will get an exception: "AttributeError: __exit__"

Any idea what to do?

Thank you.


On Tue, Dec 17, 2013 at 2:51 AM, Peter Otten <__peter__@web.de> wrote:
> Igor Korot wrote:
>
>> Hi, guys,
>>
>> On Tue, Dec 17, 2013 at 12:55 AM, Peter Otten <__peter__@web.de> wrote:
>>> Peter Otten wrote:
>>>
>>>> You are still reading the complete csv file. Assuming
>>>>
>>>> (1) the first row of the csv contains the column names
>>>> (2) you want to skip the first five rows of data
>>
>> Looking at the Peter's reply I realized I missed very important piece:
>>
>> The first row may or may not contain column names.
>> If it does not, the first row will just contain some text, i.e. "abc"
>> and the column names will be located on the row 6.
>>
>> I know if does complicate things but I am deeply sorry.
>> The csv file is generated by some program run and I guess depending on
>> the switches passed to
>> that program it either creates the header in the csv (report name,
>> time slice it ran at, user it ran under
>> and some other info.
>> Or it can be run without such switch and then it generates a normal csv.
>>
>> The report it generates is huge: it has about 30+ fields and I need to
>> read this report, parse it and
>> push accordingly to the database of mySQL.
>>
>> Thank you for any suggestions and sorry for not posting complete task.
>
> Try the following (without the mock-ups of course):
>
>
> $ cat csv_skip_header.py
> import csv
> import sys
> from contextlib import contextmanager
>
> filename = "ignored"
>
> @contextmanager
> def open(*args):
>     "mock-up, replace with open() built-in"
>     from StringIO import StringIO
>     lines = range(10)
>     if len(sys.argv) > 1 and sys.argv[1] == "--skip":
>         lines[0] = "skipped"
>         lines[6] = "field1-from-line6,field2-from-line6"
>     else:
>         lines[0] = "field1-from-line1,field2-from-line1"
>     yield StringIO("\r\n".join(map(str, lines)))
>
> def is_arbitrary_text(fieldnames):
>     "mock-up, replace with the actual check"
>     return "skipped" in fieldnames
>
> with open(filename, "rb") as f:
>     reader = csv.DictReader(f)
>     if is_arbitrary_text(reader.fieldnames):
>         for _ in range(5):
>             next(reader, None)
>         reader._fieldnames = None # underscore necessary,
>                                   # fieldnames setter doesn't work
>         reader.fieldnames # used for its side-effect
>     for row in reader:
>         print row
> $ python csv_skip_header.py
> {'field2-from-line1': None, 'field1-from-line1': '1'}
> {'field2-from-line1': None, 'field1-from-line1': '2'}
> {'field2-from-line1': None, 'field1-from-line1': '3'}
> {'field2-from-line1': None, 'field1-from-line1': '4'}
> {'field2-from-line1': None, 'field1-from-line1': '5'}
> {'field2-from-line1': None, 'field1-from-line1': '6'}
> {'field2-from-line1': None, 'field1-from-line1': '7'}
> {'field2-from-line1': None, 'field1-from-line1': '8'}
> {'field2-from-line1': None, 'field1-from-line1': '9'}
> $ python csv_skip_header.py --skip
> {'field1-from-line6': '7', 'field2-from-line6': None}
> {'field1-from-line6': '8', 'field2-from-line6': None}
> {'field1-from-line6': '9', 'field2-from-line6': None}
>
> You may find the following a bit cleaner:
>
> with open(filename, "rb") as f:
>     reader = csv.reader(f)
>     fieldnames = next(reader)
>     if is_arbitrary_text(fieldnames):
>         for _ in range(5):
>             next(reader, None)
>         fieldnames = None
>     reader = csv.DictReader(f, fieldnames=fieldnames)
>     for row in reader:
>         print row
>
> Or you do the skipping on the file (only if the rows don't have embedded
> newlines).
>
> --
> https://mail.python.org/mailman/listinfo/python-list