Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!news.albasani.net!news.stack.nl!newsfeed.xs4all.nl!newsfeed4.news.xs4all.nl!xs4all!news.tele.dk!news.tele.dk!small.news.tele.dk!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.000 X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'else:': 0.03; 'column': 0.07; 'none,': 0.07; 'subject:file': 0.07; 'sys': 0.07; 'assuming': 0.09; 'complicate': 0.09; 'filename': 0.09; 'mysql.': 0.09; 'none)': 0.09; 'report,': 0.09; 'rows': 0.09; 'underscore': 0.09; 'cc:addr:python-list': 0.11; 'python': 0.11; 'def': 0.12; 'missed': 0.12; 'creates': 0.14; '"r")': 0.16; '>>': 0.16; '(without': 0.16; 'arg': 0.16; 'at,': 0.16; 'csv': 0.16; 'exception:': 0.16; 'filename,': 0.16; 'guys,': 0.16; 'igor': 0.16; 'open()': 0.16; 'skipping': 0.16; 'stringio': 0.16; 'subject:Reading': 0.16; 'suggestion.': 0.16; 'switches': 0.16; 'task.': 0.16; 'wrote:': 0.18; 'bit': 0.19; 'have:': 0.19; 'not,': 0.20; '>>>': 0.22; 'import': 0.22; 'email addr:gmail.com>': 0.22; 'cc:addr:python.org': 0.22; 'print': 0.22; 'cc:2**1': 0.23; '>>>': 0.24; 'necessary.': 0.24; 'parse': 0.24; 'replace': 0.24; 'skip': 0.24; 'skip:{ 20': 0.24; 'text,': 0.24; 'header': 0.24; 'file.': 0.24; 'cc:no real name:2**0': 0.24; '>': 0.26; 'push': 0.26; 'skip:" 30': 0.26; 'switch': 0.26; 'pass': 0.26; 'header:In-Reply-To:1': 0.27; 'tried': 0.27; 'idea': 0.28; 'am,': 0.29; "doesn't": 0.30; 'dec': 0.30; 'skip:@ 10': 0.30; 'message-id:@mail.gmail.com': 0.30; 'url:mailman': 0.30; 'code': 0.31; 'lines': 0.31; 'posting': 0.31; '>>>>': 0.31; 'info.': 0.31; 'names.': 0.31; 'file': 0.32; 'class': 0.32; 'run': 0.32; 'url:python': 0.33; 'guess': 0.33; 'reader': 0.33; 'actual': 0.34; 'comment': 0.34; 'skip:_ 10': 0.34; 'december': 0.35; 'skip:s 30': 0.35; '(2)': 0.35; 'but': 0.35; 'received:google.com': 0.35; 'located': 0.36; 'belong': 0.36; 'i.e.': 0.36; 'yield': 0.36; 'url:listinfo': 0.36; "didn't": 0.36; 'hi,': 0.36; 'url:org': 0.36; 'implement': 0.38; 'skip:& 10': 0.38; 'thank': 0.38; 'skip:& 20': 0.39; 'does': 0.39; 'embedded': 0.39; 'either': 0.39; 'url:mail': 0.40; 'skip:u 10': 0.60; 'read': 0.60; 'first': 0.61; 'you.': 0.62; 'complete': 0.62; 'such': 0.63; 'great': 0.65; 'to:addr:gmail.com': 0.65; 'reply': 0.66; 'realized': 0.68; 'deeply': 0.69; "'1'}": 0.84; "'2'}": 0.84; 'otten': 0.84; 'peter,': 0.84; 'sorry.': 0.91; '2013': 0.98 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=kQWt5U1/88hpi9bcYn9TZzDDb1ikLFLH1VVqMzSGCHI=; b=C4cHwDSKvQ1aanBKLQ5FL1e6x0H/CIhea90HCV17oHWX34gETrNgBUq8fIxAFWSRs2 1NSJgQoPZcw3cQEt0/gHPjHma6oPA85uWpfoAp/Hw28Zso7VpkUIPkPnNgCjKUIFYQLx YrTSVIyvKJwFrMyK+DSNdsTlITrnwpnXeVq/VXGbjFDwj96lXlK4jBYRIp6FPoX+RrG7 FioDE6eO/GE+2dXikf/vu7WNIbbVO+KnHxHuLAOZopI/L1cogDvYpoUHBZi4g9B0t+hc nGpMWXZ6FeI+V/+MrTfDGGX9AFTQjmDqXQIp8ln0Xrcf2oeLC/5pmX6Qn3fqIXqI9IXe qYMA== MIME-Version: 1.0 X-Received: by 10.66.234.131 with SMTP id ue3mr713319pac.35.1387449022564; Thu, 19 Dec 2013 02:30:22 -0800 (PST) In-Reply-To: References: Date: Thu, 19 Dec 2013 10:30:22 +0000 Subject: Re: Reading csv file From: Scott Jewloszewicz-Clarke To: Igor Korot Content-Type: multipart/alternative; boundary=047d7b15a2b12251c204ede0a6b1 X-Mailman-Approved-At: Thu, 19 Dec 2013 11:55:27 +0100 Cc: python-list@python.org X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 365 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1387450528 news.xs4all.nl 2857 [2001:888:2000:d::a6]:54882 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:62393 --047d7b15a2b12251c204ede0a6b1 Content-Type: text/plain; charset=ISO-8859-1 You didn't pass in self as the first arg to open. This is necessary. S On 19 December 2013 09:22, Igor Korot wrote: > Hi, Peter, > Thank you for the great suggestion. > > I tried to implement you code but failed. > > Here's what I have: > > class FileReader: > def __init__(self, filename, isSkip): > self.path = filename > self.isSkip = isSkip > > @contextmanager > def open(*args): > from StringIO import StringIO > lines = range(10) > if self.isSkip: > lines[0] = "skipped" > lines[6] = "field1-from-line6,field2-from-line6" > else: > lines[0] = "field1-from-line1,field2-from-line1" > yield StringIO("\r\n".join(map(str, lines))) > > def is_arbitrary_text(self,fieldnames): > return "skipped" in fieldnames > > def readData(self): > with self.open(self.path, "r") as f: > reader = csv.DictReader(f) > if self.is_arbitrary_text(reader.fieldnames): > for _ in range(5): > next(reader, None) > reader._fieldnames = None > for row in reader: > print row > > Unfortunately this does not work as "def open()" does not belong to my > class and if I comment the "@contextmanager" line > I will get an exception: "AttributeError: __exit__" > > Any idea what to do? > > Thank you. > > > On Tue, Dec 17, 2013 at 2:51 AM, Peter Otten <__peter__@web.de> wrote: > > Igor Korot wrote: > > > >> Hi, guys, > >> > >> On Tue, Dec 17, 2013 at 12:55 AM, Peter Otten <__peter__@web.de> wrote: > >>> Peter Otten wrote: > >>> > >>>> You are still reading the complete csv file. Assuming > >>>> > >>>> (1) the first row of the csv contains the column names > >>>> (2) you want to skip the first five rows of data > >> > >> Looking at the Peter's reply I realized I missed very important piece: > >> > >> The first row may or may not contain column names. > >> If it does not, the first row will just contain some text, i.e. "abc" > >> and the column names will be located on the row 6. > >> > >> I know if does complicate things but I am deeply sorry. > >> The csv file is generated by some program run and I guess depending on > >> the switches passed to > >> that program it either creates the header in the csv (report name, > >> time slice it ran at, user it ran under > >> and some other info. > >> Or it can be run without such switch and then it generates a normal csv. > >> > >> The report it generates is huge: it has about 30+ fields and I need to > >> read this report, parse it and > >> push accordingly to the database of mySQL. > >> > >> Thank you for any suggestions and sorry for not posting complete task. > > > > Try the following (without the mock-ups of course): > > > > > > $ cat csv_skip_header.py > > import csv > > import sys > > from contextlib import contextmanager > > > > filename = "ignored" > > > > @contextmanager > > def open(*args): > > "mock-up, replace with open() built-in" > > from StringIO import StringIO > > lines = range(10) > > if len(sys.argv) > 1 and sys.argv[1] == "--skip": > > lines[0] = "skipped" > > lines[6] = "field1-from-line6,field2-from-line6" > > else: > > lines[0] = "field1-from-line1,field2-from-line1" > > yield StringIO("\r\n".join(map(str, lines))) > > > > def is_arbitrary_text(fieldnames): > > "mock-up, replace with the actual check" > > return "skipped" in fieldnames > > > > with open(filename, "rb") as f: > > reader = csv.DictReader(f) > > if is_arbitrary_text(reader.fieldnames): > > for _ in range(5): > > next(reader, None) > > reader._fieldnames = None # underscore necessary, > > # fieldnames setter doesn't work > > reader.fieldnames # used for its side-effect > > for row in reader: > > print row > > $ python csv_skip_header.py > > {'field2-from-line1': None, 'field1-from-line1': '1'} > > {'field2-from-line1': None, 'field1-from-line1': '2'} > > {'field2-from-line1': None, 'field1-from-line1': '3'} > > {'field2-from-line1': None, 'field1-from-line1': '4'} > > {'field2-from-line1': None, 'field1-from-line1': '5'} > > {'field2-from-line1': None, 'field1-from-line1': '6'} > > {'field2-from-line1': None, 'field1-from-line1': '7'} > > {'field2-from-line1': None, 'field1-from-line1': '8'} > > {'field2-from-line1': None, 'field1-from-line1': '9'} > > $ python csv_skip_header.py --skip > > {'field1-from-line6': '7', 'field2-from-line6': None} > > {'field1-from-line6': '8', 'field2-from-line6': None} > > {'field1-from-line6': '9', 'field2-from-line6': None} > > > > You may find the following a bit cleaner: > > > > with open(filename, "rb") as f: > > reader = csv.reader(f) > > fieldnames = next(reader) > > if is_arbitrary_text(fieldnames): > > for _ in range(5): > > next(reader, None) > > fieldnames = None > > reader = csv.DictReader(f, fieldnames=fieldnames) > > for row in reader: > > print row > > > > Or you do the skipping on the file (only if the rows don't have embedded > > newlines). > > > > -- > > https://mail.python.org/mailman/listinfo/python-list > -- > https://mail.python.org/mailman/listinfo/python-list > --047d7b15a2b12251c204ede0a6b1 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
You didn't pass in self as the first arg to open. This= is necessary.

S
<= br>
On 19 December 2013 09:22, Igor Korot <= ikorot01@gmail.com> wrote:
Hi, Peter,
Thank you for the great suggestion.

I tried to implement you code but failed.

Here's what I have:

class FileReader:
=A0 =A0 =A0 =A0 def __init__(self, filename, isSkip):
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 self.path =3D filename
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 self.isSkip =3D isSkip

=A0 =A0 =A0 =A0 @contextmanager
=A0 =A0 =A0 =A0 def open(*args):
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 from StringIO import StringIO
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 lines =3D range(10)
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 if self.isSkip:
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 lines[0] =3D "skipped&= quot;
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 lines[6] =3D "field1-f= rom-line6,field2-from-line6"
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 else:
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 lines[0] =3D "field1-f= rom-line1,field2-from-line1"
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 yield StringIO("\r\n".join(map(st= r, lines)))

=A0 =A0 =A0 =A0 def is_arbitrary_text(self,fieldnames):
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 return "skipped" in fieldnames
=A0 =A0 =A0 =A0 def readData(self):
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 with self.open(self.path, "r") as= f:
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 reader =3D csv.DictReader(f= )
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 if self.is_arbitrary_text(r= eader.fieldnames):
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 for _ in ra= nge(5):
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 next(reader, None)
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 reader._fie= ldnames =3D None
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 for row in reader:
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 print row
Unfortunately this does not work as "def open()" does not belong = to my
class and if I comment the "@contextmanager" line
I will get an exception: "AttributeError: __exit__"

Any idea what to do?

Thank you.


On Tue, Dec 17, 2013 at 2:51 AM, Peter Otten <__peter__@web.de> wrote:
> Igor Korot wrote:
>
>> Hi, guys,
>>
>> On Tue, Dec 17, 2013 at 12:55 AM, Peter Otten <__peter__@web.de> wrote:
>>> Peter Otten wrote:
>>>
>>>> You are still reading the complete csv file. Assuming
>>>>
>>>> (1) the first row of the csv contains the column names
>>>> (2) you want to skip the first five rows of data
>>
>> Looking at the Peter's reply I realized I missed very importan= t piece:
>>
>> The first row may or may not contain column names.
>> If it does not, the first row will just contain some text, i.e. &q= uot;abc"
>> and the column names will be located on the row 6.
>>
>> I know if does complicate things but I am deeply sorry.
>> The csv file is generated by some program run and I guess dependin= g on
>> the switches passed to
>> that program it either creates the header in the csv (report name,=
>> time slice it ran at, user it ran under
>> and some other info.
>> Or it can be run without such switch and then it generates a norma= l csv.
>>
>> The report it generates is huge: it has about 30+ fields and I nee= d to
>> read this report, parse it and
>> push accordingly to the database of mySQL.
>>
>> Thank you for any suggestions and sorry for not posting complete t= ask.
>
> Try the following (without the mock-ups of course):
>
>
> $ cat csv_skip_header.py
> import csv
> import sys
> from contextlib import contextmanager
>
> filename =3D "ignored"
>
> @contextmanager
> def open(*args):
> =A0 =A0 "mock-up, replace with open() built-in"
> =A0 =A0 from StringIO import StringIO
> =A0 =A0 lines =3D range(10)
> =A0 =A0 if len(sys.argv) > 1 and sys.argv[1] =3D=3D "--skip&qu= ot;:
> =A0 =A0 =A0 =A0 lines[0] =3D "skipped"
> =A0 =A0 =A0 =A0 lines[6] =3D "field1-from-line6,field2-from-line6= "
> =A0 =A0 else:
> =A0 =A0 =A0 =A0 lines[0] =3D "field1-from-line1,field2-from-line1= "
> =A0 =A0 yield StringIO("\r\n".join(map(str, lines)))
>
> def is_arbitrary_text(fieldnames):
> =A0 =A0 "mock-up, replace with the actual check"
> =A0 =A0 return "skipped" in fieldnames
>
> with open(filename, "rb") as f:
> =A0 =A0 reader =3D csv.DictReader(f)
> =A0 =A0 if is_arbitrary_text(reader.fieldnames):
> =A0 =A0 =A0 =A0 for _ in range(5):
> =A0 =A0 =A0 =A0 =A0 =A0 next(reader, None)
> =A0 =A0 =A0 =A0 reader._fieldnames =3D None # underscore necessary, > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 # = fieldnames setter doesn't work
> =A0 =A0 =A0 =A0 reader.fieldnames # used for its side-effect
> =A0 =A0 for row in reader:
> =A0 =A0 =A0 =A0 print row
> $ python csv_skip_header.py
> {'field2-from-line1': None, 'field1-from-line1': '= 1'}
> {'field2-from-line1': None, 'field1-from-line1': '= 2'}
> {'field2-from-line1': None, 'field1-from-line1': '= 3'}
> {'field2-from-line1': None, 'field1-from-line1': '= 4'}
> {'field2-from-line1': None, 'field1-from-line1': '= 5'}
> {'field2-from-line1': None, 'field1-from-line1': '= 6'}
> {'field2-from-line1': None, 'field1-from-line1': '= 7'}
> {'field2-from-line1': None, 'field1-from-line1': '= 8'}
> {'field2-from-line1': None, 'field1-from-line1': '= 9'}
> $ python csv_skip_header.py --skip
> {'field1-from-line6': '7', 'field2-from-line6'= : None}
> {'field1-from-line6': '8', 'field2-from-line6'= : None}
> {'field1-from-line6': '9', 'field2-from-line6'= : None}
>
> You may find the following a bit cleaner:
>
> with open(filename, "rb") as f:
> =A0 =A0 reader =3D csv.reader(f)
> =A0 =A0 fieldnames =3D next(reader)
> =A0 =A0 if is_arbitrary_text(fieldnames):
> =A0 =A0 =A0 =A0 for _ in range(5):
> =A0 =A0 =A0 =A0 =A0 =A0 next(reader, None)
> =A0 =A0 =A0 =A0 fieldnames =3D None
> =A0 =A0 reader =3D csv.DictReader(f, fieldnames=3Dfieldnames)
> =A0 =A0 for row in reader:
> =A0 =A0 =A0 =A0 print row
>
> Or you do the skipping on the file (only if the rows don't have em= bedded
> newlines).
>
> --
> https://mail.python.org/mailman/listinfo/python-list
--
https://mail.python.org/mailman/listinfo/python-list

--047d7b15a2b12251c204ede0a6b1--