Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #62393

Re: Reading csv file

Show key headers only | View raw

[Multipart message — attachments visible in raw view] - view raw

You didn't pass in self as the first arg to open. This is necessary.

S


On 19 December 2013 09:22, Igor Korot <ikorot01@gmail.com> wrote:

> Hi, Peter,
> Thank you for the great suggestion.
>
> I tried to implement you code but failed.
>
> Here's what I have:
>
> class FileReader:
>         def __init__(self, filename, isSkip):
>                 self.path = filename
>                 self.isSkip = isSkip
>
>         @contextmanager
>         def open(*args):
>                 from StringIO import StringIO
>                 lines = range(10)
>                 if self.isSkip:
>                         lines[0] = "skipped"
>                         lines[6] = "field1-from-line6,field2-from-line6"
>                 else:
>                         lines[0] = "field1-from-line1,field2-from-line1"
>                 yield StringIO("\r\n".join(map(str, lines)))
>
>         def is_arbitrary_text(self,fieldnames):
>                 return "skipped" in fieldnames
>
>         def readData(self):
>                 with self.open(self.path, "r") as f:
>                         reader = csv.DictReader(f)
>                         if self.is_arbitrary_text(reader.fieldnames):
>                                 for _ in range(5):
>                                         next(reader, None)
>                                 reader._fieldnames = None
>                         for row in reader:
>                                 print row
>
> Unfortunately this does not work as "def open()" does not belong to my
> class and if I comment the "@contextmanager" line
> I will get an exception: "AttributeError: __exit__"
>
> Any idea what to do?
>
> Thank you.
>
>
> On Tue, Dec 17, 2013 at 2:51 AM, Peter Otten <__peter__@web.de> wrote:
> > Igor Korot wrote:
> >
> >> Hi, guys,
> >>
> >> On Tue, Dec 17, 2013 at 12:55 AM, Peter Otten <__peter__@web.de> wrote:
> >>> Peter Otten wrote:
> >>>
> >>>> You are still reading the complete csv file. Assuming
> >>>>
> >>>> (1) the first row of the csv contains the column names
> >>>> (2) you want to skip the first five rows of data
> >>
> >> Looking at the Peter's reply I realized I missed very important piece:
> >>
> >> The first row may or may not contain column names.
> >> If it does not, the first row will just contain some text, i.e. "abc"
> >> and the column names will be located on the row 6.
> >>
> >> I know if does complicate things but I am deeply sorry.
> >> The csv file is generated by some program run and I guess depending on
> >> the switches passed to
> >> that program it either creates the header in the csv (report name,
> >> time slice it ran at, user it ran under
> >> and some other info.
> >> Or it can be run without such switch and then it generates a normal csv.
> >>
> >> The report it generates is huge: it has about 30+ fields and I need to
> >> read this report, parse it and
> >> push accordingly to the database of mySQL.
> >>
> >> Thank you for any suggestions and sorry for not posting complete task.
> >
> > Try the following (without the mock-ups of course):
> >
> >
> > $ cat csv_skip_header.py
> > import csv
> > import sys
> > from contextlib import contextmanager
> >
> > filename = "ignored"
> >
> > @contextmanager
> > def open(*args):
> >     "mock-up, replace with open() built-in"
> >     from StringIO import StringIO
> >     lines = range(10)
> >     if len(sys.argv) > 1 and sys.argv[1] == "--skip":
> >         lines[0] = "skipped"
> >         lines[6] = "field1-from-line6,field2-from-line6"
> >     else:
> >         lines[0] = "field1-from-line1,field2-from-line1"
> >     yield StringIO("\r\n".join(map(str, lines)))
> >
> > def is_arbitrary_text(fieldnames):
> >     "mock-up, replace with the actual check"
> >     return "skipped" in fieldnames
> >
> > with open(filename, "rb") as f:
> >     reader = csv.DictReader(f)
> >     if is_arbitrary_text(reader.fieldnames):
> >         for _ in range(5):
> >             next(reader, None)
> >         reader._fieldnames = None # underscore necessary,
> >                                   # fieldnames setter doesn't work
> >         reader.fieldnames # used for its side-effect
> >     for row in reader:
> >         print row
> > $ python csv_skip_header.py
> > {'field2-from-line1': None, 'field1-from-line1': '1'}
> > {'field2-from-line1': None, 'field1-from-line1': '2'}
> > {'field2-from-line1': None, 'field1-from-line1': '3'}
> > {'field2-from-line1': None, 'field1-from-line1': '4'}
> > {'field2-from-line1': None, 'field1-from-line1': '5'}
> > {'field2-from-line1': None, 'field1-from-line1': '6'}
> > {'field2-from-line1': None, 'field1-from-line1': '7'}
> > {'field2-from-line1': None, 'field1-from-line1': '8'}
> > {'field2-from-line1': None, 'field1-from-line1': '9'}
> > $ python csv_skip_header.py --skip
> > {'field1-from-line6': '7', 'field2-from-line6': None}
> > {'field1-from-line6': '8', 'field2-from-line6': None}
> > {'field1-from-line6': '9', 'field2-from-line6': None}
> >
> > You may find the following a bit cleaner:
> >
> > with open(filename, "rb") as f:
> >     reader = csv.reader(f)
> >     fieldnames = next(reader)
> >     if is_arbitrary_text(fieldnames):
> >         for _ in range(5):
> >             next(reader, None)
> >         fieldnames = None
> >     reader = csv.DictReader(f, fieldnames=fieldnames)
> >     for row in reader:
> >         print row
> >
> > Or you do the skipping on the file (only if the rows don't have embedded
> > newlines).
> >
> > --
> > https://mail.python.org/mailman/listinfo/python-list
> --
> https://mail.python.org/mailman/listinfo/python-list
>

Back to comp.lang.python | Previous | Next | Find similar | Unroll thread

Thread

Re: Reading csv file Scott Jewloszewicz-Clarke <scottwjclarke@gmail.com> - 2013-12-19 10:30 +0000

csiph-web