Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #75362

Re: Load a CSV with different row lengths

Show all headers | View raw

Miki Tebeka wrote:

> Greetings,
> 
>> I should've mentioned that I want to import my csv as a data frame or
>> numpy array or as a table.
> If you know the max length of a row, then you can do something like:
>     def gen_rows(stream, max_length):
>         for row in csv.reader(stream):
>             yield row + ([None] * (max_length - len(line))
> 
>     max_length = 10
>     with open('data.csv') as fo:
>         df = pd.DataFrame.from_records(gen_rows(fo, max_length))

With the help of the search engine that must not be named and some trial and 
error I also found a way to use pandas.read_csv():

$ cat data.csv
a,b
a,b,c,d
a,b,c
$ python3
Python 3.3.2+ (default, Feb 28 2014, 00:52:16) 
[GCC 4.8.1] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
>>> pandas.read_csv("data.csv", names=list(range(4)))
   0  1    2    3
0  a  b  NaN  NaN
1  a  b    c    d
2  a  b    c  NaN

And if the maximum row length is not known here's a modification of Miki's 
recipe:

def gen_rows(stream, max_length=None):
      rows = csv.reader(stream)
      if max_length is None:
          rows = list(rows)
          max_length = max(len(row) for row in rows)
      for row in rows:
          yield row + [None] * (max_length - len(row))

with open('data.csv') as f:
    df = pd.DataFrame.from_records(list(gen_rows(f))) # my version of pandas
                                                      # does not accept a
                                                      # generator

Back to comp.lang.python | Previous | Next — Previous in thread | Find similar | Unroll thread

Thread

Re: Load a CSV with different row lengths Ryan de Vera <ryan.devera.03@gmail.com> - 2014-07-29 09:08 -0400
  Re: Load a CSV with different row lengths Miki Tebeka <miki.tebeka@gmail.com> - 2014-07-29 22:00 -0700
    Re: Load a CSV with different row lengths Peter Otten <__peter__@web.de> - 2014-07-30 10:16 +0200

csiph-web