Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #75349 > unrolled thread

Re: Load a CSV with different row lengths

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Load a CSV with different row lengths Ryan de Vera <ryan.devera.03@gmail.com> - 2014-07-29 09:08 -0400
    Re: Load a CSV with different row lengths Miki Tebeka <miki.tebeka@gmail.com> - 2014-07-29 22:00 -0700
      Re: Load a CSV with different row lengths Peter Otten <__peter__@web.de> - 2014-07-30 10:16 +0200

#75349 — Re: Load a CSV with different row lengths

[Multipart message — attachments visible in raw view] — view raw

Hey skip,

I should've mentioned that I want to import my csv as a data frame or numpy
array or as a table.

Best regards,

Ryan


On Tue, Jul 29, 2014 at 6:29 AM, Skip Montanaro <skip@pobox.com> wrote:

> > How can I load this into python? I tried using both NumPy and Pandas.
>
> To add to Peter's response, I would be very surprised if numpy or
> Pandas couldn't be coaxed into loading your CSV file, but you didn't
> provide any details about what you expected and what you got. I've
> used Pandas to read CSV files a lot recently, and run into any
> trouble. (I suspect all but a few have equal length rows, but in cases
> where data are missing, I've found it generally inserts NaNs.)
>
> In general, you'll get more useful feedback with more complete
> questions. I'm not saying you need to necessarily provide code, but a
> traceback or unexpected output would be helpful.
>
> Skip
>

[toc] | [next] | [standalone]

#75358

Greetings,

> I should've mentioned that I want to import my csv as a data frame or numpy array or as a table.
If you know the max length of a row, then you can do something like:
    def gen_rows(stream, max_length):
        for row in csv.reader(stream):
            yield row + ([None] * (max_length - len(line))

    max_length = 10
    with open('data.csv') as fo:
        df = pd.DataFrame.from_records(gen_rows(fo, max_length))


HTH,
Miki

[toc] | [prev] | [next] | [standalone]

#75362

Miki Tebeka wrote:

> Greetings,
> 
>> I should've mentioned that I want to import my csv as a data frame or
>> numpy array or as a table.
> If you know the max length of a row, then you can do something like:
>     def gen_rows(stream, max_length):
>         for row in csv.reader(stream):
>             yield row + ([None] * (max_length - len(line))
> 
>     max_length = 10
>     with open('data.csv') as fo:
>         df = pd.DataFrame.from_records(gen_rows(fo, max_length))

With the help of the search engine that must not be named and some trial and 
error I also found a way to use pandas.read_csv():

$ cat data.csv
a,b
a,b,c,d
a,b,c
$ python3
Python 3.3.2+ (default, Feb 28 2014, 00:52:16) 
[GCC 4.8.1] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
>>> pandas.read_csv("data.csv", names=list(range(4)))
   0  1    2    3
0  a  b  NaN  NaN
1  a  b    c    d
2  a  b    c  NaN

And if the maximum row length is not known here's a modification of Miki's 
recipe:

def gen_rows(stream, max_length=None):
      rows = csv.reader(stream)
      if max_length is None:
          rows = list(rows)
          max_length = max(len(row) for row in rows)
      for row in rows:
          yield row + [None] * (max_length - len(row))

with open('data.csv') as f:
    df = pd.DataFrame.from_records(list(gen_rows(f))) # my version of pandas
                                                      # does not accept a
                                                      # generator

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web