Path: csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!aioe.org!feeder.news-service.com!newsfeed.xs4all.nl!newsfeed5.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Subject: Re: losing-end-of-row values when manipulating CSV input
Mime-Version: 1.0 (Apple Message framework v1082)
Content-Type: text/plain; charset=us-ascii
From: Neil Berg <nberg@atmos.ucla.edu>
In-Reply-To: <4E1E0B75.3000304@stoneleaf.us>
Date: Wed, 13 Jul 2011 14:19:21 -0700
Content-Transfer-Encoding: quoted-printable
References: <CC0B2F67-6C7A-4D29-9B07-508C37EDE6F6@atmos.ucla.edu> <4E1E0B75.3000304@stoneleaf.us>
To: Ethan Furman <ethan@stoneleaf.us>
Cc: python-list@python.org
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.1004.1310591889.1164.python-list@python.org>
Lines: 69
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: x330-a1.tempe.blueboxinc.net comp.lang.python:9440

Ethan,

Thank you for this tip- you are correct that I saved the data as rich =
text.  I remade the files in VI instead of TextEdit, which allowed me to =
write a true csv file and the script works as expected now. =20

Thanks again,
Neil

On Jul 13, 2011, at 2:17 PM, Ethan Furman wrote:

> Neil Berg wrote:
>> Hello all,
>>=20
>> I am having an issue with my attempts to accurately filter some data =
from a CSV file I am importing.  I have attached both a sample of the =
CSV data and my script. =20
>>=20
>> The attached CSV file contains two rows and 27 columns of data.  The =
first column is the station ID "BLS", the second column is the sensor =
number "4", the third column is the date, and the remaining 24 columns =
are hourly temperature readings.=20
>>=20
>> In my attached script, I read in row[3:] to extract just the =
temperatures, do a sanity check to make sure there are 24 values, remove =
any missing or "m" values, and then append the non-missing values into =
the "hour_list". =20
>>=20
>> Strangely the the first seven rows appear to be empty after reading =
into the CSV file, so that's what I had to incorporate the if len(temps) =
=3D=3D 24 statement. =20
>>=20
>> But the real issue is that for days with no missing values, for =
example the second row of data, the length of the hour_list should be =
24.  My script, however, is returning 23.  I think this is because the =
end-of-row-values have a trailing "\". This must mark these numbers as =
non-digits and are lost in my "isdig" filter line.  I've tried several =
ways to remove this trailing "\", but to no success.=20
>>=20
>> Do you have any suggestions on how to fix this issue?
>>=20
>> Many thanks in advance,
>>=20
>> Neil Berg
>>=20
>>=20
> Your 'csv' file isn't -- it looks like rich text.  An actual csv file=20=

> looks more like this (which is probably what you're seeing in notepad,=20=

> but is not what is stored on disk):
>=20
> =
8<------------------------------------------------------------------------=
------------
> =
BLS,4,19981101,37,m,36,34,36,35,34,34,35,36,38,39,43,42,42,42,38,36,34,32,=
33,33,35,34
> =
BLS,4,19981102,34,32,33,32,34,32,33,32,34,38,40,41,44,47,43,42,39,36,35,35=
,36,36,35,33
> =
8<------------------------------------------------------------------------=
------------
>=20
> Try saving in text-only format.
>=20
> ~Ethan~