Path: csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!selfless.tophat.at!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
From: Neil Berg <nberg@atmos.ucla.edu>
Content-Type: multipart/mixed; boundary=Apple-Mail-3--578610568
Subject: losing-end-of-row values when manipulating CSV input
Date: Wed, 13 Jul 2011 13:22:34 -0700
To: python-list@python.org
Mime-Version: 1.0 (Apple Message framework v1082)
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.1000.1310588475.1164.python-list@python.org>
Lines: 92
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: x330-a1.tempe.blueboxinc.net comp.lang.python:9433


--Apple-Mail-3--578610568
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

Hello all,

I am having an issue with my attempts to accurately filter some data =
from a CSV file I am importing.  I have attached both a sample of the =
CSV data and my script. =20

The attached CSV file contains two rows and 27 columns of data.  The =
first column is the station ID "BLS", the second column is the sensor =
number "4", the third column is the date, and the remaining 24 columns =
are hourly temperature readings.=20

In my attached script, I read in row[3:] to extract just the =
temperatures, do a sanity check to make sure there are 24 values, remove =
any missing or "m" values, and then append the non-missing values into =
the "hour_list". =20

Strangely the the first seven rows appear to be empty after reading into =
the CSV file, so that's what I had to incorporate the if len(temps) =3D=3D=
 24 statement. =20

But the real issue is that for days with no missing values, for example =
the second row of data, the length of the hour_list should be 24.  My =
script, however, is returning 23.  I think this is because the =
end-of-row-values have a trailing "\". This must mark these numbers as =
non-digits and are lost in my "isdig" filter line.  I've tried several =
ways to remove this trailing "\", but to no success.=20

Do you have any suggestions on how to fix this issue?

Many thanks in advance,

Neil Berg


--Apple-Mail-3--578610568
Content-Disposition: attachment;
	filename=csv_test.py
Content-Type: text/x-python-script;
	name="csv_test.py"
Content-Transfer-Encoding: 7bit

# Purpose: read in a CSV file containing hourly temps. at each station,
# then append non-missing hourly data into a list and find the maximum
# value of that list
#---------------------------------------------------------------------
import csv
from numpy import *

f = csv.reader(open('csv_sample.csv','rb'))

for row in f:
	temps= row[3:] #extract hourly temps, neglect station ID,sensor ID, and date 
	#print temps # you see here that the first seven rows are empty 
	if len(temps) == 24: #only keep rows with 24 temps in them  
		hour_list = [] #empty list of all integer hourly temps, i.e. exclude missing "m" values
		for val in temps:	
			#print val #here you can see that the end-of-row values have a trailing "\"
			#--------------------------------------------------------------------
			# This is where I want to strip the trailing "\" before removing any
			# missing or "m" values
			#--------------------------------------------------------------------
			isdig = str.isdigit(val) 
			if isdig is True: 
				hour_list.append(val) 
		print len(hour_list) #should be 24 for rows with no missing values, but it's 23 as is



--Apple-Mail-3--578610568
Content-Disposition: attachment;
	filename=csv_sample.csv
Content-Type: text/csv;
	name="csv_sample.csv"
Content-Transfer-Encoding: 7bit

{\rtf1\ansi\ansicpg1252\cocoartf1038\cocoasubrtf350
{\fonttbl\f0\fmodern\fcharset0 Courier;}
{\colortbl;\red255\green255\blue255;}
\margl1440\margr1440\vieww14540\viewh7680\viewkind0
\deftab720
\pard\pardeftab720\ql\qnatural

\f0\fs24 \cf0 BLS,4,19981101,37,m,36,34,36,35,34,34,35,36,38,39,43,42,42,42,38,36,34,32,33,33,35,34\
BLS,4,19981102,34,32,33,32,34,32,33,32,34,38,40,41,44,47,43,42,39,36,35,35,36,36,35,33\
}
--Apple-Mail-3--578610568--