Path: csiph.com!newsfeed.hal-mli.net!feeder3.hal-mli.net!newsfeed.hal-mli.net!feeder1.hal-mli.net!newsfeed.xs4all.nl!newsfeed2.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
To: python-list@python.org
From: Dave Angel <davea@davea.name>
Subject: Re: *.csv to *.txt after adding columns
Date: Wed, 18 Sep 2013 07:55:52 +0000 (UTC)
References: <41edba54-31d3-48ad-a50f-41f87f32d251@googlegroups.com> <c7b5595f-b847-4172-9a1d-86287e0804d5@googlegroups.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
User-Agent: XPN/1.2.6 (Street Spirit ; Linux)
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.107.1379490982.18130.python-list@python.org>
Lines: 63
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:54362

On 17/9/2013 22:28, Bryan Britten wrote:

> Dave -
>
> I can't print the output because there are close to 1,000,000 records. It would be extremely inefficient and resource intensive to look at every row.

Not if you made a sample directory with about 3 files, each containing
half a dozen lines.

> Like I said, when I take just one file and run the code over the first
> few records I get what I'd expect to see. Here's an example(non-redacted code):
>
> INPUT:
>
> import csv
>
> fileHandle = 'C:/Users/Bryan/Data Analysis/Crime Analysis/Data/'

Now, that directory specification ends with a slash.  So "+" will work
correctly.  But your original did not.

>
> varNames = 'ID\tCaseNum\tDate\tTime\tBlock\tIUCR\tPrimaryType\tDescription\tLocDesc\tArrest\tDomestic\tBeat\tDistrict\tWard\tCommArea\tFBICode\tXCoord\tYCoord\tYear\tUpdatedOn\tLat\tLong\tLoc\n'
>
> outFile = open(fileHandle + 'ChiCrime01_02.txt', 'w')
> inFile = open(fileHandle + 'ChiCrime01_02.csv', 'rb')

Instead of changing this code, you could have switched to a directory
containing only one file.

>
> Like I said, the output is exactly what I want, but it doesn't seem to be writing to the file and I don't know why. I said I didn't know if it was raising an exception because I'm new to Python and I didn't know if there were some methods that included "silent" errors where it would continue the code but produce the wrong results, such as not writing my files. 
>

The only "silent" exception I know of is the one triggered by
sys.exit(), used to terminate the process.

> Lastly, why does everyone seem to push for os.path.join versus the method I have used? Is it just a 'standard' that people like to see?
>

Because os.path.join will be smart enough to add the slashes only if
they are necessary.   That can be useful, especially if the directory
you're using as a prefix came from the user.


I think Peter's suggestion is probably right on;  you don't limit your
infiles to *.csv, so you will be processing *.txt files the second time.

Another useful debugging aid would have been to add print statements
right after opening the files, perhaps something like:

print "Reading :", infile.name
print "Writing:", outfile.name

If those names had been missing slashes, I would have been vindicated,
while if they were the same, you'd know Peter had nailed it.



-- 
DaveA