Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]


Groups > comp.lang.python > #14937

Re: CSV writer question

Path csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!aioe.org!feeder.news-service.com!newsfeed.xs4all.nl!newsfeed5.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Return-Path <python-python-list@m.gmane.org>
X-Original-To python-list@python.org
Delivered-To python-list@mail.python.org
X-Spam-Status OK 0.000
X-Spam-Evidence '*H*': 1.00; '*S*': 0.00; 'instance': 0.05; 'suppose': 0.05; '__name__': 0.09; 'csv': 0.09; 'flush': 0.09; 'instance.': 0.09; 'received:80.91': 0.09; 'received:80.91.229': 0.09; 'received:80.91.229.12': 0.09; 'received:gmane.org': 0.09; 'received:list': 0.09; 'received:lo.gmane.org': 0.09; 'subclass': 0.09; 'def': 0.15; '"__main__":': 0.16; '(anywhere': 0.16; '0))': 0.16; 'buffering': 0.16; 'chunks': 0.16; 'contextlib': 0.16; 'lot!': 0.16; 'received:dip.t-dialin.net': 0.16; 'received:t-dialin.net': 0.16; 'subject:CSV': 0.16; 'written': 0.16; 'wrote:': 0.16; 'solution.': 0.17; 'subject:question': 0.18; 'say,': 0.19; '(but': 0.21; 'jason': 0.21; "doesn't": 0.22; 'disabling': 0.23; 'welcome.': 0.23; 'missed': 0.24; 'changed': 0.24; 'command': 0.24; 'statement': 0.25; "i'm": 0.27; '(the': 0.28; 'import': 0.28; 'bound': 0.29; 'temporary': 0.29; 'yield': 0.29; '(and': 0.29; 'binding': 0.30; 'from:addr:web.de': 0.30; 'writer': 0.30; 'thanks': 0.30; 'class': 0.30; "isn't": 0.33; 'there': 0.33; 'to:addr:python-list': 0.33; 'that,': 0.33; 'copying': 0.33; "i've": 0.34; 'end.': 0.34; 'explicit': 0.34; 'skip:@ 10': 0.34; 'thus,': 0.34; 'header:X-Complaints-To:1': 0.35; 'reference': 0.35; 'data,': 0.35; 'object': 0.35; 'unless': 0.36; 'file': 0.36; 'question': 0.36; 'another': 0.37; 'convenient': 0.37; 'but': 0.37; 'open': 0.37; 'could': 0.38; 'think': 0.38; 'received:org': 0.38; 'hello,': 0.38; 'subject:: ': 0.39; 'files,': 0.39; 'manually': 0.39; 'suggestions': 0.39; 'header:Mime-Version:1': 0.39; "there's": 0.39; 'data': 0.39; 'to:addr:python.org': 0.39; "i'd": 0.40; 'hit': 0.40; 'might': 0.40; 'kind': 0.61; 'full': 0.63; 'comments,': 0.64; 'therefore,': 0.66; 'due': 0.66; 'taking': 0.66; 'reference?': 0.84
X-Injected-Via-Gmane http://gmane.org/
To python-list@python.org
From Peter Otten <__peter__@web.de>
Subject Re: CSV writer question
Date Mon, 24 Oct 2011 23:09:16 +0200
Organization None
References <CAEk9e3rzxQruEO4J_xc8pxw9y4mq6QK2HK=nTX7TdECm2+T8+g@mail.gmail.com>
Mime-Version 1.0
Content-Type text/plain; charset="ISO-8859-1"
Content-Transfer-Encoding 7Bit
X-Gmane-NNTP-Posting-Host p5084a5fb.dip.t-dialin.net
X-BeenThere python-list@python.org
X-Mailman-Version 2.1.12
Precedence list
List-Id General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe <http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive <http://mail.python.org/pipermail/python-list>
List-Post <mailto:python-list@python.org>
List-Help <mailto:python-list-request@python.org?subject=help>
List-Subscribe <http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups comp.lang.python
Message-ID <mailman.2193.1319490549.27778.python-list@python.org> (permalink)
Lines 62
NNTP-Posting-Host 2001:888:2000:d::a6
X-Trace 1319490549 news.xs4all.nl 6880 [2001:888:2000:d::a6]:44511
X-Complaints-To abuse@xs4all.nl
Xref x330-a1.tempe.blueboxinc.net comp.lang.python:14937

Show key headers only | View raw


Jason Swails wrote:

> Hello,
> 
> I have a question about a csv.writer instance.  I have a utility that I
> want to write a full CSV file from lots of data, but due to performance
> (and memory) considerations, there's no way I can write the data
> sequentially. Therefore, I write the data in chunks to temporary files,
> then combine them
> all at the end.  For convenience, I declare each writer instance via a
> statement like
> 
> my_csv = csv.writer(open('temp.1.csv', 'wb'))
> 
> so the open file object isn't bound to any explicit reference, and I don't
> know how to reference it inside the writer class (the documentation
> doesn't
> say, unless I've missed the obvious).  Thus, the only way I can think of
> to make sure that all of the data is written before I start copying these
> files sequentially into the final file is to unbuffer them so the above
> command is changed to
> 
> my_csv = csv.writer(open('temp.1.csv', 'wb', 0))
> 
> unless, of course, I add an explicit reference to track the open file
> object
> and manually close or flush it (but I'd like to avoid it if possible).  My
> question is 2-fold.  Is there a way to do that directly via the CSV API,
> or is the approach I'm taking the only way without binding the open file
> object
> to another reference?  Secondly, if these files are potentially very large
> (anywhere from ~1KB to 20 GB depending on the amount of data present),
> what kind of performance hit will I be looking at by disabling buffering
> on these types of files?
> 
> Tips, answers, comments, and/or suggestions are all welcome.
> 
> Thanks a lot!
> Jason
> 
> As an afterthought, I suppose I could always subclass the csv.writer class
> and add the reference I want to that, which I may do if there's no other
> convenient solution.

A contextmanager might help:

import csv
from contextlib import contextmanager

@contextmanager
def filewriter(filename):
    with open(filename, "wb") as outstream:
        yield csv.writer(outstream)


if __name__ == "__main__":
    with filewriter("tmp.csv") as writer:
        writer.writerows([
                ["alpha", "beta"],
                ["gamma", "delta"]])

Back to comp.lang.python | Previous | Next | Find similar | Unroll thread


Thread

Re: CSV writer question Peter Otten <__peter__@web.de> - 2011-10-24 23:09 +0200

csiph-web