Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.lang.python > #14937
| Path | csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!aioe.org!feeder.news-service.com!newsfeed.xs4all.nl!newsfeed5.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail |
|---|---|
| Return-Path | <python-python-list@m.gmane.org> |
| X-Original-To | python-list@python.org |
| Delivered-To | python-list@mail.python.org |
| X-Spam-Status | OK 0.000 |
| X-Spam-Evidence | '*H*': 1.00; '*S*': 0.00; 'instance': 0.05; 'suppose': 0.05; '__name__': 0.09; 'csv': 0.09; 'flush': 0.09; 'instance.': 0.09; 'received:80.91': 0.09; 'received:80.91.229': 0.09; 'received:80.91.229.12': 0.09; 'received:gmane.org': 0.09; 'received:list': 0.09; 'received:lo.gmane.org': 0.09; 'subclass': 0.09; 'def': 0.15; '"__main__":': 0.16; '(anywhere': 0.16; '0))': 0.16; 'buffering': 0.16; 'chunks': 0.16; 'contextlib': 0.16; 'lot!': 0.16; 'received:dip.t-dialin.net': 0.16; 'received:t-dialin.net': 0.16; 'subject:CSV': 0.16; 'written': 0.16; 'wrote:': 0.16; 'solution.': 0.17; 'subject:question': 0.18; 'say,': 0.19; '(but': 0.21; 'jason': 0.21; "doesn't": 0.22; 'disabling': 0.23; 'welcome.': 0.23; 'missed': 0.24; 'changed': 0.24; 'command': 0.24; 'statement': 0.25; "i'm": 0.27; '(the': 0.28; 'import': 0.28; 'bound': 0.29; 'temporary': 0.29; 'yield': 0.29; '(and': 0.29; 'binding': 0.30; 'from:addr:web.de': 0.30; 'writer': 0.30; 'thanks': 0.30; 'class': 0.30; "isn't": 0.33; 'there': 0.33; 'to:addr:python-list': 0.33; 'that,': 0.33; 'copying': 0.33; "i've": 0.34; 'end.': 0.34; 'explicit': 0.34; 'skip:@ 10': 0.34; 'thus,': 0.34; 'header:X-Complaints-To:1': 0.35; 'reference': 0.35; 'data,': 0.35; 'object': 0.35; 'unless': 0.36; 'file': 0.36; 'question': 0.36; 'another': 0.37; 'convenient': 0.37; 'but': 0.37; 'open': 0.37; 'could': 0.38; 'think': 0.38; 'received:org': 0.38; 'hello,': 0.38; 'subject:: ': 0.39; 'files,': 0.39; 'manually': 0.39; 'suggestions': 0.39; 'header:Mime-Version:1': 0.39; "there's": 0.39; 'data': 0.39; 'to:addr:python.org': 0.39; "i'd": 0.40; 'hit': 0.40; 'might': 0.40; 'kind': 0.61; 'full': 0.63; 'comments,': 0.64; 'therefore,': 0.66; 'due': 0.66; 'taking': 0.66; 'reference?': 0.84 |
| X-Injected-Via-Gmane | http://gmane.org/ |
| To | python-list@python.org |
| From | Peter Otten <__peter__@web.de> |
| Subject | Re: CSV writer question |
| Date | Mon, 24 Oct 2011 23:09:16 +0200 |
| Organization | None |
| References | <CAEk9e3rzxQruEO4J_xc8pxw9y4mq6QK2HK=nTX7TdECm2+T8+g@mail.gmail.com> |
| Mime-Version | 1.0 |
| Content-Type | text/plain; charset="ISO-8859-1" |
| Content-Transfer-Encoding | 7Bit |
| X-Gmane-NNTP-Posting-Host | p5084a5fb.dip.t-dialin.net |
| X-BeenThere | python-list@python.org |
| X-Mailman-Version | 2.1.12 |
| Precedence | list |
| List-Id | General discussion list for the Python programming language <python-list.python.org> |
| List-Unsubscribe | <http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe> |
| List-Archive | <http://mail.python.org/pipermail/python-list> |
| List-Post | <mailto:python-list@python.org> |
| List-Help | <mailto:python-list-request@python.org?subject=help> |
| List-Subscribe | <http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe> |
| Newsgroups | comp.lang.python |
| Message-ID | <mailman.2193.1319490549.27778.python-list@python.org> (permalink) |
| Lines | 62 |
| NNTP-Posting-Host | 2001:888:2000:d::a6 |
| X-Trace | 1319490549 news.xs4all.nl 6880 [2001:888:2000:d::a6]:44511 |
| X-Complaints-To | abuse@xs4all.nl |
| Xref | x330-a1.tempe.blueboxinc.net comp.lang.python:14937 |
Show key headers only | View raw
Jason Swails wrote:
> Hello,
>
> I have a question about a csv.writer instance. I have a utility that I
> want to write a full CSV file from lots of data, but due to performance
> (and memory) considerations, there's no way I can write the data
> sequentially. Therefore, I write the data in chunks to temporary files,
> then combine them
> all at the end. For convenience, I declare each writer instance via a
> statement like
>
> my_csv = csv.writer(open('temp.1.csv', 'wb'))
>
> so the open file object isn't bound to any explicit reference, and I don't
> know how to reference it inside the writer class (the documentation
> doesn't
> say, unless I've missed the obvious). Thus, the only way I can think of
> to make sure that all of the data is written before I start copying these
> files sequentially into the final file is to unbuffer them so the above
> command is changed to
>
> my_csv = csv.writer(open('temp.1.csv', 'wb', 0))
>
> unless, of course, I add an explicit reference to track the open file
> object
> and manually close or flush it (but I'd like to avoid it if possible). My
> question is 2-fold. Is there a way to do that directly via the CSV API,
> or is the approach I'm taking the only way without binding the open file
> object
> to another reference? Secondly, if these files are potentially very large
> (anywhere from ~1KB to 20 GB depending on the amount of data present),
> what kind of performance hit will I be looking at by disabling buffering
> on these types of files?
>
> Tips, answers, comments, and/or suggestions are all welcome.
>
> Thanks a lot!
> Jason
>
> As an afterthought, I suppose I could always subclass the csv.writer class
> and add the reference I want to that, which I may do if there's no other
> convenient solution.
A contextmanager might help:
import csv
from contextlib import contextmanager
@contextmanager
def filewriter(filename):
with open(filename, "wb") as outstream:
yield csv.writer(outstream)
if __name__ == "__main__":
with filewriter("tmp.csv") as writer:
writer.writerows([
["alpha", "beta"],
["gamma", "delta"]])
Back to comp.lang.python | Previous | Next | Find similar | Unroll thread
Re: CSV writer question Peter Otten <__peter__@web.de> - 2011-10-24 23:09 +0200
csiph-web