Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.lang.python > #57497
| From | Stefan Behnel <stefan_ml@behnel.de> |
|---|---|
| Subject | Re: Processing large CSV files - how to maximise throughput? |
| Date | 2013-10-25 08:39 +0200 |
| References | <b4737555-cb4f-457b-aed7-a1e6553fe6a5@googlegroups.com> <mailman.1494.1382667030.18130.python-list@python.org> <5269e6f6$0$29972$c3e8da3$5496439d@news.astraweb.com> <l4cq6t$oq6$1@ger.gmane.org> <CAPTjJmqvjMMqd-JzaL3BtVu3=bwgYCdpFdMSHEa8kf5kdpVJyA@mail.gmail.com> |
| Newsgroups | comp.lang.python |
| Message-ID | <mailman.1499.1382683179.18130.python-list@python.org> (permalink) |
Chris Angelico, 25.10.2013 08:13: > On Fri, Oct 25, 2013 at 2:57 PM, Dave Angel wrote: >> But I would concur -- probably they'll both give about the same speedup. >> I just detest the pain that multithreading can bring, and tend to avoid >> it if at all possible. > > I don't have a history of major pain from threading. Is this a Python > thing, or have I just been really really fortunate Likely the latter. Threads are ok if what they do is essentially what you could easily use multiple processes for as well, i.e. process independent data, maybe from/to independent files etc., using dedicated channels for communication. As soon as you need them to share any state, however, it's really easy to get it wrong and to run into concurrency issues that are difficult to reproduce and debug. Basically, with multiple processes, you start with independent systems and add connections specifically where needed, whereas with threads, you start with completely shared state and then prune away interdependencies and concurrency until it seems to work safely. That approach makes it essentially impossible to prove that threading is safe in a given setup, except for the really trivial cases. Stefan
Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread
Processing large CSV files - how to maximise throughput? Victor Hooi <victorhooi@gmail.com> - 2013-10-24 18:38 -0700
Re: Processing large CSV files - how to maximise throughput? Dave Angel <davea@davea.name> - 2013-10-25 02:10 +0000
Re: Processing large CSV files - how to maximise throughput? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-10-25 03:35 +0000
Re: Processing large CSV files - how to maximise throughput? Dave Angel <davea@davea.name> - 2013-10-25 03:57 +0000
Re: Processing large CSV files - how to maximise throughput? Chris Angelico <rosuav@gmail.com> - 2013-10-25 17:13 +1100
Re: Processing large CSV files - how to maximise throughput? Stefan Behnel <stefan_ml@behnel.de> - 2013-10-25 08:39 +0200
Re: Processing large CSV files - how to maximise throughput? Chris Angelico <rosuav@gmail.com> - 2013-10-25 18:26 +1100
Re: Processing large CSV files - how to maximise throughput? Dave Angel <davea@davea.name> - 2013-10-25 11:24 +0000
Re: Processing large CSV files - how to maximise throughput? Chris Angelico <rosuav@gmail.com> - 2013-10-25 22:42 +1100
Re: Processing large CSV files - how to maximise throughput? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-10-25 03:19 +0000
Re: Processing large CSV files - how to maximise throughput? Mark Lawrence <breamoreboy@yahoo.co.uk> - 2013-10-25 04:46 +0100
Re: Processing large CSV files - how to maximise throughput? Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2013-10-25 19:44 -0400
Re: Processing large CSV files - how to maximise throughput? Roy Smith <roy@panix.com> - 2013-10-25 20:22 -0400
Re: Processing large CSV files - how to maximise throughput? Walter Hurry <walterhurry@lavabit.com> - 2013-10-26 08:53 +0000
csiph-web