Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.016 X-Spam-Evidence: '*H*': 0.97; '*S*': 0.00; 'essentially': 0.04; 'needed,': 0.07; 'received:80.91': 0.09; 'received:80.91.229': 0.09; 'received:gmane.org': 0.09; 'received:list': 0.09; 'subject:files': 0.09; 'python': 0.11; 'concur': 0.16; 'concurrency': 0.16; 'from:addr:behnel.de': 0.16; 'from:addr:stefan_ml': 0.16; 'from:name:stefan behnel': 0.16; 'received:80.91.229.3': 0.16; 'received:plane.gmane.org': 0.16; 'reproduce': 0.16; 'safely.': 0.16; 'subject:CSV': 0.16; 'threads,': 0.16; 'wrote:': 0.18; 'stefan': 0.19; 'seems': 0.21; 'header:User-Agent:1': 0.23; 'setup,': 0.24; 'tend': 0.24; 'header:X-Complaints-To:1': 0.27; 'header:In-Reply-To:1': 0.27; 'specifically': 0.29; 'chris': 0.29; '25,': 0.31; "they'll": 0.31; 'trivial': 0.31; 'probably': 0.32; 'run': 0.32; 'fri,': 0.33; 'maybe': 0.34; 'could': 0.34; 'except': 0.35; 'possible.': 0.35; 'received:84': 0.35; 'but': 0.35; 'add': 0.35; 'really': 0.36; 'data,': 0.36; 'i.e.': 0.36; 'subject:?': 0.36; 'wrong': 0.37; 'easily': 0.37; 'connections': 0.38; 'to:addr:python-list': 0.38; 'files': 0.38; 'pm,': 0.38; 'to:addr:python.org': 0.39; 'received:org': 0.40; 'major': 0.40; 'easy': 0.60; 'dave': 0.60; 'soon': 0.63; 'channels': 0.68; 'safe': 0.72; 'fortunate': 0.84; 'pain': 0.84; 'received:arcor-ip.net': 0.84; 'received:pools .arcor-ip.net': 0.84; 'angel': 0.91; 'processes,': 0.91; 'thing,': 0.91; 'whereas': 0.91; '2013': 0.98 X-Injected-Via-Gmane: http://gmane.org/ To: python-list@python.org From: Stefan Behnel Subject: Re: Processing large CSV files - how to maximise throughput? Date: Fri, 25 Oct 2013 08:39:22 +0200 References: <5269e6f6$0$29972$c3e8da3$5496439d@news.astraweb.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Gmane-NNTP-Posting-Host: dslb-084-056-040-079.pools.arcor-ip.net User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.0 In-Reply-To: X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 28 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1382683179 news.xs4all.nl 15866 [2001:888:2000:d::a6]:57761 X-Complaints-To: abuse@xs4all.nl Path: csiph.com!usenet.pasdenom.info!news.franciliens.net!feed.ac-versailles.fr!nerim.net!novso.com!newsfeed.xs4all.nl!newsfeed3.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail Xref: csiph.com comp.lang.python:57497 Chris Angelico, 25.10.2013 08:13: > On Fri, Oct 25, 2013 at 2:57 PM, Dave Angel wrote: >> But I would concur -- probably they'll both give about the same speedup. >> I just detest the pain that multithreading can bring, and tend to avoid >> it if at all possible. > > I don't have a history of major pain from threading. Is this a Python > thing, or have I just been really really fortunate Likely the latter. Threads are ok if what they do is essentially what you could easily use multiple processes for as well, i.e. process independent data, maybe from/to independent files etc., using dedicated channels for communication. As soon as you need them to share any state, however, it's really easy to get it wrong and to run into concurrency issues that are difficult to reproduce and debug. Basically, with multiple processes, you start with independent systems and add connections specifically where needed, whereas with threads, you start with completely shared state and then prune away interdependencies and concurrency until it seems to work safely. That approach makes it essentially impossible to prove that threading is safe in a given setup, except for the really trivial cases. Stefan