Path: csiph.com!newsfeed.hal-mli.net!feeder3.hal-mli.net!newsfeed.hal-mli.net!feeder1.hal-mli.net!newsfeed.xs4all.nl!newsfeed2.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.007 X-Spam-Evidence: '*H*': 0.99; '*S*': 0.00; 'subject:Python': 0.06; 'subtle': 0.09; 'api': 0.11; 'python': 0.11; 'thread': 0.14; '(there': 0.16; 'from:addr:cs': 0.16; 'from:addr:zip.com.au': 0.16; 'from:name:cameron simpson': 0.16; 'message- id:@cskk.homeip.net': 0.16; 'qualifying': 0.16; 'received:211.29': 0.16; 'received:211.29.132': 0.16; 'received:cskk.homeip.net': 0.16; 'received:homeip.net': 0.16; 'received:optusnet.com.au': 0.16; 'received:syd.optusnet.com.au': 0.16; 'reedy': 0.16; 'simpson': 0.16; 'subject:Java': 0.16; 'wrote:': 0.18; 'library': 0.18; 'written': 0.21; 'header:User-Agent:1': 0.23; 'him.': 0.24; 'cheers,': 0.24; 'looks': 0.24; 'holds': 0.26; 'header:In-Reply- To:1': 0.27; 'am,': 0.29; "i'm": 0.30; 'contrast,': 0.31; 'jean': 0.31; 'trivial': 0.31; 'class': 0.32; 'interface': 0.32; 'not.': 0.33; 'third': 0.33; 'but': 0.35; 'add': 0.35; 'library.': 0.36; 'module.': 0.36; 'received:com.au': 0.36; 'done': 0.36; 'charset :us-ascii': 0.36; 'should': 0.36; 'received:211': 0.38; 'to:addr :python-list': 0.38; 'to:addr:python.org': 0.39; 'problems.': 0.60; 'content-disposition:inline': 0.62; 'jobs': 0.68; 'safe': 0.72; 'therefore': 0.72; 'heavy': 0.81; 'friends': 0.81; 'lifting': 0.84; 'suited': 0.93; 'race': 0.95 Date: Sat, 12 Oct 2013 08:35:26 +1100 From: Cameron Simpson To: python-list@python.org Subject: Re: Multi-threading in Python vs Java MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) References: X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.1 cv=YYGEuWhf c=1 sm=1 tr=0 a=YuQlxtEQCowy2cfE5kc7TA==:117 a=YuQlxtEQCowy2cfE5kc7TA==:17 a=ZtCCktOnAAAA:8 a=PO7r1zJSAAAA:8 a=LcaDllckn3IA:10 a=dI4LPTJexOgA:10 a=kj9zAlcOel0A:10 a=vrnE16BAAAAA:8 a=8AHkEIZyAAAA:8 a=jcJtVDayNoYA:10 a=wfYmOVlN25FtdHir3JMA:9 a=CjuIK1q_8ugA:10 X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 29 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1381527331 news.xs4all.nl 15864 [2001:888:2000:d::a6]:48058 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:56713 On 11Oct2013 15:53, Terry Reedy wrote: > On 10/11/2013 4:41 AM, Peter Cacioppi wrote: > >I should add that the computational heavy lifting is done in a third party library. So a worker thread looks roughly like this (there is a subtle race condition I'm glossing over). > > > >while len(jobs) : > > job = jobs.pop() > > model = Model(job) # Model is py interface for a lib written in C > > newJobs = model.solve() # This will take a long time > > for each newJob in newJobs : > > jobs.add(newJob) > > > >Here jobs is a thread safe object that is shared across each worker thread. It holds a priority queue of jobs that can be solved in parallel. > > > >Model is a py class that provides the API to a 3rd party library written in C.I know model.solve() will be the bottleneck operation for all but trivial problems. [...] > Given that model.solve takes a 'long time' (seconds, at least), the > extra time to start a process over the time to start a thread will > be inconsequential. I would therefore look at the multiprocessing > module. And, for contrast, I would not. Threads are my friends and Python threads seem eminently suited to the above scenario. Cheers, -- Cameron Simpson [Alain] had been looking at his dashboard, and had not seen me, so I ran into him. - Jean Alesi on his qualifying prang at Imola '93