Path: csiph.com!backup.pasdenom.info!usenet.pasdenom.info!gegeweb.org!usenet-fr.net!nerim.net!novso.com!newsfeed.xs4all.nl!newsfeed4.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
To: python-list@python.org
From: Terry Reedy <tjreedy@udel.edu>
Subject: Re: Multi-threading in Python vs Java
Date: Fri, 11 Oct 2013 15:53:41 -0400
References: <46669eab-49f4-4daf-a410-abfbe9e87fc3@googlegroups.com> <6d35b728-85c2-4fc0-bbc8-6c033bdfcfb0@googlegroups.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.0
In-Reply-To: <6d35b728-85c2-4fc0-bbc8-6c033bdfcfb0@googlegroups.com>
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.1014.1381521241.18130.python-list@python.org>
Lines: 26
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:56704

On 10/11/2013 4:41 AM, Peter Cacioppi wrote:

> I should add that the computational heavy lifting is done in a third party library. So a worker thread looks roughly like this (there is a subtle race condition I'm glossing over).
>
> while len(jobs) :
>     job = jobs.pop()
>     model = Model(job)      # Model is py interface for a lib written in C
>     newJobs = model.solve() # This will take a long time
>     for each newJob in newJobs :
>       jobs.add(newJob)
>
> Here jobs is a thread safe object that is shared across each worker thread. It holds a priority queue of jobs that can be solved in parallel.
>
> Model is a py class that provides the API to a 3rd party library written in C.I know model.solve() will be the bottleneck operation for all but trivial problems.
>
> So, my hope is that the GIL restrictions won't be problematic here. That is to say, I don't need **Python** code to ever run concurrently. I just need Python to allow a different Python worker thread to execute when all the other worker threads are blocking on the model.solve() task. Once the algorithm is in full swing, it is typical for all the worker threads should be blocking on model.Solve() at the same time.
>
> It's a nice algorithm for high level languages. Java worked well here, I'm hoping py can be nearly as fast with a much more elegant and readable code.

Given that model.solve takes a 'long time' (seconds, at least), the 
extra time to start a process over the time to start a thread will be 
inconsequential. I would therefore look at the multiprocessing module.

-- 
Terry Jan Reedy