Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #56704

Re: Multi-threading in Python vs Java

Show key headers only | View raw

On 10/11/2013 4:41 AM, Peter Cacioppi wrote:

> I should add that the computational heavy lifting is done in a third party library. So a worker thread looks roughly like this (there is a subtle race condition I'm glossing over).
>
> while len(jobs) :
>     job = jobs.pop()
>     model = Model(job)      # Model is py interface for a lib written in C
>     newJobs = model.solve() # This will take a long time
>     for each newJob in newJobs :
>       jobs.add(newJob)
>
> Here jobs is a thread safe object that is shared across each worker thread. It holds a priority queue of jobs that can be solved in parallel.
>
> Model is a py class that provides the API to a 3rd party library written in C.I know model.solve() will be the bottleneck operation for all but trivial problems.
>
> So, my hope is that the GIL restrictions won't be problematic here. That is to say, I don't need **Python** code to ever run concurrently. I just need Python to allow a different Python worker thread to execute when all the other worker threads are blocking on the model.solve() task. Once the algorithm is in full swing, it is typical for all the worker threads should be blocking on model.Solve() at the same time.
>
> It's a nice algorithm for high level languages. Java worked well here, I'm hoping py can be nearly as fast with a much more elegant and readable code.

Given that model.solve takes a 'long time' (seconds, at least), the 
extra time to start a process over the time to start a thread will be 
inconsequential. I would therefore look at the multiprocessing module.

-- 
Terry Jan Reedy

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Multi-threading in Python vs Java Peter Cacioppi <peter.cacioppi@gmail.com> - 2013-10-10 23:01 -0700
  Re: Multi-threading in Python vs Java Cameron Simpson <cs@zip.com.au> - 2013-10-11 17:53 +1100
    Re: Multi-threading in Python vs Java Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-10-11 09:30 +0000
  Re: Multi-threading in Python vs Java Peter Cacioppi <peter.cacioppi@gmail.com> - 2013-10-11 01:41 -0700
    Re: Multi-threading in Python vs Java Chris Angelico <rosuav@gmail.com> - 2013-10-11 19:48 +1100
      Re: Multi-threading in Python vs Java Piet van Oostrum <piet@vanoostrum.org> - 2013-10-11 10:55 -0400
    Re: Multi-threading in Python vs Java Terry Reedy <tjreedy@udel.edu> - 2013-10-11 15:53 -0400
  Re: Multi-threading in Python vs Java Peter Cacioppi <peter.cacioppi@gmail.com> - 2013-10-11 13:10 -0700

csiph-web