Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #13121

Re: Deadlock problem using multiprocessing

Show key headers only | View raw

2011/9/11 蓝色基因 <bluegene8210@gmail.com>:
> This is my first touch on the multiprocessing module, and I admit not
> having a deep understanding of parallel programming, forgive me if
> there's any obvious error. This is my test code:
>
> # deadlock.py
>
> import multiprocessing
>
> class MPTask:
>        def __init__(self):
>                self._tseq= range(10)   # task sequence
>                self._pool= multiprocessing.Pool(2)     # process pool
>        def _exe(self, num):
>                return num**2
>        def run(self):
>                result= self._pool.map_async(self._exe, self._tseq)
>                return result.get()
>
> result= MPTask().run()
> print(result)
>
> And seemingly it creates a deadlock, I have to manually kill the
> processes in the system monitor, yet it would be OK if the _exe()
> function was defined outside the MPTask class:
>
> # no_deadlock.py
> import multiprocessing
>
> def _exe(num):
>        return num**2
>
> class MPTask:
>        def __init__(self):
>                self._tseq= range(10)   # task sequence
>                self._pool= multiprocessing.Pool(2)     # process pool
>        def run(self):
>                result= self._pool.map_async(_exe, self._tseq)
>                return result.get()
>
> result= MPTask().run()
> print(result)
>
> This would give the correct answer without any deadlock. My questions:
>
> 1. Why is this, where did the deadlock come from?
>
> 2. Besides, is there any material I can refer to about how to avoid
> deadlocks when using multiple processes in the program?
>

I get this exception when I run the first program:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.1/threading.py", line 516, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.1/threading.py", line 469, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.1/multiprocessing/pool.py", line 231, in _handle_tasks
    put(task)
  File "/usr/lib/python3.1/pickle.py", line 1349, in dumps
    Pickler(f, protocol, fix_imports=fix_imports).dump(obj)
_pickle.PicklingError: Can't pickle <class 'method'>: attribute lookup
builtins.method failed

There is no deadlock.  You are hitting this problem:
http://stackoverflow.com/questions/1816958/cant-pickle-type-instancemethod-when-using-pythons-multiprocessing-pool-map

-- 
regards,
kushal

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Deadlock problem using multiprocessing 蓝色基因 <bluegene8210@gmail.com> - 2011-09-10 20:54 -0700
  Re: Deadlock problem using multiprocessing Kushal Kumaran <kushal.kumaran+python@gmail.com> - 2011-09-11 17:02 +0530
    Re: Deadlock problem using multiprocessing Jacky Liu <bluegene8210@gmail.com> - 2011-09-11 08:46 -0700

csiph-web