Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #13108 > unrolled thread

Deadlock problem using multiprocessing

Back to article view | Back to comp.lang.python

Contents

  Deadlock problem using multiprocessing 蓝色基因 <bluegene8210@gmail.com> - 2011-09-10 20:54 -0700
    Re: Deadlock problem using multiprocessing Kushal Kumaran <kushal.kumaran+python@gmail.com> - 2011-09-11 17:02 +0530
      Re: Deadlock problem using multiprocessing Jacky Liu <bluegene8210@gmail.com> - 2011-09-11 08:46 -0700

#13108 — Deadlock problem using multiprocessing

This is my first touch on the multiprocessing module, and I admit not
having a deep understanding of parallel programming, forgive me if
there's any obvious error. This is my test code:

# deadlock.py

import multiprocessing

class MPTask:
	def __init__(self):
		self._tseq= range(10)	# task sequence
		self._pool= multiprocessing.Pool(2)	# process pool
	def _exe(self, num):
		return num**2
	def run(self):
		result= self._pool.map_async(self._exe, self._tseq)
		return result.get()

result= MPTask().run()
print(result)

And seemingly it creates a deadlock, I have to manually kill the
processes in the system monitor, yet it would be OK if the _exe()
function was defined outside the MPTask class:

# no_deadlock.py
import multiprocessing

def _exe(num):
	return num**2

class MPTask:
	def __init__(self):
		self._tseq= range(10)	# task sequence
		self._pool= multiprocessing.Pool(2)	# process pool
	def run(self):
		result= self._pool.map_async(_exe, self._tseq)
		return result.get()

result= MPTask().run()
print(result)

This would give the correct answer without any deadlock. My questions:

1. Why is this, where did the deadlock come from?

2. Besides, is there any material I can refer to about how to avoid
deadlocks when using multiple processes in the program?

Thanks!

[toc] | [next] | [standalone]

#13121

2011/9/11 蓝色基因 <bluegene8210@gmail.com>:
> This is my first touch on the multiprocessing module, and I admit not
> having a deep understanding of parallel programming, forgive me if
> there's any obvious error. This is my test code:
>
> # deadlock.py
>
> import multiprocessing
>
> class MPTask:
>        def __init__(self):
>                self._tseq= range(10)   # task sequence
>                self._pool= multiprocessing.Pool(2)     # process pool
>        def _exe(self, num):
>                return num**2
>        def run(self):
>                result= self._pool.map_async(self._exe, self._tseq)
>                return result.get()
>
> result= MPTask().run()
> print(result)
>
> And seemingly it creates a deadlock, I have to manually kill the
> processes in the system monitor, yet it would be OK if the _exe()
> function was defined outside the MPTask class:
>
> # no_deadlock.py
> import multiprocessing
>
> def _exe(num):
>        return num**2
>
> class MPTask:
>        def __init__(self):
>                self._tseq= range(10)   # task sequence
>                self._pool= multiprocessing.Pool(2)     # process pool
>        def run(self):
>                result= self._pool.map_async(_exe, self._tseq)
>                return result.get()
>
> result= MPTask().run()
> print(result)
>
> This would give the correct answer without any deadlock. My questions:
>
> 1. Why is this, where did the deadlock come from?
>
> 2. Besides, is there any material I can refer to about how to avoid
> deadlocks when using multiple processes in the program?
>

I get this exception when I run the first program:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.1/threading.py", line 516, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.1/threading.py", line 469, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.1/multiprocessing/pool.py", line 231, in _handle_tasks
    put(task)
  File "/usr/lib/python3.1/pickle.py", line 1349, in dumps
    Pickler(f, protocol, fix_imports=fix_imports).dump(obj)
_pickle.PicklingError: Can't pickle <class 'method'>: attribute lookup
builtins.method failed

There is no deadlock.  You are hitting this problem:
http://stackoverflow.com/questions/1816958/cant-pickle-type-instancemethod-when-using-pythons-multiprocessing-pool-map

-- 
regards,
kushal

[toc] | [prev] | [next] | [standalone]

#13130

>
> I get this exception when I run the first program:
>
> Exception in thread Thread-1:
> Traceback (most recent call last):
>   File "/usr/lib/python3.1/threading.py", line 516, in _bootstrap_inner
>     self.run()
>   File "/usr/lib/python3.1/threading.py", line 469, in run
>     self._target(*self._args, **self._kwargs)
>   File "/usr/lib/python3.1/multiprocessing/pool.py", line 231, in _handle_tasks
>     put(task)
>   File "/usr/lib/python3.1/pickle.py", line 1349, in dumps
>     Pickler(f, protocol, fix_imports=fix_imports).dump(obj)
> _pickle.PicklingError: Can't pickle <class 'method'>: attribute lookup
> builtins.method failed
>
> There is no deadlock.  You are hitting this problem:http://stackoverflow.com/questions/1816958/cant-pickle-type-instancem...
>
> --
> regards,
> kushal


You're right. I just realized that I was running the program through
my Vim plug-in which would use the subprocess module to open the
process, wait for its termination and get the output. It seems that
multi-process program would act quite differently such that my Vim
plug-in had been halted without getting any error information, so I
was to falsely take it as a deadlock.

The thread in stackoverflow you referenced is great, now I'm trying to
fit the solution in my own program, thanks a lot!

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web