Path: csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!gegeweb.org!de-l.enfer-du-nord.net!feeder1.enfer-du-nord.net!feeder.news-service.com!news2.euro.net!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
MIME-Version: 1.0
Sender: kushal.kumaran@gmail.com
In-Reply-To: <5ccb9d98-f3f8-4f86-b53c-cda4522b40fc@14g2000prv.googlegroups.com>
References: <5ccb9d98-f3f8-4f86-b53c-cda4522b40fc@14g2000prv.googlegroups.com>
From: Kushal Kumaran <kushal.kumaran+python@gmail.com>
Date: Sun, 11 Sep 2011 17:02:18 +0530
Subject: Re: Deadlock problem using multiprocessing
To: =?UTF-8?B?6JOd6Imy5Z+65Zug?= <bluegene8210@gmail.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Cc: python-list@python.org
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.981.1315740760.27778.python-list@python.org>
Lines: 82
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: x330-a1.tempe.blueboxinc.net comp.lang.python:13121

2011/9/11 =E8=93=9D=E8=89=B2=E5=9F=BA=E5=9B=A0 <bluegene8210@gmail.com>:
> This is my first touch on the multiprocessing module, and I admit not
> having a deep understanding of parallel programming, forgive me if
> there's any obvious error. This is my test code:
>
> # deadlock.py
>
> import multiprocessing
>
> class MPTask:
> =C2=A0 =C2=A0 =C2=A0 =C2=A0def __init__(self):
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0self._tseq=3D rang=
e(10) =C2=A0 # task sequence
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0self._pool=3D mult=
iprocessing.Pool(2) =C2=A0 =C2=A0 # process pool
> =C2=A0 =C2=A0 =C2=A0 =C2=A0def _exe(self, num):
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0return num**2
> =C2=A0 =C2=A0 =C2=A0 =C2=A0def run(self):
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0result=3D self._po=
ol.map_async(self._exe, self._tseq)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0return result.get(=
)
>
> result=3D MPTask().run()
> print(result)
>
> And seemingly it creates a deadlock, I have to manually kill the
> processes in the system monitor, yet it would be OK if the _exe()
> function was defined outside the MPTask class:
>
> # no_deadlock.py
> import multiprocessing
>
> def _exe(num):
> =C2=A0 =C2=A0 =C2=A0 =C2=A0return num**2
>
> class MPTask:
> =C2=A0 =C2=A0 =C2=A0 =C2=A0def __init__(self):
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0self._tseq=3D rang=
e(10) =C2=A0 # task sequence
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0self._pool=3D mult=
iprocessing.Pool(2) =C2=A0 =C2=A0 # process pool
> =C2=A0 =C2=A0 =C2=A0 =C2=A0def run(self):
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0result=3D self._po=
ol.map_async(_exe, self._tseq)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0return result.get(=
)
>
> result=3D MPTask().run()
> print(result)
>
> This would give the correct answer without any deadlock. My questions:
>
> 1. Why is this, where did the deadlock come from?
>
> 2. Besides, is there any material I can refer to about how to avoid
> deadlocks when using multiple processes in the program?
>

I get this exception when I run the first program:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.1/threading.py", line 516, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.1/threading.py", line 469, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.1/multiprocessing/pool.py", line 231, in _handle_t=
asks
    put(task)
  File "/usr/lib/python3.1/pickle.py", line 1349, in dumps
    Pickler(f, protocol, fix_imports=3Dfix_imports).dump(obj)
_pickle.PicklingError: Can't pickle <class 'method'>: attribute lookup
builtins.method failed

There is no deadlock.  You are hitting this problem:
http://stackoverflow.com/questions/1816958/cant-pickle-type-instancemethod-=
when-using-pythons-multiprocessing-pool-map

--=20
regards,
kushal