Path: csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!gegeweb.org!de-l.enfer-du-nord.net!feeder2.enfer-du-nord.net!cs.uu.nl!news.stack.nl!newsfeed.xs4all.nl!newsfeed5.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
From: DPalao <dpalao.python@gmail.com>
To: python-list@python.org
Subject: 70% [* SPAM *] Re: multiprocessing.Queue blocks when sending large object
Date: Mon, 5 Dec 2011 19:28:22 +0100
User-Agent: KMail/1.13.7 (Linux/2.6.39-gentoo-r3; KDE/4.6.5; x86_64; ; )
References: <23330_1322593803_pATJ9jRv011464_201111292009.44527.dpalao.python@gmail.com> <jbidcg$4p3$1@dough.gmane.org>
In-Reply-To: <jbidcg$4p3$1@dough.gmane.org>
MIME-Version: 1.0
Content-Type: Text/Plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Cc: Lie Ryan <lie.1296@gmail.com>
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.3314.1323109714.27778.python-list@python.org>
Lines: 75
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: x330-a1.tempe.blueboxinc.net comp.lang.python:16677

Hi Lie,
Thank you for the reply.

El Lunes Diciembre 5 2011, Lie Ryan escribi=F3:
> On 11/30/2011 06:09 AM, DPalao wrote:
> > Hello,
> > I'm trying to use multiprocessing to parallelize a code. There is a
> > number of tasks (usually 12) that can be run independently. Each task
> > produces a numpy array, and at the end, those arrays must be combined.
> > I implemented this using Queues (multiprocessing.Queue): one for input
> > and another for output.
> > But the code blocks. And it must be related to the size of the item I p=
ut
> > on the Queue: if I put a small array, the code works well; if the array
> > is realistically large (in my case if can vary from 160kB to 1MB), the
> > code blocks apparently forever.
> > I have tried this:
> > http://www.bryceboe.com/2011/01/28/the-python-multiprocessing-queue-and=
=2Dl
> > arge- objects/
> > but it didn't work (especifically I put a None sentinel at the end for
> > each worker).
> >=20
> > Before I change the implementation,
> > is there a way to bypass this problem with  multiprocessing.Queue?
> > Should I post the code (or a sketchy version of it)?
>=20
> Transferring data over multiprocessing.Queue involves copying the whole
> object across an inter-process pipe, so you need to have a reasonably
> large workload in the processes to justify the cost of the copying to
> benefit from running the workload in parallel.
>=20
> You may try to avoid the cost of copying by using shared memory
> (http://docs.python.org/library/multiprocessing.html#sharing-state-betwee=
n-
> processes); you can use Queue for communicating when a new data comes in =
or
> when a task is done, but put the large data in shared memory. Be careful
> not to access the data from multiple processes concurrently.
>=20

Yep, that was my first thought, but the arrays's elements are complex64 (or=
=20
complex in general), and I don't know how to easily convert from=20
multiprocessing.Array to/from numpy.array when the type is complex. Doing t=
hat=20
would require some extra conversions forth and back which make the solution=
=20
not very attractive to me.
I tried with a Manager too, but the array cannot be modified from within th=
e=20
worker processes.
=20
In principle, the array I need to share is expected to be, at most, ~2MB in=
=20
size, and typically should be only <200kB. So, in principle, there is no hu=
ge=20
extra workload. But that could change, and I'd like to be prepared for it, =
so=20
any idea about using an Array or a Manager or another shared memory thing=20
would be great.

> In any case, have you tried a multithreaded solution? numpy is a C
> extension, and I believe it releases the GIL when working, so it
> wouldn't be in your way to achieve parallelism.

That possibility I didn't know. What does exactly break the GIL? The sharin=
g=20
of a numpy array? What if I need to also share some other "standard" python=
=20
data (eg, a dictionary)?

Best regards,

David