Path: csiph.com!fu-berlin.de!uni-berlin.de!not-for-mail
From: Israel Brewster <israel@ravnalaska.net>
Newsgroups: comp.lang.python
Subject: Re: Bi-directional sub-process communication
Date: Mon, 23 Nov 2015 13:18:58 -0900
Lines: 58
Message-ID: <mailman.78.1448317143.2291.python-list@python.org>
References: <5A3A00A6-CFD2-42FC-9C87-6F8C6861F775@ravnalaska.net> <20151123214516.GA56871@cskk.homeip.net>
Mime-Version: 1.0 (Mac OS X Mail 9.1 \(3096.5\))
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
In-Reply-To: <20151123214516.GA56871@cskk.homeip.net>
Precedence: list
Xref: csiph.com comp.lang.python:99285

On Nov 23, 2015, at 12:45 PM, Cameron Simpson <cs@zip.com.au> wrote:
>=20
> On 23Nov2015 12:22, Israel Brewster <israel@ravnalaska.net> wrote:
>> On Nov 23, 2015, at 11:51 AM, Ian Kelly <ian.g.kelly@gmail.com> =
wrote:
>>> Concurrency, ugh.
>=20
> I'm a big concurrency fan myself.
>=20
>>> It's probably better just to have a Condition/Event per thread and
>>> have the response thread identify the correct one to notify, rather
>>> than just notify a single shared Condition and hope the threads wake
>>> up in the right order.
>>=20
>> Tell me about it :-) I've actually never worked with conditions or =
notifications (actually even this bi-drectional type of communication is =
new to me), so I'll have to look into that and figure it out. Thanks for =
the information!
>=20
> I include a tag with every request, and have the responses include the =
tag; the request submission function records the response hander in a =
mapping by tag and the response handing thread looks up the mapping and =
passes the response to the right handler.
>=20
> Works just fine and avoids all the worrying about ordering etc.
>=20
> Israel, do you have control over the protocol between you and your =
subprocess?  If so, adding tags is easy and effective.

I do, and the basic concept makes sense. The one difficulty I am seeing =
is getting back to the thread that requested the data.  Let me know if =
this makes sense or I am thinking about it wrong:

- When a thread requests some data, it sends the request as a dictionary =
containing a tag (unique to the thread) as well as the request
- When the child processes the request, it encodes the response as a =
dictionary containing the tag and the response data
- A single, separate thread on the "master" side parses out responses as =
they come in and puts them into a dictionary keyed by tag
- The requesting threads, after putting the request into the Queue, =
would then block waiting for data to appear under their key in the =
dictionary

Of course, that last step could be interesting - implementing the block =
in such a way as to not tie up the processor, while still getting the =
data "as soon" as it is available. Unless there is some sort of built-in =
notification system I could use for that? I.e. the thread would =
"subscribe" to a notification based on its tag, and then wait for =
notification. When the master processing thread receives data with said =
tag, it adds it to the dictionary and "publishes" a notification to that =
tag. Or perhaps the notification itself could contain the payload?=20

Thanks for the information!

>=20
> Cheers,
> Cameron Simpson <cs@zip.com.au>