Path: csiph.com!news.swapon.de!fu-berlin.de!uni-berlin.de!not-for-mail
From: Israel Brewster <israel@ravnalaska.net>
Newsgroups: comp.lang.python
Subject: Re: Bi-directional sub-process communication
Date: Mon, 23 Nov 2015 14:14:11 -0900
Lines: 102
Message-ID: <mailman.80.1448320455.2291.python-list@python.org>
References: <5A3A00A6-CFD2-42FC-9C87-6F8C6861F775@ravnalaska.net> <20151123214516.GA56871@cskk.homeip.net> <F76E321F-3DE0-4FF7-90B4-5EA8A1017C71@ravnalaska.net> <CALvWhxtJkWk4_+boowdYSzNex32ckKgaGBa2zhJsrsag7wjkmQ@mail.gmail.com>
Mime-Version: 1.0 (Mac OS X Mail 9.1 \(3096.5\))
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
In-Reply-To: <CALvWhxtJkWk4_+boowdYSzNex32ckKgaGBa2zhJsrsag7wjkmQ@mail.gmail.com>
Precedence: list
Xref: csiph.com comp.lang.python:99287

On Nov 23, 2015, at 1:43 PM, Chris Kaynor <ckaynor@zindagigames.com> =
wrote:
>=20
> On Mon, Nov 23, 2015 at 2:18 PM, Israel Brewster =
<israel@ravnalaska.net>
> wrote:
>=20
>> Of course, that last step could be interesting - implementing the =
block in
>> such a way as to not tie up the processor, while still getting the =
data "as
>> soon" as it is available. Unless there is some sort of built-in
>> notification system I could use for that? I.e. the thread would =
"subscribe"
>> to a notification based on its tag, and then wait for notification. =
When
>> the master processing thread receives data with said tag, it adds it =
to the
>> dictionary and "publishes" a notification to that tag. Or perhaps the
>> notification itself could contain the payload?
>=20
>=20
> There are a few ways I could see handling this, without having the =
threads
> spinning and consuming CPU:
>=20
>   1. Don't worry about having the follow-up code run in the same =
thread,
>   and use a simple callback. This callback could be dispatched to a =
thread
>   via a work queue, however you may not get the same thread as the one =
that
>   made the request. This is probably the most efficient method to use, =
as the
>   threads can continue doing other work while waiting for a reply, =
rather
>   than blocking. It does make it harder to maintain state between the =
pre-
>   and post-request functions, however.
>   2. Have a single, global, event variable that wakes all threads =
waiting
>   on a reply, each of which then checks to see if the reply is for it, =
or
>   goes back to sleep. This is good if most of the time, only a few =
threads
>   will be waiting for a reply, and checking if the correct reply came =
in is
>   cheap. This is probably good enough, unless you have a LOT of =
threads
>   (hundreds).
>   3. Have an event per thread. This will use less CPU than the second
>   option, however does require more memory and OS resources, and so =
will not
>   be viable for huge numbers of threads, though if you hit the limit, =
you are
>   probably using threads wrong.
>   4. Have an event per request. This is only better than #3 if a =
single
>   thread may make multiple requests at once, and can do useful work =
when any
>   of them get a reply back (if they need all, it will make no =
difference).
>=20
> Generally, I would use option #1 or #2. Option 2 has the advantage of
> making it easy to write the functions that use the functionality, =
while
> option 1 will generally use fewer resources, and allows threads to =
continue
> to be used while waiting for replies. How much of a benefit that is =
depends
> on exactly what you are doing.

While I would agree with #1 in general, the threads, in this case, are =
CherryPy threads, so I need to get the data and return it to the client =
in the same function call, which of course means the thread needs to =
block until the data is ready - it can't return and let the result be =
processed "later".

Essentially there are times that the web client needs some information =
that only the Child process has. So the web client requests the data =
from the master process, and the master process then turns around and =
requests the data from the child, but it needs to get the data back =
before it can return it to the web client. So it has to block waiting =
for the data.

Thus we come to option #2 (or 3), which sounds good but I have no clue =
how to implement :-) Maybe something like http://pubsub.sourceforge.net =
? I'll dig into that.

>=20
> Option #4 would probably be better implemented using option #1 in all =
cases
> to avoid problems with running out of OS memory - threading features
> generally require more limited OS resources than memory. Option #3 =
will
> also often run into the same issues as option #4 in the cases it will
> provide any benefit over option #2.
>=20
> Chris
> --=20
> https://mail.python.org/mailman/listinfo/python-list