Path: csiph.com!fu-berlin.de!uni-berlin.de!not-for-mail
From: Israel Brewster <israel@ravnalaska.net>
Newsgroups: comp.lang.python
Subject: Re: Bi-directional sub-process communication
Date: Mon, 23 Nov 2015 12:22:33 -0900
Lines: 81
Message-ID: <mailman.76.1448313758.2291.python-list@python.org>
References: <B819AD72-C0F9-4E6C-992A-5E5556979883@ravnalaska.net> <CALwzidkz6LdFeXsRZW1sAYxXvJGRX5KA5XsM4T0dEOb34C9HDQ@mail.gmail.com> <CALwzidkFoar6mtiH8xhHPta=HbZdKpaiMPxjH2HegXHAS1jLAw@mail.gmail.com>
Mime-Version: 1.0 (Mac OS X Mail 9.1 \(3096.5\))
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
In-Reply-To: <CALwzidkFoar6mtiH8xhHPta=HbZdKpaiMPxjH2HegXHAS1jLAw@mail.gmail.com>
Precedence: list
Xref: csiph.com comp.lang.python:99283

On Nov 23, 2015, at 11:51 AM, Ian Kelly <ian.g.kelly@gmail.com> wrote:
>=20
> On Mon, Nov 23, 2015 at 12:55 PM, Ian Kelly <ian.g.kelly@gmail.com> =
wrote:
>> On Mon, Nov 23, 2015 at 10:54 AM, Israel Brewster =
<israel@ravnalaska.net> wrote:
>>> Concern: Since the master process is multi-threaded, it seems likely =
enough that multiple threads on the master side would make requests at =
the same time. I understand that the Queue class has locks that make =
this fine (one thread will complete posting the message before the next =
is allowed to start), and since the child process only has a single =
thread processing messages from the queue, it should process them in =
order and post the responses (if any) to the master_queue in order. But =
now I have multiple master processes all trying to read master_queue at =
the same time. Again, the locks will take care of this and prevent any =
overlapping reads, but am I guaranteed that the threads will obtain the =
lock and therefore read the responses in the right order? Or is there a =
possibility that, say, thread three will get the response that should =
have been for thread one? Is this something I need to take into =
consideration, and if so, how?
>>=20
>> Yes, if multiple master threads are waiting on the queue, it's
>> possible that a master thread could get a response that was not
>> intended for it. As far as I know there's no guarantee that the
>> waiting threads will be woken up in the order that they called get(),
>> but even if there are, consider this case:
>>=20
>> Thread A enqueues a request.
>> Thread B preempts A and enqueues a request.
>> Thread B calls get on the response queue.
>> Thread A calls get on the response queue.
>> The response from A's request arrives and is given to B.
>>=20
>> Instead of having the master threads pull objects off the response
>> queue directly, you might create another thread whose sole purpose is
>> to handle the response queue. That could look like this:
>>=20
>>=20
>> request_condition =3D threading.Condition()
>> response_global =3D None
>>=20
>> def master_thread():
>>    global response_global
>>    with request_condition:
>>        request_queue.put(request)
>>        request_condition.wait()
>>        # Note: the Condition should remain acquired until
>> response_global is reset.
>>        response =3D response_global
>>        response_global =3D None
>>    if wrong_response(response):
>>        raise RuntimeError("got a response for the wrong request")
>>    handle_response(response)
>>=20
>> def response_thread():
>>    global response_global
>>    while True:
>>        response =3D response_queue.get()
>>        with request_condition:
>>            response_global =3D response
>>            request_condition.notify()
>=20
> Actually I realized that this fails because if two threads get
> notified at about the same time, they could reacquire the Condition in
> the wrong order and so get the wrong responses.
>=20
> Concurrency, ugh.
>=20
> It's probably better just to have a Condition/Event per thread and
> have the response thread identify the correct one to notify, rather
> than just notify a single shared Condition and hope the threads wake
> up in the right order.

Tell me about it :-) I've actually never worked with conditions or =
notifications (actually even this bi-drectional type of communication is =
new to me), so I'll have to look into that and figure it out. Thanks for =
the information!

> --=20
> https://mail.python.org/mailman/listinfo/python-list