Path: csiph.com!fu-berlin.de!uni-berlin.de!not-for-mail
From: Israel Brewster <israel@ravnalaska.net>
Newsgroups: comp.lang.python
Subject: Bi-directional sub-process communication
Date: Mon, 23 Nov 2015 08:54:38 -0900
Lines: 91
Message-ID: <mailman.73.1448305462.2291.python-list@python.org>
Mime-Version: 1.0 (Mac OS X Mail 9.1 \(3096.5\))
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Precedence: list
Xref: csiph.com comp.lang.python:99279

I have a multi-threaded python app (CherryPy WebApp to be exact) that =
launches a child process that it then needs to communicate with =
bi-driectionally. To implement this, I have used a pair of Queues: a =
child_queue which I use for master->child communication, and a =
master_queue which is used for child->master communication.

The way I have the system set up, the child queue runs a loop in a tread =
that waits for messages on child_queue, and when received responds =
appropriately depending on the message received, which sometimes =
involves posting a message to master_queue.

On the master side, when it needs to communicate with the child process, =
it posts a message to child_queue, and if the request requires a =
response it will then immediately start waiting for a message on =
master_queue, typically with a timeout.

While this process works well in testing, I do have one concern (maybe =
unfounded) and a real-world issue

Concern: Since the master process is multi-threaded, it seems likely =
enough that multiple threads on the master side would make requests at =
the same time. I understand that the Queue class has locks that make =
this fine (one thread will complete posting the message before the next =
is allowed to start), and since the child process only has a single =
thread processing messages from the queue, it should process them in =
order and post the responses (if any) to the master_queue in order. But =
now I have multiple master processes all trying to read master_queue at =
the same time. Again, the locks will take care of this and prevent any =
overlapping reads, but am I guaranteed that the threads will obtain the =
lock and therefore read the responses in the right order? Or is there a =
possibility that, say, thread three will get the response that should =
have been for thread one? Is this something I need to take into =
consideration, and if so, how?

Real-world problem: While as I said this system worked well in testing, =
Now that I have gotten it out into production I've occasionally run into =
a problem where the master thread waiting for a response on master_queue =
times out while waiting. This causes a (potentially) two-fold problem, =
in that first off the master process doesn't get the information it had =
requested, and secondly that I *could* end up with an "orphaned" message =
on the queue that could cause problems the next time I try to read =
something from it.

I currently have the timeout set to 3 seconds. I can, of course, =
increase that, but that could lead to a bad user experience - and might =
not even help the situation if something else is going on. The actual =
exchange is quite simple:

On the master side, I have this code:

config.socket_queue.put('GET_PORT')
try:
    port =3D config.master_queue.get(timeout=3D3)  #wait up to three =
seconds for a response
except Empty:
     port =3D 5000  # default. Can't hurt to try.

Which, as you might have been able to guess, tries to ask the child =
process (an instance of a tornado server, btw) what port it is listening =
on. The child process then, on getting this message from the queue, runs =
the following code:

elif item =3D=3D 'GET_PORT':
      port =3D utils.config.getint('global', 'tornado.port')
      master_queue.put(port)

So nothing that should take any significant time. Of course, since this =
is a single thread handling any number of requests, it is possible that =
the thread is tied up responding to a different request (or that the GIL =
is preventing the thread from running at all, since another thread might =
be commandeering the processor), but I find it hard to believe that it =
could be tied up for more than three seconds.

So is there a better way to do sub-process bi-directional communication =
that would avoid these issues? Or do I just need to increase the timeout =
(or remove it altogether, at the risk of potentially causing the thread =
to hang if no message is posted)? And is my concern justified, or just =
paranoid? Thanks for any information that can be provided!

-----------------------------------------------
Israel Brewster
Systems Analyst II
Ravn Alaska
5245 Airport Industrial Rd
Fairbanks, AK 99709
(907) 450-7293
-----------------------------------------------