Path: csiph.com!usenet.pasdenom.info!aioe.org!news.stack.nl!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
MIME-Version: 1.0
In-Reply-To: <f1de8e1a-d836-4741-8c06-2e6adabbeed5@googlegroups.com>
References: <db59479c-198f-468d-9e73-40b1b992895c@googlegroups.com> <mailman.3673.1345657626.4697.python-list@python.org> <f1de8e1a-d836-4741-8c06-2e6adabbeed5@googlegroups.com>
From: Ian Kelly <ian.g.kelly@gmail.com>
Date: Wed, 22 Aug 2012 15:15:10 -0600
Subject: Re: How to properly implement worker processes
To: Dennis Jacobfeuerborn <djacobfeuerborn@gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Cc: python-list@python.org, comp.lang.python@googlegroups.com
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.3688.1345670143.4697.python-list@python.org>
Lines: 40
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:27685

On Wed, Aug 22, 2012 at 1:40 PM, Dennis Jacobfeuerborn
<djacobfeuerborn@gmail.com> wrote:
> I was thinking about something like that but the issue is that this reall=
y only works when you don't do any actual blocking work. I may be able to g=
et around the sleep() but then I have to fetch the URL or do some other wor=
k that might block for a while so the get() trick doesn't work.

At a lower level, it is possible to poll on both the pipe and the
socket simultaneously.  At this point though you might want to start
looking at an asynchronous or event-driven framework like twisted or
gevent.

> Also the child process might not be able to deal with such an exit comman=
d at all for one reason or another so the only safe way to get rid of it is=
 for the parent to kill it.

I think you mean that it is the most "reliable" way.  In general, the
only "safe" way to cause a process to exit is the cooperative
approach, because it may otherwise leave external resources such as
file data in an unexpected state that could cause problems later.

> The better option would be to not use a shared queue for communication an=
d instead use only dedicated pipes/queues for each child process but the do=
esn't seem to be a way to wait for a message from multiple queues/pipes. If=
 that were the case then I could simply kill the child and get rid of the r=
espective pipes/queues without affecting the other processes or communicati=
on channels.

Assuming that you're using a Unix system:

from select import select

while True:
    ready, _, _ =3D select(pipes, [], [], timeout)
    if not ready:
        # process timeout
    else:
        for pipe in ready:
            message =3D pipe.get()
            # process message