Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder4.news.weretis.net!news.mixmin.net!feed.xsnews.nl!border-1.ams.xsnews.nl!newsfeed.xs4all.nl!newsfeed5.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
MIME-Version: 1.0
In-Reply-To: <db59479c-198f-468d-9e73-40b1b992895c@googlegroups.com>
References: <db59479c-198f-468d-9e73-40b1b992895c@googlegroups.com>
From: Ian Kelly <ian.g.kelly@gmail.com>
Date: Wed, 22 Aug 2012 11:46:34 -0600
Subject: Re: How to properly implement worker processes
To: Dennis Jacobfeuerborn <djacobfeuerborn@gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Cc: python-list@python.org
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.3673.1345657626.4697.python-list@python.org>
Lines: 38
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:27665

On Wed, Aug 22, 2012 at 11:29 AM, Dennis Jacobfeuerborn
<djacobfeuerborn@gmail.com> wrote:
> Hi,
> I'm trying to implement a system for periodically checking URLs and I've =
run into problems with some of the implementation details. The URLs are sup=
posed to be checked continuously until the config for an URL is explicitly =
removed.
>
> The plan is to spawn a worker process for each URL that sends the status =
of the last check to its parent which keeps track of the state of all URLs.=
 When a URL is no longer supposed to be checked the parent process should s=
hutdown/kill the respective worker process.
>
> What I've been going for so far is that the parent process creates a glob=
al queue that is passed to all children upon creation which they use to sen=
d status messages to the parent. Then for each process a dedicated queue is=
 created that the parent uses to issue commands to the child.
>
> The issue is that since the child processes spent some time in sleep() wh=
en a command from the parent comes they cannot respond immediately which is=
 rather undesirable. What I would rather like to do is have the parent simp=
ly kill the child instead which is instantaneous and more reliable.
>
> My problem is that according to the multiprocessing docs if I kill the ch=
ild while it uses the queue to send a status to the parent then the queue b=
ecomes corrupted and since that queue is shared that means the whole thing =
pretty much stops working.
>
> How can I get around this problem and receive status updates from all chi=
ldren efficiently without a shared queue and with the ability to simply kil=
l the child process when it's no longer needed?

The usual approach to killing worker processes safely is to send them
an "exit" command, which they should respond to by terminating
cleanly.  Instead of using sleep(), have the workers do a blocking
get() on the queue with a timeout.  This way they'll receive the
"exit" message immediately as desired, but they'll still wake up at
the desired intervals in order to do their work.