Path: csiph.com!usenet.pasdenom.info!gegeweb.org!de-l.enfer-du-nord.net!feeder1.enfer-du-nord.net!border1.nntp.ams2.giganews.com!border3.nntp.ams.giganews.com!nntp.giganews.com!feed.xsnews.nl!border-3.ams.xsnews.nl!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Newsgroups: comp.lang.python
Date: Wed, 22 Aug 2012 12:40:25 -0700 (PDT)
In-Reply-To: <mailman.3673.1345657626.4697.python-list@python.org>
Complaints-To: groups-abuse@google.com
Injection-Info: glegroupsg2000goo.googlegroups.com; posting-host=93.207.121.228; posting-account=_b6A3QoAAABPwmSQu8QIyLK7idqo6rsL
References: <db59479c-198f-468d-9e73-40b1b992895c@googlegroups.com> <mailman.3673.1345657626.4697.python-list@python.org>
User-Agent: G2/1.0
MIME-Version: 1.0
Subject: Re: How to properly implement worker processes
From: Dennis Jacobfeuerborn <djacobfeuerborn@gmail.com>
To: comp.lang.python@googlegroups.com
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Cc: python-list@python.org, Dennis Jacobfeuerborn <djacobfeuerborn@gmail.com>
Precedence: list
Message-ID: <mailman.3684.1345664434.4697.python-list@python.org>
Lines: 77
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:27679

On Wednesday, August 22, 2012 7:46:34 PM UTC+2, Ian wrote:
> On Wed, Aug 22, 2012 at 11:29 AM, Dennis Jacobfeuerborn
>=20
> <djacobfeuerborn@gmail.com> wrote:
>=20
> > Hi,
>=20
> > I'm trying to implement a system for periodically checking URLs and I'v=
e run into problems with some of the implementation details. The URLs are s=
upposed to be checked continuously until the config for an URL is explicitl=
y removed.
>=20
> >
>=20
> > The plan is to spawn a worker process for each URL that sends the statu=
s of the last check to its parent which keeps track of the state of all URL=
s. When a URL is no longer supposed to be checked the parent process should=
 shutdown/kill the respective worker process.
>=20
> >
>=20
> > What I've been going for so far is that the parent process creates a gl=
obal queue that is passed to all children upon creation which they use to s=
end status messages to the parent. Then for each process a dedicated queue =
is created that the parent uses to issue commands to the child.
>=20
> >
>=20
> > The issue is that since the child processes spent some time in sleep() =
when a command from the parent comes they cannot respond immediately which =
is rather undesirable. What I would rather like to do is have the parent si=
mply kill the child instead which is instantaneous and more reliable.
>=20
> >
>=20
> > My problem is that according to the multiprocessing docs if I kill the =
child while it uses the queue to send a status to the parent then the queue=
 becomes corrupted and since that queue is shared that means the whole thin=
g pretty much stops working.
>=20
> >
>=20
> > How can I get around this problem and receive status updates from all c=
hildren efficiently without a shared queue and with the ability to simply k=
ill the child process when it's no longer needed?
>=20
>=20
>=20
> The usual approach to killing worker processes safely is to send them
>=20
> an "exit" command, which they should respond to by terminating
>=20
> cleanly.  Instead of using sleep(), have the workers do a blocking
>=20
> get() on the queue with a timeout.  This way they'll receive the
>=20
> "exit" message immediately as desired, but they'll still wake up at
>=20
> the desired intervals in order to do their work.

I was thinking about something like that but the issue is that this really =
only works when you don't do any actual blocking work. I may be able to get=
 around the sleep() but then I have to fetch the URL or do some other work =
that might block for a while so the get() trick doesn't work.
Also the child process might not be able to deal with such an exit command =
at all for one reason or another so the only safe way to get rid of it is f=
or the parent to kill it.

The better option would be to not use a shared queue for communication and =
instead use only dedicated pipes/queues for each child process but the does=
n't seem to be a way to wait for a message from multiple queues/pipes. If t=
hat were the case then I could simply kill the child and get rid of the res=
pective pipes/queues without affecting the other processes or communication=
 channels.

Regards,
  Dennis