Path: csiph.com!newsfeed.hal-mli.net!feeder3.hal-mli.net!newsfeed.hal-mli.net!feeder1.hal-mli.net!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Newsgroups: comp.lang.python
Date: Wed, 22 Aug 2012 19:28:23 -0700 (PDT)
In-Reply-To: <mailman.3688.1345670143.4697.python-list@python.org>
Complaints-To: groups-abuse@google.com
Injection-Info: glegroupsg2000goo.googlegroups.com; posting-host=93.228.171.93; posting-account=_b6A3QoAAABPwmSQu8QIyLK7idqo6rsL
References: <db59479c-198f-468d-9e73-40b1b992895c@googlegroups.com> <mailman.3673.1345657626.4697.python-list@python.org> <f1de8e1a-d836-4741-8c06-2e6adabbeed5@googlegroups.com> <mailman.3688.1345670143.4697.python-list@python.org>
User-Agent: G2/1.0
MIME-Version: 1.0
Subject: Re: How to properly implement worker processes
From: Dennis Jacobfeuerborn <djacobfeuerborn@gmail.com>
To: comp.lang.python@googlegroups.com
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Cc: python-list@python.org, Dennis Jacobfeuerborn <djacobfeuerborn@gmail.com>
Precedence: list
Message-ID: <mailman.3692.1345688905.4697.python-list@python.org>
Lines: 90
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:27692

On Wednesday, August 22, 2012 11:15:10 PM UTC+2, Ian wrote:
> On Wed, Aug 22, 2012 at 1:40 PM, Dennis Jacobfeuerborn
>=20
> <djacobfeuerborn@gmail.com> wrote:
>=20
> > I was thinking about something like that but the issue is that this rea=
lly only works when you don't do any actual blocking work. I may be able to=
 get around the sleep() but then I have to fetch the URL or do some other w=
ork that might block for a while so the get() trick doesn't work.
>=20
>=20
>=20
> At a lower level, it is possible to poll on both the pipe and the
>=20
> socket simultaneously.  At this point though you might want to start
>=20
> looking at an asynchronous or event-driven framework like twisted or
>=20
> gevent.
>=20

I was looking at twisted and while the Agent class would allow me to make a=
sync request it doesn't seem to support setting a timeout or aborting the r=
unning request. That's really the important part since the http request is =
really the only thing that might block for a while. If I can make the reque=
st asynchronously and abort it when I receive a QUIT command from the paren=
t then this would pretty much solve the issue.

>=20
> > Also the child process might not be able to deal with such an exit comm=
and at all for one reason or another so the only safe way to get rid of it =
is for the parent to kill it.
>=20
>=20
>=20
> I think you mean that it is the most "reliable" way.  In general, the
>=20
> only "safe" way to cause a process to exit is the cooperative
>=20
> approach, because it may otherwise leave external resources such as
>=20
> file data in an unexpected state that could cause problems later.
>=20

True but the child is doing nothing but making http requests and reporting =
the result to the parent so killing the process shouldn't be too much of a =
deal in this case. A segfault in an Apache worker process is very similar i=
n that it's an uncontrolled termination of the process and that works out f=
ine.

>=20
> > The better option would be to not use a shared queue for communication =
and instead use only dedicated pipes/queues for each child process but the =
doesn't seem to be a way to wait for a message from multiple queues/pipes. =
If that were the case then I could simply kill the child and get rid of the=
 respective pipes/queues without affecting the other processes or communica=
tion channels.
>=20
>=20
>=20
> Assuming that you're using a Unix system:
>=20
>=20
>=20
> from select import select
>=20
>=20
>=20
> while True:
>=20
>     ready, _, _ =3D select(pipes, [], [], timeout)
>=20
>     if not ready:
>=20
>         # process timeout
>=20
>     else:
>=20
>         for pipe in ready:
>=20
>             message =3D pipe.get()
>=20
>             # process message

That looks like a workable solution. When I decide to kill a worker process=
 I can remove the pipe from the pipes list and discard it since it's not sh=
ared.

Regards,
  Dennis