Path: csiph.com!usenet.pasdenom.info!aioe.org!news.stack.nl!newsfeed.xs4all.nl!newsfeed1.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
MIME-Version: 1.0
In-Reply-To: <733559EE-CBDC-406A-8018-78CD41E2C445@mac.com>
References: <733559EE-CBDC-406A-8018-78CD41E2C445@mac.com>
From: Oscar Benjamin <oscar.j.benjamin@gmail.com>
Date: Wed, 24 Apr 2013 23:37:28 +0100
Subject: Re: Finding the source of an exception in a python multiprocessing program
To: William Ray Wing <wrw@mac.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Cc: "python-list@python.org" <python-list@python.org>
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.1035.1366843071.3114.python-list@python.org>
Lines: 69
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:44296

On 24 April 2013 20:25, William Ray Wing <wrw@mac.com> wrote:
> I run a bit of python code that monitors my connection to the greater Int=
ernet.  It checks connectivity to the requested target IP addresses, loggin=
g both successes and failures, once every 15 seconds.  I see failures quite=
 regularly, predictably on Sunday nights after midnight when various networ=
ks are undergoing maintenance.  I'm trying to use python's multiprocessing =
library to run multiple copies in parallel to check connectivity to differe=
nt parts of the country (they in no way interact with each other).
>
> On rare occasions (maybe once every couple of months) I get the following=
 exception and traceback:
>
> Traceback (most recent call last):
>   File "./CM_Harness.py", line 12, in <module>
>     Foo =3D pool.map(monitor, targets)    # and hands off two targets
>   File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/m=
ultiprocessing/pool.py", line 227, in map
>     return self.map_async(func, iterable, chunksize).get()
>   File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/m=
ultiprocessing/pool.py", line 528, in get
>     raise self._value
> IndexError: list index out of range
>
> The code where the traceback occurs is:
>
> #!/usr/bin/env python
>
> """ Harness to call multiple parallel copies
>     of the basic monitor program
> """
>
> from multiprocessing import Pool
> from Connection_Monitor import monitor
>
> targets =3D ["8.8.8.8", "www.ncsa.edu"]
> pool =3D Pool(processes=3D2)            # start 2 worker processes
> Foo =3D pool.map(monitor, targets)    # and hands off two targets
>
>
> Line 12, in my code is simply the line that launches the underlying monit=
or code.  I'm assuming that the real error is occurring in the monitor prog=
ram that is being launched, but I'm at a loss as to what to do to get a bet=
ter handle on what's going wrong. Since, as I said, I see failures quite re=
gularly, typically on Sunday nights after midnight when various networks ar=
e undergoing maintenance, I don't _think_ the exception is being triggered =
by that sort of failure.
>
> When I look at the pool module, the error is occurring in get(self, timeo=
ut=3DNone) on the line after the final else:
>
>     def get(self, timeout=3DNone):
>         self.wait(timeout)
>         if not self._ready:
>             raise TimeoutError
>         if self._success:
>             return self._value
>         else:
>             raise self._value
>
>
> Python v 2.7.3, from Python.org, running on Mac OS-X 10.8.3

This looks to me like a bug in multiprocessing but I'm not very
experienced with it. Perhaps it would be good to open an issue on the
tracker. It might not be solvable without an easier way of reproducing
it though.


Oscar