Path: csiph.com!usenet.pasdenom.info!aioe.org!news.stack.nl!newsfeed.xs4all.nl!newsfeed4.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
From: William Ray Wing <wrw@mac.com>
Content-type: text/plain; charset=us-ascii
Content-transfer-encoding: quoted-printable
Subject: Problem in Multiprocessing module
Date: Fri, 11 Oct 2013 10:53:41 -0400
To: python-list@python.org
MIME-version: 1.0 (Mac OS X Mail 6.6 \(1510\))
Cc: William Ray Wing <wrw@mac.com>
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.1007.1381506855.18130.python-list@python.org>
Lines: 77
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:56690

I'm running into a problem in the multiprocessing module.

My code is running four parallel processes which are doing network =
access completely independently of each other (gathering data from =
different remote sources).  On rare circumstances, the code blows up =
when one of my processes has do start doing some error recovery.  I =
strongly suspect it is because there is a time-out that isn't being =
caught in the multiprocessing lib, and that in turn is exposing the =
TypeError.  Note that the error is "cannot concatenate 'str' and =
'NoneType' objects and it is occurring way down in the multiprocessing =
library.

I'd really appreciate it if someone more knowledgeable about =
multiprocessing could confirm (or refute) my suspicion and then tell me =
how to fix things up.=20

I'm running python 2.7.5 on a Mac OS-X 10.8.5

The traceback I get is:

TypeError: cannot concatenate 'str' and 'NoneType' objects
File =
"/Users/wrw/Dev/Python/Connection_Monitor/Version3.0/CM_Harness.py", =
line 20, in <module>
 my_pool =3D pool.map(monitor, targets)    # and hands off to four =
targets
File =
"/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiproc=
essing/pool.py", line 250, in map
 return self.map_async(func, iterable, chunksize).get()
File =
"/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiproc=
essing/pool.py", line 554, in get
 raise self._value

To save you-all some time:

The "get" function at line 554 in pool.py (which is in the =
multiprocessing lib) is:

   def get(self, timeout=3DNone):
       self.wait(timeout)
       if not self._ready:
           raise TimeoutError
       if self._success:
           return self._value
       else:
           raise self._value

And the map function (also in pool in the multiprocessing lib) is:

   def map(self, func, iterable, chunksize=3DNone):
       '''
       Equivalent of `map()` builtin
       '''
       assert self._state =3D=3D RUN
       return self.map_async(func, iterable, chunksize).get()

Finally, my code that calls all this is pretty simple (note that the =
targets are dummies here):

#!/usr/bin/env python

""" Harness to call multiple parallel copies
   of the basic monitor program
"""

targets =3D ["www.sdsc.edu", "www.ncsa.edu", "www.uiuc.edu", =
"www.berkeley.edu"]

pool =3D Pool(processes=3D4)                # start 4 worker processes
my_pool =3D pool.map(monitor, targets)    # and hands off to four =
targets

TiA
Bill