Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #56690 > unrolled thread

Problem in Multiprocessing module

Back to article view | Back to comp.lang.python

Contents

  Problem in Multiprocessing module William Ray Wing <wrw@mac.com> - 2013-10-11 10:53 -0400

#56690 — Problem in Multiprocessing module

I'm running into a problem in the multiprocessing module.

My code is running four parallel processes which are doing network access completely independently of each other (gathering data from different remote sources).  On rare circumstances, the code blows up when one of my processes has do start doing some error recovery.  I strongly suspect it is because there is a time-out that isn't being caught in the multiprocessing lib, and that in turn is exposing the TypeError.  Note that the error is "cannot concatenate 'str' and 'NoneType' objects and it is occurring way down in the multiprocessing library.

I'd really appreciate it if someone more knowledgeable about multiprocessing could confirm (or refute) my suspicion and then tell me how to fix things up. 

I'm running python 2.7.5 on a Mac OS-X 10.8.5

The traceback I get is:

TypeError: cannot concatenate 'str' and 'NoneType' objects
File "/Users/wrw/Dev/Python/Connection_Monitor/Version3.0/CM_Harness.py", line 20, in <module>
 my_pool = pool.map(monitor, targets)    # and hands off to four targets
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 250, in map
 return self.map_async(func, iterable, chunksize).get()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 554, in get
 raise self._value

To save you-all some time:

The "get" function at line 554 in pool.py (which is in the multiprocessing lib) is:

   def get(self, timeout=None):
       self.wait(timeout)
       if not self._ready:
           raise TimeoutError
       if self._success:
           return self._value
       else:
           raise self._value

And the map function (also in pool in the multiprocessing lib) is:

   def map(self, func, iterable, chunksize=None):
       '''
       Equivalent of `map()` builtin
       '''
       assert self._state == RUN
       return self.map_async(func, iterable, chunksize).get()

Finally, my code that calls all this is pretty simple (note that the targets are dummies here):

#!/usr/bin/env python

""" Harness to call multiple parallel copies
   of the basic monitor program
"""

targets = ["www.sdsc.edu", "www.ncsa.edu", "www.uiuc.edu", "www.berkeley.edu"]

pool = Pool(processes=4)                # start 4 worker processes
my_pool = pool.map(monitor, targets)    # and hands off to four targets

TiA
Bill

[toc] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web