Path: csiph.com!fu-berlin.de!uni-berlin.de!not-for-mail
From: MRAB <python@mrabarnett.plus.com>
Newsgroups: comp.lang.python
Subject: Re: multiprocessing not quite working
Date: Wed, 25 May 2016 00:39:38 +0100
Lines: 83
Message-ID: <mailman.71.1464133185.20402.python-list@python.org>
References: <f803eff5-786e-703f-4625-69f5e64c6a6e@enabled.com> <5ec4b860-e38d-55f4-1d54-d1f944bfeb33@mrabarnett.plus.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.1.0
In-Reply-To: <f803eff5-786e-703f-4625-69f5e64c6a6e@enabled.com>
Precedence: list
Xref: csiph.com comp.lang.python:109093

On 2016-05-24 23:17, Noah wrote:
>
> Hi,
>
> I am using this example:
> http://spartanideas.msu.edu/2014/06/20/an-introduction-to-parallel-programming-using-pythons-multiprocessing-module/
>
> I am sending and receiving communication from the worker processes.
>
> Two issues. the join is only getting to the process and waiting.
> When I comment out the .join() process the output.get() appends the
> previous process and therefore the returned output keeps getting longer
> and longer after each process returns its output.
>
> hostnames is an array of hostnames.
>
> here is my code from main():
>
>      # Define an output queue
>      output = mp.Queue()
>
>      # Setup a list of processes that we want to run
>      processes = [mp.Process(target=worker, args=(hostnames[x], output))
> for x in range(len(hostnames))]
>
>      # Run processes
>      for p in processes:
>          print "start: {}".format(p)
>          p.start()
>
>      time.sleep(6)
>      print "processes: {}".format(processes)
>
>      # Exit the completed processes
>      '''for p in processes:
>          print "join: {}".format(p)
>          p.join()'''
>
>      print "got here"
>      # Get process results from the output queue
>      # results = [output.get() for p in processes]
>
>      io = StringIO()
>      count = 0
>      for p in processes:
>          found_output = output.get()
>          print "returned {}".format(p)
>          io.write (found_output)
>          zipArchive.writestr(hostnames[count] + "." +
> content['sitename'] + '.config.txt', io.getvalue())
>          count = count + 1
>      io.close()
>
>
> def worker(hostname, output):
> .
> .
> .
>          output.put(template_output)
>
>
It says in the docs (Python 2.7, 16.6.2.2. Pipes and Queues):

"""Warning
As mentioned above, if a child process has put items on a queue (and it 
has not used JoinableQueue.cancel_join_thread), then that process will 
not terminate until all buffered items have been flushed to the pipe.

This means that *if you try joining that process you may get a deadlock 
unless you are sure that all items which have been put on the queue have 
been consumed*. Similarly, if the child process is non-daemonic then the 
parent process may hang on exit when it tries to join all its 
non-daemonic children.

Note that a queue created using a manager does not have this issue. See 
Programming guidelines.
"""

(Asterisks added to highlight.)

You should try to consume the output from a process before trying to 
join it (or, at least, join it without a timeout).