Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #109091 > unrolled thread

multiprocessing not quite working

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  multiprocessing not quite working Noah <noah-list@enabled.com> - 2016-05-24 15:17 -0700

#109091 — multiprocessing not quite working

Hi,

I am using this example: 
http://spartanideas.msu.edu/2014/06/20/an-introduction-to-parallel-programming-using-pythons-multiprocessing-module/

I am sending and receiving communication from the worker processes.

Two issues. the join is only getting to the process and waiting.
When I comment out the .join() process the output.get() appends the 
previous process and therefore the returned output keeps getting longer 
and longer after each process returns its output.

hostnames is an array of hostnames.

here is my code from main():

     # Define an output queue
     output = mp.Queue()

     # Setup a list of processes that we want to run
     processes = [mp.Process(target=worker, args=(hostnames[x], output)) 
for x in range(len(hostnames))]

     # Run processes
     for p in processes:
         print "start: {}".format(p)
         p.start()

     time.sleep(6)
     print "processes: {}".format(processes)

     # Exit the completed processes
     '''for p in processes:
         print "join: {}".format(p)
         p.join()'''

     print "got here"
     # Get process results from the output queue
     # results = [output.get() for p in processes]

     io = StringIO()
     count = 0
     for p in processes:
         found_output = output.get()
         print "returned {}".format(p)
         io.write (found_output)
         zipArchive.writestr(hostnames[count] + "." + 
content['sitename'] + '.config.txt', io.getvalue())
         count = count + 1
     io.close()


def worker(hostname, output):
.
.
.
         output.put(template_output)


Cheers

[toc] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web