Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #109091

multiprocessing not quite working

Show all headers | View raw

Hi,

I am using this example: 
http://spartanideas.msu.edu/2014/06/20/an-introduction-to-parallel-programming-using-pythons-multiprocessing-module/

I am sending and receiving communication from the worker processes.

Two issues. the join is only getting to the process and waiting.
When I comment out the .join() process the output.get() appends the 
previous process and therefore the returned output keeps getting longer 
and longer after each process returns its output.

hostnames is an array of hostnames.

here is my code from main():

     # Define an output queue
     output = mp.Queue()

     # Setup a list of processes that we want to run
     processes = [mp.Process(target=worker, args=(hostnames[x], output)) 
for x in range(len(hostnames))]

     # Run processes
     for p in processes:
         print "start: {}".format(p)
         p.start()

     time.sleep(6)
     print "processes: {}".format(processes)

     # Exit the completed processes
     '''for p in processes:
         print "join: {}".format(p)
         p.join()'''

     print "got here"
     # Get process results from the output queue
     # results = [output.get() for p in processes]

     io = StringIO()
     count = 0
     for p in processes:
         found_output = output.get()
         print "returned {}".format(p)
         io.write (found_output)
         zipArchive.writestr(hostnames[count] + "." + 
content['sitename'] + '.config.txt', io.getvalue())
         count = count + 1
     io.close()


def worker(hostname, output):
.
.
.
         output.put(template_output)


Cheers

Back to comp.lang.python | Previous | Next | Find similar | Unroll thread

Thread

multiprocessing not quite working Noah <noah-list@enabled.com> - 2016-05-24 15:17 -0700

csiph-web