Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #7399

parallel computations: subprocess.Popen(...).communicate()[0] does not work with multiprocessing.Pool

Show key headers only | View raw

Hi,
I am having an issue when making a shell call from within a
multiprocessing.Process().  Here is the story: i tried to parallelize
the computations in 800-ish Matlab scripts and then save the results
to MySQL.   The non-parallel/serial version has been running fine for
about 2 years.  However, in the parallel version via multiprocessing
that i'm working on, it appears that the Matlab scripts have never
been kicked off and nothing happened with subprocess.Popen.  The debug
printing below does not show up either.

Moreover, even if i replace the Matlab invocation with some trivial
"sed" call, still nothing happens.

Is it possible that the Python interpreter i'm using (version 2.6
released on Oct. 1, 2008) is too old?   Nevertheless, i would like to
make sure the basic framework i've now is not blatantly wrong.

Below is a skeleton of my Python program:

----------------------------------------------
import subprocess
from multiprocessing import Pool

def worker(DBrow,config):
   #  run one Matlab script
   cmd1 = "/usr/local/bin/matlab  ...  myMatlab.1.m"
   subprocess.Popen([cmd1], shell=True, stdout=subprocess.PIPE).communicate()[0]
   print "this does not get printed"

   cmd2 = "sed ..."
   print subprocess.Popen(cmd2, shell=True,
stdout=subprocess.PIPE).communicate()[0]
   print "this does not get printed either"
   sys.stdout.flush()

###   main program below
......
# kick off parallel processing
pool = Pool()
for DBrow in DBrows: pool.apply_async(worker,(DBrow,config))
pool.close()
pool.join()
......
----------------------------------------------

Furthermore, i also tried adding the following:
  multiprocessing.current_process().curr_proc.daemon = False
at the beginning of the "worker" function above but to no avail.

Any help would really be appreciated.

Back to comp.lang.python | Previous | Next — Next in thread | Find similar | Unroll thread

Thread

parallel computations: subprocess.Popen(...).communicate()[0] does not work with multiprocessing.Pool Hseu-Ming Chen <hseuming@gmail.com> - 2011-06-10 16:23 -0400
  Re: parallel computations: subprocess.Popen(...).communicate()[0] does        not work with multiprocessing.Pool Chris Torek <nospam@torek.net> - 2011-06-12 22:00 +0000

csiph-web