Path: csiph.com!x330-a1.tempe.blueboxinc.net!newsfeed.hal-mli.net!feeder1.hal-mli.net!weretis.net!feeder1.news.weretis.net!feeder.erje.net!newsfeed.xs4all.nl!newsfeed5.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.007 X-Spam-Evidence: '*H*': 0.99; '*S*': 0.00; 'debug': 0.03; 'interpreter': 0.07; 'mysql.': 0.07; 'python': 0.08; 'either.': 0.09; 'invocation': 0.09; 'matlab': 0.09; 'subprocess': 0.09; 'scripts': 0.10; 'def': 0.12; '"sed': 0.16; '###': 0.16; '(version': 0.16; 'happens.': 0.16; 'kick': 0.16; 'subject: \n ': 0.16; 'subject:] ': 0.16; 'shell': 0.19; 'subject:skip:s 10': 0.19; 'appears': 0.21; 'framework': 0.22; 'fine': 0.22; 'moreover,': 0.23; 'subject:not': 0.23; '"this': 0.25; 'function': 0.25; 'script': 0.27; 'tried': 0.27; "i'm": 0.27; 'message- id:@mail.gmail.com': 0.28; 'skip:p 30': 0.28; 'import': 0.29; 'version': 0.29; '2008)': 0.30; 'hi,': 0.31; 'print': 0.31; 'skip:- 40': 0.32; 'does': 0.33; 'to:addr:python-list': 0.33; 'on,': 0.33; 'skip:" 20': 0.33; 'too': 0.33; "i've": 0.33; '...': 0.34; 'however,': 0.34; '2.6': 0.35; 'using': 0.35; 'running': 0.37; 'beginning': 0.37; 'received:google.com': 0.37; 'issue': 0.37; 'received:209.85': 0.37; 'parallel': 0.37; 'processing': 0.38; 'run': 0.38; 'but': 0.38; 'subject:: ': 0.38; 'some': 0.38; 'skip:s 20': 0.39; 'received:209': 0.39; 'happened': 0.39; 'printed': 0.39; 'subject:with': 0.39; 'to:addr:python.org': 0.39; 'basic': 0.40; 'appreciated.': 0.40; 'following:': 0.40; 'really': 0.40; 'help': 0.40; 'results': 0.60; 'within': 0.60; 'below': 0.61; 'released': 0.63; 'show': 0.66; 'here': 0.66; 'making': 0.67; 'received:209.85.216.174': 0.73; '.......': 0.84; 'avail.': 0.84; 'oct.': 0.84; 'story:': 0.84 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:date:message-id:subject:from:to :content-type; bh=Q14ZIIFtNt6G9Eiu5SIzMOl5F63JcGbdg/cHO6kJhYU=; b=mgZ7MVqCgLHKxtcmxPGN38HQnK0eKo+BPlYV2zmwTQ3cUnAimbFo5Kmqr+KdMdlVMR 2ZB2FRKArs+jwIzs8S40d/y/4eNTzW1CCOi3PIyAwZAeWTl8D6gtzbhR7ZfgL2YJiedr meNhESjRrLfJkJHbrJlLJOZCBykOiU6/wTqis= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=MH7/LmgNcCgSRz4YihJbxGhgmURLy0jSSYT8tUA9EctFePY50sPez9I4LJ3FaQ2vwd dYv92u4AqpbxW5Be3NXZ/4mU3SQ4FwzbS6q99Vxbm0x/MI1ZpTUbEhfFtnwVOW2t5ERt LxnT1coE25HAO9Z/z5HVzuAUY49/K28sl9ouo= MIME-Version: 1.0 Date: Fri, 10 Jun 2011 16:23:19 -0400 Subject: parallel computations: subprocess.Popen(...).communicate()[0] does not work with multiprocessing.Pool From: Hseu-Ming Chen To: python-list@python.org Content-Type: text/plain; charset=ISO-8859-1 X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 50 NNTP-Posting-Host: 82.94.164.166 X-Trace: 1307737402 news.xs4all.nl 49179 [::ffff:82.94.164.166]:53636 X-Complaints-To: abuse@xs4all.nl Xref: x330-a1.tempe.blueboxinc.net comp.lang.python:7399 Hi, I am having an issue when making a shell call from within a multiprocessing.Process(). Here is the story: i tried to parallelize the computations in 800-ish Matlab scripts and then save the results to MySQL. The non-parallel/serial version has been running fine for about 2 years. However, in the parallel version via multiprocessing that i'm working on, it appears that the Matlab scripts have never been kicked off and nothing happened with subprocess.Popen. The debug printing below does not show up either. Moreover, even if i replace the Matlab invocation with some trivial "sed" call, still nothing happens. Is it possible that the Python interpreter i'm using (version 2.6 released on Oct. 1, 2008) is too old? Nevertheless, i would like to make sure the basic framework i've now is not blatantly wrong. Below is a skeleton of my Python program: ---------------------------------------------- import subprocess from multiprocessing import Pool def worker(DBrow,config): # run one Matlab script cmd1 = "/usr/local/bin/matlab ... myMatlab.1.m" subprocess.Popen([cmd1], shell=True, stdout=subprocess.PIPE).communicate()[0] print "this does not get printed" cmd2 = "sed ..." print subprocess.Popen(cmd2, shell=True, stdout=subprocess.PIPE).communicate()[0] print "this does not get printed either" sys.stdout.flush() ### main program below ...... # kick off parallel processing pool = Pool() for DBrow in DBrows: pool.apply_async(worker,(DBrow,config)) pool.close() pool.join() ...... ---------------------------------------------- Furthermore, i also tried adding the following: multiprocessing.current_process().curr_proc.daemon = False at the beginning of the "worker" function above but to no avail. Any help would really be appreciated.