Re: subprocess.Popen and multiprocessing fails to execute external program

Path	csiph.com!newsfeed.hal-mli.net!feeder3.hal-mli.net!newsfeed.hal-mli.net!feeder1.hal-mli.net!newsfeed.xs4all.nl!newsfeed2.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Return-Path	<d@davea.name>
X-Original-To	python-list@python.org
Delivered-To	python-list@mail.python.org
X-Spam-Status	OK 0.029
X-Spam-Evidence	'H': 0.94; 'S': 0.00; 'doug': 0.05; 'subject:skip:s 10': 0.05; 'url:blog': 0.09; 'subject:skip:m 10': 0.09; 'result.': 0.15; 'core.': 0.16; 'crashed': 0.16; 'externals': 0.16; 'subject: \n ': 0.16; 'subject:fails': 0.16; 'subject:program': 0.16; 'url:html)': 0.16; 'wrote:': 0.17; 'module,': 0.17; 'module': 0.19; 'external': 0.24; 'header:In- Reply-To:1': 0.25; 'header:User-Agent:1': 0.26; 'implemented': 0.27; 'core': 0.27; 'i.e.': 0.27; 'fine': 0.28; 'run': 0.28; 'consumers,': 0.29; 'far.': 0.29; 'parent': 0.29; 'could': 0.32; 'instances': 0.33; 'to:addr:python-list': 0.33; 'another': 0.33; 'data,': 0.35; 'sequence': 0.35; 'pm,': 0.35; 'there': 0.35; 'problems': 0.36; 'data': 0.37; 'subject:: ': 0.38; 'some': 0.38; 'sure': 0.38; 'to:addr:python.org': 0.39; 'received:192': 0.39; 'easily': 0.39; 'received:192.168': 0.40; 'your': 0.60; 'more': 0.63; 'header:Reply-To:1': 0.68; 'programs,': 0.71; 'received:74.208': 0.71; 'increase': 0.72; 'reply-to:no real name:2**0': 0.72; 'berliner': 0.84; 'consumer,': 0.84; 'handing': 0.84; 'pipeline': 0.84
Date	Thu, 10 Jan 2013 00:56:20 -0500
From	Dave Angel <d@davea.name>
User-Agent	Mozilla/5.0 (X11; Linux x86_64; rv:16.0) Gecko/20121011 Thunderbird/16.0.1
MIME-Version	1.0
To	python-list@python.org
Subject	Re: subprocess.Popen and multiprocessing fails to execute external program
References	<CAJ4spfuNm+4-dqoOCWPQen-jCo2mQD__nM7a-ij+1Dqxpc3OOw@mail.gmail.com>
In-Reply-To	<CAJ4spfuNm+4-dqoOCWPQen-jCo2mQD__nM7a-ij+1Dqxpc3OOw@mail.gmail.com>
Content-Type	text/plain; charset=ISO-8859-1
Content-Transfer-Encoding	7bit
X-Provags-ID	V02:K0:rUuXwMLKP4WMFraEa6n5vpcQzp4EDSWZKPKJqsENIz6 jNneWRyTkuYwraOF7hwmgpo08fDyqO7n1mDHuh7TMNjHVvFPhp wwcMe50SJuzqbWTvjGFPIJC/DLn8tzdHynU9HhztjaUEk4+QYH 1jh+/+VkrXWNQM2EXHfzujDbbhGBixqq5AJZxvmgeF9uhuLbzt COGf4LnIKv88UjdU3egeT5QL+ef/dmanN6DI3IesI3+Igqt5tn oDWQVQWKmGzOrRYVgFlax/qE4lzMrF83ftAsxxja8WjYYcrAoj nzmyMKgeqbH8Jt5jXa+mBB8ZGWgT19/rlIclZIBtfsjmijrTg= =
X-BeenThere	python-list@python.org
X-Mailman-Version	2.1.15
Precedence	list
Reply-To	d@davea.name
List-Id	General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe	<http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive	<http://mail.python.org/pipermail/python-list/>
List-Post	<mailto:python-list@python.org>
List-Help	<mailto:python-list-request@python.org?subject=help>
List-Subscribe	<http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups	comp.lang.python
Message-ID	<mailman.364.1357797401.2939.python-list@python.org> (permalink)
Lines	29
NNTP-Posting-Host	2001:888:2000:d::a6
X-Trace	1357797401 news.xs4all.nl 6855 [2001:888:2000:d::a6]:54678
X-Complaints-To	abuse@xs4all.nl
Xref	csiph.com comp.lang.python:36563

Show key headers only | View raw

On 01/09/2013 11:08 PM, Niklas Berliner wrote:
> I have a pipline that involves processing some data, handing the data to an
> external program (t_coffee used for sequence alignments in bioinformatics),
> and postprocessing the result. Since I have a lot of data, I need to run my
> pipeline in parallel which I implemented using the multiprocessing module
> following Doug Hellmanns blog (
> http://blog.doughellmann.com/2009/04/pymotw-multiprocessing-part-1.html).
>
> My pipeline works perfectly fine when I run it with the multiprocessing
> implementation and one consumer, i.e. on one core. If I increase the number
> of consumers, i.e. that multiple instances of my pipeline run in parallel
> the external program fails with a core dump.
>

Could it be that the external program is not designed to have multiple
simultaneous instances?  There are many such programs, some of which
check for an existing process before allowing another one to get far.

When using the multiprocessing module, always make sure your externals
are well-behaved before looking for problems in your multi-code.

To put it more strongly, a well-written program cannot easily be crashed
by the parent that launched it.

-- 

DaveA

Back to comp.lang.python | Previous | Next | Find similar | Unroll thread

Thread

Re: subprocess.Popen and multiprocessing fails to execute external program Dave Angel <d@davea.name> - 2013-01-10 00:56 -0500

csiph-web