Re: Python-list Digest, Vol 112, Issue 79

Path	csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!goblin2!goblin.stu.neva.ru!newsfeed.xs4all.nl!newsfeed2.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Return-Path	<niklas.berliner@gmail.com>
X-Original-To	python-list@python.org
Delivered-To	python-list@mail.python.org
X-Spam-Status	OK 0.007
X-Spam-Evidence	'H': 0.99; 'S': 0.00; 'subject:Python': 0.05; 'doug': 0.05; '-0500': 0.07; 'bash': 0.07; 'executed': 0.07; 'wrapper': 0.07; 'scripts': 0.09; 'url:blog': 0.09; 'instances.': 0.09; '(the': 0.15; 'result.': 0.15; 'core.': 0.16; 'crashed': 0.16; 'externals': 0.16; 'wrote:': 0.17; 'module,': 0.17; 'subject:Issue': 0.17; 'thu,': 0.17; 'jan': 0.18; 'module': 0.19; 'cheers,': 0.23; '>': 0.23; 'external': 0.24; 'machine': 0.24; 'script': 0.24; 'header:In-Reply-To:1': 0.25; 'developers': 0.26; '----------': 0.26; 'implemented': 0.27; 'core': 0.27; 'i.e.': 0.27; 'message-id:@mail.gmail.com': 0.27; 'subject:list': 0.28; 'fine': 0.28; 'run': 0.28; 'consumers,': 0.29; 'far.': 0.29; 'parent': 0.29; 'python).': 0.29; 'date:': 0.29; 'asking': 0.32; 'could': 0.32; 'instances': 0.33; 'to:addr:python-list': 0.33; 'another': 0.33; 'received:google.com': 0.34; 'data,': 0.35; 'fail': 0.35; 'sequence': 0.35; 'pm,': 0.35; 'there': 0.35; 'subject:': 0.36; 'but': 0.36; 'received:74.125': 0.36; 'email addr:python.org': 0.36; 'problems': 0.36; 'execute': 0.37; 'data': 0.37; 'subject:: ': 0.38; 'from:': 0.38; 'some': 0.38; 'sure': 0.38; 'to:addr:python.org': 0.39; 'easily': 0.39; 'subject:-': 0.40; 'header:Received:5': 0.40; 'help': 0.40; 'your': 0.60; 'subject:, ': 0.61; 'email name:python-list': 0.62; 'more': 0.63; 'here': 0.65; 'programs,': 0.71; 'increase': 0.72; '2013': 0.84; 'berliner': 0.84; 'consumer,': 0.84; 'handing': 0.84; 'pipeline': 0.84; '\xa0there': 0.91; 'angel': 0.93
DKIM-Signature	v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=gvhzJZcXKtkw0JPmJLvSOZjDo5z0fK8CJbm22Rz5vCI=; b=tdC0kmG4NDiU7+lXHARFQltSU7Xws/Lbo0Txveq3RFqhuOLcQUHJb2FokhyYGHsmAB Q1wdY/hn1LQHD5XOz+0yfPJWNFlymMT7zw6bMcs6gfW3DSvBDe8XZtteKe9TngfL3LQH EHlLgZBcjE3xIc8xy+TWoBcVsARhLhFY63s5e3/X3lTG3FVnsH0dOqFxSd5sk/b7K/ZJ axKZf9O+S74cdVYlx/j7xxglNLH9iTh24bzZq3oaHdStvY0jkfaN8TiSVLLrhBh9/GkL l/NDcQYzEiKzFnjI2zdfAoY589eYlnaOtPFHqoa6jCKV9JFS1VWZ3o8z5Aye2pRgTXwS D1Mw==
MIME-Version	1.0
In-Reply-To	<mailman.1480.1357797688.2938.python-list@python.org>
References	<mailman.1480.1357797688.2938.python-list@python.org>
Date	Thu, 10 Jan 2013 09:34:43 -0500
Subject	Re: Python-list Digest, Vol 112, Issue 79
From	Niklas Berliner <niklas.berliner@gmail.com>
To	python-list@python.org
Content-Type	multipart/alternative; boundary=047d7b3a7ff8696b6504d2f01435
X-BeenThere	python-list@python.org
X-Mailman-Version	2.1.15
Precedence	list
List-Id	General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe	<http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive	<http://mail.python.org/pipermail/python-list/>
List-Post	<mailto:python-list@python.org>
List-Help	<mailto:python-list-request@python.org?subject=help>
List-Subscribe	<http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups	comp.lang.python
Message-ID	<mailman.368.1357828490.2939.python-list@python.org> (permalink)
Lines	124
NNTP-Posting-Host	2001:888:2000:d::a6
X-Trace	1357828490 news.xs4all.nl 6951 [2001:888:2000:d::a6]:46647
X-Complaints-To	abuse@xs4all.nl
Xref	csiph.com comp.lang.python:36571

Show key headers only | View raw

[Multipart message — attachments visible in raw view] - view raw

>
>
>
>
> ---------- Weitergeleitete Nachricht ----------
> From: Dave Angel <d@davea.name>
> To: python-list@python.org
> Cc:
> Date: Thu, 10 Jan 2013 00:56:20 -0500
> Subject: Re: subprocess.Popen and multiprocessing fails to execute
> external program
> On 01/09/2013 11:08 PM, Niklas Berliner wrote:
> > I have a pipline that involves processing some data, handing the data to
> an
> > external program (t_coffee used for sequence alignments in
> bioinformatics),
> > and postprocessing the result. Since I have a lot of data, I need to run
> my
> > pipeline in parallel which I implemented using the multiprocessing module
> > following Doug Hellmanns blog (
> > http://blog.doughellmann.com/2009/04/pymotw-multiprocessing-part-1.html
> ).
> >
> > My pipeline works perfectly fine when I run it with the multiprocessing
> > implementation and one consumer, i.e. on one core. If I increase the
> number
> > of consumers, i.e. that multiple instances of my pipeline run in parallel
> > the external program fails with a core dump.
> >
>
> Could it be that the external program is not designed to have multiple
> simultaneous instances?  There are many such programs, some of which
> check for an existing process before allowing another one to get far.
>
> When using the multiprocessing module, always make sure your externals
> are well-behaved before looking for problems in your multi-code.
>
> To put it more strongly, a well-written program cannot easily be crashed
> by the parent that launched it.
>
>
> --
>
> DaveA
>
>

Hi Dave,

the developers of the external program said that they are using the program
with multiple simultaneous instances. Also, when I execute multiple
simultaneous instances of the external program using a bash wrapper script
on my machine it works (the same wrapper scripts that fail when executed
through python).
Before asking here I have contacted the developers of the external program
but they couldn't help me any further.

Cheers,
Niklas

Back to comp.lang.python | Previous | Next | Find similar | Unroll thread

Thread

Re: Python-list Digest, Vol 112, Issue 79 Niklas Berliner <niklas.berliner@gmail.com> - 2013-01-10 09:34 -0500

csiph-web