Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!goblin2!goblin.stu.neva.ru!newsfeed.xs4all.nl!newsfeed2.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.007 X-Spam-Evidence: '*H*': 0.99; '*S*': 0.00; 'subject:Python': 0.05; 'doug': 0.05; '-0500': 0.07; 'bash': 0.07; 'executed': 0.07; 'wrapper': 0.07; 'scripts': 0.09; 'url:blog': 0.09; 'instances.': 0.09; '(the': 0.15; 'result.': 0.15; 'core.': 0.16; 'crashed': 0.16; 'externals': 0.16; 'wrote:': 0.17; 'module,': 0.17; 'subject:Issue': 0.17; 'thu,': 0.17; 'jan': 0.18; 'module': 0.19; 'cheers,': 0.23; '>': 0.23; 'external': 0.24; 'machine': 0.24; 'script': 0.24; 'header:In-Reply-To:1': 0.25; 'developers': 0.26; '----------': 0.26; 'implemented': 0.27; 'core': 0.27; 'i.e.': 0.27; 'message-id:@mail.gmail.com': 0.27; 'subject:list': 0.28; 'fine': 0.28; 'run': 0.28; 'consumers,': 0.29; 'far.': 0.29; 'parent': 0.29; 'python).': 0.29; 'date:': 0.29; 'asking': 0.32; 'could': 0.32; 'instances': 0.33; 'to:addr:python-list': 0.33; 'another': 0.33; 'received:google.com': 0.34; 'data,': 0.35; 'fail': 0.35; 'sequence': 0.35; 'pm,': 0.35; 'there': 0.35; 'subject:': 0.36; 'but': 0.36; 'received:74.125': 0.36; 'email addr:python.org': 0.36; 'problems': 0.36; 'execute': 0.37; 'data': 0.37; 'subject:: ': 0.38; 'from:': 0.38; 'some': 0.38; 'sure': 0.38; 'to:addr:python.org': 0.39; 'easily': 0.39; 'subject:-': 0.40; 'header:Received:5': 0.40; 'help': 0.40; 'your': 0.60; 'subject:, ': 0.61; 'email name:python-list': 0.62; 'more': 0.63; 'here': 0.65; 'programs,': 0.71; 'increase': 0.72; '2013': 0.84; 'berliner': 0.84; 'consumer,': 0.84; 'handing': 0.84; 'pipeline': 0.84; '\xa0there': 0.91; 'angel': 0.93 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=gvhzJZcXKtkw0JPmJLvSOZjDo5z0fK8CJbm22Rz5vCI=; b=tdC0kmG4NDiU7+lXHARFQltSU7Xws/Lbo0Txveq3RFqhuOLcQUHJb2FokhyYGHsmAB Q1wdY/hn1LQHD5XOz+0yfPJWNFlymMT7zw6bMcs6gfW3DSvBDe8XZtteKe9TngfL3LQH EHlLgZBcjE3xIc8xy+TWoBcVsARhLhFY63s5e3/X3lTG3FVnsH0dOqFxSd5sk/b7K/ZJ axKZf9O+S74cdVYlx/j7xxglNLH9iTh24bzZq3oaHdStvY0jkfaN8TiSVLLrhBh9/GkL l/NDcQYzEiKzFnjI2zdfAoY589eYlnaOtPFHqoa6jCKV9JFS1VWZ3o8z5Aye2pRgTXwS D1Mw== MIME-Version: 1.0 In-Reply-To: References: Date: Thu, 10 Jan 2013 09:34:43 -0500 Subject: Re: Python-list Digest, Vol 112, Issue 79 From: Niklas Berliner To: python-list@python.org Content-Type: multipart/alternative; boundary=047d7b3a7ff8696b6504d2f01435 X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 124 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1357828490 news.xs4all.nl 6951 [2001:888:2000:d::a6]:46647 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:36571 --047d7b3a7ff8696b6504d2f01435 Content-Type: text/plain; charset=ISO-8859-1 > > > > > ---------- Weitergeleitete Nachricht ---------- > From: Dave Angel > To: python-list@python.org > Cc: > Date: Thu, 10 Jan 2013 00:56:20 -0500 > Subject: Re: subprocess.Popen and multiprocessing fails to execute > external program > On 01/09/2013 11:08 PM, Niklas Berliner wrote: > > I have a pipline that involves processing some data, handing the data to > an > > external program (t_coffee used for sequence alignments in > bioinformatics), > > and postprocessing the result. Since I have a lot of data, I need to run > my > > pipeline in parallel which I implemented using the multiprocessing module > > following Doug Hellmanns blog ( > > http://blog.doughellmann.com/2009/04/pymotw-multiprocessing-part-1.html > ). > > > > My pipeline works perfectly fine when I run it with the multiprocessing > > implementation and one consumer, i.e. on one core. If I increase the > number > > of consumers, i.e. that multiple instances of my pipeline run in parallel > > the external program fails with a core dump. > > > > Could it be that the external program is not designed to have multiple > simultaneous instances? There are many such programs, some of which > check for an existing process before allowing another one to get far. > > When using the multiprocessing module, always make sure your externals > are well-behaved before looking for problems in your multi-code. > > To put it more strongly, a well-written program cannot easily be crashed > by the parent that launched it. > > > -- > > DaveA > > Hi Dave, the developers of the external program said that they are using the program with multiple simultaneous instances. Also, when I execute multiple simultaneous instances of the external program using a bash wrapper script on my machine it works (the same wrapper scripts that fail when executed through python). Before asking here I have contacted the developers of the external program but they couldn't help me any further. Cheers, Niklas --047d7b3a7ff8696b6504d2f01435 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable



---------- Weitergeleitete Nachricht ----------
From:=A0Dave Ang= el <d@davea.name>
To:=A0python-list@python.org
Cc:=A0Date:=A0Thu, 10 Jan 2013 00:56:20 -0500
Subject:=A0Re: subprocess.Popen and multiprocessing fails to execute extern= al program
On 01/09/2013 11:08 PM, Niklas Berliner wrote:
> I have a pipline that involves processing some data, handing the data = to an
> external program (t_coffee used for sequence alignments in bioinformat= ics),
> and postprocessing the result. Since I have a lot of data, I need to r= un my
> pipeline in parallel which I implemented using the multiprocessing mod= ule
> following Doug Hellmanns blog (
> http://blog.doughellmann.com/2009/04/pymotw= -multiprocessing-part-1.html).
>
> My pipeline works perfectly fine when I run it with the multiprocessin= g
> implementation and one consumer, i.e. on one core. If I increase the n= umber
> of consumers, i.e. that multiple instances of my pipeline run in paral= lel
> the external program fails with a core dump.
>

Could it be that the external program is not designed to have multiple
simultaneous instances? =A0There are many such programs, some of which
check for an existing process before allowing another one to get far.

When using the multiprocessing module, always make sure your externals
are well-behaved before looking for problems in your multi-code.

To put it more strongly, a well-written program cannot easily be crashed by the parent that launched it.


--

DaveA



Hi Dave,

the developers of the externa= l program said that they are using the program with multiple simultaneous i= nstances. Also, when I execute multiple simultaneous instances of the exter= nal program using a bash wrapper script on my machine it works (the same wr= apper scripts that fail when executed through python).
Before asking here I have contacted the developers of the external program = but they couldn't help me any further.

Cheers,
Niklas
--047d7b3a7ff8696b6504d2f01435--