Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #2696 > unrolled thread

Trapping the segfault of a subprocess.Popen

Back to article view | Back to comp.lang.python

Contents

  Trapping the segfault of a subprocess.Popen Pierre GM <pierregmcode@gmail.com> - 2011-04-06 02:20 -0700
    Re: Trapping the segfault of a subprocess.Popen Nobody <nobody@nowhere.com> - 2011-04-07 00:58 +0100
      Re: Trapping the segfault of a subprocess.Popen Terry Reedy <tjreedy@udel.edu> - 2011-04-06 23:12 -0400
        Re: Trapping the segfault of a subprocess.Popen Pierre GM <pierregmcode@gmail.com> - 2011-04-07 01:45 -0700
      Re: Trapping the segfault of a subprocess.Popen Pierre GM <pierregmcode@gmail.com> - 2011-04-07 01:44 -0700

#2696 — Trapping the segfault of a subprocess.Popen

All,

I need to run a third-party binary from a python script and retrieve
its output (and its error messages). I use something like
>>> process = subprocess.Popen(options, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>> (info_out, info_err) = process.communicate()
That works fine, except that the third-party binary in question
doesn't behave very nicely and tend to segfaults without returning any
error. In that case, `process.communicate` hangs for ever.

I thought about calling a `threading.Timer` that would call
`process.terminate` if `process.wait` doesn't return after a given
time... But it's not really a solution: the process in question can
sometimes take a long time to run, and I wouldn't want to kill a
process still running.
I also thought about polling every x s and stopping when the result of
a subprocess.Popen(["ps","-p",str(initialprocess.pid)],
stdout=subprocess.PIPE) becomes only the header line, but my script
needs to run on Windows as well (and no ps over there)...

Any suggestion welcome,
Thx in advance
P.

[toc] | [next] | [standalone]

#2729

On Wed, 06 Apr 2011 02:20:22 -0700, Pierre GM wrote:

> I need to run a third-party binary from a python script and retrieve
> its output (and its error messages). I use something like
>>>> process = subprocess.Popen(options, stdout=subprocess.PIPE,
> stderr=subprocess.PIPE)
>>>> (info_out, info_err) = process.communicate()
> That works fine, except that the third-party binary in question doesn't
> behave very nicely and tend to segfaults without returning any error. In
> that case, `process.communicate` hangs for ever.

Odd. The .communicate method should return once both stdout and stderr
report EOF and the process is no longer running. Whether it terminates
normally or on a signal makes no difference.

The only thing I can think of which would cause the situation which you
describe is if the child process spawns a child of its own before
terminating. In that case, stdout/stderr won't report EOF until any
processes which inherited them have terminated.

> I thought about calling a `threading.Timer` that would call
> `process.terminate` if `process.wait` doesn't return after a given
> time... But it's not really a solution: the process in question can
> sometimes take a long time to run, and I wouldn't want to kill a
> process still running.
> I also thought about polling every x s and stopping when the result of
> a subprocess.Popen(["ps","-p",str(initialprocess.pid)],
> stdout=subprocess.PIPE) becomes only the header line, but my script
> needs to run on Windows as well (and no ps over there)...

It should suffice to call .poll() on the process. In case that doesn't
work, the usual alternative would be to send signal 0 to the process (this
will fail with ESRCH if the process doesn't exist and do nothing
otherwise), e.g.:

import os
import errno

def exists(process):
    try:
        os.kill(process.pid, 0)
    except OSError, e:
        if e.errno == errno.ESRCH:
            return False
        raise
    return True

You might need to take a different approach for Windows, but the above is
preferable to trying to parse "ps" output. Note that this only tells you
if /some/ process exists with the given PID, not whether the original
process exists; that information can only be obtained from the Popen
object.

[toc] | [prev] | [next] | [standalone]

#2738

On 4/6/2011 7:58 PM, Nobody wrote:
> On Wed, 06 Apr 2011 02:20:22 -0700, Pierre GM wrote:
>
>> I need to run a third-party binary from a python script and retrieve
>> its output (and its error messages). I use something like
>>>>> process = subprocess.Popen(options, stdout=subprocess.PIPE,
>> stderr=subprocess.PIPE)
>>>>> (info_out, info_err) = process.communicate()
>> That works fine, except that the third-party binary in question doesn't
>> behave very nicely and tend to segfaults without returning any error. In
>> that case, `process.communicate` hangs for ever.

I am not sure this will help you now, but....
Victor Stinner has added a new module to Python 3.3 that tries to catch 
segfaults and other fatal signals and produce a traceback before Python 
disappears.

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#2764

On Apr 7, 5:12 am, Terry Reedy <tjre...@udel.edu> wrote:
> On 4/6/2011 7:58 PM, Nobody wrote:
>
> > On Wed, 06 Apr 2011 02:20:22 -0700, Pierre GM wrote:
>
> >> I need to run a third-party binary from a python script and retrieve
> >> its output (and its error messages). I use something like
> >>>>> process = subprocess.Popen(options, stdout=subprocess.PIPE,
> >> stderr=subprocess.PIPE)
> >>>>> (info_out, info_err) = process.communicate()
> >> That works fine, except that the third-party binary in question doesn't
> >> behave very nicely and tend to segfaults without returning any error. In
> >> that case, `process.communicate` hangs for ever.
>
> I am not sure this will help you now, but....
> Victor Stinner has added a new module to Python 3.3 that tries to catch
> segfaults and other fatal signals and produce a traceback before Python
> disappears.

Unfortunately, I'm limited to Python 2.5.x for this project. But good
to know, thanks...

[toc] | [prev] | [next] | [standalone]

#2763

On Apr 7, 1:58 am, Nobody <nob...@nowhere.com> wrote:
> On Wed, 06 Apr 2011 02:20:22 -0700, Pierre GM wrote:
> > I need to run a third-party binary from a python script and retrieve
> > its output (and its error messages). I use something like
> >>>> process = subprocess.Popen(options, stdout=subprocess.PIPE,
> > stderr=subprocess.PIPE)
> >>>> (info_out, info_err) = process.communicate()
> > That works fine, except that the third-party binary in question doesn't
> > behave very nicely and tend to segfaults without returning any error. In
> > that case, `process.communicate` hangs for ever.
>
> Odd. The .communicate method should return once both stdout and stderr
> report EOF and the process is no longer running. Whether it terminates
> normally or on a signal makes no difference.
>
> The only thing I can think of which would cause the situation which you
> describe is if the child process spawns a child of its own before
> terminating. In that case, stdout/stderr won't report EOF until any
> processes which inherited them have terminated.

I think you nailed it. Running the incriminating command line in a
terminal doesn't return to the prompt. In fact, a ps shows that the
process is sleeping in the foreground. Guess I should change the
subject of this thread...



> > I thought about calling a `threading.Timer` that would call
> > `process.terminate` if `process.wait` doesn't return after a given
> > time... But it's not really a solution: the process in question can
> > sometimes take a long time to run, and I wouldn't want to kill a
> > process still running.
> > I also thought about polling every x s and stopping when the result of
> > a subprocess.Popen(["ps","-p",str(initialprocess.pid)],
> > stdout=subprocess.PIPE) becomes only the header line, but my script
> > needs to run on Windows as well (and no ps over there)...
>
> It should suffice to call .poll() on the process. In case that doesn't
> work, the usual alternative would be to send signal 0 to the process (this
> will fail with ESRCH if the process doesn't exist and do nothing
> otherwise), e.g.:
>
> import os
> import errno
>
> def exists(process):
>     try:
>         os.kill(process.pid, 0)
>     except OSError, e:
>         if e.errno == errno.ESRCH:
>             return False
>         raise
>     return True

OK, gonna try that, thx.


> You might need to take a different approach for Windows, but the above is
> preferable to trying to parse "ps" output. Note that this only tells you
> if /some/ process exists with the given PID, not whether the original
> process exists; that information can only be obtained from the Popen
> object.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web