Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #11098 > unrolled thread

Re: subprocess.Popen and thread module

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: subprocess.Popen and thread module Chris Rebert <clp2@rebertia.com> - 2011-08-09 23:53 -0700
    Re: subprocess.Popen and thread module Nobody <nobody@nowhere.com> - 2011-08-11 09:15 +0100

#11098 — Re: subprocess.Popen and thread module

> On Tue, Aug 9, 2011 at 11:38 PM, Danny Wong (dannwong)
> <dannwong@cisco.com> wrote:
>> Hi All,
>>   I'm trying to execute some external commands from multiple database.
>> I'm using threads and subprocess.Popen ( from docs, all the popen*
>> functions are deprecated and I was told to use subprocess.Popen) to
>> execute the external commands in parallel, but the commands seems to
>> hang.
>
> What's your Popen() call look like?

On Tue, Aug 9, 2011 at 11:48 PM, Danny Wong (dannwong)
<dannwong@cisco.com> wrote:
>    try:
>        cmd_output = subprocess.Popen(['scm', 'load', '--force', '-r', nickname, '-d', directory, project], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>        status = cmd_output.wait()

Er, did you read the warning about Popen.wait() in the docs? (emphasis added):
"""
Popen.wait()
    Wait for child process to terminate. Set and return returncode attribute.
    Warning: ***This will deadlock*** when using stdout=PIPE and/or
stderr=PIPE and the child process generates enough output to a pipe
such that it blocks waiting for the OS pipe buffer to accept more
data. Use communicate() to avoid that.
"""
– http://docs.python.org/library/subprocess.html#subprocess.Popen.wait

Cheers,
Chris

[toc] | [next] | [standalone]

#11195

Danny Wong (dannwong) <dannwong@cisco.com> wrote:

>        cmd_output = subprocess.Popen(['scm', 'load', '--force',
> '-r', nickname, '-d', directory, project], stdout=subprocess.PIPE,
> stderr=subprocess.PIPE)
>       status = cmd_output.wait()

If you redirect stdout and/or stderr to a pipe, you must wait for EOF
before calling wait(), otherwise you risk deadlock (the process blocks
waiting for Python to read data from the pipe while Python is blocked
waiting for the process to terminate).

And if you redirect both stdout and stderr to pipes, you must either use
multiple threads or non-blocking I/O, otherwise you risk deadlock (the
process blocks waiting for Python to read data from one of the pipes while
Python is blocked waiting for the process to write to the other pipe).

I suggest looking at the .communicate() method of the Popen class.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web