Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #53266 > unrolled thread

Re: subprocess.Popen instance hangs

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: subprocess.Popen instance hangs Tim Johnson <tim@akwebsoft.com> - 2013-08-29 17:00 -0800
    Re: subprocess.Popen instance hangs Nobody <nobody@nowhere.com> - 2013-08-30 15:47 +0100
      Re: subprocess.Popen instance hangs Tim Johnson <tim@akwebsoft.com> - 2013-08-30 07:32 -0800
      Re: subprocess.Popen instance hangs Jerry Hill <malaclypse2@gmail.com> - 2013-08-30 11:38 -0400
      Re: subprocess.Popen instance hangs Tim Johnson <tim@akwebsoft.com> - 2013-08-30 10:43 -0800

#53266 — Re: subprocess.Popen instance hangs

* Tim Johnson <tim@akwebsoft.com> [130829 10:51]:
> using Python 2.7.1 on OS X 10.7.5
> 
> I'm managing a process of drush using an instance of subprocess.Popen
<...> 
## This appears to be what works. 
    def __exec(self,args) :
        """Run the process with arguments"""
       p = subprocess.Popen(args,stderr=subprocess.PIPE,stdout=subprocess.PIPE)
       while 1 :
           output = p.stdout.read()
           if output :
               print(output)
           else : break
       errmsg = p.communicate()[1]
       if errmsg :
           self.__err('Process terminated with error:',errmsg)
## Thanks again, gnarly one for me. I am eternally a noob!

-- 
Tim 
tim at tee jay forty nine dot com or akwebsoft dot com
http://www.akwebsoft.com

[toc] | [next] | [standalone]

#53298

On Thu, 29 Aug 2013 17:00:21 -0800, Tim Johnson wrote:

> ## This appears to be what works.
>     def __exec(self,args) :
>         """Run the process with arguments"""
>        p =
>        subprocess.Popen(args,stderr=subprocess.PIPE,stdout=subprocess.PIPE)
>        while 1 :
>            output = p.stdout.read()

If the process tries to write more than a pipe's worth of data to stderr,
before closing stdout, it will block indefinitely.

If you want to process both stdout and stderr, you have to be able to
consume the data in whatever order the process generates it, which means
either using multiple threads or (on Unix) select/poll or non-blocking
I/O. This is what the .communicate() method does (threads on Windows,
select/poll on Unix).

The alternative is to merge both streams with stderr=subprocess.STDOUT, or
redirect one of them to a file (or /dev/null, etc).

[toc] | [prev] | [next] | [standalone]

#53301

* Nobody <nobody@nowhere.com> [130830 06:55]:
> On Thu, 29 Aug 2013 17:00:21 -0800, Tim Johnson wrote:
> 
> > ## This appears to be what works.
> >     def __exec(self,args) :
> >         """Run the process with arguments"""
> >        p =
> >        subprocess.Popen(args,stderr=subprocess.PIPE,stdout=subprocess.PIPE)
> >        while 1 :
> >            output = p.stdout.read()
> 
> If the process tries to write more than a pipe's worth of data to stderr,
> before closing stdout, it will block indefinitely.
> 
> If you want to process both stdout and stderr, you have to be able to
> consume the data in whatever order the process generates it, which means
> either using multiple threads or (on Unix) select/poll or non-blocking
> I/O. This is what the .communicate() method does (threads on Windows,
> select/poll on Unix).
> 
> The alternative is to merge both streams with stderr=subprocess.STDOUT, or
> redirect one of them to a file (or /dev/null, etc).
  In earlier code I, I was merging them...
  :) Like I said: gnarly! What if I were to do something like:
  ## code
  while 1: 
  	output = p.stout.read()
	err = p.stderr.read() ## trapping for AttributeError, etc..
	....
  ## /code
	break'ing if either no output or value in `err' 
 ?? 
  The objective is to display all output, but to also separate error
  messages from normal output.

  thank you
-- 
Tim 
tim at tee jay forty nine dot com or akwebsoft dot com
http://www.akwebsoft.com

[toc] | [prev] | [next] | [standalone]

#53302

On Fri, Aug 30, 2013 at 11:32 AM, Tim Johnson <tim@akwebsoft.com> wrote:
>   The objective is to display all output, but to also separate error
>   messages from normal output.

I still think you want to use communicate().  Like this:

p = subprocess.Popen(args,stderr=subprocess.PIPE,stdout=subprocess.PIPE)
output, err = p.communicate()

That's it.  No need for a loop, or manually handling the fact that
stderr and/or stdout could end up with a full buffer and start to
block.

-- 
Jerry

[toc] | [prev] | [next] | [standalone]

#53311

* Jerry Hill <malaclypse2@gmail.com> [130830 07:48]:
> On Fri, Aug 30, 2013 at 11:32 AM, Tim Johnson <tim@akwebsoft.com> wrote:
> >   The objective is to display all output, but to also separate error
> >   messages from normal output.
> 
> I still think you want to use communicate().  Like this:
> 
> p = subprocess.Popen(args,stderr=subprocess.PIPE,stdout=subprocess.PIPE)
> output, err = p.communicate()
> 
> That's it.  No need for a loop, or manually handling the fact that
> stderr and/or stdout could end up with a full buffer and start to
> block.
  The following code :
p = subprocess.Popen(args,stderr=subprocess.PIPE,stdout=subprocess.PIPE)
errmsg,output = p.communicate()
... Hangs
this code :
p = subprocess.Popen(args,stderr=subprocess.PIPE,stdout=subprocess.PIPE)
while 1 :
	output = p.stdout.read()
	if output :
		print(output)
	else : break
... works
-- 
Tim 
tim at tee jay forty nine dot com or akwebsoft dot com
http://www.akwebsoft.com

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web