Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #73810 > unrolled thread

Success with subprocess communicate on Windows?

Back to article view | Back to comp.lang.python

Contents

  Success with subprocess communicate on Windows? Terry Reedy <tjreedy@udel.edu> - 2014-07-01 17:05 -0400
    Re: Success with subprocess communicate on Windows? Tim Roberts <timr@probo.com> - 2014-07-01 21:33 -0700
      Re: Success with subprocess communicate on Windows? Terry Reedy <tjreedy@udel.edu> - 2014-07-02 05:05 -0400
      Re: Success with subprocess communicate on Windows? Wolfgang Maier <wolfgang.maier@biologie.uni-freiburg.de> - 2014-07-02 19:31 +0200
      Re: Success with subprocess communicate on Windows? Wolfgang Maier <wolfgang.maier@biologie.uni-freiburg.de> - 2014-07-02 19:37 +0200
      Re: Success with subprocess communicate on Windows? Terry Reedy <tjreedy@udel.edu> - 2014-07-02 19:14 -0400
      Re: Success with subprocess communicate on Windows? Wolfgang Maier <wolfgang.maier@biologie.uni-freiburg.de> - 2014-07-02 23:22 +0000
      Re: Success with subprocess communicate on Windows? Ethan Furman <ethan@stoneleaf.us> - 2014-07-02 16:54 -0700
      Re: Success with subprocess communicate on Windows? Terry Reedy <tjreedy@udel.edu> - 2014-07-03 00:09 -0400
      Re: Success with subprocess communicate on Windows? Wolfgang Maier <wolfgang.maier@biologie.uni-freiburg.de> - 2014-07-03 10:03 +0200
      Re: Success with subprocess communicate on Windows? Wolfgang Maier <wolfgang.maier@biologie.uni-freiburg.de> - 2014-07-03 11:22 +0200

#73810 — Success with subprocess communicate on Windows?

I am mentoring a GSOC student, Saimadhav Heblikar, who is working on 
adding the following feature to Idle: submit the editor text to an 
external program, such as pyflakes, and display the result in an 
OutputWindow. If it reports line numbers with problems, as pyflakes 
does, users will be able to jump to the line in the file (this exists 
already for multiple-file grep output).

The 'obvious' implementation is to use subprocess.Open.communicate to 
run the process and capture the output. The prototype code at 
http://bugs.python.org/issue21880 works on linux. The key part of the 
new code at http://bugs.python.org/file35819/3rdpartychecker-v2.diff is

+from subprocess import Popen, PIPE
...
+            proc = Popen(args, stdout=PIPE, stderr=PIPE)
+            proc.wait()
+            output, error = map(lambda b:b.decode('utf-8'), 
proc.communicate())

It does not work on Windows. As I reported on 
http://bugs.python.org/issue8631, msg222053,
 >>> subprocess.check_output("pyflakes -h")
works in the interpreter and Idle shell, while
 >>> s.check_output("pyflakes c:\programs\python34\lib\turtle.py")
gives bizarre output in the interpreter and hangs in the idle shell, as 
does the code above.

My question is whether anyone reading that has had success getting 
subprocess output capture to work consistently on Windows?

-- 
Terry Jan Reedy

[toc] | [next] | [standalone]

#73823

Terry Reedy <tjreedy@udel.edu> wrote:
>
>It does not work on Windows. As I reported on 
>http://bugs.python.org/issue8631, msg222053,
> >>> subprocess.check_output("pyflakes -h")
>works in the interpreter and Idle shell, while
> >>> s.check_output("pyflakes c:\programs\python34\lib\turtle.py")
>gives bizarre output in the interpreter and hangs in the idle shell, as 
>does the code above.

Right.  What do you think \t is in a string?  (Hint: it's only one byte.)

You need to use
    s.check_output("pyflakes c:\\programs\\python34\\lib\\turtle.py")
or
    s.check_output(r"pyflakes c:\programs\python34\lib\turtle.py")
-- 
Tim Roberts, timr@probo.com
Providenza & Boekelheide, Inc.

[toc] | [prev] | [next] | [standalone]

#73829

On 7/2/2014 12:33 AM, Tim Roberts wrote:
> Terry Reedy <tjreedy@udel.edu> wrote:
>>
>> It does not work on Windows. As I reported on
>> http://bugs.python.org/issue8631, msg222053,
>>>>> subprocess.check_output("pyflakes -h")
>> works in the interpreter and Idle shell, while
>>>>> s.check_output("pyflakes c:\programs\python34\lib\turtle.py")
>> gives bizarre output in the interpreter and hangs in the idle shell, as
>> does the code above.
>
> Right.  What do you think \t is in a string?  (Hint: it's only one byte.)

Yes, how could I forget that. But my use of a string literal here does 
not explain the problems when used string variables, as generated by os. 
I should try printing out the list of args passed to subprocess.

While remembering what \t is explains the command interpreter output 
(recursing through /lib), it does not explain the difference when 
invoked from Idle. There seems to be some bad interaction when calling 
subprocesses in the user subprocess.

> You need to use
>      s.check_output("pyflakes c:\\programs\\python34\\lib\\turtle.py")
> or
>      s.check_output(r"pyflakes c:\programs\python34\lib\turtle.py")

Now I get "Command 'pyflakes c:\programs\python34\lib\turtle.py' returns 
non-zero exit status 1" on both. On Idle, as least, a command-prompt 
window is flashed/displayed. It makes no sense to me that in the command 
interpreter,
'pyflakes c:\\programs\\python34\\lib' works and
'pyflakes c:\\programs\\python34\\lib\\turtle.py' returns status 1.
whereas both (with quotes elided and undoubled \) work at the command 
prompt.

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#73842

On 02.07.2014 11:05, Terry Reedy wrote:
> On 7/2/2014 12:33 AM, Tim Roberts wrote:
>> Terry Reedy <tjreedy@udel.edu> wrote:
>>
>> You need to use
>>      s.check_output("pyflakes c:\\programs\\python34\\lib\\turtle.py")
>> or
>>      s.check_output(r"pyflakes c:\programs\python34\lib\turtle.py")
>
> Now I get "Command 'pyflakes c:\programs\python34\lib\turtle.py' returns
> non-zero exit status 1" on both. On Idle, as least, a command-prompt
> window is flashed/displayed. It makes no sense to me that in the command
> interpreter,
> 'pyflakes c:\\programs\\python34\\lib' works and
> 'pyflakes c:\\programs\\python34\\lib\\turtle.py' returns status 1.
> whereas both (with quotes elided and undoubled \) work at the command
> prompt.
>

I am not 100% sure whether that is the problem, but from what I gather 
from the subprocess module docs the args string is passed to the Windows 
CreateProcess function as a single string.
To me this seems to imply that it is passed as the lpCommandLine 
parameter (with Null for the lpApplicationName parameter).
This is what Microsoft says about this case 
(http://msdn.microsoft.com/en-us/library/windows/desktop/ms682425%28v=vs.85%29.aspx):

<quote>
If lpApplicationName is NULL, the first white space–delimited token of 
the command line specifies the module name. If you are using a long file 
name that contains a space, use quoted strings to indicate where the 
file name ends and the arguments begin (see the explanation for the 
lpApplicationName parameter). If the file name does not contain an 
extension, .exe is appended. Therefore, if the file name extension is 
.com, this parameter must include the .com extension. If the file name 
ends in a period (.) with no extension, or if the file name contains a 
path, .exe is not appended. If the file name does not contain a 
directory path, the system searches for the executable file in the 
following sequence.
</quote>

So in your case the default behavior would be to add an .exe extension 
to pyflakes, which is probably not what you intended ?

Best,
Wolfgang

[toc] | [prev] | [next] | [standalone]

#73844

On 02.07.2014 19:31, Wolfgang Maier wrote:
>
> I am not 100% sure whether that is the problem, but from what I gather
> from the subprocess module docs the args string is passed to the Windows
> CreateProcess function as a single string.
> To me this seems to imply that it is passed as the lpCommandLine
> parameter (with Null for the lpApplicationName parameter).
> This is what Microsoft says about this case
> (http://msdn.microsoft.com/en-us/library/windows/desktop/ms682425%28v=vs.85%29.aspx):
>
>
> <quote>
> If lpApplicationName is NULL, the first white space–delimited token of
> the command line specifies the module name. If you are using a long file
> name that contains a space, use quoted strings to indicate where the
> file name ends and the arguments begin (see the explanation for the
> lpApplicationName parameter). If the file name does not contain an
> extension, .exe is appended. Therefore, if the file name extension is
> .com, this parameter must include the .com extension. If the file name
> ends in a period (.) with no extension, or if the file name contains a
> path, .exe is not appended. If the file name does not contain a
> directory path, the system searches for the executable file in the
> following sequence.
> </quote>
>
> So in your case the default behavior would be to add an .exe extension
> to pyflakes, which is probably not what you intended ?
>
> Best,
> Wolfgang
>

Ah, I forgot about your pyflakes -h example, which works, so my 
explanation must be wrong.
Sorry.

[toc] | [prev] | [next] | [standalone]

#73855

On 7/2/2014 1:37 PM, Wolfgang Maier wrote:
> On 02.07.2014 19:31, Wolfgang Maier wrote:
>>
>> I am not 100% sure whether that is the problem, but from what I gather
>> from the subprocess module docs the args string is passed to the Windows
>> CreateProcess function as a single string.
>> To me this seems to imply that it is passed as the lpCommandLine
>> parameter (with Null for the lpApplicationName parameter).
>> This is what Microsoft says about this case
>> (http://msdn.microsoft.com/en-us/library/windows/desktop/ms682425%28v=vs.85%29.aspx):
>>
>>
>>
>> <quote>
>> If lpApplicationName is NULL, the first white space–delimited token of
>> the command line specifies the module name. If you are using a long file
>> name that contains a space, use quoted strings to indicate where the
>> file name ends and the arguments begin (see the explanation for the
>> lpApplicationName parameter). If the file name does not contain an
>> extension, .exe is appended. Therefore, if the file name extension is
>> .com, this parameter must include the .com extension. If the file name
>> ends in a period (.) with no extension, or if the file name contains a
>> path, .exe is not appended. If the file name does not contain a
>> directory path, the system searches for the executable file in the
>> following sequence.
>> </quote>
>>
>> So in your case the default behavior would be to add an .exe extension
>> to pyflakes, which is probably not what you intended ?
>>
>> Best,
>> Wolfgang
>>
>
> Ah, I forgot about your pyflakes -h example, which works, so my
> explanation must be wrong.

I appreciate the attempt. I hope there is someone who has been 
successful who can answer ;-).

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#73858

Terry Reedy <tjreedy <at> udel.edu> writes:

> 
> On 7/2/2014 12:33 AM, Tim Roberts wrote:
> > Terry Reedy <tjreedy <at> udel.edu> wrote:
> >>
> > You need to use
> >      s.check_output("pyflakes c:\\programs\\python34\\lib\\turtle.py")
> > or
> >      s.check_output(r"pyflakes c:\programs\python34\lib\turtle.py")
> 
> Now I get "Command 'pyflakes c:\programs\python34\lib\turtle.py' returns 
> non-zero exit status 1" on both. On Idle, as least, a command-prompt 
> window is flashed/displayed. It makes no sense to me that in the command 
> interpreter,
> 'pyflakes c:\\programs\\python34\\lib' works and
> 'pyflakes c:\\programs\\python34\\lib\\turtle.py' returns status 1.
> whereas both (with quotes elided and undoubled \) work at the command 
> prompt.
> 

Finally found out what the problem is:
When I'm running your command using the cmd console, I get this output:

c:\python34\lib\turtle.py:571: local variable 'rgb' is assigned to but never
used
c:\python34\lib\turtle.py:2936: local variable 'a21' is assigned to but
never used
c:\python34\lib\turtle.py:3590: local variable 'dummy' is assigned to but
never used
c:\python34\lib\turtle.py:3786: undefined name 'mainloop'
c:\python34\lib\turtle.py:3969: undefined name 'mainloop'
c:\python34\lib\turtle.py:3973: undefined name 'isdown'
c:\python34\lib\turtle.py:3974: undefined name 'pu'
c:\python34\lib\turtle.py:3976: undefined name 'pd'
..

now look at the exit code:
echo %errorlevel%
1

ah, pyflakes does exit with a non-zero exit code when it finds errors in
your file!!

Now, using subprocess.check_output in IDLE:

>>> msg=subprocess.check_output(r'pyflakes c:\python34\lib\turtle.py')
Traceback (most recent call last):
  File "<pyshell#45>", line 1, in <module>
    msg=subprocess.check_output(r'pyflakes c:\python34\lib\turtle.py')
  File "C:\Python34\lib\subprocess.py", line 618, in check_output
    raise CalledProcessError(retcode, process.args, output=output)
subprocess.CalledProcessError: Command 'pyflakes c:\python34\lib\turtle.py'
returned non-zero exit status 1

as you said, but logical since (taken from the docs):

"subprocess.check_output(args, *, input=None, stdin=None, stderr=None,
shell=False, universal_newlines=False, timeout=None) 
Run command with arguments and return its output.

If the return code was non-zero it raises a CalledProcessError. The
CalledProcessError object will have the return code in the returncode
attribute and any output in the output attribute."

and in fact:
>>> try:
	msg=subprocess.check_output(r'pyflakes c:\python34\lib\turtle.py')
except subprocess.CalledProcessError as e:
	print (e.output[:81])

	
b"c:\\python34\\lib\\turtle.py:571: local variable 'rgb' is assigned to but
never used"

So, everything's just fine except that it may be more convenient to use
Popen().communicate() to avoid raising the error in the first place :)

Hope it helps this time,
Wolfgang

[toc] | [prev] | [next] | [standalone]

#73861

On 07/02/2014 04:22 PM, Wolfgang Maier wrote:
>
> So, everything's just fine except that it may be more convenient to use
> Popen().communicate() to avoid raising the error in the first place :)

Nice sleuthing!  :)

--
~Ethan~

[toc] | [prev] | [next] | [standalone]

#73874

On 7/2/2014 7:22 PM, Wolfgang Maier wrote:

> Finally found out what the problem is:
> When I'm running your command using the cmd console, I get this output:
>
> c:\python34\lib\turtle.py:571: local variable 'rgb' is assigned to but never
> used
> c:\python34\lib\turtle.py:2936: local variable 'a21' is assigned to but
> never used
> c:\python34\lib\turtle.py:3590: local variable 'dummy' is assigned to but
> never used
> c:\python34\lib\turtle.py:3786: undefined name 'mainloop'
> c:\python34\lib\turtle.py:3969: undefined name 'mainloop'
> c:\python34\lib\turtle.py:3973: undefined name 'isdown'
> c:\python34\lib\turtle.py:3974: undefined name 'pu'
> c:\python34\lib\turtle.py:3976: undefined name 'pd'
> ..
>
> now look at the exit code:
> echo %errorlevel%
> 1

I did not know about this.

> ah, pyflakes does exit with a non-zero exit code when it finds errors in
> your file!!

I guessed that and confirmed it in pyflakes.api.main().

The same is true with >pyflakes c:/programs/python34/lib (path for my 
machine), except that then there is 20 or more times as much output.

> Now, using subprocess.check_output in IDLE:
>
>>>> msg=subprocess.check_output(r'pyflakes c:\python34\lib\turtle.py')
> Traceback (most recent call last):
>    File "<pyshell#45>", line 1, in <module>
>      msg=subprocess.check_output(r'pyflakes c:\python34\lib\turtle.py')
>    File "C:\Python34\lib\subprocess.py", line 618, in check_output
>      raise CalledProcessError(retcode, process.args, output=output)
> subprocess.CalledProcessError: Command 'pyflakes c:\python34\lib\turtle.py'
> returned non-zero exit status 1

Yes, but what puzzled me is that running
subprocess.check_output(r'pyflakes c:\programs\python34\lib')
in the regular interpreter *does* produce output instead of the error 
message. My guess is that it fills up the pipe, so that check_output 
starts reading the pipe long before pyflakes exits with status 1.

Hmm. I tried it again, and I see some but not all of the output I got at 
the command line *and* I see the exit status message. So the subprocess 
must get some of the output but then stop when it sees the exit status. 
  Thanks.

> as you said, but logical since (taken from the docs):
>
> "subprocess.check_output(args, *, input=None, stdin=None, stderr=None,
> shell=False, universal_newlines=False, timeout=None)
> Run command with arguments and return its output.
>
> If the return code was non-zero it raises a CalledProcessError. The
> CalledProcessError object will have the return code in the returncode
> attribute and any output in the output attribute."
>
> and in fact:
>>>> try:
> 	msg=subprocess.check_output(r'pyflakes c:\python34\lib\turtle.py')
> except subprocess.CalledProcessError as e:
> 	print (e.output[:81])
>
> 	
> b"c:\\python34\\lib\\turtle.py:571: local variable 'rgb' is assigned to but
> never used"
>
> So, everything's just fine except that it may be more convenient to use
> Popen().communicate() to avoid raising the error in the first place :)

As I believe I said in my first post, that is where we started -- but in 
Idle. I did not realize then that at least some of the problem is 
specific to Idle on Windows and not Python on Windows. It may have 
something to do with running in pythonw.exe

> Hope it helps this time,

Yes, it did, to focus my attention. I will see if I can get all of the 
output with communicate in the normal interpreter. Having it work in 
Idle is a different matter, as before it hung indefinitely without 
showing anything with
subprocess.check_output(r'pyflakes c:\programs\python34\lib')
or with the original Popen.communicate.
But I should start experiments in Idle with something that works right 
in the normal interpreter.
---
Doing that, this works on 2.7.7, 3.4.1+, and 3.5.0a
(and would work in posix also).

import subprocess as s
p = s.Popen([r'pyflakes', r'c:\programs\python34\lib\turtle.py'],
             stdout=s.PIPE, stderr=s.PIPE)
o,e = p.communicate()
for line in o.decode().splitlines(): print(line)

Next I will reapply the patch that did not work, add a debug print, and 
see how what did not work Windows (but worked on linux) is different 
from the above.

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#73886

On 07/03/2014 06:09 AM, Terry Reedy wrote:
>
> Yes, but what puzzled me is that running
> subprocess.check_output(r'pyflakes c:\programs\python34\lib')
> in the regular interpreter *does* produce output instead of the error
> message. My guess is that it fills up the pipe, so that check_output
> starts reading the pipe long before pyflakes exits with status 1.
>
> Hmm. I tried it again, and I see some but not all of the output I got at
> the command line *and* I see the exit status message. So the subprocess
> must get some of the output but then stop when it sees the exit status.
>   Thanks.
>

For a partial explanation try this:

from the command line (again my path is slightly different):

pyflakes C:\Python34\lib > stdout.txt

this will still give you a few lines of output and these should be the 
same ones you're seeing from the python interpreter when you're doing:

subprocess.check_output(r'pyflakes C:\Python34\lib')

==> pyflakes sends these lines to stderr instead of stdout !!

confirmation:

subprocess.check_output(r'pyflakes C:\Python34\lib', stderr=subprocess.PIPE)

and the output is gone.

So the remaining questions are:
- why on earth is pyflakes sending these lines (and only these) to stderr ?

- what is happening to the stderr output when run in IDLE ? I guess it 
is caught and suppressed somewhere, but to add to your observations the 
check_output call doesn't hang on IDLE, but finishes eventually with no 
output other than the traceback.

Best wishes,
Wolfgang

[toc] | [prev] | [next] | [standalone]

#73889

On 07/03/2014 10:03 AM, Wolfgang Maier wrote:
> On 07/03/2014 06:09 AM, Terry Reedy wrote:
>
> - what is happening to the stderr output when run in IDLE ? I guess it
> is caught and suppressed somewhere, but to add to your observations the
> check_output call doesn't hang on IDLE, but finishes eventually with no
> output other than the traceback.

stderr output not being displayed in IDLE is mentioned in 
http://bugs.python.org/issue13582 .

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web