Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #90077 > unrolled thread

PEP idea: On Windows, subprocess should implicitly support .bat and .cmd scripts by using FindExecutable from win32 API

Back to article view | Back to comp.lang.python

Contents

  PEP idea: On Windows, subprocess should implicitly support .bat and .cmd scripts by using FindExecutable from win32 API Stefan Zimmermann <zimmermann.code@gmail.com> - 2015-05-06 15:11 -0700
    Re: PEP idea: On Windows, subprocess should implicitly support .bat and .cmd scripts by using FindExecutable from win32 API Dave Angel <davea@davea.name> - 2015-05-06 20:58 -0400
    Re: PEP idea: On Windows, subprocess should implicitly support .bat and .cmd scripts by using FindExecutable from win32 API Chris Angelico <rosuav@gmail.com> - 2015-05-07 12:19 +1000
      Re: PEP idea: On Windows, subprocess should implicitly support .bat and .cmd scripts by using FindExecutable from win32 API Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-05-07 13:33 +1000
        Re: PEP idea: On Windows, subprocess should implicitly support .bat and .cmd scripts by using FindExecutable from win32 API Chris Angelico <rosuav@gmail.com> - 2015-05-07 13:57 +1000
        Re: PEP idea: On Windows, subprocess should implicitly support .bat and .cmd scripts by using FindExecutable from win32 API Gisle Vanem <gvanem@yahoo.no> - 2015-05-07 09:15 +0200
          Re: PEP idea: On Windows, subprocess should implicitly support .bat and .cmd scripts by using FindExecutable from win32 API Stefan Zimmermann <zimmermann.code@gmail.com> - 2015-05-07 02:38 -0700
            Re: PEP idea: On Windows, subprocess should implicitly support .bat and .cmd scripts by using FindExecutable from win32 API Stefan Zimmermann <zimmermann.code@gmail.com> - 2015-05-07 03:03 -0700
              Re: PEP idea: On Windows, subprocess should implicitly support .bat and .cmd scripts by using FindExecutable from win32 API Marko Rauhamaa <marko@pacujo.net> - 2015-05-07 13:10 +0300
                Re: PEP idea: On Windows, subprocess should implicitly support .bat and .cmd scripts by using FindExecutable from win32 API Chris Angelico <rosuav@gmail.com> - 2015-05-07 20:24 +1000
                Re: PEP idea: On Windows, subprocess should implicitly support .bat and .cmd scripts by using FindExecutable from win32 API Dave Angel <davea@davea.name> - 2015-05-07 07:28 -0400
                Re: PEP idea: On Windows, subprocess should implicitly support .bat and .cmd scripts by using FindExecutable from win32 API Chris Angelico <rosuav@gmail.com> - 2015-05-07 21:43 +1000
                  Re: PEP idea: On Windows, subprocess should implicitly support .bat and .cmd scripts by using FindExecutable from win32 API Marko Rauhamaa <marko@pacujo.net> - 2015-05-07 15:41 +0300
                    Re: PEP idea: On Windows, subprocess should implicitly support .bat and .cmd scripts by using FindExecutable from win32 API Chris Angelico <rosuav@gmail.com> - 2015-05-07 22:53 +1000
                      Re: PEP idea: On Windows, subprocess should implicitly support .bat and .cmd scripts by using FindExecutable from win32 API Marko Rauhamaa <marko@pacujo.net> - 2015-05-07 16:44 +0300
                        Re: PEP idea: On Windows, subprocess should implicitly support .bat and .cmd scripts by using FindExecutable from win32 API Chris Angelico <rosuav@gmail.com> - 2015-05-08 00:03 +1000
                          Re: PEP idea: On Windows, subprocess should implicitly support .bat and .cmd scripts by using FindExecutable from win32 API Marko Rauhamaa <marko@pacujo.net> - 2015-05-07 18:24 +0300
                            Re: PEP idea: On Windows, subprocess should implicitly support .bat and .cmd scripts by using FindExecutable from win32 API Stefan Zimmermann <zimmermann.code@gmail.com> - 2015-05-07 08:45 -0700
                              Re: PEP idea: On Windows, subprocess should implicitly support .bat and .cmd scripts by using FindExecutable from win32 API Marko Rauhamaa <marko@pacujo.net> - 2015-05-07 21:13 +0300
                                Re: PEP idea: On Windows, subprocess should implicitly support .bat and .cmd scripts by using FindExecutable from win32 API Stefan Zimmermann <zimmermann.code@gmail.com> - 2015-05-07 16:27 -0700
                            Re: PEP idea: On Windows, subprocess should implicitly support .bat and .cmd scripts by using FindExecutable from win32 API Chris Angelico <rosuav@gmail.com> - 2015-05-08 11:50 +1000
                            Re: PEP idea: On Windows, subprocess should implicitly support .bat and .cmd scripts by using FindExecutable from win32 API Ben Finney <ben+python@benfinney.id.au> - 2015-05-08 12:26 +1000
                              Re: PEP idea: On Windows, subprocess should implicitly support .bat and .cmd scripts by using FindExecutable from win32 API Marko Rauhamaa <marko@pacujo.net> - 2015-05-08 09:14 +0300
                        Re: PEP idea: On Windows, subprocess should implicitly support .bat and .cmd scripts by using FindExecutable from win32 API Ian Kelly <ian.g.kelly@gmail.com> - 2015-05-07 09:14 -0600
                        Re: PEP idea: On Windows, subprocess should implicitly support .bat and .cmd scripts by using FindExecutable from win32 API Chris Angelico <rosuav@gmail.com> - 2015-05-08 11:46 +1000

Page 1 of 2  [1] 2  Next page →

#90077 — PEP idea: On Windows, subprocess should implicitly support .bat and .cmd scripts by using FindExecutable from win32 API

Hi.

I don't like that subprocess.Popen(['command']) only works on Windows if there is a command.exe in %PATH%. As a Windows user you would normally expect that also command.bat and command.cmd can be run that way.

There are simple workarounds like Popen(..., shell=True) but that is a heavy overhead for .exe files.

Currently I use pywin32 and call Popen([win32api.FindExecutable('command')[1]]) as a workaround. This has zero overhead.

It should be default for Popen to call FindExecutable internally.

Was this discussed before?
Is it worth a PEP?
Or at least an issue?

Cheers,
Stefan

[toc] | [next] | [standalone]

#90081

On 05/06/2015 06:11 PM, Stefan Zimmermann wrote:
> Hi.
>
> I don't like that subprocess.Popen(['command']) only works on Windows if there is a command.exe in %PATH%.

  As a Windows user you would normally expect that also command.bat and 
command.cmd can be run that way.
>

and command.com.

If it's really unfortunate that you picked "command" for your sample 
program name.  Since command.com was the shell in MSDOS, I was about to 
point you to COMSPEC to address your problem.

There's nothing Windows-specific about that behaviour.  In Linux, there 
are bash commands that can only be run by using shell=True.  Fortunately 
Popen didn't make the mistake of pretending it's a shell.

There is lots more to running a batch file than launching it.  The whole 
syntax of the rest of the commandline differs when you're doing that.

> There are simple workarounds like Popen(..., shell=True) but that is a heavy overhead for .exe files.

And the reason there's such an overhead is because you're requesting the 
services of the shell.  If you don't need those services, use shell=False.

>
> Currently I use pywin32 and call Popen([win32api.FindExecutable('command')[1]]) as a workaround. This has zero overhead.
>
> It should be default for Popen to call FindExecutable internally.
>
> Was this discussed before?
> Is it worth a PEP?
> Or at least an issue?
>
> Cheers,
> Stefan
>


-- 
DaveA

[toc] | [prev] | [next] | [standalone]

#90085

On Thu, May 7, 2015 at 10:58 AM, Dave Angel <davea@davea.name> wrote:
> There's nothing Windows-specific about that behaviour.  In Linux, there are
> bash commands that can only be run by using shell=True.  Fortunately Popen
> didn't make the mistake of pretending it's a shell.

But bash commands aren't the same as shell scripts. For instance, if
you want to enumerate bash aliases, you can't exec() to the 'alias'
command, because there isn't one. But shell scripts *can* be exec'd:

$ grep $ exec_demo.*
exec_demo.c:#include <stdio.h>
exec_demo.c:#include <unistd.h>
exec_demo.c:int main()
exec_demo.c:{
exec_demo.c: printf("This part is coming from C code.\n");
exec_demo.c: int err=execl("./exec_demo.sh", 0);
exec_demo.c: printf("exec() failed! %d\n",err);
exec_demo.c:}
exec_demo.sh:#!/bin/sh
exec_demo.sh:echo This part ran from the shell.
exec_demo.sh:echo Hello, world!
$ ./a.out
This part is coming from C code.
This part ran from the shell.
Hello, world!
$ pike -e 'Process.exec("./exec_demo.sh");'
This part ran from the shell.
Hello, world!
$ python -c 'import subprocess; subprocess.call(["./exec_demo.sh"])'
This part ran from the shell.
Hello, world!
(Python doesn't seem to have any way to 'exec', but a subprocess comes
to the same thing.)

I don't know about Windows, but it seems reasonable to be able to be
able to run many types of program equally, including batch files. But
maybe Windows is just weak that way.

ChrisA

[toc] | [prev] | [next] | [standalone]

#90087

On Thursday 07 May 2015 12:19, Chris Angelico wrote:

> On Thu, May 7, 2015 at 10:58 AM, Dave Angel <davea@davea.name> wrote:
>> There's nothing Windows-specific about that behaviour.  In Linux, there
>> are
>> bash commands that can only be run by using shell=True.  Fortunately
>> Popen didn't make the mistake of pretending it's a shell.
> 
> But bash commands aren't the same as shell scripts. For instance, if
> you want to enumerate bash aliases, you can't exec() to the 'alias'
> command, because there isn't one. But shell scripts *can* be exec'd:

Um, are we still talking about Python here? exec("alias") fails with 
NameError on all the versions of Python I know. *semi-wink*

I'm guessing you're taking about some other exec, it might be a good idea 
that on a Python mailing list you don't assume that we're all going to 
understand the context :-)


> $ grep $ exec_demo.*
> exec_demo.c:#include <stdio.h>
> exec_demo.c:#include <unistd.h>
> exec_demo.c:int main()
> exec_demo.c:{
> exec_demo.c: printf("This part is coming from C code.\n");
> exec_demo.c: int err=execl("./exec_demo.sh", 0);
> exec_demo.c: printf("exec() failed! %d\n",err);
> exec_demo.c:}
> exec_demo.sh:#!/bin/sh
> exec_demo.sh:echo This part ran from the shell.
> exec_demo.sh:echo Hello, world!
> $ ./a.out
> This part is coming from C code.
> This part ran from the shell.
> Hello, world!

> $ pike -e 'Process.exec("./exec_demo.sh");'
> This part ran from the shell.
> Hello, world!

Okay, so C code can call the shell. So can Pike.


> $ python -c 'import subprocess; subprocess.call(["./exec_demo.sh"])'
> This part ran from the shell.
> Hello, world!

And so can Python. I'm not entirely sure what point you are trying to make 
here, or how it relates to the OP's problem that when he calls 

subprocess.Popen(['foo'])

he expects it to run any of foo.exe, foo.cmd, foo.bat (and possibly any 
other number of executable files). Are you agreeing with him or disagreeing?

Apart from any other number of problems, surely having "foo" alone run 
foo.exe, foo.bat etc. is at best confusing and at worst a security risk? 
What if you have *both* foo.exe and foo.bat in the same directory?


> (Python doesn't seem to have any way to 'exec', but a subprocess comes
> to the same thing.)

According to `man exec` on my Linux system, I don't think that is correct. 
The exec* family of functions "replaces the current process image with a new 
process image", they don't run in a subprocess.

I think the Python equivalent of Unix exec* commands are the various 
os.exec* functions.


> I don't know about Windows, but it seems reasonable to be able to be
> able to run many types of program equally, including batch files. But
> maybe Windows is just weak that way.


Hmmm. I'm not sure if this is relevant, or if I'm going off on a tangent, 
but if I write a short bash script and set the execute permission:

steve@runes:~$ chmod u+x test.sh
steve@runes:~$ cat test.sh 
echo "Running shell script"


subprocess.call fails unless I set shell=True:


py> p = subprocess.Popen('./test.sh', shell=True)
py> Running shell script

py> p = subprocess.Popen('./test.sh', shell=False)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/local/lib/python2.7/subprocess.py", line 1308, in 
_execute_child
    raise child_exception
OSError: [Errno 8] Exec format error


How is this any different from needing to specify shell=True for .bat and 
.cmd files under Windows? This is not a rhetorical question, I actually want 
to know.



-- 
Steve

[toc] | [prev] | [next] | [standalone]

#90088

On Thu, May 7, 2015 at 1:33 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> On Thursday 07 May 2015 12:19, Chris Angelico wrote:
>
>> On Thu, May 7, 2015 at 10:58 AM, Dave Angel <davea@davea.name> wrote:
>>> There's nothing Windows-specific about that behaviour.  In Linux, there
>>> are
>>> bash commands that can only be run by using shell=True.  Fortunately
>>> Popen didn't make the mistake of pretending it's a shell.
>>
>> But bash commands aren't the same as shell scripts. For instance, if
>> you want to enumerate bash aliases, you can't exec() to the 'alias'
>> command, because there isn't one. But shell scripts *can* be exec'd:
>
> Um, are we still talking about Python here? exec("alias") fails with
> NameError on all the versions of Python I know. *semi-wink*
>
> I'm guessing you're taking about some other exec, it might be a good idea
> that on a Python mailing list you don't assume that we're all going to
> understand the context :-)

Fair point. I'm talking about the underlying exec* family of
functions, which are used on Unix-like systems to do normal process
execution. More on that later.

>> $ grep $ exec_demo.*
>> exec_demo.c:#include <stdio.h>
>> exec_demo.c:#include <unistd.h>
>> exec_demo.c:int main()
>> exec_demo.c:{
>> exec_demo.c: printf("This part is coming from C code.\n");
>> exec_demo.c: int err=execl("./exec_demo.sh", 0);
>> exec_demo.c: printf("exec() failed! %d\n",err);
>> exec_demo.c:}
>> exec_demo.sh:#!/bin/sh
>> exec_demo.sh:echo This part ran from the shell.
>> exec_demo.sh:echo Hello, world!
>> $ ./a.out
>> This part is coming from C code.
>> This part ran from the shell.
>> Hello, world!
>
>> $ pike -e 'Process.exec("./exec_demo.sh");'
>> This part ran from the shell.
>> Hello, world!
>
> Okay, so C code can call the shell. So can Pike.
>
>
>> $ python -c 'import subprocess; subprocess.call(["./exec_demo.sh"])'
>> This part ran from the shell.
>> Hello, world!
>
> And so can Python. I'm not entirely sure what point you are trying to make
> here, or how it relates to the OP's problem that when he calls
>
> subprocess.Popen(['foo'])
>
> he expects it to run any of foo.exe, foo.cmd, foo.bat (and possibly any
> other number of executable files). Are you agreeing with him or disagreeing?

I'm stating that this works on Unix with shell=False. Consequently,
the OP's request that it work thusly on Windows ought to be possible,
and ought to be reasonable.

> Apart from any other number of problems, surely having "foo" alone run
> foo.exe, foo.bat etc. is at best confusing and at worst a security risk?
> What if you have *both* foo.exe and foo.bat in the same directory?

There's a specific search order. Back in the days of DOS, it was
simply "com, then exe, then bat", but on modern Windowses, I think
it's governed by an environment variable (similarly to PATH for the
directories to search in). It's identical security risk to the
possibility of putting something earlier in $PATH; if you insist on
running a specific executable, you put the entire path, and then
there's no problem. (Actually, I'd consider this to be a feature, not
a bug - it's the equivalent of shadowing built-ins in a Python module.
When you want it, it's really useful.)

>> (Python doesn't seem to have any way to 'exec', but a subprocess comes
>> to the same thing.)
>
> According to `man exec` on my Linux system, I don't think that is correct.
> The exec* family of functions "replaces the current process image with a new
> process image", they don't run in a subprocess.
>
> I think the Python equivalent of Unix exec* commands are the various
> os.exec* functions.

Oh, I forgot about os.exec. That's a better equivalent:

$ python3 -c 'import os; os.execl("./exec_demo.sh","exec_demo.sh")'
This part ran from the shell.
Hello, world!

When you call subprocess.* to run something, what happens is generally
that Python forks and exec's to the new process. The parent of the
fork keeps running (or maybe waits on the child immediately), the
child uses one of the exec family of functions to replace itself with
the new process. This is different from Windows, where there's a
standard API function for "start this program in a new process".
Advantages of the fork/exec model include that you can do stuff in
between the two steps - for instance, you fork, then you change your
user credentials, root directory, current directory, nice value, usage
limits, etc, etc, etc, etc, prior to exec'ing. Unix systems don't need
a way to say "run this process with this root directory", because it
can all be built up from primitives.

So yes, in terms of the ability to locate other executables, it makes
no difference whether you fork first or not - the exec call is the
same.

> Hmmm. I'm not sure if this is relevant, or if I'm going off on a tangent,
> but if I write a short bash script and set the execute permission:
>
> steve@runes:~$ chmod u+x test.sh
> steve@runes:~$ cat test.sh
> echo "Running shell script"
>
> subprocess.call fails unless I set shell=True:
>
> py> p = subprocess.Popen('./test.sh', shell=True)
> py> Running shell script
>
> py> p = subprocess.Popen('./test.sh', shell=False)
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "/usr/local/lib/python2.7/subprocess.py", line 711, in __init__
>     errread, errwrite)
>   File "/usr/local/lib/python2.7/subprocess.py", line 1308, in
> _execute_child
>     raise child_exception
> OSError: [Errno 8] Exec format error
>
>
> How is this any different from needing to specify shell=True for .bat and
> .cmd files under Windows? This is not a rhetorical question, I actually want
> to know.

Hmm... hm... Ha! Found the difference. I had an explicit shebang on my
script; yours just starts out with shell commands. That means that
your shell script wasn't truly executable, and thus requires a shell
to execute it. Try adding "#!/bin/sh" to the top and rerun that - at
that point, it becomes kernel-executable instead of just
shell-executable.

ChrisA

[toc] | [prev] | [next] | [standalone]

#90089

Chris Angelico wrote:

> There's a specific search order. Back in the days of DOS, it was
> simply "com, then exe, then bat", but on modern Windowses, I think
> it's governed by an environment variable.

You probably mean '%PATHEXT'. Mine is:
  .COM;.EXE;.BAT;.BTM;.CMD;.JS;.JSE;.WSF;.WSH;.MSC;.tcl;.py;.pyw;.pl;.htm;.html

In my favourite shell 4NT, I simply can have:
   set .py=python

Instead of the Explorer associations that the Python-installer puts
in my registry. Revealed from my shell:
   c:\> assoc .py
    .py=py_auto_file
   c:\> ftype py_auto_file
    py_auto_file="F:\ProgramFiler\Python27\python.exe" "%1"

In ShellExecuteEx(), what program gets launched for "py_auto_file" in this
case, seems to be determined by the 'SHELLEXECUTEINFO:lpClass' member.
I fail to see that Python uses this structure anywhere.

-- 
--gv

[toc] | [prev] | [next] | [standalone]

#90092

Nice to see that my topic gains that interest :)
And I see that I should have gone more into detail about what I'm actually trying to point out.

Chris Angelico wrote:
> Hmm... hm... Ha! Found the difference. I had an explicit shebang on my
> script; yours just starts out with shell commands. That means that
> your shell script wasn't truly executable, and thus requires a shell
> to execute it. Try adding "#!/bin/sh" to the top and rerun that - at
> that point, it becomes kernel-executable instead of just
> shell-executable.

That's the big advantage of Unix. You can write an kernel-executable script without any file extension, just by putting a shebang to the beginning of that file. And for the caller it makes no difference if 'command' is a binary or a script and Popen('command') works in both cases, without the shell=True overhead.

Steven D'Aprano wrote:
> Apart from any other number of problems, surely having "foo" alone run 
> foo.exe, foo.bat etc. is at best confusing and at worst a security risk? 
> What if you have *both* foo.exe and foo.bat in the same directory?

On Unix you can shadow any binary with a wrapper script of the same name located in a path appearing earlier in $PATH. Any caller will automatically run your script instead of the original binary. An that's usually seen as a big advantage on Unix.
On Windows executability depends on the file extension and if you want to wrap some command.exe you usually write a command.bat in a path with higher precedence. And in Windows it's standard that .exe, .com, .bat and .cmd files should be callable without writing the file extension.
And as already mentioned, there is a defined precedence order if they are in the same directory.
That's not more or less security risky as shadowing binaries with scripts on Unix.

My point is that compared to Unix it's just a big disadvantage on Windows that the subprocess.Popen(['command']) can only call command.exe implicitly, which makes it impossible to work with custom wrapper .bat or .cmd scripts without the shell=True overhead.
And this is acutally confusing for a Windows user.
You write a wrapper .bat for some .exe and are wondering why your Python script doesn't use it.

And the FindExecutable() function from the win32 API would just be the perfect solution for implementing this.

[toc] | [prev] | [next] | [standalone]

#90094

And last but not least, Popen behavior on Windows makes it difficult to write OS-independent Python code which calls external commands that are not binary by default:

2 examples:

1. I wrote a coffeetools package which wraps the CoffeeScript compiler in a Python API. The 'coffee' command is a Node.js script. And under Windows it is installed with a 'coffee.cmd' wrapper to make it callable. So to make Popen work you have to switch and call 'coffee' on Unix and 'coffee.cmd' on Windows. But from the Windows shell you can just call 'coffee'. Maybe in the future the .cmd changes to .bat ...

2. I the embedded portable git from SourceTree instead of the standard Windows git installation. It has a git.bat wrapper which calls the internal git.exe (which must be in the same dir with a lot of other included ported Unix tools and therefore not recommended to add that dir to PATH). That made the dulwich package unworkable for me because it just tries to Popen(['git', ...]). And I am currently trying to make the dulwich developers accept my pull request with a workaround...

[toc] | [prev] | [next] | [standalone]

#90095

Stefan Zimmermann <zimmermann.code@gmail.com>:

> And last but not least, Popen behavior on Windows makes it difficult
> to write OS-independent Python code which calls external commands that
> are not binary by default:

Then, write OS-dependent Python code.

I don't think it's Python's job to pave over OS differences. Java does
that by not offering precious system facilities -- very painful. Python
is taking steps in that direction, but I hope it won't go too far.


Marko

[toc] | [prev] | [next] | [standalone]

#90096

On Thu, May 7, 2015 at 8:10 PM, Marko Rauhamaa <marko@pacujo.net> wrote:
> Stefan Zimmermann <zimmermann.code@gmail.com>:
>
>> And last but not least, Popen behavior on Windows makes it difficult
>> to write OS-independent Python code which calls external commands that
>> are not binary by default:
>
> Then, write OS-dependent Python code.
>
> I don't think it's Python's job to pave over OS differences. Java does
> that by not offering precious system facilities -- very painful. Python
> is taking steps in that direction, but I hope it won't go too far.

On the contrary, I think it *is* a high level language's job to pave
over those differences. Portable C code generally has to have a
whopping 'configure' script that digs into your hardware, OS, library,
etc availabilities, and lets you figure out which way to do things.
Python code shouldn't need to worry about that. You don't need to care
whether you're on a 32-bit or 64-bit computer; you don't need to care
whether it's an Intel chip or a RISCy one; you shouldn't have to
concern yourself with the difference between BSD networking and
WinSock. There'll be a handful of times when you do care, and for
those, it's nice to have some facilities exposed; but the bulk of code
shouldn't need to know about the platform it's running on.

Java went for a philosophy of "write once, run anywhere" in its early
days, and while that hasn't exactly been stuck to completely, it's
still the reasoning behind the omission of certain system facilities.
Python accepts and understands that there will be differences, so you
can't call os.getuid() on Windows, and there are a few restrictions on
the subprocess module if you want maximum portability, but the bulk of
your code won't be any different on Linux, Windows, Mac OS, OS/2,
Amiga, OS/400, Solaris, or a MicroPython board.

ChrisA

[toc] | [prev] | [next] | [standalone]

#90097

On 05/07/2015 06:24 AM, Chris Angelico wrote:
> On Thu, May 7, 2015 at 8:10 PM, Marko Rauhamaa <marko@pacujo.net> wrote:
>> Stefan Zimmermann <zimmermann.code@gmail.com>:
>>
>>> And last but not least, Popen behavior on Windows makes it difficult
>>> to write OS-independent Python code which calls external commands that
>>> are not binary by default:
>>
>> Then, write OS-dependent Python code.
>>
>> I don't think it's Python's job to pave over OS differences. Java does
>> that by not offering precious system facilities -- very painful. Python
>> is taking steps in that direction, but I hope it won't go too far.
>
> On the contrary, I think it *is* a high level language's job to pave
> over those differences. Portable C code generally has to have a
> whopping 'configure' script that digs into your hardware, OS, library,
> etc availabilities, and lets you figure out which way to do things.
> Python code shouldn't need to worry about that. You don't need to care
> whether you're on a 32-bit or 64-bit computer; you don't need to care
> whether it's an Intel chip or a RISCy one; you shouldn't have to
> concern yourself with the difference between BSD networking and
> WinSock. There'll be a handful of times when you do care, and for
> those, it's nice to have some facilities exposed; but the bulk of code
> shouldn't need to know about the platform it's running on.
>
> Java went for a philosophy of "write once, run anywhere" in its early
> days, and while that hasn't exactly been stuck to completely, it's
> still the reasoning behind the omission of certain system facilities.
> Python accepts and understands that there will be differences, so you
> can't call os.getuid() on Windows, and there are a few restrictions on
> the subprocess module if you want maximum portability, but the bulk of
> your code won't be any different on Linux, Windows, Mac OS, OS/2,
> Amiga, OS/400, Solaris, or a MicroPython board.
>
> ChrisA
>

It's a nice goal.  But these aren't OS features in Windows, they're 
shell features.  And there are several shells.  If the user has 
installed a different shell, is it Python's job to ignore it and 
simulate what cmd.exe does?

Seems to me that's what shell=True is for.  it signals Python that we're 
willing to trust the shell to do whatever magic it chooses, from adding 
extensions, to calling interpreters, to changing search order, to 
parsing the line in strange ways, to setting up temporary environment 
contexts, etc.

If there were just one shell, it might make sense to emulate its 
features.  Or it might make sense to contort its features to look like a 
Unix shell.  But with multiple possibilities, seems that's more like 
space for a 3rd party library.

-- 
DaveA

[toc] | [prev] | [next] | [standalone]

#90098

On Thu, May 7, 2015 at 9:28 PM, Dave Angel <davea@davea.name> wrote:
> It's a nice goal.  But these aren't OS features in Windows, they're shell
> features.  And there are several shells.  If the user has installed a
> different shell, is it Python's job to ignore it and simulate what cmd.exe
> does?

It might be an unattainable goal (in fact, it almost certainly is),
but I was specifically disagreeing with the notion that it's right and
normal to write a bunch of platform-specific code in Python. That
should be the rarity.

ChrisA

[toc] | [prev] | [next] | [standalone]

#90099

Chris Angelico <rosuav@gmail.com>:

> I was specifically disagreeing with the notion that it's right and
> normal to write a bunch of platform-specific code in Python. That
> should be the rarity.

Why is that?

Code is written for a specific need and environment. Often trying to
write generic solutions leads to cumbersome and clunky results on *all*
platforms.

A software system is defined through its interfaces. Natural system
interfaces are very different under different operating systems. The
chosen programming language for whatever component is often an
afterthought. I'm glad I can still write native Linux code using Python.
I couldn't do that with Java, which doesn't have things like os.fork(),
file descriptors or process ids.


Marko

[toc] | [prev] | [next] | [standalone]

#90100

On Thu, May 7, 2015 at 10:41 PM, Marko Rauhamaa <marko@pacujo.net> wrote:
> Chris Angelico <rosuav@gmail.com>:
>
>> I was specifically disagreeing with the notion that it's right and
>> normal to write a bunch of platform-specific code in Python. That
>> should be the rarity.
>
> Why is that?
>
> Code is written for a specific need and environment. Often trying to
> write generic solutions leads to cumbersome and clunky results on *all*
> platforms.
>
> A software system is defined through its interfaces.

And the most important interface is with a human. Humans are the same
whether you're running under Windows, Linux, or anything else. If you
want to write single-platform code, go for it; but if you want to
write cross-platform code, the best way is to let someone else take
care of the differences, abstracting them away into a nice tidy thing
that we call a high-level language.

I don't need forking, file descriptors, or process IDs, to describe
how a person uses my code. Those are *implementation details*. Now, it
might be that I have to concern myself with some of them. Maybe I want
to get optimal performance out of something, and that means using
multiple processes and managing them properly. Maybe I need to
interface with systemd, respond to dozens of different process-level
signals, use directory notifications, and do a bunch of other
Linux-only things, so maybe it's just completely impractical to
consider supporting even BSD-based Unixes, much less Windows. So be
it. But to the greatest extent possible, Python should let me write
code that doesn't care about any of that.

ChrisA

[toc] | [prev] | [next] | [standalone]

#90101

Chris Angelico <rosuav@gmail.com>:

>> A software system is defined through its interfaces.
>
> And the most important interface is with a human.

I barely ever program anything for the human interface.

> If you want to write single-platform code, go for it; but if you want
> to write cross-platform code, the best way is to let someone else take
> care of the differences, abstracting them away into a nice tidy thing
> that we call a high-level language.

You suggested most software should be platform-agnostic. Now you are
qualifying the statement.

But still, I challenge the notion that you could write a web site, game
or application that feels natural on the XBox, iPhone, Windows PC and
LXDE at the same time without significant amounts of
platform-conditioned parts.

> I don't need forking, file descriptors, or process IDs, to describe
> how a person uses my code. Those are *implementation details*.

Even if I programmed for the human and the UI experience were
more-or-less identical between platforms, the system interfaces can be
conceptually quite different. Heroic attempts have been made to overcome
those differences with generic APIs. However, Python should stay out of
that crusade.

Whole programming cultures, idioms and "right ways" differ between
platforms. What's the right way to write a service (daemon)? That's
probably completely different between Windows and Linux. Linux itself is
undergoing a biggish transformation there: an exemplary daemon of last
year will likely be deprecated within a few years.


Marko

[toc] | [prev] | [next] | [standalone]

#90102

On Thu, May 7, 2015 at 11:44 PM, Marko Rauhamaa <marko@pacujo.net> wrote:
> Chris Angelico <rosuav@gmail.com>:
>
>>> A software system is defined through its interfaces.
>>
>> And the most important interface is with a human.
>
> I barely ever program anything for the human interface.
>
>> If you want to write single-platform code, go for it; but if you want
>> to write cross-platform code, the best way is to let someone else take
>> care of the differences, abstracting them away into a nice tidy thing
>> that we call a high-level language.
>
> You suggested most software should be platform-agnostic. Now you are
> qualifying the statement.

I'm qualifying it because it's impossible to write 100%
platform-agnostic code without restricting yourself far too much; but
that doesn't mean that it isn't a worthwhile aim.

> But still, I challenge the notion that you could write a web site, game
> or application that feels natural on the XBox, iPhone, Windows PC and
> LXDE at the same time without significant amounts of
> platform-conditioned parts.

Hmm, you're picking up some very different things there. When a human
picks up an iPhone, s/he expects to use it with a touch-based
interface; I don't know what the normal UI for an Xbox is, but Xbox
users would; and the most normal interface for LXDE would be a
mouse+keyboard. The ideal UI for each of them will differ. This is the
same as coding your application differently if you expect a blind
person to use it, or if you want to make it possible to use your
program in a theatre without disturbing the audience, or any other UI
constraint you wish to concoct. That's nothing to do with platform. If
you write a program for Ubuntu, it might go onto a tablet or a
desktop, and the ideal UI for those is (in my opinion, though not
apparently in Unity's) different.

But if you design your program to be used with the same fundamental
human interface - say, a mouse and a keyboard - then you should be
able to do that the same way on many platforms. I've seen libraries
that let you build an ncurses-like interface or a full GUI window,
using exactly the same application code. It's not difficult.

>> I don't need forking, file descriptors, or process IDs, to describe
>> how a person uses my code. Those are *implementation details*.
>
> Even if I programmed for the human and the UI experience were
> more-or-less identical between platforms, the system interfaces can be
> conceptually quite different. Heroic attempts have been made to overcome
> those differences with generic APIs. However, Python should stay out of
> that crusade.
>
> Whole programming cultures, idioms and "right ways" differ between
> platforms. What's the right way to write a service (daemon)? That's
> probably completely different between Windows and Linux. Linux itself is
> undergoing a biggish transformation there: an exemplary daemon of last
> year will likely be deprecated within a few years.

And that's where a library function can be really awesome. What's the
right way to daemonize? "import daemonize; daemonize.daemonize()"
seems good to me. Maybe there's platform-specific code in the
*implementation* of that, but in your application, no. That's the job
of a layer underneath you.

Incidentally, the way I'm seeing things shift these days is mainly
toward *not* daemonizing your services at all. That makes life a lot
easier; instead of writing special code to put yourself in the
background, you just write your code to the standard basic "glass
teletype" model, and then add a little config file that makes it run
in the background. But a Python module could provide a generic
"install as service" function, which will create a systemd config
file, or a Windows service whatever-it-is, or the equivalent on a Mac,
or an Upstart job file, or whatever it detects. Same difference. A
library takes care of all of that.

In Python, we have the 'subprocess' module. Due to Windows
limitations, you have to restrict yourself to having an importable
main file if you want perfect cross-platform compatibility, but that
doesn't affect how your code runs on Linux or Mac OS. What's the best
way to farm work off to a bunch of processes and have them communicate
their results back? You use the subprocess module, and then it doesn't
matter whether they use Unix sockets, named pipes, physical files on
the disk, or miniature nuclear explosions, they'll communicate back
just fine. And when someone develops a new platform that uses nuclear
fusion instead of fission for interprocess communication, Python's
standard library gets enhanced, and your code instantly works - you
don't have to specifically handle the new case.

That's Python's job. Abstracting away all those differences so you
don't have to look at them.

ChrisA

[toc] | [prev] | [next] | [standalone]

#90104

Chris Angelico <rosuav@gmail.com>:

> What's the best way to farm work off to a bunch of processes and have
> them communicate their results back? You use the subprocess module,
> and then it doesn't matter whether they use Unix sockets, named pipes,
> physical files on the disk, or miniature nuclear explosions, they'll
> communicate back just fine. And when someone develops a new platform
> that uses nuclear fusion instead of fission for interprocess
> communication, Python's standard library gets enhanced, and your code
> instantly works - you don't have to specifically handle the new case.
>
> That's Python's job. Abstracting away all those differences so you
> don't have to look at them.

That's the difference between our opinions: you want Python to work the
same on different OS's. I want Python's system programming facilities to
closely mirror those of C.

Take your example of subprocess.Popen. It may be essential to know that
the communication channel is a pipe with standard pipe semantics. The
child program might not be written in Python, after all. In fact, at
system design level you shouldn't care what language you use as long as
the communication interfaces are specified.

Java is lousy at system programming (while excellent in many other
respects). It has traditionally tried to avoid the associated issues by
effectively mandating that all parts of a system be written in Java.
Gladly, Python hasn't (yet) made the same mistake.


Marko

[toc] | [prev] | [next] | [standalone]

#90105

This discussion is getting really interesting and far beyond the actual topic :)
I want to add some additional thoughts on Popen:

Marko Rauhamaa wrote:
> Stefan Zimmermann:
> 
> > And last but not least, Popen behavior on Windows makes it difficult
> > to write OS-independent Python code which calls external commands that
> > are not binary by default:
> 
> Then, write OS-dependent Python code.

Then you even have to write tool-distribution-dependent code.
Especially Unix tools are often distributed in many different variants for Windows. Some installers expose .exe, some .bat, some .cmd files to the user. So you always have to explicitly support any variant. Or run everything through 'cmd /c ...', which is as mentioned a real overhead for .exe files. Or you always have to manually use the win32 API, just to call a simple external tool. 

Calling an external command should be one of the simplest tasks in a high level scripting language like Python. And that should not involve any OS-specific differences, unless you want to use some advanced process handling features which are only supported by some specific OS.

Meanwhile I checked Ruby and Perl regarding this feature. Both support it. In both langs every standard function that calls external commands (like Perl's exec() or system() or Ruby's exec() or system() or IO.popen()), whether they invoke a shell or call it directly, support running 'tool.bat' or 'tool.cmd' by just writing 'tool'. Python almost seems to be the only major scripting language which does not support this implicitly.

Dave Angel wrote:
> It's a nice goal.  But these aren't OS features in Windows, they're 
> shell features.  And there are several shells.  If the user has 
> installed a different shell, is it Python's job to ignore it and 
> simulate what cmd.exe does?

In fact, it's something between OS and shell. Yes, .bat and .cmd files are always run thgrough cmd.exe. But on the OS level they are also condsidered executable files. And that doesn't depend on %PATHEXT% or any registered applications for file extensions on the explorer or shell level. On the OS level .exe, .com, .bat and .cmd is the exclusive set of extensions which are considered as executable. Not more and not less. When you search for the path of an executable with the win32 API, you can call FindExecutable with only 'tool' and you will get '...\tool.exe', '.com', '.bat' or '.cmd'. Whatever is found first according to PATH and precedence. Not more and not less. You can try that via pywin32 using win32api.FindExecutable().

And interestingly, Popen can call .bat and .cmd scripts directly if you explicitly specify the extension, also with shell=False. But it can't call any other file types even if that works in the shell because some application is registered for them on the explorer level, unless shell=True.

On Windows .bat and .cmd scripts have a special status beginning on the lowest OS level and a Windows user normally expects that any scripting language should be able to run them without explicit extension. Other major languages do it. Why not Python, too?

[toc] | [prev] | [next] | [standalone]

#90106

Stefan Zimmermann <zimmermann.code@gmail.com>:

> Calling an external command should be one of the simplest tasks in a
> high level scripting language like Python.

Actually, that's quite a tricky operation in any OS. For example, bash's
simplicity is a trap that claims a lot of victims.

Anyway, Python has os.system() that does the quick and dirty thing you
might be looking for.

> And that should not involve any OS-specific differences, unless you
> want to use some advanced process handling features which are only
> supported by some specific OS.

I can't speak for Windows, but under Linux, it gets advanced pretty
quickly (what gets inherited, how about zombies, how about signals, etc
etc).

> Meanwhile I checked Ruby and Perl regarding this feature. Both support
> it. In both langs every standard function that calls external commands
> (like Perl's exec() or system() or Ruby's exec() or system() or
> IO.popen()), whether they invoke a shell or call it directly, support
> running 'tool.bat' or 'tool.cmd' by just writing 'tool'. Python almost
> seems to be the only major scripting language which does not support
> this implicitly.

I'm not against subprocess.Popen() doing its work under Windows the way
Windows system programmers would expect. I'm against trying to force
Windows and Linux into the same mold where there are genuine
differences.

If I were a Windows developer, I'd expect Python to support something
analogous to what I'd have in C++ or C#. If pipes are natural IPC
channels under Windows, then subprocess.Popen() is probably pretty close
to its mark. However, most of the IPC facilities listed here:

   <URL: https://msdn.microsoft.com/en-us/library/windows/desktop/aa36
   5574%28v=vs.85%29.aspx>

seem to be absent in Python (clipboard, COM, data copy, DDE, file
mapping, mailslots, rpc).


Marko

[toc] | [prev] | [next] | [standalone]

#90108

Marko Rauhamaa wrote:
> Anyway, Python has os.system() that does the quick and dirty thing you
> might be looking for.

Always invokes shell ==> overhead for .exe files

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | comp.lang.python

csiph-web