Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #37279 > unrolled thread

Re: Retrieving the full command line

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Retrieving the full command line Tim Golden <mail@timgolden.me.uk> - 2013-01-22 15:07 +0000
    Re: Retrieving the full command line Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-01-22 23:46 +0000
      Re: Retrieving the full command line Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2013-01-23 00:53 +0000
        Re: Retrieving the full command line Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-01-23 03:58 +0000
          Re: Retrieving the full command line Tim Golden <mail@timgolden.me.uk> - 2013-01-23 09:58 +0000
          Re: Retrieving the full command line Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2013-01-23 10:01 +0000
            Re: Retrieving the full command line Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-01-24 04:49 +0000
              Re: Retrieving the full command line Chris Angelico <rosuav@gmail.com> - 2013-01-24 16:06 +1100
              Re: Retrieving the full command line Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2013-01-24 10:06 +0000
              Re: Retrieving the full command line Tim Golden <mail@timgolden.me.uk> - 2013-01-24 10:56 +0000
              Re: Retrieving the full command line Tim Golden <mail@timgolden.me.uk> - 2013-01-24 11:04 +0000
              Re: Retrieving the full command line Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2013-01-24 11:30 +0000
              Re: Retrieving the full command line Tim Golden <mail@timgolden.me.uk> - 2013-01-24 13:45 +0000
              Re: Retrieving the full command line Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2013-01-24 15:28 +0000
              Re: Retrieving the full command line Tim Golden <mail@timgolden.me.uk> - 2013-01-24 15:51 +0000
              Re: Retrieving the full command line Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2013-01-24 16:08 +0000
              Re: Retrieving the full command line Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2013-01-24 16:53 +0000
              Re: Retrieving the full command line Tim Golden <mail@timgolden.me.uk> - 2013-01-24 17:13 +0000
              Re: Retrieving the full command line Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2013-01-24 20:01 +0000
              Re: Retrieving the full command line Tim Golden <mail@timgolden.me.uk> - 2013-01-24 20:54 +0000

#37279 — Re: Retrieving the full command line

On 22/01/2013 14:53, Terry Reedy wrote:
> On 1/22/2013 4:24 AM, Tim Golden wrote:
>> [Python 2.7/3.3 (and hg tip) running on Windows. Not Windows-specific,
>> though].
>>
>> I use the python -mpackage incantation to run a package which has a
>> __main__.py module and which uses relative imports internally.
>>
>> I'm developing under cherrypy which includes a reloader for development.
>> The reloader attempts to rebuild the original
>> command line by combining sys.executable and sys.argv and then does an
>> execv.
>>
>> There does not appear to be any way within Python of determining the
>> command line I used. The combination of sys.executable and sys.argv in
>> this case will look like: "c:\python33\python.exe app/__main__.py". But
>> running this precludes the use of package-relative imports.
> 
> If I understand right, the reloader should be updated to translate
> 'x/__main__.py' to '-m x'. Filenames of form'__x__' are reserved, in a
> sense, like similar identifiers in programs, and '__main__.py' should
> not be used for a file meant to executed directly.

To be clear: it's Python itself, not the reloader, which is coming up
with __main__.py. sys.executable is "c:\python33\python.exe" and
sys.argv is ['c:\path\to\__main__.py'] for a program which has been
started by "c:\python33\python.exe -mpath\to".

Obviously, there is any number of ways around this specific issue,
including what you suggest: a canonical rewrite of "python
path\to\__main__.py" into "python -mpath\to". But it's not clear to me
that this rewrite should be the responsibility of calling code.

TJG

[toc] | [next] | [standalone]

#37360

On Tue, 22 Jan 2013 15:07:18 +0000, Tim Golden wrote:

> On 22/01/2013 14:53, Terry Reedy wrote:
>> On 1/22/2013 4:24 AM, Tim Golden wrote:
>>> [Python 2.7/3.3 (and hg tip) running on Windows. Not Windows-specific,
>>> though].
>>>
>>> I use the python -mpackage incantation to run a package which has a
>>> __main__.py module and which uses relative imports internally.
>>>
>>> I'm developing under cherrypy which includes a reloader for
>>> development. The reloader attempts to rebuild the original command
>>> line by combining sys.executable and sys.argv and then does an execv.
>>>
>>> There does not appear to be any way within Python of determining the
>>> command line I used. The combination of sys.executable and sys.argv in
>>> this case will look like: "c:\python33\python.exe app/__main__.py".
>>> But running this precludes the use of package-relative imports.
>> 
>> If I understand right, the reloader should be updated to translate
>> 'x/__main__.py' to '-m x'. Filenames of form'__x__' are reserved, in a
>> sense, like similar identifiers in programs, and '__main__.py' should
>> not be used for a file meant to executed directly.
> 
> To be clear: it's Python itself, not the reloader, which is coming up
> with __main__.py. sys.executable is "c:\python33\python.exe" and
> sys.argv is ['c:\path\to\__main__.py'] for a program which has been
> started by "c:\python33\python.exe -mpath\to".

I don't believe you can give direct paths to the -m flag. It uses the 
normal import mechanism to locate a module or package, so you have to 
give it a name which would be importable.

c:\python33\python.exe -m app

would work, where "app" is either a package or module:

C:\something\on\PYTHONPATH\app\__main__.py
C:\something\on\PYTHONPATH\app.py


> Obviously, there is any number of ways around this specific issue,
> including what you suggest: a canonical rewrite of "python
> path\to\__main__.py" into "python -mpath\to". But it's not clear to me
> that this rewrite should be the responsibility of calling code.


I am a bit disturbed that you cannot distinguish between:

python C:\something\on\pythonpath\app\__main__.py

python -m app


by inspecting the command line. I consider it a bug, or at least a 
misfeature, if Python transforms the command line before making it 
available in sys.argv.


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#37388

On 22 January 2013 23:46, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
[SNIP]
>
> I am a bit disturbed that you cannot distinguish between:
>
> python C:\something\on\pythonpath\app\__main__.py
>
> python -m app
>
>
> by inspecting the command line. I consider it a bug, or at least a
> misfeature, if Python transforms the command line before making it
> available in sys.argv.

The purpose of the -m option is that you can run a script that is
located via the Python import path instead of an explicit file path.
The idea is that if '/path/to/somewhere' is in sys.path then:
    python -m script arg1 arg2
is equivalent to
    python /path/to/somewhere/script.py arg1 arg2

If Python didn't modify sys.argv then 'script.py' would need to be
rewritten to understand that sys.argv would be in a different format
when it was invoked using the -m option.

I believe that it has previously been proposed to include something
like sys.raw_argv, although perhaps for different reasons. Although
something that might be more useful to the OP (and that I at one point
wanted) would be a function similar to execfile but that would launch
a separate process using the same interpreter as the current process.


Oscar

[toc] | [prev] | [next] | [standalone]

#37409

On Wed, 23 Jan 2013 00:53:21 +0000, Oscar Benjamin wrote:

> On 22 January 2013 23:46, Steven D'Aprano
> <steve+comp.lang.python@pearwood.info> wrote: [SNIP]
>>
>> I am a bit disturbed that you cannot distinguish between:
>>
>> python C:\something\on\pythonpath\app\__main__.py
>>
>> python -m app
>>
>>
>> by inspecting the command line. I consider it a bug, or at least a
>> misfeature, if Python transforms the command line before making it
>> available in sys.argv.
> 
> The purpose of the -m option is that you can run a script that is
> located via the Python import path instead of an explicit file path. The
> idea is that if '/path/to/somewhere' is in sys.path then:
>     python -m script arg1 arg2
> is equivalent to
>     python /path/to/somewhere/script.py arg1 arg2
> 
> If Python didn't modify sys.argv then 'script.py' would need to be
> rewritten to understand that sys.argv would be in a different format
> when it was invoked using the -m option.

I don't think that it would be in a different format. Normally people 
only care about sys.argv[1:], the actual arguments. argv[0], the name of 
the script, already comes in multiple formats: absolute or relative paths.

Currently, if I have a package __main__.py that prints sys.argv, I get 
results like this:

steve@runes:~$ python3.3 /home/steve/python/testpackage/__main__.py ham 
spam eggs
['/home/steve/python/testpackage/__main__.py', 'ham', 'spam', 'eggs']


which is correct, that's what I gave on the command line. But:

steve@runes:~$ python3.3 -m testpackage ham spam eggs
['/home/steve/python/testpackage/__main__.py', 'ham', 'spam', 'eggs']


The second example is lying. It should say:

['-m testpackage', 'ham', 'spam', 'eggs']


If you are one of the few people who care about argv[0], then you are 
already dealing with the fact that the name of the executable script is 
not always an absolute path and therefore can vary greatly from one call 
to another. Hell, if you are on a system with soft links, the name of the 
script in the command line is not even necessarily the name of the 
module. So there's not much more effort involved in dealing with one 
extra case:

# assuming you care, which most people don't
if sys.argv[0].startswith('-m'):
    do_something()
elif os.path.isabs(sys.argv[0]):
    do_this()
else:  # relative path
    do_that()



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#37437

On 23/01/2013 03:58, Steven D'Aprano wrote:
> Currently, if I have a package __main__.py that prints sys.argv, I get 
> results like this:
> 
> steve@runes:~$ python3.3 /home/steve/python/testpackage/__main__.py ham 
> spam eggs
> ['/home/steve/python/testpackage/__main__.py', 'ham', 'spam', 'eggs']
> 
> 
> which is correct, that's what I gave on the command line. But:
> 
> steve@runes:~$ python3.3 -m testpackage ham spam eggs
> ['/home/steve/python/testpackage/__main__.py', 'ham', 'spam', 'eggs']
> 
> 
> The second example is lying. It should say:
> 
> ['-m testpackage', 'ham', 'spam', 'eggs']

Thanks for the input, Steven & Oscar.

Apologies for the confusion over my initial example. Of course, -m runs
something on sys.path, not something in the filesystem as such. I
confused myself because, running on Windows where the current directory
is on the path, I sit in c:\path\to and do python -mapp

Now I look harder, this discussion is basically issue14208:

  http://bugs.python.org/issue14208

so I'll probably go and contribute over there.

TJG

[toc] | [prev] | [next] | [standalone]

#37438

On 23 January 2013 03:58, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> On Wed, 23 Jan 2013 00:53:21 +0000, Oscar Benjamin wrote:
>
>> On 22 January 2013 23:46, Steven D'Aprano
>> <steve+comp.lang.python@pearwood.info> wrote: [SNIP]
>>>
>> The purpose of the -m option is that you can run a script that is
>> located via the Python import path instead of an explicit file path. The
>> idea is that if '/path/to/somewhere' is in sys.path then:
>>     python -m script arg1 arg2
>> is equivalent to
>>     python /path/to/somewhere/script.py arg1 arg2
>>
>> If Python didn't modify sys.argv then 'script.py' would need to be
>> rewritten to understand that sys.argv would be in a different format
>> when it was invoked using the -m option.
>
> I don't think that it would be in a different format. Normally people
> only care about sys.argv[1:], the actual arguments. argv[0], the name of
> the script, already comes in multiple formats: absolute or relative paths.
>
> Currently, if I have a package __main__.py that prints sys.argv, I get
> results like this:
>
> steve@runes:~$ python3.3 /home/steve/python/testpackage/__main__.py ham
> spam eggs
> ['/home/steve/python/testpackage/__main__.py', 'ham', 'spam', 'eggs']
>
>
> which is correct, that's what I gave on the command line. But:
>
> steve@runes:~$ python3.3 -m testpackage ham spam eggs
> ['/home/steve/python/testpackage/__main__.py', 'ham', 'spam', 'eggs']
>
>
> The second example is lying. It should say:
>
> ['-m testpackage', 'ham', 'spam', 'eggs']

I don't know why you would expect this. I imagined that you would want

['-m', 'testpackage', 'ham', 'spam', 'eggs']

If the two were combined into one string I would expect it to at least
be a valid argument list:

['-mtestpackage', 'ham', 'spam', 'eggs']

>
>
> If you are one of the few people who care about argv[0], then you are
> already dealing with the fact that the name of the executable script is
> not always an absolute path and therefore can vary greatly from one call
> to another. Hell, if you are on a system with soft links, the name of the
> script in the command line is not even necessarily the name of the
> module. So there's not much more effort involved in dealing with one
> extra case:

Unless I've missed something sys.argv[0] is always a valid path to the
script. Whether it is absolute or not shouldn't matter. For imported
modules the path is available from __name__. For a script that is
executed rather than imported __name__ == "__main__" but the path is
accessible from sys.argv[0]. If you are one of those people who cares
about sys.argv[0] then this is probably the value that you wanted it
to contain.

If it were important for sys.argv to show how exactly the script was
located and executed, then why not also include the 'python3.3'
command line argument (the real argv[0])? sys.argv emulates the argv
that e.g. a C program would get. The real command line used is not
exactly the same since a Python script is not a directly executable
binary, so Python processes the argument list before passing it
through.

In the OP's case, the script is never invoked without -m so the following works:

cmdline = [sys.executable, '-m', __package__] + sys.argv[1:]

In the more general case it would be better to have an API
specifically for this purpose.


Oscar

[toc] | [prev] | [next] | [standalone]

#37535

On Wed, 23 Jan 2013 10:01:24 +0000, Oscar Benjamin wrote:

> On 23 January 2013 03:58, Steven D'Aprano
> <steve+comp.lang.python@pearwood.info> wrote:
>> On Wed, 23 Jan 2013 00:53:21 +0000, Oscar Benjamin wrote:
>>
>>> On 22 January 2013 23:46, Steven D'Aprano
>>> <steve+comp.lang.python@pearwood.info> wrote: [SNIP]
>>>>
>>> The purpose of the -m option is that you can run a script that is
>>> located via the Python import path instead of an explicit file path.
>>> The idea is that if '/path/to/somewhere' is in sys.path then:
>>>     python -m script arg1 arg2
>>> is equivalent to
>>>     python /path/to/somewhere/script.py arg1 arg2
>>>
>>> If Python didn't modify sys.argv then 'script.py' would need to be
>>> rewritten to understand that sys.argv would be in a different format
>>> when it was invoked using the -m option.
>>
>> I don't think that it would be in a different format. Normally people
>> only care about sys.argv[1:], the actual arguments. argv[0], the name
>> of the script, already comes in multiple formats: absolute or relative
>> paths.
>>
>> Currently, if I have a package __main__.py that prints sys.argv, I get
>> results like this:
>>
>> steve@runes:~$ python3.3 /home/steve/python/testpackage/__main__.py ham
>> spam eggs
>> ['/home/steve/python/testpackage/__main__.py', 'ham', 'spam', 'eggs']
>>
>>
>> which is correct, that's what I gave on the command line. But:
>>
>> steve@runes:~$ python3.3 -m testpackage ham spam eggs
>> ['/home/steve/python/testpackage/__main__.py', 'ham', 'spam', 'eggs']
>>
>>
>> The second example is lying. It should say:
>>
>> ['-m testpackage', 'ham', 'spam', 'eggs']
> 
> I don't know why you would expect this. I imagined that you would want
> 
> ['-m', 'testpackage', 'ham', 'spam', 'eggs']


No. argv[0] is intended to be the script being called, argv[1:] for the 
arguments to the script. Given the two choices:

1) Break every Python script that expects argv[1:] to be the arguments
   to the script, forcing them to decide whether they should look at
   argv[1:] or argv[2:] according to whether or not argv[0] == '-m'; 

or 

2) don't break anything, but make a very small addition to the semantics
   of argv[0] (was: "the path to the script", add "or -m and the name of
   module/package") that won't break anyone's code;


there's practically no choice in the matter.



> If the two were combined into one string I would expect it to at least
> be a valid argument list:
> 
> ['-mtestpackage', 'ham', 'spam', 'eggs']

Okay, fair point. I didn't consider that.

Note however that there is an ambiguity between calling "python -mspam" 
and calling a script literally named "-mspam". But that same ambiguity 
exists in the shell, so I don't consider it a problem. You cannot call a 
script named -mspam unless you use something like this "python ./-mspam".


>> If you are one of the few people who care about argv[0], then you are
>> already dealing with the fact that the name of the executable script is
>> not always an absolute path and therefore can vary greatly from one
>> call to another. Hell, if you are on a system with soft links, the name
>> of the script in the command line is not even necessarily the name of
>> the module. So there's not much more effort involved in dealing with
>> one extra case:
> 
> Unless I've missed something sys.argv[0] is always a valid path to the
> script. Whether it is absolute or not shouldn't matter. 

Sure. But if you care about argv[0] (say, you want to pull out the name 
of the script at runtime, instead of hard-coding it), then you need to be 
aware that you could be given an absolute path, a relative path, a bare 
script name, or the path of a softlink to the file you actually care 
about. Adding one more trivially simple case is not a large burden.

People hardly ever care about argv[0]. At least, I don't think I ever 
have. But the OP does, and Python mangling argv[0] is causing him grief 
because it lies, claiming to have called the __main__.py of his package 
directly when in fact he called it with -m.


> For imported
> modules the path is available from __name__. For a script that is
> executed rather than imported __name__ == "__main__" but the path is
> accessible from sys.argv[0]. If you are one of those people who cares
> about sys.argv[0] then this is probably the value that you wanted it to
> contain.

I'm wary about guessing what people "probably" want, and therefore lying 
about what they actually got. That's DWIM coding, and that almost always 
ends in tears.


> If it were important for sys.argv to show how exactly the script was
> located and executed, then why not also include the 'python3.3' command
> line argument (the real argv[0])? sys.argv emulates the argv that e.g. a
> C program would get. The real command line used is not exactly the same
> since a Python script is not a directly executable binary, so Python
> processes the argument list before passing it through.

Also a good point. To some degree, we're constrained by backwards 
compatibility -- there's only so much change we can do without breaking 
code, and setting argv[0] to the python executable instead of the script 
is too big a change.

In any case, you can get that information using sys.executable, or at 
least you can get the path of the actual Python binary, or you can use 
sys.version (or equivalent) to determine which version of Python you're 
using. 

This does mean that you can't play dirty hacks like some C binaries do, 
where they change their behaviour depending on whether you call them via 
one path or another path. vi does that. But that's hardly a big loss.

Contrariwise, I don't believe that there is currently *any* way to 
distinguish between running a script with or without -m. That should be 
fixed.


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#37538

On Thu, Jan 24, 2013 at 3:49 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> Note however that there is an ambiguity between calling "python -mspam"
> and calling a script literally named "-mspam". But that same ambiguity
> exists in the shell, so I don't consider it a problem. You cannot call a
> script named -mspam unless you use something like this "python ./-mspam".

Another spanner for your works: "python -- -mspam" succeeds. That sets
argv[0] to '-mspam'.

> People hardly ever care about argv[0]. At least, I don't think I ever
> have. But the OP does, and Python mangling argv[0] is causing him grief
> because it lies, claiming to have called the __main__.py of his package
> directly when in fact he called it with -m.

Usually when I reference argv[0], $0, or any equivalent, it's for a
usage display - eg:

USAGE: $0 [opts] infile [outfile]
--foo    Fooify the file
--bar    Burn your computer to the ground

So I don't particularly care about symlinks or relative paths (if it
worked once, it'll probably work another time). But ambiguities may be
an issue.

ChrisA

[toc] | [prev] | [next] | [standalone]

#37548

On 24 January 2013 04:49, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
[SNIP]
>
> Contrariwise, I don't believe that there is currently *any* way to
> distinguish between running a script with or without -m. That should be
> fixed.

As I said earlier in the thread, the __package__ module global
distinguishes the two cases:

~$ mkdir pkg
~$ touch pkg/__init__.py
~$ vim pkg/__main__.py
~$ cat pkg/__main__.py
import sys
if __package__ is None:
    cmdline = [sys.executable] + sys.argv
else:
    cmdline = [sys.executable, '-m', __package__] + sys.argv[1:]
print(cmdline)
~$ python pkg/__main__.py arg1 arg2
['q:\\tools\\Python27\\python.exe', 'pkg/__main__.py', 'arg1', 'arg2']
~$ python -m pkg arg1 arg2
['q:\\tools\\Python27\\python.exe', '-m', 'pkg', 'arg1', 'arg2']


Oscar

[toc] | [prev] | [next] | [standalone]

#37557

On 24/01/2013 10:06, Oscar Benjamin wrote:
> On 24 January 2013 04:49, Steven D'Aprano
> <steve+comp.lang.python@pearwood.info> wrote:
> [SNIP]
>>
>> Contrariwise, I don't believe that there is currently *any* way to
>> distinguish between running a script with or without -m. That should be
>> fixed.
> 
> As I said earlier in the thread, the __package__ module global
> distinguishes the two cases:
> 
> ~$ mkdir pkg
> ~$ touch pkg/__init__.py
> ~$ vim pkg/__main__.py
> ~$ cat pkg/__main__.py
> import sys
> if __package__ is None:
>     cmdline = [sys.executable] + sys.argv
> else:
>     cmdline = [sys.executable, '-m', __package__] + sys.argv[1:]
> print(cmdline)
> ~$ python pkg/__main__.py arg1 arg2
> ['q:\\tools\\Python27\\python.exe', 'pkg/__main__.py', 'arg1', 'arg2']
> ~$ python -m pkg arg1 arg2
> ['q:\\tools\\Python27\\python.exe', '-m', 'pkg', 'arg1', 'arg2']

Reasonable (and thanks for the clear example), but it doesn't work
if the package which is reconstructing the command line the package
which was the target of the original command line. In my case,
I'm making use of the cherrypy reloader, whose __package__ is
cherrypy.process. But the command which invoked the program was
python -m myapp.

ie I'm issuing "python -m myapp". In myapp.__main__ I'm importing
cherrypy, itself a package, and somewhere in cherrypy.whatever there is
code which attempts to reconstruct the command line.

TJG

[toc] | [prev] | [next] | [standalone]

#37560

On 24/01/2013 10:56, Tim Golden wrote:
> if the package which is reconstructing the command line the package
> which was the target of the original command line.


Sorry:

  if the package which is reconstructing the command line *is not*
  the package which was the target of the original command line.

TJG

[toc] | [prev] | [next] | [standalone]

#37570

On 24 January 2013 10:56, Tim Golden <mail@timgolden.me.uk> wrote:
> On 24/01/2013 10:06, Oscar Benjamin wrote:
>> On 24 January 2013 04:49, Steven D'Aprano
>> <steve+comp.lang.python@pearwood.info> wrote:
>> [SNIP]
>>>
>>> Contrariwise, I don't believe that there is currently *any* way to
>>> distinguish between running a script with or without -m. That should be
>>> fixed.
>>
>> As I said earlier in the thread, the __package__ module global
>> distinguishes the two cases:
>>
>> ~$ mkdir pkg
>> ~$ touch pkg/__init__.py
>> ~$ vim pkg/__main__.py
>> ~$ cat pkg/__main__.py
>> import sys
>> if __package__ is None:
>>     cmdline = [sys.executable] + sys.argv
>> else:
>>     cmdline = [sys.executable, '-m', __package__] + sys.argv[1:]
>> print(cmdline)
>> ~$ python pkg/__main__.py arg1 arg2
>> ['q:\\tools\\Python27\\python.exe', 'pkg/__main__.py', 'arg1', 'arg2']
>> ~$ python -m pkg arg1 arg2
>> ['q:\\tools\\Python27\\python.exe', '-m', 'pkg', 'arg1', 'arg2']
>
> Reasonable (and thanks for the clear example), but it doesn't work
> if the package which is reconstructing the command line the package
> which was the target of the original command line. In my case,
> I'm making use of the cherrypy reloader, whose __package__ is
> cherrypy.process. But the command which invoked the program was
> python -m myapp.
>
> ie I'm issuing "python -m myapp". In myapp.__main__ I'm importing
> cherrypy, itself a package, and somewhere in cherrypy.whatever there is
> code which attempts to reconstruct the command line.

Easy enough:

~$ mkdir pkg
~$ touch pkg/__init__.py
~$ vim pkg/__main__.py
~$ cat pkg/__main__.py
import pkg.whatever
~$ vim pkg/whatever.py
~$ cat pkg/whatever.py
import sys
import pkg.__main__ as main
cmdline = [sys.executable, '-m', main.__package__] + sys.argv[1:]
print(cmdline)
~$ python -m pkg
['q:\\tools\\Python27\\python.exe', '-m', 'pkg']
~$ python -m pkg arg1 arg32
['q:\\tools\\Python27\\python.exe', '-m', 'pkg', 'arg1', 'arg32']

I don't really understand what your spec is. Why do you need to
inspect this information from sys.argv? Can you not just always use
'python -m pkg' as your entry point?


Oscar

[toc] | [prev] | [next] | [standalone]

#37590

On 24/01/2013 11:30, Oscar Benjamin wrote:
> I don't really understand what your spec is. Why do you need to
> inspect this information from sys.argv? Can you not just always use
> 'python -m pkg' as your entry point?

Sorry about the confusion. I think my original point was simply one
of surprise that sys.argv wouldn't essentially mirror the elements
of the command line which I used to get there.
The specifics of my use-case weren't really too important.

For completeness, I'm talking about the cherrypy Autoreloader which
attempts to restart (via execv) whatever process was responsible for
loading it in the first place, via an identical or equivalent command
line. The current (cherrypy) code simply joins sys.executable and
sys.argv but this fails in the face of python -m as we have seen.

The cherrypy package has no especial knowledge of the structure of the
application which imported it and so must piece together the command
line somehow. Clearly, I can take various approaches of the sort
which you've outlined, or subclass the reloader, or fetch the original
command line from the OS, etc. It's not that this is a showstopper,
merely slightly surprising. (To me).

TJG

[toc] | [prev] | [next] | [standalone]

#37599

On 24 January 2013 13:45, Tim Golden <mail@timgolden.me.uk> wrote:
> On 24/01/2013 11:30, Oscar Benjamin wrote:
>> I don't really understand what your spec is. Why do you need to
>> inspect this information from sys.argv? Can you not just always use
>> 'python -m pkg' as your entry point?
>
[SNIP]
>
> For completeness, I'm talking about the cherrypy Autoreloader which
> attempts to restart (via execv) whatever process was responsible for
> loading it in the first place, via an identical or equivalent command
> line. The current (cherrypy) code simply joins sys.executable and
> sys.argv but this fails in the face of python -m as we have seen.
>
> The cherrypy package has no especial knowledge of the structure of the
> application which imported it and so must piece together the command
> line somehow. Clearly, I can take various approaches of the sort
> which you've outlined, or subclass the reloader, or fetch the original
> command line from the OS, etc. It's not that this is a showstopper,
> merely slightly surprising. (To me).

Ok I understand. Then I guess you want:

import __main__
pkg = __main__.__package__


Oscar

[toc] | [prev] | [next] | [standalone]

#37603

On 24/01/2013 15:28, Oscar Benjamin wrote:
> On 24 January 2013 13:45, Tim Golden <mail@timgolden.me.uk> wrote:
>> On 24/01/2013 11:30, Oscar Benjamin wrote:
>>> I don't really understand what your spec is. Why do you need to
>>> inspect this information from sys.argv? Can you not just always use
>>> 'python -m pkg' as your entry point?
>>
> [SNIP]
>>
>> For completeness, I'm talking about the cherrypy Autoreloader which
>> attempts to restart (via execv) whatever process was responsible for
>> loading it in the first place, via an identical or equivalent command
>> line. The current (cherrypy) code simply joins sys.executable and
>> sys.argv but this fails in the face of python -m as we have seen.
>>
>> The cherrypy package has no especial knowledge of the structure of the
>> application which imported it and so must piece together the command
>> line somehow. Clearly, I can take various approaches of the sort
>> which you've outlined, or subclass the reloader, or fetch the original
>> command line from the OS, etc. It's not that this is a showstopper,
>> merely slightly surprising. (To me).
> 
> Ok I understand. Then I guess you want:
> 
> import __main__
> pkg = __main__.__package__

Brilliant. Never thought of importing __main__. Thanks.

For the benefit of anyone still watching, the code (which has to be
compatible back to 2.3) looks something like this:

<code>
import __main__

# [.. .snip ...]

try:
    is_package = bool(__main__.__package__)
except NameError:
    is_package = False
if is_package:
    args = [sys.executable, '-m', __main__.__package__] + sys.argv[1:]
else:
    args = [sys.executable] + sys.argv

os.chdir(_startup_cwd) # avoids relative/absolute issues
os.execv(args[0], args)

</code>

I don't pretend it's foolproot, but it certainly works for my particular
case. Nor have I considered it against all the cases identified in PEP
432: http://www.python.org/dev/peps/pep-0432/#configuring-sys-argv

TJG

[toc] | [prev] | [next] | [standalone]

#37606

On 24 January 2013 15:51, Tim Golden <mail@timgolden.me.uk> wrote:
> On 24/01/2013 15:28, Oscar Benjamin wrote:
>> On 24 January 2013 13:45, Tim Golden <mail@timgolden.me.uk> wrote:
>>> On 24/01/2013 11:30, Oscar Benjamin wrote:
>>>> I don't really understand what your spec is. Why do you need to
>>>> inspect this information from sys.argv? Can you not just always use
>>>> 'python -m pkg' as your entry point?
>>>
>> [SNIP]
>>>
>>> For completeness, I'm talking about the cherrypy Autoreloader which
>>> attempts to restart (via execv) whatever process was responsible for
>>> loading it in the first place, via an identical or equivalent command
>>> line. The current (cherrypy) code simply joins sys.executable and
>>> sys.argv but this fails in the face of python -m as we have seen.
>>>
>>> The cherrypy package has no especial knowledge of the structure of the
>>> application which imported it and so must piece together the command
>>> line somehow. Clearly, I can take various approaches of the sort
>>> which you've outlined, or subclass the reloader, or fetch the original
>>> command line from the OS, etc. It's not that this is a showstopper,
>>> merely slightly surprising. (To me).
>>
>> Ok I understand. Then I guess you want:
>>
>> import __main__
>> pkg = __main__.__package__
>
> Brilliant. Never thought of importing __main__. Thanks.
>
> For the benefit of anyone still watching, the code (which has to be
> compatible back to 2.3) looks something like this:
>
> <code>
> import __main__
>
> # [.. .snip ...]
>
> try:
>     is_package = bool(__main__.__package__)
> except NameError:
>     is_package = False
> if is_package:
>     args = [sys.executable, '-m', __main__.__package__] + sys.argv[1:]
> else:
>     args = [sys.executable] + sys.argv
>
> os.chdir(_startup_cwd) # avoids relative/absolute issues
> os.execv(args[0], args)
>
> </code>
>
> I don't pretend it's foolproot, but it certainly works for my particular
> case. Nor have I considered it against all the cases identified in PEP
> 432: http://www.python.org/dev/peps/pep-0432/#configuring-sys-argv

Does it work if you use the -m option to run a module rather than a script?


Oscar

[toc] | [prev] | [next] | [standalone]

#37608

On 24 January 2013 16:08, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
> On 24 January 2013 15:51, Tim Golden <mail@timgolden.me.uk> wrote:
>> On 24/01/2013 15:28, Oscar Benjamin wrote:
>>> On 24 January 2013 13:45, Tim Golden <mail@timgolden.me.uk> wrote:
>>>> On 24/01/2013 11:30, Oscar Benjamin wrote:
>>>>> I don't really understand what your spec is. Why do you need to
>>>>> inspect this information from sys.argv? Can you not just always use
>>>>> 'python -m pkg' as your entry point?
>>>>
>>> [SNIP]
>>>>
>>>> For completeness, I'm talking about the cherrypy Autoreloader which
>>>> attempts to restart (via execv) whatever process was responsible for
>>>> loading it in the first place, via an identical or equivalent command
>>>> line. The current (cherrypy) code simply joins sys.executable and
>>>> sys.argv but this fails in the face of python -m as we have seen.
>>>>
>>>> The cherrypy package has no especial knowledge of the structure of the
>>>> application which imported it and so must piece together the command
>>>> line somehow. Clearly, I can take various approaches of the sort
>>>> which you've outlined, or subclass the reloader, or fetch the original
>>>> command line from the OS, etc. It's not that this is a showstopper,
>>>> merely slightly surprising. (To me).
>>>
>>> Ok I understand. Then I guess you want:
>>>
>>> import __main__
>>> pkg = __main__.__package__
>>
>> Brilliant. Never thought of importing __main__. Thanks.
>>
>> For the benefit of anyone still watching, the code (which has to be
>> compatible back to 2.3) looks something like this:
>>
>> <code>
>> import __main__
>>
>> # [.. .snip ...]
>>
>> try:
>>     is_package = bool(__main__.__package__)
>> except NameError:
>>     is_package = False
>> if is_package:
>>     args = [sys.executable, '-m', __main__.__package__] + sys.argv[1:]
>> else:
>>     args = [sys.executable] + sys.argv
>>
>> os.chdir(_startup_cwd) # avoids relative/absolute issues
>> os.execv(args[0], args)
>>
>> </code>
>>
>> I don't pretend it's foolproot, but it certainly works for my particular
>> case. Nor have I considered it against all the cases identified in PEP
>> 432: http://www.python.org/dev/peps/pep-0432/#configuring-sys-argv
>
> Does it work if you use the -m option to run a module rather than a script?

Sorry that was written incorrectly. I meant to say: does it work when
a module is directly on sys.path rather than as a submodule of a
package? In this case __package__ is set to the empty string if run
with -m or None if run with a direct path. So the check needs to be
"__package__ is not None" rather than "bool(__package__)".


Oscar

[toc] | [prev] | [next] | [standalone]

#37611

On 24/01/2013 16:53, Oscar Benjamin wrote:
>> Does it work if you use the -m option to run a module rather than a script?
> 
> Sorry that was written incorrectly. I meant to say: does it work when
> a module is directly on sys.path rather than as a submodule of a
> package? In this case __package__ is set to the empty string if run
> with -m or None if run with a direct path. So the check needs to be
> "__package__ is not None" rather than "bool(__package__)".

The answer is: it depends. Given the code I outlined earlier:

A package-based module run via -m (python -m package.module) works
as described (including the implicit __main__ module, my
primary use-case).

A module run from the filesystem (python c:\path\to\module.py) works
by dropping through through to the not is_package logic branch.

A module run via -m (python -m module) actually works by accident,
because it too drops through to the not is_package branch and is
rerun with its full filesystem path. This doesn't have the same
problems as running a package from the filesystem because relative
imports aren't an issue. I don't know if there are any other differences
between python -mmodule and python c:\path\to\module.py.

As you say, a more refined check could determine a blank __package__
as opposed to a None __package__. But this code (cherrypy) must also
cope with version of Python before 2.6 which didn't even have a
__package__ attribute, muddying the waters that little bit further.

TJG

[toc] | [prev] | [next] | [standalone]

#37623

On 24 January 2013 17:13, Tim Golden <mail@timgolden.me.uk> wrote:
> On 24/01/2013 16:53, Oscar Benjamin wrote:
>>> Does it work if you use the -m option to run a module rather than a script?
>>
>> Sorry that was written incorrectly. I meant to say: does it work when
>> a module is directly on sys.path rather than as a submodule of a
>> package? In this case __package__ is set to the empty string if run
>> with -m or None if run with a direct path. So the check needs to be
>> "__package__ is not None" rather than "bool(__package__)".
>
> The answer is: it depends. Given the code I outlined earlier:
>
> A package-based module run via -m (python -m package.module) works
> as described (including the implicit __main__ module, my
> primary use-case).

Does it work in the "python -m package.module" case? It looks to me as
if the code below will run "python -m package" instead. I think you
need to split the module name out of sys.argv[0] and put that into the
command line as well.

> import __main__
>
> # [.. .snip ...]
>
> try:
>     is_package = bool(__main__.__package__)
> except NameError:
>     is_package = False
> if is_package:
>     args = [sys.executable, '-m', __main__.__package__] + sys.argv[1:]
> else:
>     args = [sys.executable] + sys.argv
>
> os.chdir(_startup_cwd) # avoids relative/absolute issues
> os.execv(args[0], args)

I believe "python -m package" and "python -m package.__main__" are
equivalent so it doesn't matter in that case.

>
> A module run from the filesystem (python c:\path\to\module.py) works
> by dropping through through to the not is_package logic branch.
>
> A module run via -m (python -m module) actually works by accident,
> because it too drops through to the not is_package branch and is
> rerun with its full filesystem path. This doesn't have the same
> problems as running a package from the filesystem because relative
> imports aren't an issue. I don't know if there are any other differences
> between python -mmodule and python c:\path\to\module.py.

There is a (probably pathological) case in which running the script
via a file path and running it via -m are not equivalent. "python
dir/script.py" places "dir" at the top of sys.path. "python -m script"
only works if "dir" is in sys.path but it needn't be at the top. This
means that the order of sys.path is changed and import conflicts may
be resolved differently in the two cases:

~$ cat ~/.local/lib/python2.7/site-packages/script.py
print("Running script.py")
import optparse
~$ cat ~/.local/lib/python2.7/site-packages/optparse.py
raise ImportError('Wrong optparse!')
~$ python ~/.local/lib/python2.7/site-packages/script.py
Running script.py
Traceback (most recent call last):
  File "/home/oscar/.local/lib/python2.7/site-packages/script.py",
line 2, in <module>
    import optparse
  File "/home/oscar/.local/lib/python2.7/site-packages/optparse.py",
line 1, in <module>
    raise ImportError('Wrong optparse!')
ImportError: Wrong optparse!
~$ python -m script
Running script.py
~$ python ~/.local/lib/python2.7/site-packages/script.py
Running script.py
Traceback (most recent call last):
  File "/home/oscar/.local/lib/python2.7/site-packages/script.py",
line 2, in <module>
    import optparse
  File "/home/oscar/.local/lib/python2.7/site-packages/optparse.py",
line 1, in <module>
    raise ImportError('Wrong optparse!')
ImportError: Wrong optparse!
~$ python -m script
Running script.py

>
> As you say, a more refined check could determine a blank __package__
> as opposed to a None __package__. But this code (cherrypy) must also
> cope with version of Python before 2.6 which didn't even have a
> __package__ attribute, muddying the waters that little bit further.

Then you won't be able to use this method to get the -m switch to work
on Python < 2.6. If it's ok to just never try the -m switch on those
versions then it's as simple as doing:
pkg = getattr(__main__, '__package__', None)


Oscar

[toc] | [prev] | [next] | [standalone]

#37630

On 24/01/2013 20:01, Oscar Benjamin wrote:
> On 24 January 2013 17:13, Tim Golden <mail@timgolden.me.uk> wrote:
>> A package-based module run via -m (python -m package.module) works
>> as described (including the implicit __main__ module, my
>> primary use-case).
>
> Does it work in the "python -m package.module" case? It looks to me as
> if the code below will run "python -m package" instead. I think you
> need to split the module name out of sys.argv[0] and put that into the
> command line as well.

Good catch. Unless I am, once again, mistaken, I can't see any way
of recreating a dotted module path which is running as a main module
without the somewhat fragile expedient of walking back up the
filepath and hoping that nothing strange has happened to the imports
on the way down? (pywin32: I'm looking at you!)

And we haven't even touched on zipped archives or other invented
loaders (pulling modules from a database or direct from the internet).

I think, though, that the point which with I started this thread is
still valid: that to reconstruct the command line with which Python
was started, you need an increasingly awkward set of conditions and
checks. As opposed to asking the interpreter: please give me the command
line which started you.

How common that particular requirement actually is, I couldn't say.
But the two cases which clearly could use it are: the reloader
setup we've been discussing here; and the apply-new-version-and-restart
which I'm hoping to use in another setup at work.

Thanks for all the help and input on this, Oscar

TJG

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web