Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #81284

Re: PyWart: "Python's import statement and the history of external dependencies"

Show key headers only | View raw

On Sat, Nov 22, 2014 at 11:25 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> Ian Kelly wrote:
>
> - It's hard to keep track of what modules are in the standard library. Which
> of the following is *not* in Python 3.3's std lib? No cheating by looking
> them up.)
>
>     os2emxpath, wave, sndheader, statslib, poplist, plist,
>     pickletools, picklelib, path, cgi, cgitb, copylib, xpath

Okay, here's my guesses.

os2emxpath: In the stdlib, but more often accessed as "os.path" while
running under OS/2
wave: Not in the stdlib, though I'd avoid the name anyway.
sndheader: Not in the stdlib - probably on PyPI though
poplist, plist, pickletools, picklelib: I suspect PyPI, not stdlib,
but could be wrong
path: Not in the stdlib (there's os.path and I doubt there'd be both)
cgi, cgitb: In the stdlib
copylib: No idea, could be either way.
xpath: I'll guess this as not being present.

I'm probably pretty wrong, though.

>>> # Contrary to popular belief, sys.path is *NOT* a module,  #
>>> # no, it's a global!                                       #
>>
>> I really doubt that this is a popular belief.
>
> I'm not aware of anyone who believes that sys.path is a module.
> But yes, sys.path is not just global, but process-wide global. *All* modules
> share the same sys.path.

Even leaving aside Rick's sloppy language, I still doubt that it's
popular belief that sys.path be module-specific. You're modifying
something in a different module, and Python's always maintained that
two instances of "import sys" will give two references to the exact
same module object.

> That would be horrible. But here's an alternative which is less horrible and
> maybe even useful.
>
> There's still a single process-wide search path, but there's a second
> per-module search path which is searched first. By default it's empty.
>
> So a module can define it's own extra search path:
>
> __path__ = ['look/here', 'and/here']
> import something
>
> without affecting any other modules.

That's what Rick said first, and then said that if you're going to be
explicit, you should do the job properly and not have _any_ implicit
paths.

Thing is, though, it still breaks the sys.modules concept. Either
__path__ is ignored if the module was found in sys.modules, or it's
possible to have multiple entries with the same name (which would make
it hard to have a module replace itself in sys.modules, currently a
supported thing). Although I suppose all it'd require is that
sys.modules be keyed by __file__ rather than __name__, so they're
identified by fully qualified path and file name. (What does that do
in the face of .pyc files?)

>> And after all that, it would still fail if you happened to want to
>> import both "calendar" modules into the same module.
>
> __path__ = []
> import calendar
> __path__ = ['my/python/modules']
> import calendar as mycalendar

Frankly, if you actually want this, I think it's time to turn to an
uglier-but-more-flexible method.like poking around in importlib. (I'm
not sure off-hand how you'd go about it, it's not instantly obvious
from help(importlib).) I'm more concerned about the possibility of
your import succeeding or failing depending on the order of other
imports:

# foo.py
import calendar

# bar.py
__path__ = ['my/python/modules']
import foo
import calendar

How's that one to be resolved? That's what I don't like.

So long as sys.modules is (a) process-wide and (b) keyed by module
name rather than file name, sys.path MUST be process-wide too, and
MUST be set on startup, or as soon as possible afterwards. Any module
imported prior to altering sys.path will be fetched based on the
previous search path - and you have to import sys to change sys.path,
which means the minimum set of unalterable modules is, on Python 3.5:

rosuav@sikorsky:~$ cat showmods.py
import sys
print(", ".join(sorted(sys.modules.keys())))
rosuav@sikorsky:~$ python3 showmods.py
__main__, _codecs, _collections_abc, _frozen_importlib, _imp, _io,
_signal, _sitebuiltins, _stat, _sysconfigdata, _thread, _warnings,
_weakref, _weakrefset, abc, builtins, codecs, encodings,
encodings.aliases, encodings.latin_1, encodings.utf_8, errno,
genericpath, io, marshal, os, os.path, posix, posixpath, site, stat,
sys, sysconfig, zipimport

... that's a decent lot of modules you can't fiddle with. Hence
PYTHONPATH, which presumably is processed by the interpreter prior to
loading any modules.

ChrisA

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

PyWart: "Python's import statement and the history of external dependencies" Rick Johnson <rantingrickjohnson@gmail.com> - 2014-11-15 21:01 -0800
  Re: PyWart: "Python's import statement and the history of external dependencies" Chris Angelico <rosuav@gmail.com> - 2014-11-16 16:16 +1100
  Re: PyWart: "Python's import statement and the history of external dependencies" alex23 <wuwei23@gmail.com> - 2014-11-21 10:14 +1000
    Re: PyWart: "Python's import statement and the history of external dependencies" Chris Angelico <rosuav@gmail.com> - 2014-11-21 11:31 +1100
    Re: PyWart: "Python's import statement and the history of external dependencies" Rick Johnson <rantingrickjohnson@gmail.com> - 2014-11-20 19:53 -0800
      Re: PyWart: "Python's import statement and the history of external dependencies" Tim Chase <python.list@tim.thechases.com> - 2014-11-21 04:26 -0600
        Re: PyWart: "Python's import statement and the history of external dependencies" Rick Johnson <rantingrickjohnson@gmail.com> - 2014-11-21 07:52 -0800
          Re: PyWart: "Python's import statement and the history of external dependencies" Tim Chase <python.list@tim.thechases.com> - 2014-11-21 10:12 -0600
          Re: PyWart: "Python's import statement and the history of external dependencies" Ian Kelly <ian.g.kelly@gmail.com> - 2014-11-21 10:42 -0700
      Re: PyWart: "Python's import statement and the history of external dependencies" Ned Batchelder <ned@nedbatchelder.com> - 2014-11-21 06:32 -0500
      Re: PyWart: "Python's import statement and the history of external dependencies" Ian Kelly <ian.g.kelly@gmail.com> - 2014-11-21 08:33 -0700
        Re: PyWart: "Python's import statement and the history of external dependencies" Rick Johnson <rantingrickjohnson@gmail.com> - 2014-11-21 10:24 -0800
          Re: PyWart: "Python's import statement and the history of external dependencies" Chris Angelico <rosuav@gmail.com> - 2014-11-22 05:34 +1100
          Re: PyWart: "Python's import statement and the history of external dependencies" Michael Torrie <torriem@gmail.com> - 2014-11-21 12:05 -0700
            Re: PyWart: "Python's import statement and the history of external dependencies" Rick Johnson <rantingrickjohnson@gmail.com> - 2014-11-21 12:29 -0800
              Re: PyWart: "Python's import statement and the history of external dependencies" Michael Torrie <torriem@gmail.com> - 2014-11-21 15:54 -0700
          Re: PyWart: "Python's import statement and the history of external dependencies" Ian Kelly <ian.g.kelly@gmail.com> - 2014-11-21 12:23 -0700
            Re: PyWart: "Python's import statement and the history of external dependencies" Rick Johnson <rantingrickjohnson@gmail.com> - 2014-11-21 14:25 -0800
              Re: PyWart: "Python's import statement and the history of external dependencies" Rick Johnson <rantingrickjohnson@gmail.com> - 2014-11-21 15:21 -0800
                Re: PyWart: "Python's import statement and the history of external dependencies" Chris Angelico <rosuav@gmail.com> - 2014-11-22 10:52 +1100
                Re: PyWart: "Python's import statement and the history of external dependencies" Rick Johnson <rantingrickjohnson@gmail.com> - 2014-11-21 17:07 -0800
                Re: PyWart: "Python's import statement and the history of external dependencies" Chris Angelico <rosuav@gmail.com> - 2014-11-22 13:36 +1100
                Re: PyWart: "Python's import statement and the history of external dependencies" Ian Kelly <ian.g.kelly@gmail.com> - 2014-11-22 02:27 -0700
                Re: PyWart: "Python's import statement and the history of external dependencies" sohcahtoa82@gmail.com - 2014-11-21 16:33 -0800
                Import a module from a specific file path (was: PyWart: "Python's import statement and the history of external dependencies") Ben Finney <ben+python@benfinney.id.au> - 2014-11-22 11:37 +1100
                Re: Import a module from a specific file path Marko Rauhamaa <marko@pacujo.net> - 2014-11-22 08:28 +0200
                Re: Import a module from a specific file path (was: PyWart: "Python's import statement and the history of external dependencies") Chris Angelico <rosuav@gmail.com> - 2014-11-22 13:32 +1100
                Re: Import a module from a specific file path Ben Finney <ben+python@benfinney.id.au> - 2014-11-22 17:46 +1100
              Re: PyWart: "Python's import statement and the history of external dependencies" Ian Kelly <ian.g.kelly@gmail.com> - 2014-11-21 17:32 -0700
                Re: PyWart: "Python's import statement and the history of external dependencies" Rick Johnson <rantingrickjohnson@gmail.com> - 2014-11-21 17:48 -0800
          Re: PyWart: "Python's import statement and the history of external dependencies" Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2014-11-21 15:38 -0500
        Re: PyWart: "Python's import statement and the history of external dependencies" Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-11-22 23:25 +1100
          Re: PyWart: "Python's import statement and the history of external dependencies" Chris Angelico <rosuav@gmail.com> - 2014-11-23 00:00 +1100
            Re: PyWart: "Python's import statement and the history of external dependencies" Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-11-23 17:30 +1100
              Re: PyWart: "Python's import statement and the history of external dependencies" Rustom Mody <rustompmody@gmail.com> - 2014-11-22 22:44 -0800
              Re: PyWart: "Python's import statement and the history of external dependencies" Chris Angelico <rosuav@gmail.com> - 2014-11-23 17:56 +1100
              Re: PyWart: "Python's import statement and the history of external dependencies" Chris Angelico <rosuav@gmail.com> - 2014-11-23 17:59 +1100
                Re: PyWart: "Python's import statement and the history of external   dependencies" Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2014-11-24 11:37 +1300
                Re: PyWart: "Python's import statement and the history of external dependencies" Rick Johnson <rantingrickjohnson@gmail.com> - 2014-11-23 22:25 -0800
                Re: PyWart: "Python's import statement and the history of external dependencies" Chris Angelico <rosuav@gmail.com> - 2014-11-24 17:34 +1100
                Re: PyWart: "Python's import statement and the history of external dependencies" Mark Lawrence <breamoreboy@yahoo.co.uk> - 2014-11-24 10:34 +0000
                Re: PyWart: "Python's import statement and the history of external dependencies" Rick Johnson <rantingrickjohnson@gmail.com> - 2014-11-24 08:51 -0800
                Re: PyWart: "Python's import statement and the history of external dependencies" Ned Batchelder <ned@nedbatchelder.com> - 2014-11-24 12:06 -0500
                Re: PyWart: "Python's import statement and the history of external dependencies" Robin Becker <robin@reportlab.com> - 2014-11-24 17:07 +0000
                Re: PyWart: "Python's import statement and the history of external dependencies" Marko Rauhamaa <marko@pacujo.net> - 2014-11-24 19:08 +0200
                Re: PyWart: "Python's import statement and the history of external dependencies" Rustom Mody <rustompmody@gmail.com> - 2014-11-24 09:38 -0800
                Re: PyWart: "Python's import statement and the history of external dependencies" Ian Kelly <ian.g.kelly@gmail.com> - 2014-11-24 15:34 -0700
              Re: PyWart: "Python's import statement and the history of external dependencies" Ben Finney <ben+python@benfinney.id.au> - 2014-11-23 18:43 +1100
              Re: PyWart: "Python's import statement and the history of external dependencies" Chris Angelico <rosuav@gmail.com> - 2014-11-23 18:51 +1100
              Re: PyWart: "Python's import statement and the history of external dependencies" Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2014-11-23 11:23 -0500
              Re: PyWart: "Python's import statement and the history of external dependencies" Chris Angelico <rosuav@gmail.com> - 2014-11-24 03:36 +1100
          Re: PyWart: "Python's import statement and the history of external dependencies" Tim Chase <python.list@tim.thechases.com> - 2014-11-22 15:27 -0600
            Re: PyWart: "Python's import statement and the history of external dependencies" Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-11-23 12:00 +1100
              Re: PyWart: "Python's import statement and the history of external dependencies" Tim Chase <python.list@tim.thechases.com> - 2014-11-22 19:33 -0600

csiph-web