Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #80850 > unrolled thread

PyWart: "Python's import statement and the history of external dependencies"

Back to article view | Back to comp.lang.python

Contents

  PyWart: "Python's import statement and the history of external dependencies" Rick Johnson <rantingrickjohnson@gmail.com> - 2014-11-15 21:01 -0800
    Re: PyWart: "Python's import statement and the history of external dependencies" Chris Angelico <rosuav@gmail.com> - 2014-11-16 16:16 +1100
    Re: PyWart: "Python's import statement and the history of external dependencies" alex23 <wuwei23@gmail.com> - 2014-11-21 10:14 +1000
      Re: PyWart: "Python's import statement and the history of external dependencies" Chris Angelico <rosuav@gmail.com> - 2014-11-21 11:31 +1100
      Re: PyWart: "Python's import statement and the history of external dependencies" Rick Johnson <rantingrickjohnson@gmail.com> - 2014-11-20 19:53 -0800
        Re: PyWart: "Python's import statement and the history of external dependencies" Tim Chase <python.list@tim.thechases.com> - 2014-11-21 04:26 -0600
          Re: PyWart: "Python's import statement and the history of external dependencies" Rick Johnson <rantingrickjohnson@gmail.com> - 2014-11-21 07:52 -0800
            Re: PyWart: "Python's import statement and the history of external dependencies" Tim Chase <python.list@tim.thechases.com> - 2014-11-21 10:12 -0600
            Re: PyWart: "Python's import statement and the history of external dependencies" Ian Kelly <ian.g.kelly@gmail.com> - 2014-11-21 10:42 -0700
        Re: PyWart: "Python's import statement and the history of external dependencies" Ned Batchelder <ned@nedbatchelder.com> - 2014-11-21 06:32 -0500
        Re: PyWart: "Python's import statement and the history of external dependencies" Ian Kelly <ian.g.kelly@gmail.com> - 2014-11-21 08:33 -0700
          Re: PyWart: "Python's import statement and the history of external dependencies" Rick Johnson <rantingrickjohnson@gmail.com> - 2014-11-21 10:24 -0800
            Re: PyWart: "Python's import statement and the history of external dependencies" Chris Angelico <rosuav@gmail.com> - 2014-11-22 05:34 +1100
            Re: PyWart: "Python's import statement and the history of external dependencies" Michael Torrie <torriem@gmail.com> - 2014-11-21 12:05 -0700
              Re: PyWart: "Python's import statement and the history of external dependencies" Rick Johnson <rantingrickjohnson@gmail.com> - 2014-11-21 12:29 -0800
                Re: PyWart: "Python's import statement and the history of external dependencies" Michael Torrie <torriem@gmail.com> - 2014-11-21 15:54 -0700
            Re: PyWart: "Python's import statement and the history of external dependencies" Ian Kelly <ian.g.kelly@gmail.com> - 2014-11-21 12:23 -0700
              Re: PyWart: "Python's import statement and the history of external dependencies" Rick Johnson <rantingrickjohnson@gmail.com> - 2014-11-21 14:25 -0800
                Re: PyWart: "Python's import statement and the history of external dependencies" Rick Johnson <rantingrickjohnson@gmail.com> - 2014-11-21 15:21 -0800
                  Re: PyWart: "Python's import statement and the history of external dependencies" Chris Angelico <rosuav@gmail.com> - 2014-11-22 10:52 +1100
                    Re: PyWart: "Python's import statement and the history of external dependencies" Rick Johnson <rantingrickjohnson@gmail.com> - 2014-11-21 17:07 -0800
                      Re: PyWart: "Python's import statement and the history of external dependencies" Chris Angelico <rosuav@gmail.com> - 2014-11-22 13:36 +1100
                      Re: PyWart: "Python's import statement and the history of external dependencies" Ian Kelly <ian.g.kelly@gmail.com> - 2014-11-22 02:27 -0700
                  Re: PyWart: "Python's import statement and the history of external dependencies" sohcahtoa82@gmail.com - 2014-11-21 16:33 -0800
                  Import a module from a specific file path (was: PyWart: "Python's import statement and the history of external dependencies") Ben Finney <ben+python@benfinney.id.au> - 2014-11-22 11:37 +1100
                    Re: Import a module from a specific file path Marko Rauhamaa <marko@pacujo.net> - 2014-11-22 08:28 +0200
                  Re: Import a module from a specific file path (was: PyWart: "Python's import statement and the history of external dependencies") Chris Angelico <rosuav@gmail.com> - 2014-11-22 13:32 +1100
                  Re: Import a module from a specific file path Ben Finney <ben+python@benfinney.id.au> - 2014-11-22 17:46 +1100
                Re: PyWart: "Python's import statement and the history of external dependencies" Ian Kelly <ian.g.kelly@gmail.com> - 2014-11-21 17:32 -0700
                  Re: PyWart: "Python's import statement and the history of external dependencies" Rick Johnson <rantingrickjohnson@gmail.com> - 2014-11-21 17:48 -0800
            Re: PyWart: "Python's import statement and the history of external dependencies" Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2014-11-21 15:38 -0500
          Re: PyWart: "Python's import statement and the history of external dependencies" Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-11-22 23:25 +1100
            Re: PyWart: "Python's import statement and the history of external dependencies" Chris Angelico <rosuav@gmail.com> - 2014-11-23 00:00 +1100
              Re: PyWart: "Python's import statement and the history of external dependencies" Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-11-23 17:30 +1100
                Re: PyWart: "Python's import statement and the history of external dependencies" Rustom Mody <rustompmody@gmail.com> - 2014-11-22 22:44 -0800
                Re: PyWart: "Python's import statement and the history of external dependencies" Chris Angelico <rosuav@gmail.com> - 2014-11-23 17:56 +1100
                Re: PyWart: "Python's import statement and the history of external dependencies" Chris Angelico <rosuav@gmail.com> - 2014-11-23 17:59 +1100
                  Re: PyWart: "Python's import statement and the history of external   dependencies" Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2014-11-24 11:37 +1300
                    Re: PyWart: "Python's import statement and the history of external dependencies" Rick Johnson <rantingrickjohnson@gmail.com> - 2014-11-23 22:25 -0800
                      Re: PyWart: "Python's import statement and the history of external dependencies" Chris Angelico <rosuav@gmail.com> - 2014-11-24 17:34 +1100
                      Re: PyWart: "Python's import statement and the history of external dependencies" Mark Lawrence <breamoreboy@yahoo.co.uk> - 2014-11-24 10:34 +0000
                        Re: PyWart: "Python's import statement and the history of external dependencies" Rick Johnson <rantingrickjohnson@gmail.com> - 2014-11-24 08:51 -0800
                          Re: PyWart: "Python's import statement and the history of external dependencies" Ned Batchelder <ned@nedbatchelder.com> - 2014-11-24 12:06 -0500
                          Re: PyWart: "Python's import statement and the history of external dependencies" Robin Becker <robin@reportlab.com> - 2014-11-24 17:07 +0000
                          Re: PyWart: "Python's import statement and the history of external dependencies" Marko Rauhamaa <marko@pacujo.net> - 2014-11-24 19:08 +0200
                            Re: PyWart: "Python's import statement and the history of external dependencies" Rustom Mody <rustompmody@gmail.com> - 2014-11-24 09:38 -0800
                    Re: PyWart: "Python's import statement and the history of external dependencies" Ian Kelly <ian.g.kelly@gmail.com> - 2014-11-24 15:34 -0700
                Re: PyWart: "Python's import statement and the history of external dependencies" Ben Finney <ben+python@benfinney.id.au> - 2014-11-23 18:43 +1100
                Re: PyWart: "Python's import statement and the history of external dependencies" Chris Angelico <rosuav@gmail.com> - 2014-11-23 18:51 +1100
                Re: PyWart: "Python's import statement and the history of external dependencies" Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2014-11-23 11:23 -0500
                Re: PyWart: "Python's import statement and the history of external dependencies" Chris Angelico <rosuav@gmail.com> - 2014-11-24 03:36 +1100
            Re: PyWart: "Python's import statement and the history of external dependencies" Tim Chase <python.list@tim.thechases.com> - 2014-11-22 15:27 -0600
              Re: PyWart: "Python's import statement and the history of external dependencies" Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-11-23 12:00 +1100
                Re: PyWart: "Python's import statement and the history of external dependencies" Tim Chase <python.list@tim.thechases.com> - 2014-11-22 19:33 -0600

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

#81261

On Friday, November 21, 2014 5:59:44 PM UTC-6, Chris Angelico wrote:
> On Sat, Nov 22, 2014 at 10:21 AM, Rick Johnson
> <rantingrickjohnson@gmail.com> wrote:
> > 1. Use the historical "implicit import" mechanism for most day
> > to day imports, and let Python do all the heavy lifting.
> >
> > 2. Use the new "explicit import" mechanism for advanced name
> > resolutions, but realize that since you are now taking
> > control of import, so you must do more work, and you must be
> > "exhaustively explicit" about *where* Python searches.
> 
> In other words, what you want is:
> 
> # today's method, import based on search path
> import sys
> # new explicit path method
> import '/usr/local/lib/python3.5/lib-dynload/math.cpython-35m.so'
> 
> [...]
> 
> There are a VERY VERY few cases where you really do want
> to import a module from a specific file path. For those
> situations, there are already techniques you can use. Why
> change the behaviour of import itself?

Chris, either take the time to read and understand my posts
*FULLY*, or don't bother to reply. I have in no way
suggested that we import via "filepaths". Stop lying.

[toc] | [prev] | [next] | [standalone]

#81263

On Sat, Nov 22, 2014 at 12:07 PM, Rick Johnson
<rantingrickjohnson@gmail.com> wrote:
> On Friday, November 21, 2014 5:59:44 PM UTC-6, Chris Angelico wrote:
>> On Sat, Nov 22, 2014 at 10:21 AM, Rick Johnson
>> <rantingrickjohnson@gmail.com> wrote:
>> > 1. Use the historical "implicit import" mechanism for most day
>> > to day imports, and let Python do all the heavy lifting.
>> >
>> > 2. Use the new "explicit import" mechanism for advanced name
>> > resolutions, but realize that since you are now taking
>> > control of import, so you must do more work, and you must be
>> > "exhaustively explicit" about *where* Python searches.
>>
>> In other words, what you want is:
>>
>> # today's method, import based on search path
>> import sys
>> # new explicit path method
>> import '/usr/local/lib/python3.5/lib-dynload/math.cpython-35m.so'
>>
>> [...]
>>
>> There are a VERY VERY few cases where you really do want
>> to import a module from a specific file path. For those
>> situations, there are already techniques you can use. Why
>> change the behaviour of import itself?
>
> Chris, either take the time to read and understand my posts
> *FULLY*, or don't bother to reply. I have in no way
> suggested that we import via "filepaths". Stop lying.

Okay. Explain to me the difference between your proposed "explicit
import" mechanism, where you specifically name the path where Python
should import something from, and "importing via filepaths".

ChrisA

[toc] | [prev] | [next] | [standalone]

#81281

On Fri, Nov 21, 2014 at 6:07 PM, Rick Johnson
<rantingrickjohnson@gmail.com> wrote:
> On Friday, November 21, 2014 5:59:44 PM UTC-6, Chris Angelico wrote:
>> In other words, what you want is:
>>
>> # today's method, import based on search path
>> import sys
>> # new explicit path method
>> import '/usr/local/lib/python3.5/lib-dynload/math.cpython-35m.so'
>>
>> [...]
>>
>> There are a VERY VERY few cases where you really do want
>> to import a module from a specific file path. For those
>> situations, there are already techniques you can use. Why
>> change the behaviour of import itself?
>
> Chris, either take the time to read and understand my posts
> *FULLY*, or don't bother to reply. I have in no way
> suggested that we import via "filepaths". Stop lying.

I see very little substantive difference between what Chris posted and
your suggestion of:

__search_path__ = ['/usr/local/lib/python3.5/lib-dynload']
import math

His comment about non-portability stands.

[toc] | [prev] | [next] | [standalone]

#81258

On Friday, November 21, 2014 3:21:31 PM UTC-8, Rick Johnson wrote:
> On Friday, November 21, 2014 4:25:49 PM UTC-6, Rick Johnson wrote:
> > ############################################################
> > #                          STEP 3                          #
> > ############################################################
> > # Make the following changes to the import machinery:      #
> > # Before Python reads a module file, Python will clear the #
> > # values in "sys.path_extra", OR, query the                #
> > # "__search_paths__" variable, if any paths exists in this #
> > # list, THEN THESE PATHS MUST BE SEARCHED, AND THEY MUST   #
> > # BE SEARCHED BEFORE ANY PATHS IN "sys.path", AND NO       #
> > # PEEKING IN "sys.modules" IS ALLOWED!                     #
> > ############################################################
> 
> 
> Actually, to be more specific, not only should the
> __search_path__ variable be unique to each file, Python
> should query the value of "__search_path__" variable each
> time it encounters an import statement.
> 
> So, if we wanted to import multiple modules who share the
> same name, but who exist in different namespaces, we would
> could do so by changing the values of __search_path__ before
> calling the import:
> 
>     #
>     # import the first coconut module Python can find using
>     # the default search path, if not raise an error
>     #
>     import coconut
> 
>     #
>     # import the first coconut module python can find using
>     # *ONLY* the explicit paths provided, or raise an error.
>     #
>     __search_path__ = [
>         'path/to/a/specific/coconut/tree',
>         'might/be/here/also'
>         ]
>     import coconut
> 
> At first i was thinking that *IF* any "explicit paths" were
> defined via __search_path__, then Python should look in all
> those paths first, then if nothing is found look in
> sys.modules, then if nothing is found, exhaust sys.modules,
> 
> BUT NOW I AM THINKING THAT IS A BAD IDEA!
> 
> When __search_path__ *IS* defined, Python should look *ONLY*
> in the provided paths, and if not module is found, raise an
> "ExplictImportError".
> 
> Because if we're going to be explicit then we need to also be
> consistent. If the programmer is requesting that Python
> search in a predefined directories, then obviously he
> doing so to avoid name clashes, so if Python went on and
> returned a module OUTSIDE of those predefined paths, Python
> would be circumventing (IMPLICITLY AGAIN!) the wishes of the
> programmer.
> 
> ============================================================
>  Succinct Summary Of My New Proposal:
> ============================================================
> 
> 1. Use the historical "implicit import" mechanism for most day
> to day imports, and let Python do all the heavy lifting.
> 
> 2. Use the new "explicit import" mechanism for advanced name
> resolutions, but realize that since you are now taking
> control of import, so you must do more work, and you must be
> "exhaustively explicit" about *where* Python searches.

I have a better idea.

Don't give modules the same name as built-in or very popular modules.

You seem like the type of person that would stick their hair into an electric egg beater and then blame the manufacturer because it ripped out their hair.  You'd probably say that the egg beater should recognize that there's hair in the beaters and turn off.

[toc] | [prev] | [next] | [standalone]

#81260 — Import a module from a specific file path (was: PyWart: "Python's import statement and the history of external dependencies")

Chris Angelico <rosuav@gmail.com> writes:

> In other words, what you want is:
>
> # today's method, import based on search path
> import sys
> # new explicit path method
> import '/usr/local/lib/python3.5/lib-dynload/math.cpython-35m.so'

I don't think I'd ever want to specify an absolute file path for the
module. But it would make my Python life immeasurably better if I could
specify *relative* file paths for importing a module.

    import '../foo/bar/beans.py' as beans

> Can you tell me, please, how this is going to work for *any* system
> other than the one it was built on?

Allowing relative paths makes this portable, so long as the
application's relative tree structure is maintained.

> There are a VERY VERY few cases where you really do want to import a
> module from a specific file path. For those situations, there are
> already techniques you can use. Why change the behaviour of import
> itself?

An example:

* I write a program, ‘fooprog’, using the hypothetical “import a module
  at a specified file path” feature::

    #! /usr/bin/python3

    import './bar/beans.py' as beans

    beans.do_stuff()

* The program ‘fooprog’ and the module ‘beans.py’ are in a sensible
  directory structure::

    foo-proj-1.0/
        foo/
            __init__.py
            fooprog
            bar/
                __init__.py
                beans.py
        tests/
            __init__.py
            test_fooprog.py
            test_bar.py

* Running the program ‘fooprog’ invokes Python (via its standard
  shebang), which invokes the file ‘fooprog’ as the ‘__main__’ module
  *with no package awareness*.

  Or, equivalently, ‘python foo/fooprog’ skips the shebang step.

* Regardless of package considerations, specifying ‘./bar/beans.py’
  locates the module ‘beans’ for import, just fine.

That example will work in the presence of this “import from a named file
location” feature.

But it will fail with the standard Python import feature today. How
would you expect this example to change so it will work with current
Python?

Solutions usually seem to entail contortions of cluttering the import
block by discovering the current path, and fussing around with
‘sys.path’, before finally doing the import::

    #! /usr/bin/python3

    import sys
    import os.path

    program_dir = os.path.dirname(__file__)
    parent_dir = os.path.dirname(program_dir)
    if not parent_dir in sys.path:
        sys.path.insert(1, parent_dir)

    import foo.bar.beans as beans

    beans.do_stuff()

Maybe you can suggest a better portable method to do this today in
Python.

Importantly, once you've come up with how to do it today in Python: Do
you really consider that superior to simply specifying a filesystem path
for a file containing the module?

-- 
 \        “I'd take the awe of understanding over the awe of ignorance |
  `\                                          any day.” —Douglas Adams |
_o__)                                                                  |
Ben Finney

[toc] | [prev] | [next] | [standalone]

#81276 — Re: Import a module from a specific file path

Ben Finney <ben+python@benfinney.id.au>:

> Solutions usually seem to entail contortions of cluttering the import
> block by discovering the current path, and fussing around with
> ‘sys.path’, before finally doing the import::
>
>     #! /usr/bin/python3
>
>     import sys
>     import os.path
>
>     program_dir = os.path.dirname(__file__)

More robustly:

   program_dir = os.path.dirname(os.path.realname(__file__))

Executables are often soft links.

> Importantly, once you've come up with how to do it today in Python: Do
> you really consider that superior to simply specifying a filesystem
> path for a file containing the module?

Your complaint is valid. Many modern programming languages (also java,
guile, elisp etc) suffer from it. Java has alleviated the problem with
jar files.

So why don't C programmers complain about the problem?

First, the #include directive does exactly what you are asking: it
supports relative paths. Secondly, you can build executables statically.


Marko

[toc] | [prev] | [next] | [standalone]

#81264 — Re: Import a module from a specific file path (was: PyWart: "Python's import statement and the history of external dependencies")

On Sat, Nov 22, 2014 at 11:37 AM, Ben Finney <ben+python@benfinney.id.au> wrote:
> I don't think I'd ever want to specify an absolute file path for the
> module. But it would make my Python life immeasurably better if I could
> specify *relative* file paths for importing a module.
>
> Allowing relative paths makes this portable, so long as the
> application's relative tree structure is maintained.
>
> Maybe you can suggest a better portable method to do this today in
> Python.

Ah, didn't think of relative paths. Yes, I can see that'd be both more
useful and less problematic.

What you suggest looks very much like you're running something from a
package, though.

> * The program ‘fooprog’ and the module ‘beans.py’ are in a sensible
>   directory structure::
>
>     foo-proj-1.0/
>         foo/
>             __init__.py
>             fooprog
>             bar/
>                 __init__.py
>                 beans.py
>         tests/
>             __init__.py
>             test_fooprog.py
>             test_bar.py

So can you simply:

$ python -m foo.fooprog

? (or 'python3', either way)

ChrisA

[toc] | [prev] | [next] | [standalone]

#81278 — Re: Import a module from a specific file path

Chris Angelico <rosuav@gmail.com> writes:

> So can you simply:
>
> $ python -m foo.fooprog
>
> ? (or 'python3', either way)

So, that's a pretty awful user interface. (The file is named ‘fooprog’
because it's an executable, intended to be run directly at the command
line.) That solution would be rather inferior to the program being able
to import modules when invoked as the top-level program.

Can you suggest solutions that allow invoking the program as ‘./fooprog’
to work?

-- 
 \         “In economics, hope and faith coexist with great scientific |
  `\      pretension and also a deep desire for respectability.” —John |
_o__)                                    Kenneth Galbraith, 1970-06-07 |
Ben Finney

[toc] | [prev] | [next] | [standalone]

#81259

On Fri, Nov 21, 2014 at 3:25 PM, Rick Johnson
<rantingrickjohnson@gmail.com> wrote:
>> The only exception is if you're doing "import calendar" from inside
>> the ricklib package, and you're using Python 2, and you don't have
>> "from __future__ import absolute_import" at the top of your module.
>> The solution to this is easy: just add that __future__ import to the
>> top of your module, and poof, implicit relative imports don't happen.
>> This is also fixed entirely in Python 3.
>
> I wish the irony of needing to know an "implicit rule" to
> solve an "implicit problem" could be funny, but it's really
> just sad. I can't help but be reminded of the Python zen.
> "If it's difficult to explain it's probably a bad idea".
> What's more difficult to explain than an implicit rule you
> have no knowledge of?

It's not so much implicit as an old wart that was fixed in a
backward-compatible way. As I said it's not an issue in Python 3, so
it's kind of pointless to be griping about it now.

>> > Anyone would expect that when *DIRECTLY* importing a
>> > package, if the __init__ file has code, then THAT code
>> > should be executed, HOWEVER,  not many would expect that
>> > merely "referencing" the package name (in order to import a
>> > more deeply nested package) would cause ALL the
>> > intermediate __init__ files to execute -- this is madness,
>> > and it prevents using an __init__ file as an "import hub"
>> > (without side-effects)!
>>
>> The whole point of the __init__.py file, in case you didn't intuit it
>> from the name, is to host any initialization code for the package. Why
>> on earth would you expect to import a module from a package without
>> initializing the package?
>
> Because you failed to notice that i was NOT importing the
> "package" which contained the __init__file, no, i was
> importing a *SUB PACKAGE* of the package that contained the
> __init__ file.

No, I understood that. My question stands.

> Why does the code in the main package need to run when i
> *explicitly* and *directly* fetched a "nested resource"
> within the package? Nothing within the __init__ file is
> affecting the code within the subpackage i wanted, and
> inversely, the package i wanted does not depend on anything
> within the __init__ file. There exists no symbiotic
> relationship between these two machinery, yet, by
> referencing one of them, the other starts doing unnecessary
> things!

It has nothing to do with the __init__ file specifically.  When you
import ham.spam.eggs, Python first imports ham, then ham.spam, then
ham.spam.eggs. The reason for this should be obvious: the module is
imported as a local variable in whatever context the code is running
in. "ham.spam.eggs" is not a legal variable name. Instead it has to
use the variable "ham", and populate it with the module ham. The
module "ham" then gets an attribute (i.e. module-level global) named
"spam" that is populated with the module ham.spam. Finally that module
gets an attribute "eggs" that is populated with the ham.spam.eggs
module. Because of this, it is impossible to import the module
ham.spam.eggs (under that name, anyway) without first importing ham
and ham.spam.

Another reason is that a package can affect the way its contents are
imported, e.g. by setting a package-level __path__ variable.

But also, the __init__.py is executed because it is considered to be
the initializer for the entire package. Would you expect to be able to
create a class instance and call one of its methods without having the
__init__ method called? The __init__.py file is called that because
it's meant to be analogous. What you seem to be asking for is a way
lump a subpackage into a package without treating that higher-level
package as a package. There is a way to do this, using namespace
packages, which have no __init__.py files.

> There is a workaround, but it's even more of a mess. In
> order to maintain a unique "import hub" without the chance
> of side effects from __init__ files, i can move all the code
> in "ricklib.subpkg1.__init__.py" (the code that does all the
> imports) into a normal file named
> ricklib.subpkg1._subpkg1.py".

Or you could just accept that if you don't want importing ham to
automatically import ham.spam, then you shouldn't put that import in
ham/__init__.py. Most __init__.py files *should* be empty.

> If Python expects me to use packages to protect my module
> names from clashing, but it gives me no method by which to
> import packages without causing side effects, then what is a
> boy to do (besides creating workarounds for workarounds)?

If you don't want any initialization to happen when you import your
modules, then don't put any initialization code in your __init__.py.
Or use namespace packages as mentioned above.

[toc] | [prev] | [next] | [standalone]

#81262

On Friday, November 21, 2014 6:33:32 PM UTC-6, Ian wrote:
> On Fri, Nov 21, 2014 at 3:25 PM, Rick Johnson
> > Why does the code in the main package need to run when i
> > *explicitly* and *directly* fetched a "nested resource"
> > within the package?[...]
> 
> It has nothing to do with the __init__ file specifically.
> When you import ham.spam.eggs [...]  Would you expect to
> be able to create a class instance and call one of its
> methods without having the __init__ method called?

Not an "instance method", but a "class method", yes! ;-)

Just as Objects have both class level *AND* instance level
scoping for methods, so too should packages -- in a
different sort of way of course. 

> If you don't want any initialization to happen when you
> import your modules, then don't put any initialization
> code in your __init__.py. Or use namespace packages as
> mentioned above.

Ha, that's what i realized after being caught in the
"__init__ nightmare" not too long ago :). But now I have my
deeply nested module namespaces working just fine under
Python 2.x. although i feel like the system is too clunky
and in need of refining. Maybe "namespace packages" are what
i need, although i see they are Python 3.x only. :-(

In any event, thanks for taking the time to explain the
details. I feel i understand the systems much better now.
Your "coconut example" was the key to my understanding. And
now, i can stand on the top of the mountain and see how the
system works -- I no longer have this damn mountain
blocking my view. 

I think Python's import and packaging systems are both very
"good ideas" that just happen to be covered in a blanket of
dust, and all we need to do it give then a good cleaning to
bring out that showroom shine -- and detailing is my
specialty!

[toc] | [prev] | [next] | [standalone]

#81250

On Fri, 21 Nov 2014 12:23:48 -0700, Ian Kelly <ian.g.kelly@gmail.com>
declaimed the following:

>On Fri, Nov 21, 2014 at 11:24 AM, Rick Johnson
><rantingrickjohnson@gmail.com> wrote:
>> Are you also going to call drivers "fools" because they bought
>> a "certain brand" of car only to have the airbag explode in
>> their face?
>
>No, but I'll call them fools if they buy a car and the engine catches
>fire because they never bothered to change the oil.
>
	It's more likely that the engine will seize from lack of lubrication...

	Now -- if they were pouring gasoline into the oil sump to thin the oil
so they can crank it over in the middle of winter, THEN I'd not be
surprised at the appearance of a flaming fool <G>
-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
    wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/

[toc] | [prev] | [next] | [standalone]

#81283

Ian Kelly wrote:

> On Thu, Nov 20, 2014 at 8:53 PM, Rick Johnson
> <rantingrickjohnson@gmail.com> wrote:
>> FOR INSTANCE: Let's say i write a module that presents a
>> reusable GUI calendar widget, and then i name the module
>> "calender.py".
>>
>> Then Later, when i try to import *MY* GUI widget named
>> "calendar", i will not get *MY* calendar widget, no, i will
>> get the Python calendar module.
> 
> Because like a fool you created a local module with a totally generic
> name like "calendar" and dumped it into the top-level namespace. 

Firstly, let me say that I am amazed that people have waded through Rick's
self-aggrandising purple prose, insults, demands and utter confusion to
find something worth engaging him on.

Having said that, it's not fair to blame the user for shadowing standard
library modules:

- All users start off as beginners, who may not be aware that this is even a
possibility;

- It's hard to keep track of what modules are in the standard library. Which
of the following is *not* in Python 3.3's std lib? No cheating by looking
them up.)

    os2emxpath, wave, sndheader, statslib, poplist, plist, 
    pickletools, picklelib, path, cgi, cgitb, copylib, xpath

- By default, sys.path is set up so that user modules have priority over std
lib modules. Given a name clash, shadowing in inevitable.


> The 
> Python import system gives you the ability to create packages, so use
> them -- create a package to contain all the widgets you create.


Packages come with their own challenges and complexities, especially if
you're trying to support both Python 2 and 3 from the same code base.
Besides, for simple uses, packages are overkill.


[...]
>> # Contrary to popular belief, sys.path is *NOT* a module,  #
>> # no, it's a global!                                       #
> 
> I really doubt that this is a popular belief.

I'm not aware of anyone who believes that sys.path is a module. If they did,
a simple import would prove that it is not:

py> import sys.path
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named path


But yes, sys.path is not just global, but process-wide global. *All* modules
share the same sys.path.


>> This is another confusing fundamental of Python, it seems *MORE*
>> intuitive that changes to the "import search path" would only
>> affect the *CURRENT* module's "import search path", but that's
>> is not the case!
> 
> This just seems like it would create a maintenance nightmare. Before
> you could import anything, you'd have to figure out what the search
> path is for the particular module you're working and whether it
> happens to include the particular package you want. You find it
> doesn't, so you make a local change to the search path. Later, you
> make the same change to other modules that need to import it. Later
> still, the package moves to a different location in the file system,
> and now you get to go through and update every one of those modules
> with the new directory. Lucky you.

That would be horrible. But here's an alternative which is less horrible and
maybe even useful.

There's still a single process-wide search path, but there's a second
per-module search path which is searched first. By default it's empty.

So a module can define it's own extra search path:

__path__ = ['look/here', 'and/here']
import something

without affecting any other modules.


> And after all that, it would still fail if you happened to want to
> import both "calendar" modules into the same module.

__path__ = []
import calendar
__path__ = ['my/python/modules']
import calendar as mycalendar



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#81284

On Sat, Nov 22, 2014 at 11:25 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> Ian Kelly wrote:
>
> - It's hard to keep track of what modules are in the standard library. Which
> of the following is *not* in Python 3.3's std lib? No cheating by looking
> them up.)
>
>     os2emxpath, wave, sndheader, statslib, poplist, plist,
>     pickletools, picklelib, path, cgi, cgitb, copylib, xpath

Okay, here's my guesses.

os2emxpath: In the stdlib, but more often accessed as "os.path" while
running under OS/2
wave: Not in the stdlib, though I'd avoid the name anyway.
sndheader: Not in the stdlib - probably on PyPI though
poplist, plist, pickletools, picklelib: I suspect PyPI, not stdlib,
but could be wrong
path: Not in the stdlib (there's os.path and I doubt there'd be both)
cgi, cgitb: In the stdlib
copylib: No idea, could be either way.
xpath: I'll guess this as not being present.

I'm probably pretty wrong, though.

>>> # Contrary to popular belief, sys.path is *NOT* a module,  #
>>> # no, it's a global!                                       #
>>
>> I really doubt that this is a popular belief.
>
> I'm not aware of anyone who believes that sys.path is a module.
> But yes, sys.path is not just global, but process-wide global. *All* modules
> share the same sys.path.

Even leaving aside Rick's sloppy language, I still doubt that it's
popular belief that sys.path be module-specific. You're modifying
something in a different module, and Python's always maintained that
two instances of "import sys" will give two references to the exact
same module object.

> That would be horrible. But here's an alternative which is less horrible and
> maybe even useful.
>
> There's still a single process-wide search path, but there's a second
> per-module search path which is searched first. By default it's empty.
>
> So a module can define it's own extra search path:
>
> __path__ = ['look/here', 'and/here']
> import something
>
> without affecting any other modules.

That's what Rick said first, and then said that if you're going to be
explicit, you should do the job properly and not have _any_ implicit
paths.

Thing is, though, it still breaks the sys.modules concept. Either
__path__ is ignored if the module was found in sys.modules, or it's
possible to have multiple entries with the same name (which would make
it hard to have a module replace itself in sys.modules, currently a
supported thing). Although I suppose all it'd require is that
sys.modules be keyed by __file__ rather than __name__, so they're
identified by fully qualified path and file name. (What does that do
in the face of .pyc files?)

>> And after all that, it would still fail if you happened to want to
>> import both "calendar" modules into the same module.
>
> __path__ = []
> import calendar
> __path__ = ['my/python/modules']
> import calendar as mycalendar

Frankly, if you actually want this, I think it's time to turn to an
uglier-but-more-flexible method.like poking around in importlib. (I'm
not sure off-hand how you'd go about it, it's not instantly obvious
from help(importlib).) I'm more concerned about the possibility of
your import succeeding or failing depending on the order of other
imports:

# foo.py
import calendar

# bar.py
__path__ = ['my/python/modules']
import foo
import calendar

How's that one to be resolved? That's what I don't like.

So long as sys.modules is (a) process-wide and (b) keyed by module
name rather than file name, sys.path MUST be process-wide too, and
MUST be set on startup, or as soon as possible afterwards. Any module
imported prior to altering sys.path will be fetched based on the
previous search path - and you have to import sys to change sys.path,
which means the minimum set of unalterable modules is, on Python 3.5:

rosuav@sikorsky:~$ cat showmods.py
import sys
print(", ".join(sorted(sys.modules.keys())))
rosuav@sikorsky:~$ python3 showmods.py
__main__, _codecs, _collections_abc, _frozen_importlib, _imp, _io,
_signal, _sitebuiltins, _stat, _sysconfigdata, _thread, _warnings,
_weakref, _weakrefset, abc, builtins, codecs, encodings,
encodings.aliases, encodings.latin_1, encodings.utf_8, errno,
genericpath, io, marshal, os, os.path, posix, posixpath, site, stat,
sys, sysconfig, zipimport

... that's a decent lot of modules you can't fiddle with. Hence
PYTHONPATH, which presumably is processed by the interpreter prior to
loading any modules.

ChrisA

[toc] | [prev] | [next] | [standalone]

#81334

Chris Angelico wrote:

> On Sat, Nov 22, 2014 at 11:25 PM, Steven D'Aprano
> <steve+comp.lang.python@pearwood.info> wrote:
>> Ian Kelly wrote:
>>
>> - It's hard to keep track of what modules are in the standard library.
>> Which of the following is *not* in Python 3.3's std lib? No cheating by
>> looking them up.)
>>
>>     os2emxpath, wave, sndheader, statslib, poplist, plist,
>>     pickletools, picklelib, path, cgi, cgitb, copylib, xpath
> 
> Okay, here's my guesses.
> 
> os2emxpath: In the stdlib, but more often accessed as "os.path" while
> running under OS/2

Correct.

> wave: Not in the stdlib, though I'd avoid the name anyway.

Incorrect. The wave module is for manipulating .wav files.

> sndheader: Not in the stdlib - probably on PyPI though

Correct. It is actually spelled "sndhdr".

> poplist, plist, pickletools, picklelib: I suspect PyPI, not stdlib,
> but could be wrong

There's a pickletools, but not picklelib. plist is actually spelled
plistlib, and there is poplib, not poplist.

> path: Not in the stdlib (there's os.path and I doubt there'd be both)
> cgi, cgitb: In the stdlib

Correct.

> copylib: No idea, could be either way.

Spelled "copy", not "copylist".

> xpath: I'll guess this as not being present.

Correct.

Not bad, but as you can see, the naming convensions are all over the place.
It's easy to accidentally shadow the std lib.



[...]
>> That would be horrible. But here's an alternative which is less horrible
>> and maybe even useful.
>>
>> There's still a single process-wide search path, but there's a second
>> per-module search path which is searched first. By default it's empty.
>>
>> So a module can define it's own extra search path:
>>
>> __path__ = ['look/here', 'and/here']
>> import something
>>
>> without affecting any other modules.
> 
> That's what Rick said first, and then said that if you're going to be
> explicit, you should do the job properly and not have _any_ implicit
> paths.

Rick should ask himself why virtually every single language, from compiled
languages like Ada, C, Pascal and Java, to interpreted languages like bash,
all use search paths instead of explicit paths.

Hint: the answer is not that every single language designer in the world is
a moron.


> Thing is, though, it still breaks the sys.modules concept. Either
> __path__ is ignored if the module was found in sys.modules

Good point.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#81338

On Sunday, November 23, 2014 12:00:15 PM UTC+5:30, Steven D'Aprano wrote:
> Rick should ask himself why virtually every single language, from compiled
> languages like Ada, C, Pascal and Java, to interpreted languages like bash,
> all use search paths instead of explicit paths.
> 
> Hint: the answer is not that every single language designer in the world is
> a moron.

In all fairness to Rick I can think of two languages -- Erlang
and Emacs-lisp.  They probably dont do what he wants
(I dont really know what he wants -- his verbiage is too much
for me to get through) however they come close.
Erlang warns about shadowing system-names
Elisp has a function -- list-load-path-shadows -- that informs about
all shadowings in the system.

In a way this makes sense because for a system to warn about shadowing is a trivial cost. For a programmer to keep track of 
which toes are being stepped on is an error-prone headache.

[toc] | [prev] | [next] | [standalone]

#81340

On Sun, Nov 23, 2014 at 5:30 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> Chris Angelico wrote:
>
>> Okay, here's my guesses.
>>
>> os2emxpath: In the stdlib, but more often accessed as "os.path" while
>> running under OS/2
>
> Correct.

I'm in a special position here, as I actually have an OS/2 system, and
have used Python on it. So I know about the particular obscure system
that you happened to pick there. Now, if you'd said "amigapath", I
wouldn't have known.

> [chomp a few correct and a few wrong, here and there]
>
> Not bad, but as you can see, the naming convensions are all over the place.
> It's easy to accidentally shadow the std lib.

Agreed. There's no way to guarantee that you aren't shadowing someone
else's module, except by namespacing your own - either putting your
name at the beginning of every file name ("fredfoobar_calendar") or by
making a package ("fredfoobar.calendar").

It would be nice, though, to be able to *detect* shadowing. Is there a
way (maybe through importlib) to ask Python to enumerate every file
that could be found when attempting to "import calendar"? Something
like this:

>>> import importlib
>>> importlib.find_all_modules("calendar")
['calendar.py', '/usr/local/lib/python3.5/calendar.py']

That would be a useful feature, if it doesn't already exist. If it
does, it could do with a bit more visibility - poking around on Google
and interactively with help(importlib) etc didn't reveal it to me.

>> That's what Rick said first, and then said that if you're going to be
>> explicit, you should do the job properly and not have _any_ implicit
>> paths.
>
> Rick should ask himself why virtually every single language, from compiled
> languages like Ada, C, Pascal and Java, to interpreted languages like bash,
> all use search paths instead of explicit paths.
>
> Hint: the answer is not that every single language designer in the world is
> a moron.

Maybe they are. Maybe Rick's just so amazingly brilliant that he,
alone among all language designers, has figured this out. Maybe it's
time for his fork of Python to sweep the world with its incredible
solutions to massively prevalent problems!

While it *is* possible for every language in the world to get
something wrong, there's usually a few around the place that get it
right. Until Python 3.3, most popular languages got Unicode support
either wrong (using UTF-16 being the most common wrongness) or
inefficient (using UTF-32), or just didn't have any at all (bytes
everywhere). Could someone look at the state of world programming
languages and say "Everyone uses UTF-16, and not every language
designer in the world is a moron, therefore UTF-16 must be the right
thing to do"? No, because Haskell, Scheme, Pike, and bash all had it
right, already. Sure, they're not the most-used languages in the
world, but they certainly are used (bash plenty, though to be honest,
not many people use bash scripts for Unicode heavy-lifting), and they
prove that UTF-16 isn't the only way to go.

So, where are the languages that use explicit paths? Show us why
they're so great, using actual production code.

ChrisA

[toc] | [prev] | [next] | [standalone]

#81341

On Sun, Nov 23, 2014 at 5:30 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
>> wave: Not in the stdlib, though I'd avoid the name anyway.
>
> Incorrect. The wave module is for manipulating .wav files.
>
> > sndheader: Not in the stdlib - probably on PyPI though
>
> Correct. It is actually spelled "sndhdr".

Just out of curiosity, why does the stdlib need modules for
manipulating .wav and other sound files, but we have to go to PyPI to
get a PostgreSQL client? It's a queer world...

ChrisA

[toc] | [prev] | [next] | [standalone]

#81404 — Re: PyWart: "Python's import statement and the history of external dependencies"

Chris Angelico wrote:
> Just out of curiosity, why does the stdlib need modules for
> manipulating .wav and other sound files, but we have to go to PyPI to
> get a PostgreSQL client?

I suspect it's mainly for historical reasons. The wave
module has been around since the very early days of
Python when the stldib was smaller and the barrier
to entry was lower. By today's standards, it probably
wouldn't make it in.

-- 
Greg

[toc] | [prev] | [next] | [standalone]

#81434

On Sunday, November 23, 2014 4:37:53 PM UTC-6, Gregory Ewing wrote:
> Chris Angelico wrote:
> > Just out of curiosity, why does the stdlib need modules
> > for manipulating .wav and other sound files, but we have
> > to go to PyPI to get a PostgreSQL client?
>
> I suspect it's mainly for historical reasons. The wave
> module has been around since the very early days of
> Python when the stldib was smaller and the barrier
> to entry was lower. By today's standards, it probably
> wouldn't make it in.

Immigration policies evolve basically in the same manner.
When a country is founded in a land which is sparsely
populated, naturally the state will welcome any and all
immigrants (as noted in the "New Colossus"), but as the
population becomes more and more dense, the natural (and
wise) reaction is to limit, or even prohibit, the influx of
migrants.

============================================================
 PSYCHOANALYSIS OF MIGRANT/STATE SYMBIOSIS:
============================================================

The majority my readers would probably agree that any state
which projected such liberal immigration policies would be
doing so based on pure "selfishness", and they would be
correct, however, most would fail to realize that both the
migrant *AND* the state are basing their decisions on pure
selfishness.

THERE IS NO SUCH THING AS A PURELY SELFISH ACT!

For the state, gaining citizens means expanding it's
military power (you know, the whole "power-in-numbers"
thing), and expanding it's economic power (money, money,
money!), and last but not least, expanding it's tax base
(hey, did i mention money yet?).

GREED IS PROPAGATED BY THE SYNERGY OF "SYMBIOTIC SELFISHNESS"!

For the migrant, he seeks to gain or benefit by some means,
because if he did not gain something *positive* from the
interaction, then he would not be a migrant, no, by
definition he would be a *masochist*! Like the old saying
goes: "When your at the bottom, the only direction you can
go is up!". So by this definition, the migrant seeks to
utilize the resources in his "new land" for he failed to
utilize in his "home land".

FLIES DON'T CARE MUCH FOR VINEGAR, BUT THEY SURE LOVE HONEY!

In the balance of this "relationship", the state is more
venerable than the migrant. The state only survives *if* it
can maintain its population (but more importantly *IF* it
can maintain a "quality" population!), but the migrant can
survive under *any* state that is willing to provide for
him, he need not pledge allegiance to any one state, no, no,,,
and if at any point the relationship becomes untenable for
him, he has proven *already* his willingness to betray his
responsibility as an equal member of this symbiosis, by
simply re-locating!

But is the migrant *really* an equal member?  No. The
migrant does not share equally the burdens of the symbiosis.
Since the migrant is free to utilize the resources of the
state, and then simply pack-up and leave when the resources
are exhausted, the migrant is in-effect the master, and the
state is the slave. And although the state does extract a
tiny amount of power from the migrant, this power is useless
when one party can nullify the contract just because the
"honey pot" ran empty.

============================================================
 CONCLUSION:
============================================================

The migrant/state symbiosis is a fine example of how we (as
*weak* emotional beings) fall into these "emotional traps"
set by "forked tongued propagandist" in hopes of diverting
our attention away from reality. No being in this universe
makes any choice based on altruistic motives, no, altruism
is a mask worn by the most cunning of con-artist -- a true
weapon of deceit!

[toc] | [prev] | [next] | [standalone]

#81435

On Mon, Nov 24, 2014 at 5:25 PM, Rick Johnson
<rantingrickjohnson@gmail.com> wrote:
> The migrant/state symbiosis is a fine example of how we (as
> *weak* emotional beings) fall into these "emotional traps"
> set by "forked tongued propagandist" in hopes of diverting
> our attention away from reality. No being in this universe
> makes any choice based on altruistic motives, no, altruism
> is a mask worn by the most cunning of con-artist -- a true
> weapon of deceit!

As CS Lewis wrote in various places, notably "The Screwtape Letters",
people who are fundamentally driven by pure selfishness are utterly
unable to even comprehend love and genuine altruism. Fortunately for
us, your email is both off-topic and useless, so we can cheerfully
ignore it.

ChrisA

[toc] | [prev] | [next] | [standalone]

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

Back to top | Article view | comp.lang.python

csiph-web