Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #80850 > unrolled thread

PyWart: "Python's import statement and the history of external dependencies"

Back to article view | Back to comp.lang.python

Contents

  PyWart: "Python's import statement and the history of external dependencies" Rick Johnson <rantingrickjohnson@gmail.com> - 2014-11-15 21:01 -0800
    Re: PyWart: "Python's import statement and the history of external dependencies" Chris Angelico <rosuav@gmail.com> - 2014-11-16 16:16 +1100
    Re: PyWart: "Python's import statement and the history of external dependencies" alex23 <wuwei23@gmail.com> - 2014-11-21 10:14 +1000
      Re: PyWart: "Python's import statement and the history of external dependencies" Chris Angelico <rosuav@gmail.com> - 2014-11-21 11:31 +1100
      Re: PyWart: "Python's import statement and the history of external dependencies" Rick Johnson <rantingrickjohnson@gmail.com> - 2014-11-20 19:53 -0800
        Re: PyWart: "Python's import statement and the history of external dependencies" Tim Chase <python.list@tim.thechases.com> - 2014-11-21 04:26 -0600
          Re: PyWart: "Python's import statement and the history of external dependencies" Rick Johnson <rantingrickjohnson@gmail.com> - 2014-11-21 07:52 -0800
            Re: PyWart: "Python's import statement and the history of external dependencies" Tim Chase <python.list@tim.thechases.com> - 2014-11-21 10:12 -0600
            Re: PyWart: "Python's import statement and the history of external dependencies" Ian Kelly <ian.g.kelly@gmail.com> - 2014-11-21 10:42 -0700
        Re: PyWart: "Python's import statement and the history of external dependencies" Ned Batchelder <ned@nedbatchelder.com> - 2014-11-21 06:32 -0500
        Re: PyWart: "Python's import statement and the history of external dependencies" Ian Kelly <ian.g.kelly@gmail.com> - 2014-11-21 08:33 -0700
          Re: PyWart: "Python's import statement and the history of external dependencies" Rick Johnson <rantingrickjohnson@gmail.com> - 2014-11-21 10:24 -0800
            Re: PyWart: "Python's import statement and the history of external dependencies" Chris Angelico <rosuav@gmail.com> - 2014-11-22 05:34 +1100
            Re: PyWart: "Python's import statement and the history of external dependencies" Michael Torrie <torriem@gmail.com> - 2014-11-21 12:05 -0700
              Re: PyWart: "Python's import statement and the history of external dependencies" Rick Johnson <rantingrickjohnson@gmail.com> - 2014-11-21 12:29 -0800
                Re: PyWart: "Python's import statement and the history of external dependencies" Michael Torrie <torriem@gmail.com> - 2014-11-21 15:54 -0700
            Re: PyWart: "Python's import statement and the history of external dependencies" Ian Kelly <ian.g.kelly@gmail.com> - 2014-11-21 12:23 -0700
              Re: PyWart: "Python's import statement and the history of external dependencies" Rick Johnson <rantingrickjohnson@gmail.com> - 2014-11-21 14:25 -0800
                Re: PyWart: "Python's import statement and the history of external dependencies" Rick Johnson <rantingrickjohnson@gmail.com> - 2014-11-21 15:21 -0800
                  Re: PyWart: "Python's import statement and the history of external dependencies" Chris Angelico <rosuav@gmail.com> - 2014-11-22 10:52 +1100
                    Re: PyWart: "Python's import statement and the history of external dependencies" Rick Johnson <rantingrickjohnson@gmail.com> - 2014-11-21 17:07 -0800
                      Re: PyWart: "Python's import statement and the history of external dependencies" Chris Angelico <rosuav@gmail.com> - 2014-11-22 13:36 +1100
                      Re: PyWart: "Python's import statement and the history of external dependencies" Ian Kelly <ian.g.kelly@gmail.com> - 2014-11-22 02:27 -0700
                  Re: PyWart: "Python's import statement and the history of external dependencies" sohcahtoa82@gmail.com - 2014-11-21 16:33 -0800
                  Import a module from a specific file path (was: PyWart: "Python's import statement and the history of external dependencies") Ben Finney <ben+python@benfinney.id.au> - 2014-11-22 11:37 +1100
                    Re: Import a module from a specific file path Marko Rauhamaa <marko@pacujo.net> - 2014-11-22 08:28 +0200
                  Re: Import a module from a specific file path (was: PyWart: "Python's import statement and the history of external dependencies") Chris Angelico <rosuav@gmail.com> - 2014-11-22 13:32 +1100
                  Re: Import a module from a specific file path Ben Finney <ben+python@benfinney.id.au> - 2014-11-22 17:46 +1100
                Re: PyWart: "Python's import statement and the history of external dependencies" Ian Kelly <ian.g.kelly@gmail.com> - 2014-11-21 17:32 -0700
                  Re: PyWart: "Python's import statement and the history of external dependencies" Rick Johnson <rantingrickjohnson@gmail.com> - 2014-11-21 17:48 -0800
            Re: PyWart: "Python's import statement and the history of external dependencies" Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2014-11-21 15:38 -0500
          Re: PyWart: "Python's import statement and the history of external dependencies" Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-11-22 23:25 +1100
            Re: PyWart: "Python's import statement and the history of external dependencies" Chris Angelico <rosuav@gmail.com> - 2014-11-23 00:00 +1100
              Re: PyWart: "Python's import statement and the history of external dependencies" Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-11-23 17:30 +1100
                Re: PyWart: "Python's import statement and the history of external dependencies" Rustom Mody <rustompmody@gmail.com> - 2014-11-22 22:44 -0800
                Re: PyWart: "Python's import statement and the history of external dependencies" Chris Angelico <rosuav@gmail.com> - 2014-11-23 17:56 +1100
                Re: PyWart: "Python's import statement and the history of external dependencies" Chris Angelico <rosuav@gmail.com> - 2014-11-23 17:59 +1100
                  Re: PyWart: "Python's import statement and the history of external   dependencies" Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2014-11-24 11:37 +1300
                    Re: PyWart: "Python's import statement and the history of external dependencies" Rick Johnson <rantingrickjohnson@gmail.com> - 2014-11-23 22:25 -0800
                      Re: PyWart: "Python's import statement and the history of external dependencies" Chris Angelico <rosuav@gmail.com> - 2014-11-24 17:34 +1100
                      Re: PyWart: "Python's import statement and the history of external dependencies" Mark Lawrence <breamoreboy@yahoo.co.uk> - 2014-11-24 10:34 +0000
                        Re: PyWart: "Python's import statement and the history of external dependencies" Rick Johnson <rantingrickjohnson@gmail.com> - 2014-11-24 08:51 -0800
                          Re: PyWart: "Python's import statement and the history of external dependencies" Ned Batchelder <ned@nedbatchelder.com> - 2014-11-24 12:06 -0500
                          Re: PyWart: "Python's import statement and the history of external dependencies" Robin Becker <robin@reportlab.com> - 2014-11-24 17:07 +0000
                          Re: PyWart: "Python's import statement and the history of external dependencies" Marko Rauhamaa <marko@pacujo.net> - 2014-11-24 19:08 +0200
                            Re: PyWart: "Python's import statement and the history of external dependencies" Rustom Mody <rustompmody@gmail.com> - 2014-11-24 09:38 -0800
                    Re: PyWart: "Python's import statement and the history of external dependencies" Ian Kelly <ian.g.kelly@gmail.com> - 2014-11-24 15:34 -0700
                Re: PyWart: "Python's import statement and the history of external dependencies" Ben Finney <ben+python@benfinney.id.au> - 2014-11-23 18:43 +1100
                Re: PyWart: "Python's import statement and the history of external dependencies" Chris Angelico <rosuav@gmail.com> - 2014-11-23 18:51 +1100
                Re: PyWart: "Python's import statement and the history of external dependencies" Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2014-11-23 11:23 -0500
                Re: PyWart: "Python's import statement and the history of external dependencies" Chris Angelico <rosuav@gmail.com> - 2014-11-24 03:36 +1100
            Re: PyWart: "Python's import statement and the history of external dependencies" Tim Chase <python.list@tim.thechases.com> - 2014-11-22 15:27 -0600
              Re: PyWart: "Python's import statement and the history of external dependencies" Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-11-23 12:00 +1100
                Re: PyWart: "Python's import statement and the history of external dependencies" Tim Chase <python.list@tim.thechases.com> - 2014-11-22 19:33 -0600

Page 1 of 3  [1] 2 3  Next page →

#80850 — PyWart: "Python's import statement and the history of external dependencies"

Python's attempt to solve the "external dependencies problem"
has yet to produce the results that many people, including
myself, would like.

Actually, Python is not alone in this deficiency, no, Python
is just *ANOTHER* language in a *STRING* of languages over
the years who has *YET AGAIN* implemented the same old
crusty design patterns, packaged them in a shiny metallic
wrapping paper with a big red bow on top, and hoped that no
one would notice the stench...

    WELL I NOTICED M.F.!

Before we can formulate a solution to this mess, we must
first obtain an "inside perspective" of the "world" in which a
Python script lives during run-time.

============================================================
 Welcome to "Python Gardens": a master planned community
============================================================

Well... urm, sort of. @_@

I like to analyze problems, when possible, in relation to
real world "tangible" examples. So for me, i like to think
of the main script (aka: __main__) as an apartment building,
and each module that runs under the main script as a single
apartment, and finally, the inhabitants of the apartments as
objects.

Furthermore, we can define the apartment building as being
"global space", and each individual apartment as being
"local space". The first law we encounter is that "global
space" is reserved for all tenants/guest, but "local space"
is by *invitation* only!

    You can think of "import" as similar to sending out an
    invitation, and requesting that a friend join you inside
    your apartment (we'll get back to "import" later).

And of course, with this being the 21st century and all, every
apartment has *local* access to *global* resources like
water and electrical.

    In Python, these "global resources" are available
    "locally" in every module via the implicit namespace of
    "__builtins__". You can think of built-ins as a "house
    keeper" robot that lives in every apartment (I call mine
    "Rosie"). It was there when you moved in --a "house
    warming" gift of sorts-- and it helps you with chores
    and makes your life a little easier. Sometimes i abuse 
    my robot but i always apologize afterwards.
    
Now that we know how the global and local spaces are defined
(in the context of modules), and what implicit/explicit
"gifts" (are supplied), and "rules" (are demanded) in our
little Python world, we have a good basis to start
understanding why Python's import mechanism is at best a
"helpful failure" that only *naively* attempts to
streamline the management of such an important language
feature as external dependencies!

Up until this point, our apartment example has not mapped
the actual *mechanics* of "import" to a real world example,
but now that we have the correct "perspective", we can tread
into the dark and damp underworld that is "import".

Remember when i said: "import is like sending out an
invitation"? Well, in actuality, import is only
*superficially* similar to "sending out an invitation".

You see, when you send an invitation in real life, the most
important two things you'll need are a *name* and an
*address* --a name so you'll know *who* to send the
invitation to, and an address so you'll know *where* to send
the invitation-- but Python's import mechanism does not work
that way. When you import an external dependency into a
Python module all you need is the name of a module --
addresses are neither required *OR* allowed!

Like almost all modern languages, Python has adopted the
ubiquitous practice of internalizing a mapping of known
directories from which to search for modules (called a
search path), so when we "import somemodule" the following
happens

(Note: I'm not going into too many details for the sake of topic):

    1. Python checks if the external resource has already been loaded

    if not 1, then

    2. Python looks in a few "special places"

    if not 2, then

    3. Python searches one-by-one the directories in sys.path

RESULT: Python finds the resource or barfs an exception.

I have no problem with step 1, however, step 2 and step 3
can be redundant, excessive, and even unreliable. I can
explain better why i levee such harsh words for "import" by
going back to our apartment building example.

Let's imagine that Python is the "lobby boy" of our apartment
building. And in our little example, one of the duties of
the "lobby boy" is to manage invitations between tenants
and guests. Each time a tenant wants to invite someone into
their apartment, they must pick up the phone, press the
import button (which connects them to the lobby boy via a
voice connection) and they say the name of the person they
want to invite.

    But *EVEN IF* they know the address of the recipient, they
    are not allowed to say the address to the lobby boy, no,
    because *THIS* building is owned by a evil tyrant,
    who has declared that any mention of an address when
    calling import is punishable by the fatal exception!

So, being a good tenant who does not want to *DIE*, they
simply mention the recipients name, roll their eyes, hang up
the phone, and hope that lobby boy can locate the correct
recipient

It is at this point that the lobby boy (aka: Python) springs
into action. The lobby boy has a gargantuan task ahead of
him, he must locate a tenant based solely on the name with
no idea of which apartment the tenant resides. And,
realizing that he has a terrible memory, and that some of
the building's pranksters have sent him on "wild goose chases"
for recipients that do not exist, his first action is to
check his daily log and ensure he did not *already* deliver
the invitation...

    DAMN! Not in the log!

So then the lobby boy checks for a few special places...

    DAMN! not in my "special places" either!

Sadly, the lobby knows that now he must check the ENTIRE
building, of which contains thousands of tenants. So he
starts on floor one (sys.path[0]), knocks on all the doors,
then proceeds to floor two (sys.path[1]), knocks on all
those doors..., and so on and so forth until he locates the
recipient or dies of exhaustion -- whichever comes first.

It is this point that i find the import machinery to be
lacking. It's not only the *needless* exhaustive searching
via such an unintelligent algorithm, but also the great
possibility of importing the wrong resource.

    "Yeah Rick, but who wants to include a file-path for
    importing modules, the whole reason for import was to
    abstract away the file-system and file-paths, etc... Are
    you proposing we go back and re-adopt the historical
    conventions?"

Absolutely not. In most day to day usage Python's import
statement works just fine, it's a cycle burner, but who
cares about a lobby boy anyway right?

I have thousands of modules on my hdrive, and many many
directories holding them, even if i could overlook the
"cycle bonfire" of import's underlying machinery, the list of
intuit-able names for modules are finite, and i'm not about
to start tagging file-names because then i'm back to the
file-system again -- but instead of slashes i'm using
underscores.

    NO THANKS!.

What the creators of "import" fail to understand is that a
trade-off has been made, and the Python programmer has been
given the short end of a very long stick, and now we're
being beaten with the very long stick and he nothing to 
defend himself with but a tooth-pick! 

Creating an "implicit name resolution system" (aka: import) 
to abstract away an "explicit name resolution system" 
(file-paths) has resulted in more problems that it can solve:

    1. Name clashes!
    2. Smaller name pool!
    3. Machinery is too implicit!
    4. Circular imports are inevitable!
    5. Much too difficult to use and/or explain!
    6. Too many "gotchas"!

Of course, if "import" is not problematic enough for you,
one you use one of the many modules like importlib, imp,
importil, ihooks, or __import__ function, and then become
more confused!! -- Python's external dependency is not only
broken, it's discombobulated and downright confusing!

In closing, there is only one thing you need to know about
Python's import statement:  it is deceptively easy to
underestimate it's ability to *EFF UP* your expectations!

[toc] | [next] | [standalone]

#80851

On Sun, Nov 16, 2014 at 4:01 PM, Rick Johnson
<rantingrickjohnson@gmail.com> wrote:
> Creating an "implicit name resolution system" (aka: import)
> to abstract away an "explicit name resolution system"
> (file-paths) has resulted in more problems that it can solve:
>
>     1. Name clashes!
>     2. Smaller name pool!
>     3. Machinery is too implicit!
>     4. Circular imports are inevitable!
>     5. Much too difficult to use and/or explain!
>     6. Too many "gotchas"!

And the ability to write cross-platform code!

ChrisA

[toc] | [prev] | [next] | [standalone]

#81149

On 16/11/2014 3:01 PM, Rick Johnson wrote:
> Python's attempt to solve the "external dependencies problem"
> has yet to produce the results that many people, including
> myself, would like.

I'd say this was an argumentum ad populum, only you didn't cite anything 
that shows the "many" you claim you speak for.

> Actually, Python is not alone in this deficiency, no, Python
> is just *ANOTHER* language in a *STRING* of languages over
> the years who has *YET AGAIN* implemented the same old
> crusty design patterns, packaged them in a shiny metallic
> wrapping paper with a big red bow on top, and hoped that no
> one would notice the stench...

Are you sure that's not just you?

>      1. Name clashes!
>      2. Smaller name pool!

Just off the top of my head, we have several solutions for this:

1) Rebinding imports

     import foo as foo2

2) Namespace packages

     import mypackage.foo

3) Filepath imports (deprecated in 3.4 but under discussion for 
reprecation or replacement)

     import importlib.machinery

     loader = importlib.machinery.SourceFileLoader("module.name",
         "/path /to/file.py")
     foo = loader.load_module()
     foo.MyClass()

4) Virtual environments

In before "but WHY so MANY different WAYS OMG PYTHON you M.F.P.O.S.": 
because they offer flexibility.

>      3. Machinery is too implicit!

See: importlib

>      4. Circular imports are inevitable!

They're also clear indication of bad design. That's not something Python 
should explicitly set out to support.

>      5. Much too difficult to use and/or explain!

Well sure, if your reaction to things you don't understand is to rant 
against them instead of asking questions, then I can see how that would 
be true.

Also: citation needed.

>      6. Too many "gotchas"!

Is this the sequel to "Too Many Cooks"? Awesome.

> In closing, there is only one thing you need to know about
> Python's import statement:  it is deceptively easy to
> underestimate it's ability to *EFF UP* your expectations!

There's your problem right there: a language conforms to its 
specification, not your random expectation.

I'm eager to see how you resolve your concerns in RickPython3000.

[toc] | [prev] | [next] | [standalone]

#81153

On Fri, Nov 21, 2014 at 11:14 AM, alex23 <wuwei23@gmail.com> wrote:
>>      1. Name clashes!
>>      2. Smaller name pool!
>
> Just off the top of my head, we have several solutions for this:
>
> 1) Rebinding imports
>
>     import foo as foo2

To be fair to Rick, this doesn't actually solve anything. If you have
two modules called "foo", you can't import one of them this way.

But part of the problem is actually before you even get as far as
importing: it's the nature of open source, forking, and the internet.
There's a module called "remix", which one Peter Sobot forked and
tweaked for his own use, and then distributed a slightly changed
version of. (I believe the original intention was for the patches to
be incorporated into trunk, but that hasn't happened as yet.) If you
want to use his app, you need to use his tweaked remix module; if you
want to use someone else's, you might need to use the original from
upstream. What do you do about that? They're both called "remix". How
do you import one or the other? How do you install one or the other?

There is fundamentally no solution to this. As soon as there are two
incompatible (even just slightly incompatible) versions of a project
in the wild, people need to cope with it. It's the same with avconv vs
ffmpeg, it's the same with wodim vs cdrecord, it's the same with all
of them - until one of them undergoes a name change, everyone's stuck.

ChrisA

[toc] | [prev] | [next] | [standalone]

#81165

On Thursday, November 20, 2014 6:15:03 PM UTC-6, alex23 wrote:
> On 16/11/2014 3:01 PM, Rick Johnson wrote:
> [...]
> > Actually, Python is not alone in this deficiency, no, Python
> > is just *ANOTHER* language in a *STRING* of languages over
> > the years who has *YET AGAIN* implemented the same old
> > crusty design patterns, packaged them in a shiny metallic
> > wrapping paper with a big red bow on top, and hoped that no
> > one would notice the stench...
>
> Are you sure that's not just you?
>
> >      1. Name clashes!
> >      2. Smaller name pool!
>
>
> Just off the top of my head, we have several solutions for
> this:
>
> 1) Rebinding imports
>
>      import foo as foo2
>
> 2) Namespace packages
>
>      import mypackage.foo

I was not talking about name clashes within the script
(which are the problem of the programmer, because he and
only he has *DIRECT* control over the contents of any
scripts he writes!) no, i was talking about two or more
modules who share the same name who then, in effect,
*SHADOW* each other (this is problem for which the
programmer *CANNOT* solve *INTUITIVELY* because he cannot
*EASILY* control the manner in which the underlying import
mechanism finds modules). If you had taken the time to read
my example utilizing a "lobby boy", then you might have
understood what i was talking about.

FOR INSTANCE: Let's say i write a module that presents a
reusable GUI calendar widget, and then i name the module
"calender.py".

Then Later, when i try to import *MY* GUI widget named
"calendar", i will not get *MY* calendar widget, no, i will
get the Python calendar module.

The reason Python finds the library module instead of my
module is because the Python library is higher in search
path than my module.

OKAY, FAIR ENOUGH...

Now, i could *TRY* to be "cleaver" and inject the directory
that holds *MY* "calendar module" as the *FIRST* directory
of sys.path, and this will work, but it's a TOTAL FREAKING
HACK, and unbeknownst to most first-time "import hackers",
the changes to sys.path does not only affect the "contents" of
sys.path within the "current module", but *ALL* modules
further down the "import chain" will suffer the changes!

############################################################
#                        BE WARNED:                        #
############################################################
# Contrary to popular belief, sys.path is *NOT* a module,  #
# no, it's a global!                                       #
############################################################

This is another confusing fundamental of Python, it seems *MORE*
intuitive that changes to the "import search path" would only
affect the *CURRENT* module's "import search path", but that's
is not the case!

############################################################
#                    Lesson Of The Day                     #
############################################################
# Try to be intuitive, and python will *break* you!        #
############################################################

And you know why, it's because our "dear leaders" have this
irrational phobia of global variables. But the joke is on
them, because they fail to realize that importing a module
into more than one namespace is creating globals!

    YOU CAN PUT LIPSTICK ON A PIG, BUT IT'S STILL A PIG!

This is what happens when your emperor stops wearing
clothes, and more proof that Python's import mechanism sucks
dingo balls!

Now some people will say: "Don't name your modules the same
as a Python library modules and problem solved!"

Sorry, but that's not good enough for me! I believe that the
current import mechanism is only *HALF* of what it should
be. Currently, import uses an "implicit" name resolution via
predefined paths in sys.path, and for 75% of the time, this
"implicit" import mechanism will work just fine -- it's a
slow algorithm, but fast enough for humans not to notice.

HOWEVER,

in the 25% of cases where *NAME SHADOWING* or where the
author knows the location and can specify the location to
save "lobby boy" from knocking on doors until his knuckles
bleed, the author should have an option defining the search
path *LOCALLY* (in a manner that will *NOT* affect the
"import search path" further down the import chain), WITHOUT
needing to import one of the many discombobulated and
confusing import tools that Python has tortured this
community with.


>
> 3) Filepath imports (deprecated in 3.4 but under discussion for
> reprecation or replacement)
>
>      import importlib.machinery
>
>      loader = importlib.machinery.SourceFileLoader("module.name",
>          "/path /to/file.py")
>      foo = loader.load_module()
>      foo.MyClass()

That's ridiculously noisy. I have an idea, if you *REALLY*
want to introduce boilerplate hell then Python should adopt
"import header files", NOT!

> 4) Virtual environments
>
> In before "but WHY so MANY different WAYS OMG PYTHON you
> M.F.P.O.S.": because they offer flexibility.
>
> >      3. Machinery is too implicit!
>
> See: importlib
>
> >      4. Circular imports are inevitable!
>
> They're also clear indication of bad design. That's not
> something Python should explicitly set out to support.

I think your confusing "circular references" with "circular
imports", and while the two have much in common, and both
should be avoid when possible, there are times when both are
unavoidable (if you want to write sane code!). But this a
discussion better suited to a thread of it's own.

> >      5. Much too difficult to use and/or explain!
>
> Well sure, if your reaction to things you don't understand
> is to rant against them instead of asking questions, then
> I can see how that would be true.

I've asked questions on this list, and you know what kind of
help i get, NONE! I can distinctly remember asking D'Aprano
to help explain the new format method and he flat out
refused, even though he time for "Devils Advocating", he had
no time to help me -- go figure!

The only time i get *real* help from people on this list is
when i ask them privately. I won't mention any names but it
would be nice to feel like a welcomed member who could
participate fully.

You may not believe me, but i have seen some major
improvements in the community since i've been here. Whilst
my complaints to the caretakers of Python.org about the
lackluster and discombobulated nature of the site were
initially taken as offensive, i was pleased to see my
recommendations fully implemented within a year or so. Heck,
i can remember a time when you had to navigate four or five
pages (from the home page) just to see the damn docs, now,
both 2.x and 3.x docs are just a single click away!

    THANKS RICK!

About three years ago i started contacting every owner of a
major Python tutorial and pleaded with them to upgrade
their tuts to Python 3.x, again, some took the advice as
offensive, but most of them finally admitted the importance
of sticking to the latest Python version.

    THANKS RICK!

Probably my proudest achievement is to see all the
neophytes who have gained the courage to interact with this
community. Before i came along, most of them hide in the
shadows and only watched, unwilling to participate for fear
of ridicule by our rabid attack dogs. But, after witnessing
the hell that i went through, and my public lynching, but
then my triumphant rise as like a "Phoenix From The Ashes",
they realized that the fearless can *NEVER* be defeated, and
that those who carry *ABSOLUTE* resolve to succeed, *WILL*
succeed -- even if they don't get *ANY* credit for their
contributions!

    THANKS RICK!

PS: You have to pat *yourself* on the back 'round here.

[toc] | [prev] | [next] | [standalone]

#81191

On 2014-11-20 19:53, Rick Johnson wrote:
> FOR INSTANCE: Let's say i write a module that presents a
> reusable GUI calendar widget, and then i name the module
> "calender.py".
> 
> Then Later, when i try to import *MY* GUI widget named
> "calendar", i will not get *MY* calendar widget, no, i will
> get the Python calendar module.
> 
> The reason Python finds the library module instead of my
> module is because the Python library is higher in search
> path than my module.

What messed-up version of Python are you running?  Or did you fail to
test your conjecture?

$ cat > calendar.py
print("This is my local calendar.py")
x=1
$ cat > importtest.py
import calendar
print(dir(calendar))
$ python2 importtest.py 
This is my local calendar.py
['__builtins__', '__doc__', '__file__', '__name__', '__package__', 'x']
$ python3 importtest.py
This is my local calendar.py
['__builtins__', '__cached__', '__doc__', '__file__', '__name__', '__package__', 'x']


It finds my local module, not the system "calendar" module.

-tkc

[toc] | [prev] | [next] | [standalone]

#81217

On Friday, November 21, 2014 4:29:48 AM UTC-6, Tim Chase wrote:

> What messed-up version of Python are you running?  
> Or did you fail to test your conjecture?
> 
> $ cat > calendar.py
> print("This is my local calendar.py")
> x=1
> $ cat > importtest.py
> import calendar
> print(dir(calendar))
> $ python2 importtest.py 
> This is my local calendar.py
> ['__builtins__', '__doc__', '__file__', '__name__', '__package__', 'x']
> $ python3 importtest.py
> This is my local calendar.py
> ['__builtins__', '__cached__', '__doc__', '__file__', '__name__', '__package__', 'x']
> 
> It finds my local module, not the system "calendar" module.

You failed to provide the needed information!
    
  1. We need to see the values in sys.path. So print them
  directly before making the call to import.
  
  2. Where did you store your custom "calendar.py" script?
  
Remember, "import" is an implicit mechanism, it works by
searching the values of 'sys.path' one-by-one (if the module
was not already loaded!) until the *FIRST* module matching
the name you requested is found. If the folder that holds
*your* custom "calendar.py" script is listed *before* the
folder containing the "python calendar" modules then of
course your module will be correctly loaded.

[toc] | [prev] | [next] | [standalone]

#81223

On 2014-11-21 07:52, Rick Johnson wrote:
> On Friday, November 21, 2014 4:29:48 AM UTC-6, Tim Chase wrote:
> 
> > What messed-up version of Python are you running?  
> > Or did you fail to test your conjecture?
> > 
> > $ cat > calendar.py
> > print("This is my local calendar.py")
> > x=1
> > $ cat > importtest.py
> > import calendar
> > print(dir(calendar))
> > $ python2 importtest.py 
> > This is my local calendar.py
> > ['__builtins__', '__doc__', '__file__', '__name__',
> > '__package__', 'x'] $ python3 importtest.py
> > This is my local calendar.py
> > ['__builtins__', '__cached__', '__doc__', '__file__', '__name__',
> > '__package__', 'x']
> > 
> > It finds my local module, not the system "calendar" module.
> 
> You failed to provide the needed information!
>     
>   1. We need to see the values in sys.path. So print them
>   directly before making the call to import.
>   
>   2. Where did you store your custom "calendar.py" script?

Had you paid attention, having watched the creation of the two files
and the invocation of stock python, you would know that they were in
the same folder, and there's no modification of sys.path in the
process.  So if you're seeing different behaviors due to a modified
sys.path, then you should be aware that *you* modified sys.path and
thus broke things.

The above was tested on Python2 and Python3, both on Linux (Debian
in this case) and Win32 (using "copy con" instead of "cat >") with the
out-of-box install.

The only time I've been stung by name overloading is in the indirect
case of creating a local email.py file and then importing smtplib
only to have things break in unforeseen ways.  If smtplib used
relative imports for $STDLIB/email.py I suspect it would ameliorate
that particular issue for me.

-tkc

[toc] | [prev] | [next] | [standalone]

#81231

On Fri, Nov 21, 2014 at 9:12 AM, Tim Chase
<python.list@tim.thechases.com> wrote:
> The only time I've been stung by name overloading is in the indirect
> case of creating a local email.py file and then importing smtplib
> only to have things break in unforeseen ways.  If smtplib used
> relative imports for $STDLIB/email.py I suspect it would ameliorate
> that particular issue for me.

Relative imports are based on package namespaces and can only be done
within a package. There's no way to do a relative import from a
separate top-level package, whether they happen to be found in the
same directory or not.

$ mkdir p1
$ touch p1/__init__.py
$ echo 'from ..p2 import m2' > p1/m1.py
$ mkdir p2
$ touch p2/__init__.py
$ touch p2/m2.py
$ python3 -c 'import p1.m1'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/ikelly/p1/m1.py", line 1, in <module>
    from ..p2 import m2
ValueError: attempted relative import beyond top-level package

[toc] | [prev] | [next] | [standalone]

#81197

On 11/20/14 10:53 PM, Rick Johnson wrote:
> If you had taken the time to read
> my example utilizing a "lobby boy", then you might have
> understood what i was talking about.

Rick, if you are frustrated that people don't know what you are talking 
about, you should try writing shorter messages, with less bombast.  I 
don't know if you are aiming for humor with your messages, but to my 
ears, you are verging on self-parody.  I'm surprised that people are 
taking you seriously.

-- 
Ned Batchelder, http://nedbatchelder.com

[toc] | [prev] | [next] | [standalone]

#81213

On Thu, Nov 20, 2014 at 8:53 PM, Rick Johnson
<rantingrickjohnson@gmail.com> wrote:
> FOR INSTANCE: Let's say i write a module that presents a
> reusable GUI calendar widget, and then i name the module
> "calender.py".
>
> Then Later, when i try to import *MY* GUI widget named
> "calendar", i will not get *MY* calendar widget, no, i will
> get the Python calendar module.

Because like a fool you created a local module with a totally generic
name like "calendar" and dumped it into the top-level namespace. The
Python import system gives you the ability to create packages, so use
them -- create a package to contain all the widgets you create. If
they're only for your own use, then you can just call it ricklib or
anything else you want.  Then your import becomes:

import ricklib.calendar  # or from ricklib import calendar

Now you can drop as much stuff in there as you like, and none of it
will ever conflict with the standard library (unless a standard
"ricklib" module is added, which is unlikely).

> The reason Python finds the library module instead of my
> module is because the Python library is higher in search
> path than my module.

Your PYTHONPATH is added to the sys.path before the Python library,
not after, so to the extent that there is an issue, it's actually the
other way around. You would get your local calendar module when trying
to import the system calendar module.

> # Contrary to popular belief, sys.path is *NOT* a module,  #
> # no, it's a global!                                       #

I really doubt that this is a popular belief.

> This is another confusing fundamental of Python, it seems *MORE*
> intuitive that changes to the "import search path" would only
> affect the *CURRENT* module's "import search path", but that's
> is not the case!

This just seems like it would create a maintenance nightmare. Before
you could import anything, you'd have to figure out what the search
path is for the particular module you're working and whether it
happens to include the particular package you want. You find it
doesn't, so you make a local change to the search path. Later, you
make the same change to other modules that need to import it. Later
still, the package moves to a different location in the file system,
and now you get to go through and update every one of those modules
with the new directory. Lucky you.

And after all that, it would still fail if you happened to want to
import both "calendar" modules into the same module.

> in the 25% of cases where *NAME SHADOWING* or where the
> author knows the location and can specify the location to
> save "lobby boy" from knocking on doors until his knuckles
> bleed, the author should have an option defining the search
> path *LOCALLY* (in a manner that will *NOT* affect the
> "import search path" further down the import chain), WITHOUT
> needing to import one of the many discombobulated and
> confusing import tools that Python has tortured this
> community with.

By doing so, the author would also make the module non-portable. This
is a major problem if they intend to distribute it, but still a
problem even if they don't. Any time the code is moved to a new system
and placed in a new location (or worse: moved to a new operating
system) the code will have to be updated. And you can pretty much
forget about hosting it on a network share.

>> 3) Filepath imports (deprecated in 3.4 but under discussion for
>> reprecation or replacement)
>>
>>      import importlib.machinery
>>
>>      loader = importlib.machinery.SourceFileLoader("module.name",
>>          "/path /to/file.py")
>>      foo = loader.load_module()
>>      foo.MyClass()
>
> That's ridiculously noisy. I have an idea, if you *REALLY*
> want to introduce boilerplate hell then Python should adopt
> "import header files", NOT!

It's three lines of code to replace one. Two if you exclude the
importlib.machinery import that doesn't need to be repeated.  Is this
really any worse than something like:

local_search_path.insert(0, "/path/to/local/module")
import my_local_module

that you are proposing?

[toc] | [prev] | [next] | [standalone]

#81234

On Friday, November 21, 2014 9:34:55 AM UTC-6, Ian wrote:
> On Thu, Nov 20, 2014 at 8:53 PM, Rick Johnson
> > FOR INSTANCE: Let's say i write a module that presents a
> > reusable GUI calendar widget, and then i name the module
> > "calender.py".
> >
> > Then Later, when i try to import *MY* GUI widget named
> > "calendar", i will not get *MY* calendar widget, no, i will
> > get the Python calendar module.
>
> Because like a fool you created a local module with a
> totally generic name like "calendar" and dumped it into
> the top-level namespace.

Are you also going to call drivers "fools" because they bought
a "certain brand" of car only to have the airbag explode in
their face? What's next, are you going to blame them for not
wearing a mask? Why am *i* the fool when it's obvious that
the creators of Python were either shortsighted and/or
careless with the designs? The only people who suffer are
those who put their trust in the designer, and not the
designer himself -- something is wrong with this picture!

> The Python import system gives you the ability to create
> packages, so use them -- create a package to contain all
> the widgets you create. If they're only for your own use,
> then you can just call it ricklib or anything else you
> want.

Of course, I did that a long time ago! But like everything
in Python, when your try to be cleaver and create a
workaround for the design flaws of this language (and there
are many!) you end up with shards of jagged metal stuck in
your face!

    GvR MUST BE A HUGE FAN OF THE JIGSAW MOVIES!

> Now you can drop as much stuff in there as you like, and
> none of it will ever conflict with the standard library
> (unless a standard "ricklib" module is added, which is
> unlikely).

Yes, and now we've solved one problem by replacing it with
it's inverse -- try importing the *python lib* calendar
module and all you will get is your local "intra-package"
version. Now, the only way to get to the lib module is by
mutilating sys.path, or using an import utility module to
"import by filepath".

    THANKS PYTHON!

You are correct though, Python's packaging system CAN BE
used to isolate "intra-package" scripts from clashing with
outside scripts, HOWEVER, there are many caveats that one
must know (which are NOT intuitive!) to use Python's
packaging system without requiring a plastic surgeon on
retainer!

Lets say i take your advice, and i want to use python
packages to protect a set of modules from the "global import
scope".

############################################################
#                     Explosive Trap 1                     #
############################################################
# Any attempt to import a stdlib module, who's name        #
# matches one of the modules within the current package,   #
# will result in importing the local "package module",     #
# forcing you to: (1) inject search paths into sys.path    #
# manually (2) import a module to correctly import the     #
# dependency                                               #
############################################################

YUCK!

############################################################
#                     Explosive Trap 2                     #
############################################################
# Any code in the __init__ file will be executed NOT ONLY  #
# when the package is imported DIRECTLY, but even when the #
# package name is referenced as part of a larger "absolute #
# import path"                                             #
############################################################

OUCH!

Anyone would expect that when *DIRECTLY* importing a
package, if the __init__ file has code, then THAT code
should be executed, HOWEVER,  not many would expect that
merely "referencing" the package name (in order to import a
more deeply nested package) would cause ALL the
intermediate __init__ files to execute -- this is madness,
and it prevents using an __init__ file as an "import hub"
(without side-effects)!

    "Yeah rick, but why would you want to use an __init__ file
     as an "import hub"

Because the alternative is messy. If i have a collection of
modules under a package, sometimes i would like to import
all the "exportable objects" into the __init__ file and use
the package as an "import hub". Imagine a package layout
like this:

    + ricklib
      __init__.py
      + subpkg1 (ricklib.subpkg1)
        __init__.py
        module1.py
        module2.py
        module3.py
        + subpkg1a (ricklib.subpkg1.subpkg1a)

And the contents of "ricklib.subpkg1.__init__.py" are as
follows:

    from module1 import Class1
    from module2 import Class2
    from module3 import Class3

Now, when i want to import a number of modules from subpkg1
i can do:

    from ricklib.subpkg1 import Class1, Class2, ...

    Instead of this:
    
    from ricklib.subpkg1.module1 import Class1
    from ricklib.subpkg1.module2 import Class2

And everything will work fine, UNTIL, i try to access
"subpkg1a"

    from ricklib.subpkg1.subpkg1a import something

By simply MENTIONING the name "subpkga" (within the
"absolute path" of "ricklib.subpkg1.subpkg1a"), Python will
execute the "ricklib.subpkg1.__init__.py" file, thereby
burning cycles importing things that don't even need at the
time! And god forbid you have some "print statements" in
there! 

    BETTER GO MAKE A SANDWICH WHILE THIS ONE LOADS!

> > This is another confusing fundamental of Python, it seems *MORE*
> > intuitive that changes to the "import search path" would only
> > affect the *CURRENT* module's "import search path", but that's
> > is not the case!
>
> This just seems like it would create a maintenance
> nightmare. Before you could import anything, you'd have to
> figure out what the search path is for the particular
> module you're working and whether it happens to include
> the particular package you want. You find it doesn't, so
> you make a local change to the search path. Later, you
> make the same change to other modules that need to import
> it. Later still, the package moves to a different location
> in the file system, and now you get to go through and
> update every one of those modules with the new directory.

But the current "global import search path" injections are
just the inverse. You make changes to sys.path in one
module, and if you fail to reset the changes before
execution moves to the next module in the "import chain",
then that module's import search path will be affected in
implicit ways that could result in importing the wrong
module.

At least with my proposed "local import search mutations",
the modifications to sys.path would NOT propagate outwards,
because each module would recieve a fresh copy of sys.path
from which to use "as-is" or to modify "as needed", but
those changes would only be relevant within the current
iteration of the import chain.

I think you fail to appreciate my proposal because you are
forgetting the importance of maintaining a "linear
structure" for external dependencies. This "idea" that a
script who is "higher" in the import chain, should be
affecting the "import search paths" defined in the all the
"lower scripts", is doomed to failure, and it will fail
hard.

> > That's ridiculously noisy. I have an idea, if you *REALLY*
> > want to introduce boilerplate hell then Python should adopt
> > "import header files", NOT!
>
> It's three lines of code to replace one. Two if you exclude the
> importlib.machinery import that doesn't need to be repeated.  Is this
> really any worse than something like:
>
> local_search_path.insert(0, "/path/to/local/module")
> import my_local_module
>
> that you are proposing?

If the changes were LOCAL, then i would have no problem to
this type of mutation, But they are not.

[toc] | [prev] | [next] | [standalone]

#81235

On Sat, Nov 22, 2014 at 5:24 AM, Rick Johnson
<rantingrickjohnson@gmail.com> wrote:
> Of course, I did that a long time ago! But like everything
> in Python, when your try to be cleaver...

Just so you know, I never try to be one of these.

http://img1.wikia.nocookie.net/__cb20110615074214/americanmcgeesalice/images/1/1a/Vorpal_Cleaver.png

Along with misuse of "your" when you mean "you", plus a few of the
all-too-common misplaced (or omitted) apostrophes, this kind of errant
English leaves me wondering if you take as much care over your Python
code as you do when writing c.l.py posts, and then blame Python for
the results.

ChrisA

[toc] | [prev] | [next] | [standalone]

#81237

On 11/21/2014 11:24 AM, Rick Johnson wrote:
> Why am *i* the fool when it's obvious that
> the creators of Python were either shortsighted and/or
> careless with the designs? The only people who suffer are
> those who put their trust in the designer, and not the
> designer himself -- something is wrong with this picture!

So if you know how to create the perfect language that has a solution to
these very difficult problems, please come forward with it.  Until such
time, it's okay to identify problems with the language as you see them,
but to cast aspersion on the developers is out of line.  Do you know
Guido personally to know that he's short-sighted or careless?  I
certainly don't, but from what I can see he is neither.  Not perfect,
but pretty darn smart.  Much smarter than I am (despite the fact I can
also recognize several of Python's flaws).  The same goes for many other
core developers.  They tend to be smart, articulate, and know how to
communicate with people.  Everything is about trade-offs and the ones
Python makes work very well for most people.

Perhaps the problem with the picture is that you are unable to see it
clearly. I assure you that Python developers not only create Python,
they use it too, for their own purposes.  Why would think they are some
sort of cabal getting kicks from leading the poor masses of Python
programmers at their whim?

[toc] | [prev] | [next] | [standalone]

#81247

On Friday, November 21, 2014 1:06:18 PM UTC-6, Michael Torrie wrote:
> On 11/21/2014 11:24 AM, Rick Johnson wrote:
> > Why am *i* the fool when it's obvious that
> > the creators of Python were either shortsighted and/or
> > careless with the designs? The only people who suffer are
> > those who put their trust in the designer, and not the
> > designer himself -- something is wrong with this picture!
> 
> So if you know how to create the perfect language that has a solution to
> these very difficult problems, please come forward with it.  Until such
> time, it's okay to identify problems with the language as you see them,
> but to cast aspersion on the developers is out of line.  Do you know
> Guido personally to know that he's short-sighted or careless?

Not personally. But how will we *ever* know if he refuses to
participate in these discussions? 

    YE SHALL *KNOW* THEM BY THEIR FRUITS!

Although GvR has created one of the best scripting languages
known to man, he has utterly failed to foster a sense of
community between the members, and no matter how great
Python might be, the fact that our community is nothing more
than a "*privileged* white-boys club" is nothing less than
self-destructive.

But even ignoring the atrocious sexual and racial diversity
of this community, outside of Python-dev and Python-ideas,
GvR has refuses to engage with the "mainstream alter boys".
And even WITHIN those "moderated groups", he's very
*selective* about who he will respond to.

    THIS COMMUNITY IS *ONLY* SUPERFICIALLY FREE!
    
I know we just *LOVE* to advertise our "commitment to
diversity"[1] but the truth is, our "commitment" is only a
friendly mask covering the evil truth. The truth is, there
exists a STRONG undercurrent of hostility towards ANY
dissent or outside influence within the "upper echelons" of
this community.

This is NOT a free and open society that prides itself on
building heterogeneous relationships to solve complex
problems, no, this is a *Theocracy*, where one man is raised
above all as though his phallus has been plated with gold!

And although he has not taken hammer and chisel to stone
tablets and carved "malevolent laws" by his own hand, by
virtue of his silence, he HAS implicitly defined a
"religious creed" which forbids all free thought and
expression  -- and those who *dare* to dissent will be
excommunicated. That's just what they did over at Python-
ideas not to long ago!

    WHY WOULD THEY DO THAT?

GvR has effectively walled himself off from the community with
the exception of a few "advisers" which he allows to enter
his "inner circle". This is NOT how you treat people who are
part of *YOUR* community, that is, IF you want to call
yourself "benevolent"!

    SO DROP THE "BENEVOLENT ACT" ALREADY, IT'S PATRONIZING!
    
If he *REALLY* wants to earn that "benevolent" badge of his,
then he needs to demand that all these forums become free
and open to all opinions and ideas. But more importantly, he
needs to start participating with the community at all
levels, not just hiding behind the coat tails of his attack
dogs and brown-nosers. 
    
   START BEARING SOME PALATABLE FRUITS!

[1]: https://www.python.org/community/diversity/

[toc] | [prev] | [next] | [standalone]

#81253

On 11/21/2014 01:29 PM, Rick Johnson wrote:
> Not personally. But how will we *ever* know if he refuses to
> participate in these discussions? 

Why should he participate in these discussions? Why should you be in
charge of said discussions?

By the way, Python has more than certainly borne fruit, and the vast
majority of it is very good indeed.

[toc] | [prev] | [next] | [standalone]

#81238

On Fri, Nov 21, 2014 at 11:24 AM, Rick Johnson
<rantingrickjohnson@gmail.com> wrote:
> Are you also going to call drivers "fools" because they bought
> a "certain brand" of car only to have the airbag explode in
> their face?

No, but I'll call them fools if they buy a car and the engine catches
fire because they never bothered to change the oil.

If you don't want to have module name collisions, then don't create
modules with names that are likely to collide when Python gives you an
excellent tool for avoiding collisions (namespaces). Don't go blaming
Python for "bad design" when you couldn't even be bothered to use the
tools made available to you.

>> Now you can drop as much stuff in there as you like, and
>> none of it will ever conflict with the standard library
>> (unless a standard "ricklib" module is added, which is
>> unlikely).
>
> Yes, and now we've solved one problem by replacing it with
> it's inverse -- try importing the *python lib* calendar
> module and all you will get is your local "intra-package"
> version. Now, the only way to get to the lib module is by
> mutilating sys.path, or using an import utility module to
> "import by filepath".

Um, no. If your calendar module is named ricklib.calendar, then
importing just calendar will import the standard library calendar.

The only exception is if you're doing "import calendar" from inside
the ricklib package, and you're using Python 2, and you don't have
"from __future__ import absolute_import" at the top of your module.
The solution to this is easy: just add that __future__ import to the
top of your module, and poof, implicit relative imports don't happen.
This is also fixed entirely in Python 3.

This is the second rather blatant error you've made about Python's
import mechanism, which makes me suspect that you don't really know
very much about it.

> Anyone would expect that when *DIRECTLY* importing a
> package, if the __init__ file has code, then THAT code
> should be executed, HOWEVER,  not many would expect that
> merely "referencing" the package name (in order to import a
> more deeply nested package) would cause ALL the
> intermediate __init__ files to execute -- this is madness,
> and it prevents using an __init__ file as an "import hub"
> (without side-effects)!

The whole point of the __init__.py file, in case you didn't intuit it
from the name, is to host any initialization code for the package. Why
on earth would you expect to import a module from a package without
initializing the package?

> Because the alternative is messy. If i have a collection of
> modules under a package, sometimes i would like to import
> all the "exportable objects" into the __init__ file and use
> the package as an "import hub".

What is the point of putting things into a hierarchical namespace in
the first place if you're just going to turn around and subvert it
like this?

> But the current "global import search path" injections are
> just the inverse. You make changes to sys.path in one
> module, and if you fail to reset the changes before
> execution moves to the next module in the "import chain",
> then that module's import search path will be affected in
> implicit ways that could result in importing the wrong
> module.

No, because the trick you describe doesn't even work. If you edit
sys.path in one file in order to import the coconut module:

sys.path.insert(0, '/path/to/island')
import coconut

And then in another module change the sys.path file and try to import
a different coconut module:

sys.path[0] = '/path/to/other/island')
import coconut

You think the second import will produce the second coconut module? It
won't, because the sys.modules cache will already contain an entry for
'coconut' that points to the first module imported. In order to make
this work, you would have to not only modify sys.path but also clear
the sys.modules cache.

Hopefully by the time you've done that you will have realized that
you're abusing the import system by having multiple modules with the
same name, and that as a general rule modules shouldn't be
manipulating the value of sys.path *at all*. Instead, set your
sys.path correctly from the PYTHONPATH environment variable, and if
you really must modify sys.path, try to do it only from the main
script.

>> It's three lines of code to replace one. Two if you exclude the
>> importlib.machinery import that doesn't need to be repeated.  Is this
>> really any worse than something like:
>>
>> local_search_path.insert(0, "/path/to/local/module")
>> import my_local_module
>>
>> that you are proposing?
>
> If the changes were LOCAL, then i would have no problem to
> this type of mutation, But they are not.

The example of a direct loader call that described as "boilerplate
hell", to which I was responding, doesn't change anything, locally or
globally. All it does is import a module from an arbitrary location.

[toc] | [prev] | [next] | [standalone]

#81252

On Friday, November 21, 2014 1:24:53 PM UTC-6, Ian wrote:
> On Fri, Nov 21, 2014 at 11:24 AM, Rick Johnson
> > Are you also going to call drivers "fools" because they bought
> > a "certain brand" of car only to have the airbag explode in
> > their face?
> 
> No, but I'll call them fools if they buy a car and the engine catches
> fire because they never bothered to change the oil.

As Dennis pointed out that's highly unlikely.

> If you don't want to have module name collisions, then don't create
> modules with names that are likely to collide when Python gives you an
> excellent tool for avoiding collisions (namespaces). Don't go blaming
> Python for "bad design" when you couldn't even be bothered to use the
> tools made available to you.

And by "namespaces" you must be talking about packages here?
Okay, hold that thought...

> >> Now you can drop as much stuff in there as you like, and
> >> none of it will ever conflict with the standard library
> >> (unless a standard "ricklib" module is added, which is
> >> unlikely).
> >
> > Yes, and now we've solved one problem by replacing it with
> > it's inverse -- try importing the *python lib* calendar
> > module and all you will get is your local "intra-package"
> > version. Now, the only way to get to the lib module is by
> > mutilating sys.path, or using an import utility module to
> > "import by filepath".
> 
> Um, no. If your calendar module is named ricklib.calendar, then
> importing just calendar will import the standard library calendar.
> 
> The only exception is if you're doing "import calendar" from inside
> the ricklib package, and you're using Python 2, and you don't have
> "from __future__ import absolute_import" at the top of your module.
> The solution to this is easy: just add that __future__ import to the
> top of your module, and poof, implicit relative imports don't happen.
> This is also fixed entirely in Python 3.

I wish the irony of needing to know an "implicit rule" to
solve an "implicit problem" could be funny, but it's really
just sad. I can't help but be reminded of the Python zen.
"If it's difficult to explain it's probably a bad idea".
What's more difficult to explain than an implicit rule you
have no knowledge of?

 NOW THERE'S SOME IRONY FOR YOU!

> > Anyone would expect that when *DIRECTLY* importing a
> > package, if the __init__ file has code, then THAT code
> > should be executed, HOWEVER,  not many would expect that
> > merely "referencing" the package name (in order to import a
> > more deeply nested package) would cause ALL the
> > intermediate __init__ files to execute -- this is madness,
> > and it prevents using an __init__ file as an "import hub"
> > (without side-effects)!
> 
> The whole point of the __init__.py file, in case you didn't intuit it
> from the name, is to host any initialization code for the package. Why
> on earth would you expect to import a module from a package without
> initializing the package?

Because you failed to notice that i was NOT importing the
"package" which contained the __init__file, no, i was
importing a *SUB PACKAGE* of the package that contained the
__init__ file.

Why does the code in the main package need to run when i
*explicitly* and *directly* fetched a "nested resource"
within the package? Nothing within the __init__ file is
affecting the code within the subpackage i wanted, and
inversely, the package i wanted does not depend on anything
within the __init__ file. There exists no symbiotic
relationship between these two machinery, yet, by
referencing one of them, the other starts doing unnecessary
things!

There is a workaround, but it's even more of a mess. In
order to maintain a unique "import hub" without the chance
of side effects from __init__ files, i can move all the code
in "ricklib.subpkg1.__init__.py" (the code that does all the
imports) into a normal file named
ricklib.subpkg1._subpkg1.py". 

 + ricklib 
   __init__.py 
   + subpkg1 (ricklib.subpkg1) 
      __init__.py 
      _subpkg1.py
      module1.py 
      module2.py 
      module3.py 
      + subpkg1a (ricklib.subpkg1.subpkg1a) 
      
Now, since the __init__ file has no global code, i can
import "ricklib.subpkg1.subpkg1s" without side effect -- but
of course, at a cost! 

Advanced Python packages are a zero sum game. You cannot
remove the problems, all you can do is move them to new
locations.

  so instead of the former (with side effects):
    
    from ricklib.subpkg1.subpkg1a import something
    
  I can do this (without side effect):

    from ricklib.subpkg1.subpkg1a._subpkg1a import something
    
  But at the cost of my sanity.
    

> > Because the alternative is messy. If i have a collection of
> > modules under a package, sometimes i would like to import
> > all the "exportable objects" into the __init__ file and use
> > the package as an "import hub".
> 
> What is the point of putting things into a hierarchical namespace in
> the first place if you're just going to turn around and subvert it
> like this?

I'm not "subverting it", i'm merely trying to organize my
*VAST* code libraries utilizing the ONLY tools that have been
made available to me -- and to be quite honest, these tools are
lacking!

If Python expects me to use packages to protect my module
names from clashing, but it gives me no method by which to
import packages without causing side effects, then what is a
boy to do (besides creating workarounds for workarounds)? 

> > But the current "global import search path" injections are
> > just the inverse. You make changes to sys.path in one
> > module, and if you fail to reset the changes before
> > execution moves to the next module in the "import chain",
> > then that module's import search path will be affected in
> > implicit ways that could result in importing the wrong
> > module.
> 
> No, because the trick you describe doesn't even work. If you edit
> sys.path in one file in order to import the coconut module:
> 
> sys.path.insert(0, '/path/to/island')
> import coconut
> 
> And then in another module change the sys.path file and try to import
> a different coconut module:
> 
> sys.path[0] = '/path/to/other/island')
> import coconut
> 
> You think the second import will produce the second coconut module? It
> won't, because the sys.modules cache will already contain an entry for
> 'coconut' that points to the first module imported. In order to make
> this work, you would have to not only modify sys.path but also clear
> the sys.modules cache.

DAMMIT! 

We have levels upon levels of esoteric, un-intuit-able, and
implicit procedures happining here. It seems that sys.path
modification is fruitless. Which is good, because i never
liked the idea of it anyway. It's a dirty hack!

Thanks for pointing this out in such a simplistic manner! I
now understand why the import mechinism is so unintuitable!
I'm gushing with exuberance because i feel as though a
"coconut of enlightenment" has been dropped on my head!

    I NOW KNOW HOW TO FIX THE PYTHON IMPORT PROBLEM!

I was aware that sys.path was global, but for some reason, i
failed to realize that sys.modules was (i don't know why, it
seems to make perfect sense now!) I was thinking that
sys.modules was specific to each script, and as such, even
though modules were being cached by name, the changes to
sys.path would circumvent the "short circuit behavior" of
"module name resolution"; implemented by a "peek" into
system modules.
    
Not only is Python going to stop searching the "file system"
for module named "coconut" when it first encounters a file
named coconut, it won't even check sys.path to see if the
programmer has injected a new directory that might contain a
*DIFFERENT* coconut file!

    '/path/to/other/island'

And that's good for performance, but bad for intuition!
Now, i'm not suggestion that Python burn up our hard-drives
with listdir() requests, however, there must be some sort of
method we can implement that will solve the "coconut import
problem".

============================================================
 THE SOLUTION TO PYTHON'S IMPLICIT IMPORT NIGHTMARE
============================================================

The only way to wrangle this beast in a manner that offer
both the ease of implicit machinery, and the power of explicit
machinery, and do both in a backwards compatible manner, is
to introduce the following:

############################################################
#                         STEP 1:                          #
############################################################
# Make sys.path *read-only*                                #
############################################################

Actually "step 1" is optional. We could leave it as is but
making it read-only would be more consistent. We need to
drive home the point that "injecting sys.path is fruitless"

############################################################
#                         STEP 2:                          #
############################################################
# Add a new listing called "sys.path_extra" (or whatever   #
# makes sense; that's a dumb name but the best i can think #
# of for now), OR (even better), allow a new variable into #
# each module namespace called "__search_path__" (or       #
# something. Again, the name is negotiable), for which     #
# will be a list of paths.                                 #
############################################################

############################################################
#                          STEP 3                          #
############################################################
# Make the following changes to the import machinery:      #
# Before Python reads a module file, Python will clear the #
# values in "sys.path_extra", OR, query the                #
# "__search_paths__" variable, if any paths exists in this #
# list, THEN THESE PATHS MUST BE SEARCHED, AND THEY MUST   #
# BE SEARCHED BEFORE ANY PATHS IN "sys.path", AND NO       #
# PEEKING IN "sys.modules" IS ALLOWED!                     #
############################################################

By this method, we can retain the existing implementation of
import, whist offering a more powerful tool to control the
machinery when needed.

[toc] | [prev] | [next] | [standalone]

#81254

On Friday, November 21, 2014 4:25:49 PM UTC-6, Rick Johnson wrote:
> ############################################################
> #                          STEP 3                          #
> ############################################################
> # Make the following changes to the import machinery:      #
> # Before Python reads a module file, Python will clear the #
> # values in "sys.path_extra", OR, query the                #
> # "__search_paths__" variable, if any paths exists in this #
> # list, THEN THESE PATHS MUST BE SEARCHED, AND THEY MUST   #
> # BE SEARCHED BEFORE ANY PATHS IN "sys.path", AND NO       #
> # PEEKING IN "sys.modules" IS ALLOWED!                     #
> ############################################################


Actually, to be more specific, not only should the
__search_path__ variable be unique to each file, Python
should query the value of "__search_path__" variable each
time it encounters an import statement.

So, if we wanted to import multiple modules who share the
same name, but who exist in different namespaces, we would
could do so by changing the values of __search_path__ before
calling the import:

    #
    # import the first coconut module Python can find using
    # the default search path, if not raise an error
    #
    import coconut

    #
    # import the first coconut module python can find using
    # *ONLY* the explicit paths provided, or raise an error.
    #
    __search_path__ = [
        'path/to/a/specific/coconut/tree',
        'might/be/here/also'
        ]
    import coconut

At first i was thinking that *IF* any "explicit paths" were
defined via __search_path__, then Python should look in all
those paths first, then if nothing is found look in
sys.modules, then if nothing is found, exhaust sys.modules,

BUT NOW I AM THINKING THAT IS A BAD IDEA!

When __search_path__ *IS* defined, Python should look *ONLY*
in the provided paths, and if not module is found, raise an
"ExplictImportError".

Because if we're going to be explicit then we need to also be
consistent. If the programmer is requesting that Python
search in a predefined directories, then obviously he
doing so to avoid name clashes, so if Python went on and
returned a module OUTSIDE of those predefined paths, Python
would be circumventing (IMPLICITLY AGAIN!) the wishes of the
programmer.

============================================================
 Succinct Summary Of My New Proposal:
============================================================

1. Use the historical "implicit import" mechanism for most day
to day imports, and let Python do all the heavy lifting.

2. Use the new "explicit import" mechanism for advanced name
resolutions, but realize that since you are now taking
control of import, so you must do more work, and you must be
"exhaustively explicit" about *where* Python searches.

[toc] | [prev] | [next] | [standalone]

#81256

On Sat, Nov 22, 2014 at 10:21 AM, Rick Johnson
<rantingrickjohnson@gmail.com> wrote:
> 1. Use the historical "implicit import" mechanism for most day
> to day imports, and let Python do all the heavy lifting.
>
> 2. Use the new "explicit import" mechanism for advanced name
> resolutions, but realize that since you are now taking
> control of import, so you must do more work, and you must be
> "exhaustively explicit" about *where* Python searches.

In other words, what you want is:

# today's method, import based on search path
import sys
# new explicit path method
import '/usr/local/lib/python3.5/lib-dynload/math.cpython-35m.so'

Can you tell me, please, how this is going to work for *any* system
other than the one it was built on? One of the reasons I write Python
code is because I expect it to work unchanged on Windows, Linux, Mac
OS, BSD, OS/2, uPy, and any other platform I care to throw it onto. If
I want to hard-code all the details, I might as well write C code and
at least take advantage of ./configure to work out the right paths for
me.

There are a VERY VERY few cases where you really do want to import a
module from a specific file path. For those situations, there are
already techniques you can use. Why change the behaviour of import
itself?

ChrisA

[toc] | [prev] | [next] | [standalone]

Page 1 of 3  [1] 2 3  Next page →

Back to top | Article view | comp.lang.python

csiph-web