Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #36606 > unrolled thread

PyWart: Import resolution order

Back to article view | Back to comp.lang.python

Contents

  PyWart: Import resolution order Rick Johnson <rantingrickjohnson@gmail.com> - 2013-01-10 22:13 -0800
    Re: PyWart: Import resolution order Chris Angelico <rosuav@gmail.com> - 2013-01-11 17:30 +1100
      Re: PyWart: Import resolution order Rick Johnson <rantingrickjohnson@gmail.com> - 2013-01-11 21:28 -0800
        Re: PyWart: Import resolution order Chris Angelico <rosuav@gmail.com> - 2013-01-12 17:03 +1100
        Re: PyWart: Import resolution order Ian Kelly <ian.g.kelly@gmail.com> - 2013-01-12 00:36 -0700
          Re: PyWart: Import resolution order 88888 Dihedral <dihedral88888@googlemail.com> - 2013-01-12 19:56 -0800
          Re: PyWart: Import resolution order 88888 Dihedral <dihedral88888@googlemail.com> - 2013-01-12 19:56 -0800
      Re: PyWart: Import resolution order Rick Johnson <rantingrickjohnson@gmail.com> - 2013-01-11 21:28 -0800
        Re: PyWart: Import resolution order alex23 <wuwei23@gmail.com> - 2013-01-12 19:23 -0800
    Re: PyWart: Import resolution order Terry Reedy <tjreedy@udel.edu> - 2013-01-11 08:35 -0500
    Re: PyWart: Import resolution order Michael Torrie <torriem@gmail.com> - 2013-01-11 10:53 -0700
    Re: PyWart: Import resolution order Rick Johnson <rantingrickjohnson@gmail.com> - 2013-01-11 20:50 -0800
      Re: PyWart: Import resolution order alex23 <wuwei23@gmail.com> - 2013-01-11 23:44 -0800

#36606 — PyWart: Import resolution order

Python's import resolution order is terrible.[1]

The fact that Python looks in the stdlib _first_ is not a good idea. It would seem more intuitive for a custom "math" module (living in the current directory) to /override/ the stlib "math" module. The proper order is as follows:

1. Current package or directory
2. stdlib
3. under the bed
4. who cares at this point

Look, if you want to keep you foolish imports starting from the stdlib, fine, just create a switch so we can change it to "package/directory" if we like. 

If /only/ the Python gods had placed all stdlib modules in a package named, i don't know, "lib" then none of this would be an issue. You could happily have a stdlib::math and a mylib::math, and when you import either module your code will reflect that path explicitly. Hmm, this last sentence reminded me of something??? Oh yes -> "Explicit is better than implicit".


[1]: Yes, yes, i know about sys.path.insert. *puke*!

[toc] | [next] | [standalone]

#36607

On Fri, Jan 11, 2013 at 5:13 PM, Rick Johnson
<rantingrickjohnson@gmail.com> wrote:
> The fact that Python looks in the stdlib _first_ is not a good idea. It would seem more intuitive for a custom "math" module (living in the current directory) to /override/ the stlib "math" module. The proper order is as follows:
>
> 1. Current package or directory
> 2. stdlib
> 3. under the bed
> 4. who cares at this point

Why is it better to import from the current directory first? Windows
has that policy for executable commands; Unix, on the other hand,
requires that you put an explicit path for anything that isn't in the
standard search path. Which of these options is the more likely to
produce security holes and/or unexpected behaviour?

Welcome back to the list, Rick. Got any demonstrable code for Python 4000 yet?

ChrisA

[toc] | [prev] | [next] | [standalone]

#36673

On Friday, January 11, 2013 12:30:27 AM UTC-6, Chris Angelico wrote:
> Why is it better to import from the current directory first?

Opps. I was not explicit enough with my explanation :). I meant, "look in the current directory FIRST when in a package". Since many times (most all times) packages will contain many sub-modules that need to be imported into the package's main.py module, and sometimes these modules will have the same name as a stdlib module, then looking in the package FIRST makes sense.


But the real solution is not to change the default resolution order. The real solution is to wrap all builtin modules into a package and use full paths to access every module. But wait, i have an even better idea... read below!

> Windows
> has that policy for executable commands; Unix, on the other hand,
> requires that you put an explicit path for anything that isn't in the
> standard search path. Which of these options is the more likely to
> produce security holes and/or unexpected behaviour?

I prefer the latter of course :). 

I think if python where *strict* about full paths for non-builtins, then we would be in a better place. But there is an even better solution! Force all python users to wrap THEIR modules in a toplevel package. Maybe even create the package for them. YES!. Call it "lib". Any "naked" modules (meaning modules that are not in a package)  would have to be accessed starting from "lib". Of course professionals like you and i are already using packages to properly nest out modules, but the newbie's won't realize the power of packaging modules for some time, so create the default "lib" package for them.

For instance you could create a package named "chris" and then have a module named math exist inside. Alternatively if you choose to be a non-professional and create a math module without a containing package, python would throw the module into the default "lib" package. The only way you could access your math module now would be by using the path "lib.math". 

So the conclusion is:

 * We encourage python programmers to use packages so they avoid any name clashes with built-in modules. 
 
 * if they refuse fine, any "naked" modules they create will be packaged in a default package (call it "lib", "userlib", or whatever) and will require them to prefix the module name with "lib." -- or "lib:" if i get my way!
 
By doing this we solve the many problems related to module name resolution orders and we create cleaner code bases. Damn i am full of good ideas!

> Welcome back to the list, Rick. Got any demonstrable code
> for Python 4000 yet?

I am working on it. Stay tuned. Rick is going to rock your little programming world /very/ soon.

[toc] | [prev] | [next] | [standalone]

#36679

On Sat, Jan 12, 2013 at 4:28 PM, Rick Johnson
<rantingrickjohnson@gmail.com> wrote:
> On Friday, January 11, 2013 12:30:27 AM UTC-6, Chris Angelico wrote:
>> Welcome back to the list, Rick. Got any demonstrable code
>> for Python 4000 yet?
>
> I am working on it. Stay tuned. Rick is going to rock your little programming world /very/ soon.

So all I have to do is paper my programming world and I win?

Sorry, can't hang around arguing about module search paths and your
recommendations to add piles of syntactic salt. Gotta finish building
and playing with today's Linux kernel (3.2.7, it's looking good so
far).

ChrisA

[toc] | [prev] | [next] | [standalone]

#36687

On Fri, Jan 11, 2013 at 10:28 PM, Rick Johnson
<rantingrickjohnson@gmail.com> wrote:
> On Friday, January 11, 2013 12:30:27 AM UTC-6, Chris Angelico wrote:
>> Why is it better to import from the current directory first?
>
> Opps. I was not explicit enough with my explanation :). I meant, "look in the current directory FIRST when in a package". Since many times (most all times) packages will contain many sub-modules that need to be imported into the package's main.py module, and sometimes these modules will have the same name as a stdlib module, then looking in the package FIRST makes sense.

And again, in Python 2.x this is already the case.  When importing in
a package, it tries to do a relative import before it even looks at
sys.path.

> I think if python where *strict* about full paths for non-builtins, then we would be in a better place.

And again, in Python 3, where implicit relative imports have been
removed from the language, it already is strict about using full
paths.  You can still do relative imports, but you have to be explicit
about them.

> For instance you could create a package named "chris" and then have a module named math exist inside. Alternatively if you choose to be a non-professional and create a math module without a containing package, python would throw the module into the default "lib" package. The only way you could access your math module now would be by using the path "lib.math".

What if I create a package named "math"?  Does that also automatically
get renamed to "lib.math"?  How is it decided what package names are
proper; is it just because it happens to clash with a stdlib name that
the package gets magically renamed?

What if I create a package, and then later a module with the same name
happens to be added to the stdlib?  My program that uses the package
just breaks because it no longer imports the correct thing?

> Damn i am full of good ideas!

Your ideas might be better if you first spent some time gaining a
better understanding of how the language works as is.

[toc] | [prev] | [next] | [standalone]

#36719

Ian於 2013年1月12日星期六UTC+8下午3時36分43秒寫道：
> On Fri, Jan 11, 2013 at 10:28 PM, Rick Johnson
> 
> <rantingrickjohnson@gmail.com> wrote:
> 
> > On Friday, January 11, 2013 12:30:27 AM UTC-6, Chris Angelico wrote:
> 
> >> Why is it better to import from the current directory first?
> 
> >
> 
> > Opps. I was not explicit enough with my explanation :). I meant, "look in the current directory FIRST when in a package". Since many times (most all times) packages will contain many sub-modules that need to be imported into the package's main.py module, and sometimes these modules will have the same name as a stdlib module, then looking in the package FIRST makes sense.
> 
> 
> 
> And again, in Python 2.x this is already the case.  When importing in
> 
> a package, it tries to do a relative import before it even looks at
> 
> sys.path.
> 
> 
> 
> > I think if python where *strict* about full paths for non-builtins, then we would be in a better place.
> 
> 
> 
> And again, in Python 3, where implicit relative imports have been
> 
> removed from the language, it already is strict about using full
> 
> paths.  You can still do relative imports, but you have to be explicit
> 
> about them.
> 
> 
> 
> > For instance you could create a package named "chris" and then have a module named math exist inside. Alternatively if you choose to be a non-professional and create a math module without a containing package, python would throw the module into the default "lib" package. The only way you could access your math module now would be by using the path "lib.math".
> 
> 
> 
> What if I create a package named "math"?  Does that also automatically
> 
> get renamed to "lib.math"?  How is it decided what package names are
> 
> proper; is it just because it happens to clash with a stdlib name that
> 
> the package gets magically renamed?
> 
> 
> 
> What if I create a package, and then later a module with the same name
> 
> happens to be added to the stdlib?  My program that uses the package
> 
> just breaks because it no longer imports the correct thing?
> 
> 
> 
> > Damn i am full of good ideas!
> 
> 
> 
> Your ideas might be better if you first spent some time gaining a
> 
> better understanding of how the language works as is.

OK, I think to develop a GUI with auto-code 
translations in an IDE  with python as the CAD/CAM scripting  language can be helpful.

But usually this kind of sotware projects is in the 
commercial part. 

[toc] | [prev] | [next] | [standalone]

#36720

Ian於 2013年1月12日星期六UTC+8下午3時36分43秒寫道：
> On Fri, Jan 11, 2013 at 10:28 PM, Rick Johnson
> 
> <rantingrickjohnson@gmail.com> wrote:
> 
> > On Friday, January 11, 2013 12:30:27 AM UTC-6, Chris Angelico wrote:
> 
> >> Why is it better to import from the current directory first?
> 
> >
> 
> > Opps. I was not explicit enough with my explanation :). I meant, "look in the current directory FIRST when in a package". Since many times (most all times) packages will contain many sub-modules that need to be imported into the package's main.py module, and sometimes these modules will have the same name as a stdlib module, then looking in the package FIRST makes sense.
> 
> 
> 
> And again, in Python 2.x this is already the case.  When importing in
> 
> a package, it tries to do a relative import before it even looks at
> 
> sys.path.
> 
> 
> 
> > I think if python where *strict* about full paths for non-builtins, then we would be in a better place.
> 
> 
> 
> And again, in Python 3, where implicit relative imports have been
> 
> removed from the language, it already is strict about using full
> 
> paths.  You can still do relative imports, but you have to be explicit
> 
> about them.
> 
> 
> 
> > For instance you could create a package named "chris" and then have a module named math exist inside. Alternatively if you choose to be a non-professional and create a math module without a containing package, python would throw the module into the default "lib" package. The only way you could access your math module now would be by using the path "lib.math".
> 
> 
> 
> What if I create a package named "math"?  Does that also automatically
> 
> get renamed to "lib.math"?  How is it decided what package names are
> 
> proper; is it just because it happens to clash with a stdlib name that
> 
> the package gets magically renamed?
> 
> 
> 
> What if I create a package, and then later a module with the same name
> 
> happens to be added to the stdlib?  My program that uses the package
> 
> just breaks because it no longer imports the correct thing?
> 
> 
> 
> > Damn i am full of good ideas!
> 
> 
> 
> Your ideas might be better if you first spent some time gaining a
> 
> better understanding of how the language works as is.

OK, I think to develop a GUI with auto-code 
translations in an IDE  with python as the CAD/CAM scripting  language can be helpful.

But usually this kind of sotware projects is in the 
commercial part. 

[toc] | [prev] | [next] | [standalone]

#36675

On Friday, January 11, 2013 12:30:27 AM UTC-6, Chris Angelico wrote:
> Why is it better to import from the current directory first?

Opps. I was not explicit enough with my explanation :). I meant, "look in the current directory FIRST when in a package". Since many times (most all times) packages will contain many sub-modules that need to be imported into the package's main.py module, and sometimes these modules will have the same name as a stdlib module, then looking in the package FIRST makes sense.


But the real solution is not to change the default resolution order. The real solution is to wrap all builtin modules into a package and use full paths to access every module. But wait, i have an even better idea... read below!

> Windows
> has that policy for executable commands; Unix, on the other hand,
> requires that you put an explicit path for anything that isn't in the
> standard search path. Which of these options is the more likely to
> produce security holes and/or unexpected behaviour?

I prefer the latter of course :). 

I think if python where *strict* about full paths for non-builtins, then we would be in a better place. But there is an even better solution! Force all python users to wrap THEIR modules in a toplevel package. Maybe even create the package for them. YES!. Call it "lib". Any "naked" modules (meaning modules that are not in a package)  would have to be accessed starting from "lib". Of course professionals like you and i are already using packages to properly nest out modules, but the newbie's won't realize the power of packaging modules for some time, so create the default "lib" package for them.

For instance you could create a package named "chris" and then have a module named math exist inside. Alternatively if you choose to be a non-professional and create a math module without a containing package, python would throw the module into the default "lib" package. The only way you could access your math module now would be by using the path "lib.math". 

So the conclusion is:

 * We encourage python programmers to use packages so they avoid any name clashes with built-in modules. 
 
 * if they refuse fine, any "naked" modules they create will be packaged in a default package (call it "lib", "userlib", or whatever) and will require them to prefix the module name with "lib." -- or "lib:" if i get my way!
 
By doing this we solve the many problems related to module name resolution orders and we create cleaner code bases. Damn i am full of good ideas!

> Welcome back to the list, Rick. Got any demonstrable code
> for Python 4000 yet?

I am working on it. Stay tuned. Rick is going to rock your little programming world /very/ soon.

[toc] | [prev] | [next] | [standalone]

#36718

On Jan 12, 3:28 pm, Rick Johnson <rantingrickjohn...@gmail.com> wrote:
> I am working on it. Stay tuned. Rick is going to rock your little programming world /very/ soon.

I am so confidant that this will never happen that if you _do_ ever
produce _anything_ that even remotely resembles your claims, I pledge
to provide you with enough funding to continue full-time development
on it for 5 years, let's say 5 years @ US$50k per year.

However, one condition for acceptance will be that you never post here
again.

[toc] | [prev] | [next] | [standalone]

#36620

On 1/11/2013 1:13 AM, Rick Johnson wrote:
>
> Python's import resolution order is terrible.[1]
>
> The fact that Python looks in the stdlib _first_ is not a good idea.

And the fact is that it does not do so. The order depends on sys.path, 
and '' is the first entry.

> It would seem more intuitive for a custom "math" module (living in
> the current directory) to /override/ the stlib "math" module.

Try it.

This is a nuisance though, when not intended. Someone writes a 
random.py, and a year later in the same directory, an unrelated 
hopscotch.py, which tries to import random.exponential. The import fails 
and they post here, having forgotten about their own random.py, which 
does not have such a function. Posts like this happen a few times a year.

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#36634

On 01/10/2013 11:13 PM, Rick Johnson wrote:
> 
> Python's import resolution order is terrible.[1]
> 
> The fact that Python looks in the stdlib _first_ is not a good idea.

Whether or not the default behavior is desirable or not, sys.path is set
by default to look in the current directory first on any Python
distribution I have looked at.  As Terry says, this fact causes a lot of
problems for newbies who accidentally override standard library modules
with their own python files and chaos ensues.

If your python installation does not search the current directory first,
then you must have changed sys.path.

[toc] | [prev] | [next] | [standalone]

#36669

On Friday, January 11, 2013 7:35:37 AM UTC-6, Terry Reedy wrote:
> On 1/11/2013 1:13 AM, Rick Johnson wrote:
> > The fact that Python looks in the stdlib _first_ is not a good idea.
> 
> And the fact is that it does not do so. The order depends on sys.path, 
> and '' is the first entry.
> 
> > It would seem more intuitive for a custom "math" module (living in
> > the current directory) to /override/ the stlib "math" module.
>
> This is a nuisance though, when not intended. Someone writes a 
> random.py, and a year later in the same directory, an unrelated 
> hopscotch.py, which tries to import random.exponential. The import fails 
> and they post here, having forgotten about their own random.py, which 
> does not have such a function. Posts like this happen a few times a year.

That's why i also mentioned the failure of Python to wrap stdlib modules in a package. If we would protect all built-in modules by placing them in a package (lib or py) then this problem would never happen. 

Of course many people will piss and moan about the extra typing. I say, you have a choice: a few extra chars or multitudes of extra headaches -- I choose the first option.

Since more time is spent /maintaining/ code bases than /writing/ them, the explicit path is always the correct path to choose. Anyone who says otherwise is either careless or selfish (aka: seeking job security).

[toc] | [prev] | [next] | [standalone]

#36689

On 12 Jan, 14:50, Rick Johnson <rantingrickjohn...@gmail.com> wrote:
> Of course many people will piss and moan about the extra typing.

You just ignored the fact that your original claim was incorrect and
kept going on with your rant anyway.

> Since more time is spent /maintaining/ code bases than /writing/ them

In your case, more time is actually spent on insisting _other_ people
maintain code bases.

Everyone: PLEASE STOP FEEDING THE TROLL

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web