Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #13093 > unrolled thread

import packet.module without importing packet.__init__ ?

Back to article view | Back to comp.lang.python

Contents

  import packet.module without importing packet.__init__ ? Gelonida N <gelonida@gmail.com> - 2011-09-11 02:05 +0200
    Re: import packet.module without importing packet.__init__ ? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-09-11 10:56 +1000
      Re: import packet.module without importing packet.__init__ ? Gelonida N <gelonida@gmail.com> - 2011-09-11 05:39 +0200
        Re: import packet.module without importing packet.__init__ ? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-09-11 14:51 +1000
          Re: import packet.module without importing packet.__init__ ? Gelonida N <gelonida@gmail.com> - 2011-09-11 16:17 +0200

#13093 — import packet.module without importing packet.__init__ ?

Hi,

I am little shaky with how exactly python imports packages / modules etc.

Is it possible to import a module from a packet without importing its
__init__.py ?


Full example:
==============

# application.py ---------------------
print "starting application"
import mypacket.module1

# mypacket.__init__.py ---------------------
print "importing __init__"


# mypacket.module1.py ---------------------
print "importing module1"



The output, that I get with python 2.6 is
$ python application.py
> starting application
> importing __init__
> importing module1

Under certain circumstances I'd like to avoid the implicit import
of __init__.py

Due to other constrains of my project I am obliged to have a
non empty __init__.py

[toc] | [next] | [standalone]

#13096

Gelonida N wrote:


> Is it possible to import a module from a packet without importing its
> __init__.py ?

Untested, but I think so. But you shouldn't. The solution (if it does work,
as I said I haven't tested it) is a hack.

Suppose you have a library like this:

modules/
+-- spam.py
+-- ham.py
+-- package/
    +-- __init__.py
    +-- cheese.py

The directory "modules" is in your Python path, so you can "import spam"
or "import package". The trick is to add "modules/package" to the Python
path so that "import cheese" will find cheese.py directly.

You can do that by manipulating sys.path, or by setting the environment
variable PYTHONPATH.

Another hack would be to add a hard link to the top level:

modules/
+-- spam.py
+-- ham.py
+-- cheese.py <------------+
+-- package/               |
    +-- __init__.py        |
    +-- cheese.py <--------+


This is not a copy, it is a hard link: the same file appears in literally
two places. (But note that not all file systems support hard linking.) Soft
links or shortcuts might also work. Now you can "import cheese".

But generally, you should design your package so that it doesn't matter
whether or not __init__.py is imported first. I see three reasnable
situations:

(1) package.cheese should rely on package, in which case it requires
package.__init__ to be imported first. (Sometimes I put code common to the
entire package in __init__.py.)

(2) Just don't worry about the fact that package.__init__ gets imported
first. Do you have any idea how many modules get imported when Python
starts up? What's one more?

(3) If cheese.py truly is independent of package, then it shouldn't be part
of package. Just make it a stand alone module outside the package
directory, and be done. (Your installer can still treat cheese.py as a
dependency of package, and ensure that they are both installed.)



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#13105

Hi Steven,

Thanks for your answer.

On 09/11/2011 02:56 AM, Steven D'Aprano wrote:
> Gelonida N wrote: 
>> Is it possible to import a module from a packet without importing its
>> __init__.py ?
> 
> Untested, but I think so. But you shouldn't. The solution (if it does work,
> as I said I haven't tested it) is a hack.
> 
> Suppose you have a library like this:
. . .
. . .
> The directory "modules" is in your Python path, so you can "import spam"
> or "import package". The trick is to add "modules/package" to the Python
> path so that "import cheese" will find cheese.py directly.
> 

Yes it's a hack, but good to know it exists. Just in case.

> Another hack would be to add a hard link to the top level:
> 

As you said: Hard links would be a little annoying on some file systems.

> 
> But generally, you should design your package so that it doesn't matter
> whether or not __init__.py is imported first. I see three reasnable
> situations:
> 
> (1) package.cheese should rely on package, in which case it requires
> package.__init__ to be imported first. (Sometimes I put code common to the
> entire package in __init__.py.)
> 
> (2) Just don't worry about the fact that package.__init__ gets imported
> first. Do you have any idea how many modules get imported when Python
> starts up? What's one more?
> 
> (3) If cheese.py truly is independent of package, then it shouldn't be part
> of package. Just make it a stand alone module outside the package
> directory, and be done. (Your installer can still treat cheese.py as a
> dependency of package, and ensure that they are both installed.)
> 


There's still something, that I am philosophycally missing.

Wy do I have to import the entire tree if I'm just interested in a leave.

this is not always desired.

let's imagine

kitchen.__init__
kitchen.knives
kitchen.pans
kitchen.pots


or alternatively
os
os.path
os.wheteverelse

Let's assume, that  kitchen/__init__.py is importing the sub modules
knives, pans and pots

then I understand of course, that the next lines would import the entire
kitchen

> import kitchen # will import all activities
> kitchen.pots.boil(egg)


But if I know that I just want to boil some eggs I would really
appreciate if I could just say

> from kitchen.pot import boil
> boil(eggs)
without having the side effect of getting pans and knives also loaded
into memory.

Moving pots out into a separate package doesn't really feel right.
I agree, that a 10 level deep package tree is not really desirable, but
having every package flat on the top level doesn't sound that good either.

So it seems, that developers of huge packages shouldn't really do a lot
in __init__.py.

Otherwise any user would always 'suck in' the entiry package with all
its sub modules.




To give more details about why I asked this question:

The real setup looks a little different and there are some issues, that
I am having:

One package in question is a huge django application.

Within the package is one module defining some constants and tiny
functions, which I would like to reuse from some command line tools,
which should start quickly without loading any of the django specific
modules. (the startup penalty and the memory overhead would be noticable)

I am using rpc4django, which expects, that __init__.py of the django
application exports some all rpc functions
which will basically import 95% of the django application and the entire
django frame work (none of which were required by my command tool,
support utility for this application)


I could of course create a separate package just for this tiny sub
module, but somehow it doesn't feel right to me.


The second issue is a little more complicated. basically I end up having
circular dependencies.

as I have two applications that  send a few customs django signals forth
an back.

If I wouldn't import __init__.py or if I moved the declaration of the
signals into a third package then circular dependencies would disappear.


So it seems, that if I can't avoid loading __init__.py in
a nice way I have to create  one or two tiny packages outside of my
django application,
or change the python path  as you proposed (increases of course name
space clashes)

or to find a way for rpc4django of loading / locating the rpc functions
without putting them  in __init__.py change the rpc functions such, that
they will only import the other sub modules when being called the first
time (proxies / delayed imports, . . .)

[toc] | [prev] | [next] | [standalone]

#13111

Gelonida N wrote:


> There's still something, that I am philosophycally missing.
> 
> Wy do I have to import the entire tree if I'm just interested in a leave.

You don't. Python just imports the branches leading to the leaf, not the
entire tree.

[...]
> But if I know that I just want to boil some eggs I would really
> appreciate if I could just say
> 
>> from kitchen.pot import boil
>> boil(eggs)
> without having the side effect of getting pans and knives also loaded
> into memory.

In your example, you stated that kitchen explicitly imports kitchen.pans and
kitchen.knives. So in that case, take it up with the designer of the
kitchen package -- it was his decision to import them, just like he
imported re and math and httplib. (For example.) 

If kitchen is your package, then it is your choice: if you want to have
control over when sub-packages get imported, then don't automatically
import them in __init__.py. If you want them to be automatically imported,
like os and os.path, then automatically import them. If you don't, don't.

But note that you cannot expect to import kitchen.pot without importing
kitchen first.


> One package in question is a huge django application.
> 
> Within the package is one module defining some constants and tiny
> functions, which I would like to reuse from some command line tools,
> which should start quickly without loading any of the django specific
> modules. (the startup penalty and the memory overhead would be noticable)
> 
> I am using rpc4django, which expects, that __init__.py of the django
> application exports some all rpc functions
> which will basically import 95% of the django application and the entire
> django frame work (none of which were required by my command tool,
> support utility for this application)
> 
> 
> I could of course create a separate package just for this tiny sub
> module, but somehow it doesn't feel right to me.

Why should it be a package? Just make it a stand-alone module.

Or use path manipulation from your command line tool to import it on its
own. It might be somewhat of a hack, but if need to do it, do so.

Or do something like this:

my_app/
+--  __init__.py  # lightweight, nearly empty
+--  cmd_tools.py
+--  module1.py
+--  module2.py
+--  rpc/
     +-- __init__.py  # huge, imports my_app.module1, my_app.module2, etc.


then point rpc4django at my_app.rpc instead of my_app.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#13127

Hi Steven,

Thanks again for your answer.


On 09/11/2011 06:51 AM, Steven D'Aprano wrote:
> Gelonida N wrote:

> 
> In your example, you stated that kitchen explicitly imports kitchen.pans and
> kitchen.knives. So in that case, take it up with the designer of the
> kitchen package -- it was his decision to import them, just like he
> imported re and math and httplib. (For example.) 

Exactly. Thus my conclusion, that too many imports (just for commodity)
in __init__.py might not be such a good idea if one wants to allow
leaf-only imports.
> 
> If kitchen is your package, then it is your choice: if you want to have
> control over when sub-packages get imported, then don't automatically
> import them in __init__.py. 

Agreed. This is however my problem: Up to my understanding rpc4django
insists on putting code into __init__.py

> 
> 
>> One package in question is a huge django application.
>>
>> Within the package is one module defining some constants and tiny
>> functions, which I would like to reuse from some command line tools,
>> which should start quickly without loading any of the django specific
>> modules. (the startup penalty and the memory overhead would be noticable)
>>
>> I am using rpc4django, which expects, that __init__.py of the django
>> application exports some all rpc functions
>> which will basically import 95% of the django application and the entire
>> django frame work (none of which were required by my command tool,
>> support utility for this application)
>>
>>
>> I could of course create a separate package just for this tiny sub
>> module, but somehow it doesn't feel right to me.
> 
> Why should it be a package? Just make it a stand-alone module.

True

> Or do something like this:
> 
> my_app/
> +--  __init__.py  # lightweight, nearly empty
> +--  cmd_tools.py
> +--  module1.py
> +--  module2.py
> +--  rpc/
>      +-- __init__.py  # huge, imports my_app.module1, my_app.module2, etc.
> 
> then point rpc4django at my_app.rpc instead of my_app.
> 

Up to my knowledge rpc4django just imports all __inits__ of all django
applications to find look for functions with a certain decorator.

I'm not sure, whether django allows nested appllications, but this might
be a solution.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web