Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #57820 > unrolled thread

Re: Organising packages/modules - importing functions from a common.py in a separate directory?

Back to article view | Back to comp.lang.python

Contents

  Re: Organising packages/modules - importing functions from a common.py in a separate directory? Jean-Michel Pichavant <jeanmichel@sequans.com> - 2013-10-28 14:12 +0100
    Re: Organising packages/modules - importing functions from a common.py in a separate directory? Victor Hooi <victorhooi@gmail.com> - 2013-10-28 17:51 -0700
      Re: Organising packages/modules - importing functions from a common.py in a separate directory? Ben Finney <ben+python@benfinney.id.au> - 2013-10-29 12:01 +1100
        Re: Organising packages/modules - importing functions from a common.py in a separate directory? Victor Hooi <victorhooi@gmail.com> - 2013-10-28 18:08 -0700
          Re: Organising packages/modules - importing functions from a common.py in a separate directory? Victor Hooi <victorhooi@gmail.com> - 2013-10-28 23:58 -0700
            Re: Organising packages/modules - importing functions from a common.py in a separate directory? Peter Otten <__peter__@web.de> - 2013-10-29 08:44 +0100
              Re: Organising packages/modules - importing functions from a common.py in a separate directory? Victor Hooi <victorhooi@gmail.com> - 2013-10-29 12:39 -0700
                Re: Organising packages/modules - importing functions from a common.py in a separate directory? Peter Otten <__peter__@web.de> - 2013-10-30 10:48 +0100
                Re: Organising packages/modules - importing functions from a common.py in a separate directory? Peter Otten <__peter__@web.de> - 2013-10-30 11:23 +0100

#57820 — Re: Organising packages/modules - importing functions from a common.py in a separate directory?

----- Original Message -----
> Hi,
> 
> I have a collection of Python scripts I'm using to load various bits
> of data into a database.
> 
> I'd like to move some of the common functions (e.g. to setup loggers,
> reading in configuration etc.) into a common file, and import them
> from there.
> 
> I've created empty __init__.py files, and my current directory
> structure looks something like this:
> 
> foo_loading/
>     __init__.py
>     common/
>         common_foo.py
>     em_load/
>         __init__.py
>         config.yaml
>         sync_em.py
>     pg_load/
>         __init__.py
>         config.yaml
>         sync_pg.py
> 
> So from within the sync_em.py script, I'm trying to import a function
> from foo_loading/common/common_foo.py.
> 
>     from ..common.common_foo import setup_foo_logging
> 
> I get the error:
> 
>     ValueError: Attempted relative import in non-package
> 
> If I change directories to the parent of "foo_loading", then run
> 
>     python -m foo_loading.em_load.sync_em sync_em.py
> 
> it works. However, this seems a bit roundabout, and I suspect I'm not
> doing things correctly.
> 
> Ideally, I want a user to be able to just run sync_em.py from it's
> own directory, and have it correctly import the logging/config
> modules from common_foo.py, and just work.
> 
> What is the correct way to achieve this?
> 
> Secondly, if I want to move all of the config.yaml files to a common
> foo_loading/config.yaml, or even foo_loading/config/config.yaml,
> what is the correct way to access this from within the scripts?
> Should I just be using "../", or is there a better way?
> 
> Cheers,
> Victor

Long story short : use absolute imports.

name properly your module with a distinct name and import that way, even inside your package:

import foo_loading.common.common_foo

Names like common, lib, setup are farely prone to collision with other badly referenced import from other modules. One way to solve this is to use a distinct namespace, in other words, prefix every import with the module name.

cheers,

JM


-- IMPORTANT NOTICE: 

The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

[toc] | [next] | [standalone]

#57865

Hi,

Ok, so I should be using absolute imports, not relative imports.

Hmm, I just tried to use absolute imports, and it can't seem to locate the modules:

In the file "foo_loading/em_load/sync_em.py", I have:

    from common.common_bex import setup_foo_logging

When I try to run that script:

    python sync_em.py

I get:

    ImportError: No module named common.common_foo

I've also tried adding "foo_loading" (the package name):

    from foo_loading.common.common_bex import setup_foo_logging

Same error:

    ImportError: No module named foo_loading.common.bex_common

Any thoughts?

Cheers,
Victor

On Tuesday, 29 October 2013 00:12:58 UTC+11, Jean-Michel Pichavant  wrote:
> ----- Original Message -----
> > Hi,
> > 
> > I have a collection of Python scripts I'm using to load various bits
> > of data into a database.
> > 
> > I'd like to move some of the common functions (e.g. to setup loggers,
> > reading in configuration etc.) into a common file, and import them
> > from there.
> > 
> > I've created empty __init__.py files, and my current directory
> > structure looks something like this:
> > 
> > foo_loading/
> >     __init__.py
> >     common/
> >         common_foo.py
> >     em_load/
> >         __init__.py
> >         config.yaml
> >         sync_em.py
> >     pg_load/
> >         __init__.py
> >         config.yaml
> >         sync_pg.py
> > 
> > So from within the sync_em.py script, I'm trying to import a function
> > from foo_loading/common/common_foo.py.
> > 
> >     from ..common.common_foo import setup_foo_logging
> > 
> > I get the error:
> > 
> >     ValueError: Attempted relative import in non-package
> > 
> > If I change directories to the parent of "foo_loading", then run
> > 
> >     python -m foo_loading.em_load.sync_em sync_em.py
> > 
> > it works. However, this seems a bit roundabout, and I suspect I'm not
> > doing things correctly.
> > 
> > Ideally, I want a user to be able to just run sync_em.py from it's
> > own directory, and have it correctly import the logging/config
> > modules from common_foo.py, and just work.
> > 
> > What is the correct way to achieve this?
> > 
> > Secondly, if I want to move all of the config.yaml files to a common
> > foo_loading/config.yaml, or even foo_loading/config/config.yaml,
> > what is the correct way to access this from within the scripts?
> > Should I just be using "../", or is there a better way?
> > 
> > Cheers,
> > Victor
> 
> Long story short : use absolute imports.
> 
> name properly your module with a distinct name and import that way, even inside your package:
> 
> import foo_loading.common.common_foo
> 
> Names like common, lib, setup are farely prone to collision with other badly referenced import from other modules. One way to solve this is to use a distinct namespace, in other words, prefix every import with the module name.
> 
> cheers,
> 
> JM
> 
> 
> -- IMPORTANT NOTICE: 
> 
> The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

[toc] | [prev] | [next] | [standalone]

#57866

Victor Hooi <victorhooi@gmail.com> writes:

> Ok, so I should be using absolute imports, not relative imports.

I'd say it is fine to use relative imports, so long as they are
explicit. (In Python 3, the default for an import is to be absolute, and
the *only* way to do a relative import is to make it explicitly
relative. So you may as well start doing so now.)

> Hmm, I just tried to use absolute imports, and it can't seem to locate
> the modules:
>
> In the file "foo_loading/em_load/sync_em.py", I have:
>
>     from common.common_bex import setup_foo_logging

So I'd recommend this be done with an explicit relative import:

    from .common.common_bex import setup_foo_logging

or, better, import a module:

    from .common import common_bex

or a whole package:

    from . import common

-- 
 \         “I went over to the neighbor's and asked to borrow a cup of |
  `\       salt. ‘What are you making?’ ‘A salt lick.’” —Steven Wright |
_o__)                                                                  |
Ben Finney

[toc] | [prev] | [next] | [standalone]

#57867

Hi,

If I try to use:

    from .common.common_foo import setup_foo_logging

I get:

    ValueError: Attempted relative import in non-package

And the absolute imports don't seem to be able to find the right modules.

Is it something to do with the fact I'm running the sync_em.py script from the "foo_loading/em_load" directory?

I thought I could just refer to the full path, and it'd find it, but evidently not...hmm.

Cheers,
Victor

On Tuesday, 29 October 2013 12:01:03 UTC+11, Ben Finney  wrote:
> Victor Hooi <victorhooi@gmail.com> writes:
> 
> 
> 
> > Ok, so I should be using absolute imports, not relative imports.
> 
> 
> 
> I'd say it is fine to use relative imports, so long as they are
> 
> explicit. (In Python 3, the default for an import is to be absolute, and
> 
> the *only* way to do a relative import is to make it explicitly
> 
> relative. So you may as well start doing so now.)
> 
> 
> 
> > Hmm, I just tried to use absolute imports, and it can't seem to locate
> 
> > the modules:
> 
> >
> 
> > In the file "foo_loading/em_load/sync_em.py", I have:
> 
> >
> 
> >     from common.common_bex import setup_foo_logging
> 
> 
> 
> So I'd recommend this be done with an explicit relative import:
> 
> 
> 
>     from .common.common_bex import setup_foo_logging
> 
> 
> 
> or, better, import a module:
> 
> 
> 
>     from .common import common_bex
> 
> 
> 
> or a whole package:
> 
> 
> 
>     from . import common
> 
> 
> 
> -- 
> 
>  \         “I went over to the neighbor's and asked to borrow a cup of |
> 
>   `\       salt. ‘What are you making?’ ‘A salt lick.’” —Steven Wright |
> 
> _o__)                                                                  |
> 
> Ben Finney

[toc] | [prev] | [next] | [standalone]

#57890

Hi,

Hmm, this post on SO seems to suggest that importing from another sibling directory in a package ins't actually possibly in Python without some ugly hacks?

http://stackoverflow.com/questions/6323860/sibling-package-imports

Did I read the above correctly?

Is there another way I can structure my code so that I can run the sync_em.py and sync_pg.py scripts, and they can pull common functions from somewhere?

Cheers,
Victor

On Tuesday, 29 October 2013 12:08:10 UTC+11, Victor Hooi  wrote:
> Hi,
> 
> 
> 
> If I try to use:
> 
> 
> 
>     from .common.common_foo import setup_foo_logging
> 
> 
> 
> I get:
> 
> 
> 
>     ValueError: Attempted relative import in non-package
> 
> 
> 
> And the absolute imports don't seem to be able to find the right modules.
> 
> 
> 
> Is it something to do with the fact I'm running the sync_em.py script from the "foo_loading/em_load" directory?
> 
> 
> 
> I thought I could just refer to the full path, and it'd find it, but evidently not...hmm.
> 
> 
> 
> Cheers,
> 
> Victor
> 
> 
> 
> On Tuesday, 29 October 2013 12:01:03 UTC+11, Ben Finney  wrote:
> 
> > Victor Hooi <victorhooi@gmail.com> writes:
> 
> > 
> 
> > 
> 
> > 
> 
> > > Ok, so I should be using absolute imports, not relative imports.
> 
> > 
> 
> > 
> 
> > 
> 
> > I'd say it is fine to use relative imports, so long as they are
> 
> > 
> 
> > explicit. (In Python 3, the default for an import is to be absolute, and
> 
> > 
> 
> > the *only* way to do a relative import is to make it explicitly
> 
> > 
> 
> > relative. So you may as well start doing so now.)
> 
> > 
> 
> > 
> 
> > 
> 
> > > Hmm, I just tried to use absolute imports, and it can't seem to locate
> 
> > 
> 
> > > the modules:
> 
> > 
> 
> > >
> 
> > 
> 
> > > In the file "foo_loading/em_load/sync_em.py", I have:
> 
> > 
> 
> > >
> 
> > 
> 
> > >     from common.common_bex import setup_foo_logging
> 
> > 
> 
> > 
> 
> > 
> 
> > So I'd recommend this be done with an explicit relative import:
> 
> > 
> 
> > 
> 
> > 
> 
> >     from .common.common_bex import setup_foo_logging
> 
> > 
> 
> > 
> 
> > 
> 
> > or, better, import a module:
> 
> > 
> 
> > 
> 
> > 
> 
> >     from .common import common_bex
> 
> > 
> 
> > 
> 
> > 
> 
> > or a whole package:
> 
> > 
> 
> > 
> 
> > 
> 
> >     from . import common
> 
> > 
> 
> > 
> 
> > 
> 
> > -- 
> 
> > 
> 
> >  \         “I went over to the neighbor's and asked to borrow a cup of |
> 
> 
> >   `\       salt. ‘What are you making?’ ‘A salt lick.’” —Steven Wright |
> 
> > 
> 
> > _o__)                                                                  |
> 
> > 
> 
> > Ben Finney

[toc] | [prev] | [next] | [standalone]

#57892

Victor Hooi wrote:

> Hi,
> 
> Hmm, this post on SO seems to suggest that importing from another sibling
> directory in a package ins't actually possibly in Python without some ugly
> hacks?
> 
> http://stackoverflow.com/questions/6323860/sibling-package-imports
> 
> Did I read the above correctly?

Yes.
 
> Is there another way I can structure my code so that I can run the
> sync_em.py and sync_pg.py scripts, and they can pull common functions from
> somewhere?

The packages you are trying to access in your original post 

> foo_loading/
>     __init__.py
>     common/
>         common_foo.py
>     em_load/
>         __init__.py
>         config.yaml
>         sync_em.py
>     pg_load/
>         __init__.py
>         config.yaml
>         sync_pg.py


aren't actually siblings in the sense of the stackoverflow topic above, they 
are subpackages of foo_loading, and as you already found out

> So from within the sync_em.py script, I'm trying to import a function from 
foo_loading/common/common_foo.py.
> 
>     from ..common.common_foo import setup_foo_logging
> 
> I get the error:
> 
>     ValueError: Attempted relative import in non-package 
> 
> If I change directories to the parent of "foo_loading", then run
> 
>     python -m foo_loading.em_load.sync_em sync_em.py
> 
> it works. However, this seems a bit roundabout, and I suspect I'm not 
doing things correctly.
> 
> Ideally, I want a user to be able to just run sync_em.py from it's own 
directory, and have it correctly import the logging/config modules from 
common_foo.py, and just work.
> 
> What is the correct way to achieve this?

you can access them as long as the *parent* directory of foo_loading is in 
sys.path through PYTHONPATH, or as the working directory, or any other 
means. However, if you step into the package, e. g.

$ cd foo_loading
$ python -c 'import common'

then from Python's point of view 'common' is a toplevel package rather than 
the intended 'foo_loading.common', and intra-package imports will break.

To preserve your sanity I therefore recommend that you 

(1) avoid to put package directories into sys.path
(1a) avoid to cd into a package
(2) put scripts you plan to invoke directly rather than import outside the 
package.

[toc] | [prev] | [next] | [standalone]

#57963

Hi,

Wait - err, subpackage != module, right? Do you think you could explain what a sub-package is please? I tried Googling, and couldn't seem to find the term in this context.

Also, so you're saying to put the actual script that I want to invoke *outside* the Python package.

Do you mean something like this:

> sync_em.py
> sync_pg.py
> foo_loading/ 
>     __init__.py 
>     common/ 
>         common_foo.py 
>     em_load/ 
>         __init__.py 
>         config.yaml 
>         em.py
>     pg_load/ 
>         __init__.py 
>         config.yaml 
>         pg.py

and the sync_em.py and sync_pg.py would just be thin wrappers pulling in things from em.py and pg.py? Is that a recommended approach to organise the code?

Would it make any difference if I actually packaged it up so you could install it in site-packages? Could I then call modules from other modules within the package?

Cheers,
Victor

On Tuesday, 29 October 2013 18:44:47 UTC+11, Peter Otten  wrote:
> Victor Hooi wrote:
> 
> 
> 
> > Hi,
> 
> > 
> 
> > Hmm, this post on SO seems to suggest that importing from another sibling
> 
> > directory in a package ins't actually possibly in Python without some ugly
> 
> > hacks?
> 
> > 
> 
> > http://stackoverflow.com/questions/6323860/sibling-package-imports
> 
> > 
> 
> > Did I read the above correctly?
> 
> 
> 
> Yes.
> 
>  
> 
> > Is there another way I can structure my code so that I can run the
> 
> > sync_em.py and sync_pg.py scripts, and they can pull common functions from
> 
> > somewhere?
> 
> 
> 
> The packages you are trying to access in your original post 
> 
> 
> 
> > foo_loading/
> 
> >     __init__.py
> 
> >     common/
> 
> >         common_foo.py
> 
> >     em_load/
> 
> >         __init__.py
> 
> >         config.yaml
> 
> >         sync_em.py
> 
> >     pg_load/
> 
> >         __init__.py
> 
> >         config.yaml
> 
> >         sync_pg.py
> 
> 
> 
> 
> 
> aren't actually siblings in the sense of the stackoverflow topic above, they 
> 
> are subpackages of foo_loading, and as you already found out
> 
> 
> 
> > So from within the sync_em.py script, I'm trying to import a function from 
> 
> foo_loading/common/common_foo.py.
> 
> > 
> 
> >     from ..common.common_foo import setup_foo_logging
> 
> > 
> 
> > I get the error:
> 
> > 
> 
> >     ValueError: Attempted relative import in non-package 
> 
> > 
> 
> > If I change directories to the parent of "foo_loading", then run
> 
> > 
> 
> >     python -m foo_loading.em_load.sync_em sync_em.py
> 
> > 
> 
> > it works. However, this seems a bit roundabout, and I suspect I'm not 
> 
> doing things correctly.
> 
> > 
> 
> > Ideally, I want a user to be able to just run sync_em.py from it's own 
> 
> directory, and have it correctly import the logging/config modules from 
> 
> common_foo.py, and just work.
> 
> > 
> 
> > What is the correct way to achieve this?
> 
> 
> 
> you can access them as long as the *parent* directory of foo_loading is in 
> 
> sys.path through PYTHONPATH, or as the working directory, or any other 
> 
> means. However, if you step into the package, e. g.
> 
> 
> 
> $ cd foo_loading
> 
> $ python -c 'import common'
> 
> 
> 
> then from Python's point of view 'common' is a toplevel package rather than 
> 
> the intended 'foo_loading.common', and intra-package imports will break.
> 
> 
> 
> To preserve your sanity I therefore recommend that you 
> 
> 
> 
> (1) avoid to put package directories into sys.path
> 
> (1a) avoid to cd into a package
> 
> (2) put scripts you plan to invoke directly rather than import outside the 
> 
> package.

[toc] | [prev] | [next] | [standalone]

#58015

Victor Hooi wrote:

> Wait - err, subpackage != module, right? Do you think you could explain
> what a sub-package is please? I tried Googling, and couldn't seem to find
> the term in this context.

In analogy to subdirectory I em_load and pg_load -- and common if you add an 
__init__.py would be sub-packages, provided only the parent of foo_loading 
is in sys.path and you import them with

import foo_loading.pg_load

etc.

> Also, so you're saying to put the actual script that I want to invoke
> *outside* the Python package.
> 
> Do you mean something like this:
> 
>> sync_em.py
>> sync_pg.py
>> foo_loading/
>>     __init__.py
>>     common/
           __init__.py
>>         common_foo.py
>>     em_load/
>>         __init__.py
>>         config.yaml
>>         em.py
>>     pg_load/
>>         __init__.py
>>         config.yaml
>>         pg.py
> 
> and the sync_em.py and sync_pg.py would just be thin wrappers pulling in
> things from em.py and pg.py? Is that a recommended approach to organise
> the code?

I don't know. I prefer it that way.
 
> Would it make any difference if I actually packaged it up so you could
> install it in site-packages? Could I then call modules from other modules
> within the package?

If you mean "import", yes, installing is one way to get it into sys.path.

[toc] | [prev] | [next] | [standalone]

#58024

Victor Hooi wrote:

> Wait - err, subpackage != module, right? Do you think you could explain
> what a sub-package is please? I tried Googling, and couldn't seem to find
> the term in this context.

[second attempt]

In analogy to the term "subdirectory" em_load and pg_load -- and common if 
you add an __init__.py -- would be sub-packages, provided only the parent of 
foo_loading is in sys.path and you import them with

import foo_loading.pg_load

etc.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web