Path: csiph.com!usenet.pasdenom.info!goblin1!goblin2!goblin.stu.neva.ru!newsfeed.xs4all.nl!newsfeed4a.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
MIME-Version: 1.0
In-Reply-To: <eb5918fb-41ea-4199-b2f2-ae28b454c983@googlegroups.com>
References: <eb5918fb-41ea-4199-b2f2-ae28b454c983@googlegroups.com>
From: Devin Jeanpierre <jeanpierreda@gmail.com>
Date: Wed, 7 Jan 2015 14:55:32 -0600
Subject: Re: pickle, modules, and ImportErrors
To: John Ladasky <john_ladasky@sbcglobal.net>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Cc: "comp.lang.python" <python-list@python.org>
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.17453.1420664176.18130.python-list@python.org>
Lines: 70
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:83302

On Wed, Jan 7, 2015 at 2:12 PM, John Ladasky <john_ladasky@sbcglobal.net> w=
rote:
> If I execute "import my_svr" in an iPython interpreter, everything works =
as I think that I should expect:
-snip-
> However, a nearly-identical program in the parent folder fails (note that=
 all I change is the relative path to the file):
> Traceback (most recent call last):
>   File "reload test different directory.py", line 6, in <module>
>     model =3D load(f)
> ImportError: No module named 'model'
>
>
> Do I need to "import my_svr.model as model" then?  Adding that line chang=
es nothing. I get the exact same "ImportError: No module named 'model'".
>
> Likewise for "import my_svr", "from my_svr import *", or even "from my_sv=
r.model import SVRModel".
>
> It is clear that I'm failing to understand something important.

in the first case, the model module was available as a top-level
module, "model". Pickles referenced that module when serialized. In
the second case, the model module was available as a submodule of the
top level my_svr package. So any pickles serialized from there would
use my_svr.model to refer to the model module. There *is* no model
module in this second case, so deserializing fails.

If you never run model directly, and only ever import it or run it as
my_svr.model, then you will be fine, and pickles will all serialize
and deserialize the same way.

For example, instead of python -i my_svr/model.py, you can use python
-im my_svr.model . (or ipython -im my_svr.model).

P.S. don't use pickle, it is a security vulnerability equivalent in
severity to using exec in your code, and an unversioned opaque
schemaless blob that is very difficult to work with when circumstances
change.

> I do not have any circular import dependencies; however, some of the file=
s in my package do need to import definitions from files earlier in my data=
 pipeline.  In order to make everything work inside the module, as well as =
making a parent-folder "import my_svr" work from a iPython,  I find myself =
needing to use statements like these inside my training.py program:
>
>
> try:
>     from model import *
>     from sampling import *
> except ImportError:
>     from .model import *
>     from .sampling import *
>
>
> This bothers me.  I don't know whether it is correct usage.  I don't know=
 whether it is causing my remaining ImportError problem.

This is a symptom of the differing ways you are importing these
modules, as above. If you only ever run them and import them as
my_svr.blahblah, then only the second set of imports are necessary.

P.S. don't use import *, and if you do use import *, don't use more
than one per file -- it makes it really hard to figure out where a
given global came from (was it defined here? was it defined in model?
was it defined in sampling?)

I hope that resolves all your questions!

-- Devin