Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #83301 > unrolled thread

pickle, modules, and ImportErrors

Back to article view | Back to comp.lang.python

Contents

  pickle, modules, and ImportErrors John Ladasky <john_ladasky@sbcglobal.net> - 2015-01-07 12:12 -0800
    Re: pickle, modules, and ImportErrors Devin Jeanpierre <jeanpierreda@gmail.com> - 2015-01-07 14:55 -0600
      Re: pickle, modules, and ImportErrors John Ladasky <john_ladasky@sbcglobal.net> - 2015-01-07 16:23 -0800
        Re: pickle, modules, and ImportErrors Chris Angelico <rosuav@gmail.com> - 2015-01-08 12:06 +1100
    Re: pickle, modules, and ImportErrors Ian Kelly <ian.g.kelly@gmail.com> - 2015-01-07 14:01 -0700

#83301 — pickle, modules, and ImportErrors

I am progressing towards organizing a recent project of mine as a proper Python package.  It is not a huge package, about 700 lines of code so far.  But it breaks into logical pieces, and I don't feel like scrolling back and forth through a 700-line file.

I am running Python 3.4.0 on Ubuntu 14.04, if it matters.  

I want a package which I can use in an iPython session, as well as in programs which share the package directory.  Compatibility within the package directory appears to be easy.  From outside the package, I am getting ImportErrors that I have not been able to fix.

Although I have used pickle often, this is the first time that I have written a package with an __init__.py.  It reads thus:


# __init__.py for my_svr module

from .database import MasterDatabase
from .sampling import Input
from .sampling import MultiSample
from .sampling import Sample
from .sampling import Target
from .sampling import DataPoint
from .model import Prediction
from .model import SVRModel
from .model import comparison
from .training import TrainingSession

__all__ = ("MasterDatabase", "Input", "MultiSample", "Sample", "Target",
           "DataPoint", "Prediction", "SVRModel", "comparison",
           "TrainingSession")


If I execute "import my_svr" in an iPython interpreter, everything works as I think that I should expect:


In [8]: dir(my_svr)
Out[8]: 
['Input',
 'MasterDatabase',
 'MultiSample',
 'Prediction',
 'SVRModel',
 'Sample',
 'Target',
 'DataPoint',
 'TrainingSession',
 '__all__',
 '__builtins__',
 '__cached__',
 '__doc__',
 '__file__',
 '__loader__',
 '__name__',
 '__package__',
 '__path__',
 '__spec__',
 'comparison',
 'database',
 'model',
 'sampling',
 'training']



My training.py program produces an SVRModel object, and pickles it to a binary file.  The following simple program will unpickle the file saved by training.py, provided that the program resides in the module's own folder:


# reload_test.py

from pickle import load

with open("../models/sample model.pkl", "rb") as f:
    model = load(f)
print(model)


And, I get the str representation of my SVRModel instance.

However, a nearly-identical program in the parent folder fails (note that all I change is the relative path to the file):


# parent_folder_reload_test.py

from pickle import load

with open("models/sample model.pkl", "rb") as f:
    model = load(f)
print(model)


Traceback (most recent call last):
  File "reload test different directory.py", line 6, in <module>
    model = load(f)
ImportError: No module named 'model'


Do I need to "import my_svr.model as model" then?  Adding that line changes nothing. I get the exact same "ImportError: No module named 'model'".  

Likewise for "import my_svr", "from my_svr import *", or even "from my_svr.model import SVRModel".

It is clear that I'm failing to understand something important.

I do not have any circular import dependencies; however, some of the files in my package do need to import definitions from files earlier in my data pipeline.  In order to make everything work inside the module, as well as making a parent-folder "import my_svr" work from a iPython,  I find myself needing to use statements like these inside my training.py program:


try:
    from model import *
    from sampling import *
except ImportError:
    from .model import *
    from .sampling import *


This bothers me.  I don't know whether it is correct usage.  I don't know whether it is causing my remaining ImportError problem.

Any advice is appreciated.  Thanks!

[toc] | [next] | [standalone]

#83302

On Wed, Jan 7, 2015 at 2:12 PM, John Ladasky <john_ladasky@sbcglobal.net> wrote:
> If I execute "import my_svr" in an iPython interpreter, everything works as I think that I should expect:
-snip-
> However, a nearly-identical program in the parent folder fails (note that all I change is the relative path to the file):
> Traceback (most recent call last):
>   File "reload test different directory.py", line 6, in <module>
>     model = load(f)
> ImportError: No module named 'model'
>
>
> Do I need to "import my_svr.model as model" then?  Adding that line changes nothing. I get the exact same "ImportError: No module named 'model'".
>
> Likewise for "import my_svr", "from my_svr import *", or even "from my_svr.model import SVRModel".
>
> It is clear that I'm failing to understand something important.

in the first case, the model module was available as a top-level
module, "model". Pickles referenced that module when serialized. In
the second case, the model module was available as a submodule of the
top level my_svr package. So any pickles serialized from there would
use my_svr.model to refer to the model module. There *is* no model
module in this second case, so deserializing fails.

If you never run model directly, and only ever import it or run it as
my_svr.model, then you will be fine, and pickles will all serialize
and deserialize the same way.

For example, instead of python -i my_svr/model.py, you can use python
-im my_svr.model . (or ipython -im my_svr.model).

P.S. don't use pickle, it is a security vulnerability equivalent in
severity to using exec in your code, and an unversioned opaque
schemaless blob that is very difficult to work with when circumstances
change.

> I do not have any circular import dependencies; however, some of the files in my package do need to import definitions from files earlier in my data pipeline.  In order to make everything work inside the module, as well as making a parent-folder "import my_svr" work from a iPython,  I find myself needing to use statements like these inside my training.py program:
>
>
> try:
>     from model import *
>     from sampling import *
> except ImportError:
>     from .model import *
>     from .sampling import *
>
>
> This bothers me.  I don't know whether it is correct usage.  I don't know whether it is causing my remaining ImportError problem.

This is a symptom of the differing ways you are importing these
modules, as above. If you only ever run them and import them as
my_svr.blahblah, then only the second set of imports are necessary.

P.S. don't use import *, and if you do use import *, don't use more
than one per file -- it makes it really hard to figure out where a
given global came from (was it defined here? was it defined in model?
was it defined in sampling?)

I hope that resolves all your questions!

-- Devin

[toc] | [prev] | [next] | [standalone]

#83308

On Wednesday, January 7, 2015 12:56:29 PM UTC-8, Devin Jeanpierre wrote:

[snip]

> If you never run model directly, and only ever import it or run it as
> my_svr.model, then you will be fine, and pickles will all serialize
> and deserialize the same way.

Thank you Devin... I re-ran TrainingSession from within ipython, and everything worked the way I hoped.  I obtained an SVRModel object which I could pickle to disk, and unpickle in a subsequent session.  I don't actually need to run the test code that I appended to training.py any more, so I won't.

[snip]

> P.S. don't use pickle, it is a security vulnerability equivalent in
> severity to using exec in your code, and an unversioned opaque
> schemaless blob that is very difficult to work with when circumstances
> change.

For all of its shortcomings, I can't live without pickle.  In this case, I am doing data mining.  My TrainingSession class commandeers seven CPU cores via Multiprocessing.Pool.  Still, even my "toy" TrainingSessions take several minutes to run.  I can't afford to re-run TrainingSession every time I need my models.  I need a persistent object.

Besides, the opportunity for mischief is low.  My code is for my own personal use.  And I trust the third-party libraries that I am using.  My SVRModel object wraps the NuSVR object from scikit-learn, which in turn wraps the libsvm binary.

> > try:
> >     from model import *
> >     from sampling import *
> > except ImportError:
> >     from .model import *
> >     from .sampling import *
> >
> >
> > This bothers me.  I don't know whether it is correct usage.  I don't know whether it is causing my remaining ImportError problem.
> 
> This is a symptom of the differing ways you are importing these
> modules, as above. If you only ever run them and import them as
> my_svr.blahblah, then only the second set of imports are necessary.

OK, I will try refactoring the import code at some point.
 
> I hope that resolves all your questions!

I think so, thanks.

[toc] | [prev] | [next] | [standalone]

#83309

On Thu, Jan 8, 2015 at 11:23 AM, John Ladasky
<john_ladasky@sbcglobal.net> wrote:
>> P.S. don't use pickle, it is a security vulnerability equivalent in
>> severity to using exec in your code, and an unversioned opaque
>> schemaless blob that is very difficult to work with when circumstances
>> change.
>
> For all of its shortcomings, I can't live without pickle.  In this case, I am doing data mining.  My TrainingSession class commandeers seven CPU cores via Multiprocessing.Pool.  Still, even my "toy" TrainingSessions take several minutes to run.  I can't afford to re-run TrainingSession every time I need my models.  I need a persistent object.
>
> Besides, the opportunity for mischief is low.  My code is for my own personal use.  And I trust the third-party libraries that I am using.  My SVRModel object wraps the NuSVR object from scikit-learn, which in turn wraps the libsvm binary.

There are several issues, not all of which are easily dodged. Devin cited two:

* Security: it's fundamentally equivalent to using 'exec'
* Unversioned: it's hard to make updates to your code and then load old data

"For your own personal use" dodges the first one, but makes the second
one even more of a concern. You can get much better persistence using
a textual format like JSON, and adding in a simple 'version' member
can make it even easier. Then, when you make changes, you can cope
with old data fairly readily.

Pickle is still there if you want it, but you do have to be aware of
its limitations. If you edit the TrainingSession class, you may well
have to rerun the training... but maybe that's not a bad thing.

ChrisA

[toc] | [prev] | [next] | [standalone]

#83303

On Wed, Jan 7, 2015 at 1:12 PM, John Ladasky <john_ladasky@sbcglobal.net> wrote:
> Do I need to "import my_svr.model as model" then?  Adding that line changes nothing. I get the exact same "ImportError: No module named 'model'".
>
> Likewise for "import my_svr", "from my_svr import *", or even "from my_svr.model import SVRModel".
>
> It is clear that I'm failing to understand something important.
>
> I do not have any circular import dependencies; however, some of the files in my package do need to import definitions from files earlier in my data pipeline.  In order to make everything work inside the module, as well as making a parent-folder "import my_svr" work from a iPython,  I find myself needing to use statements like these inside my training.py program:
>
>
> try:
>     from model import *
>     from sampling import *
> except ImportError:
>     from .model import *
>     from .sampling import *
>
>
> This bothers me.  I don't know whether it is correct usage.  I don't know whether it is causing my remaining ImportError problem.

It sounds like you have import path confusion. The first imports there
are absolute imports. If the absolute paths of the named modules are
just "model" and "sampling", then that would be correct. However, it
sounds like these modules are part of a "my_svr" package, so the
absolute paths of these modules are actually "my_svr.model" and
"my_svr.sampling". In that case, the the absolute path that you're
trying to use for the import is wrong. Either fully specify the module
path, or use the relative import you have below.

Now, why might those absolute imports appear to work even though the
paths are incorrect? Well, it sounds like you may be running a file
inside the package as your main script. This causes some problems.

1) Python doesn't realize that the script it's running is part of a
package. It just calls that module '__main__', and if something else
happens to import the module later by its real name, then you'll end
up loading a second copy of the module by that name, which can lead to
confusing behavior.

2) Python implicitly adds the directory containing the script to the
front of the sys.path list. Since the directory containing the package
is presumably already on the sys.path, this means that the internal
modules now appear in sys.path *twice*, with two different names:
"my_svr.model" and "model" refer to the same source file. Python
doesn't realize this however, and so they don't refer to the same
module. As a result, importing both "my_svr.model" and "model" will
again result in two separate copies of the module being loaded, with
two different names.

It sounds like your pickle file was created from an object from the
"model" module. This works fine when the sys.path is set such that
"model" is the absolute path of a module, i.e. when you run your main
script within the package. When running an external main script,
sys.path doesn't get set that way, and so it can't find the "model"
module.

The solution to this is to avoid executing a module within a package
as your main script. Run a script outside the package and have it
import and call into the package instead. An alternative is to run the
module using the -m flag of the Python executable; this fixes problem
2 above but not problem 1.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web