Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #4593 > unrolled thread

importing class objects from a pickled file

Back to article view | Back to comp.lang.python

Contents

  importing class objects from a pickled file Catherine Moroney <Catherine.M.Moroney@jpl.nasa.gov> - 2011-05-03 17:18 -0700
    Re: importing class objects from a pickled file James Mills <prologic@shortcircuit.net.au> - 2011-05-04 12:29 +1000
    Re: importing class objects from a pickled file Owen Jacobson <angrybaldguy@gmail.com> - 2011-05-04 01:29 -0400
    Re: importing class objects from a pickled file Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2011-05-05 14:28 +1200

#4593 — importing class objects from a pickled file

Hello,

I have an object of class X that I am writing to a pickled file.  The 
pickling part goes fine, but I am having some problems reading the 
object back out, as I get complaints about "unable to import module X".

The only way I have found around it is to run the read-file code out of 
the same directory that contains the X.py file, but this is obviously 
not a portable way of doing things.

Even when I put statements into the code such as "from Y.X import X"
where Y is the name of the python package that contains the X,py file,
the import statement works, but I am still unable to read the object 
from the pickled file, running into the same "unable to import module X"
error.

Am I explaining myself properly?  Why doesn't the code that loads the
object from the pickled file work unless I am sitting in the same 
directory?   The code that writes the pickled file has the statement
"from Y.X import X" statement" at the top, as does the reading code, but
even though that import statement succeeds, the read still fails with 
the import error.

Thanks for any help,

Catherine

[toc] | [next] | [standalone]

#4595

On Wed, May 4, 2011 at 10:18 AM, Catherine Moroney
<Catherine.M.Moroney@jpl.nasa.gov> wrote:
> Am I explaining myself properly?  Why doesn't the code that loads the
> object from the pickled file work unless I am sitting in the same directory?
>   The code that writes the pickled file has the statement
> "from Y.X import X" statement" at the top, as does the reading code, but
> even though that import statement succeeds, the read still fails with the
> import error.

pickled objects won't work this way unless you "customize" the
way in which your class gets serialized.

See: http://docs.python.org/library/pickle.html#the-pickle-protocol

Any time you want to unpickle a user class, that class must be available.

I suggest serializing to a more common format (say JSON) and re-create
your class with the data.

cheers
James

-- 
-- James Mills
--
-- "Problems are solved by method"

[toc] | [prev] | [next] | [standalone]

#4597

On 2011-05-03 20:18:33 -0400, Catherine Moroney said:

> Hello,
> 
> I have an object of class X that I am writing to a pickled file.  The 
> pickling part goes fine, but I am having some problems reading the 
> object back out, as I get complaints about "unable to import module X".
> 
> The only way I have found around it is to run the read-file code out of 
> the same directory that contains the X.py file, but this is obviously 
> not a portable way of doing things.
> 
> Even when I put statements into the code such as "from Y.X import X"
> where Y is the name of the python package that contains the X,py file,
> the import statement works, but I am still unable to read the object 
> from the pickled file, running into the same "unable to import module X"
> error.
> 
> Am I explaining myself properly?  Why doesn't the code that loads the
> object from the pickled file work unless I am sitting in the same directory?

That's not the actual requirement.

The pickled representation of an instance of a user-defined class contains:

	* the name of the class
	* the name of the class's module
	* the object's attributes, pickled recursively

You can have a look at the representation using protocol zero, which is 
printable and not terribly hard to read. Here's an example, lightly 
annotated:

                >>> import pickle
                >>> class Example(object):
                ...   def __init__(self, x):
                ...     self.x = x
                ...
                >>> one = Example(1)
                >>> print pickle.dumps(one, 0)
                ccopy_reg
                _reconstructor
                p0
module name     (c__main__
class name      Example
                p1
                c__builtin__
                object
                p2
                Ntp3
                Rp4
                (dp5
field name      S'x'
                p6
field value     I1
                sb.
                >>>

Obviously, since the pickled representation only contains the *name and 
location* of the class and not the class itself, you need to have the 
definition of the class handy when unpickling an object. This means 
that the module that defines the class must be on the module search 
path - either because it's in the current directory (or a package in 
the current directory, or..), or because it's in PYTHONPATH, or because 
it's somewhere else on sys.path.

> The code that writes the pickled file has the statement
> "from Y.X import X" statement" at the top, as does the reading code, but
> even though that import statement succeeds, the read still fails with 
> the import error.

The unpickle code uses something morally equivalent to __import__ to 
convert the module name in the pickled data stream into a module 
object. The symbols defined in the module you called pickle.load from 
don't matter at all, just like any other function defined in a separate 
module.

Pickle isn't meant for transmitting data between two unrelated 
programs. It's designed to be a low-development-effort way to preserve 
and restore objects for a single program or a group of related programs 
that share libraries. If you want to share data between unrelated 
programs, use something a little more interoperable - json, xml, or 
some other program-agnostic representation.

-o

(And if you were thinking of using pickle over the internet, forget it: 
the pickle encoding is a small stack-based programming language in its 
own right, and can be used to invoke semi-arbitrary module-scoped 
funtions.)

[toc] | [prev] | [next] | [standalone]

#4685

Catherine Moroney wrote:

> I am having some problems reading the 
> object back out, as I get complaints about "unable to import module X".
> 
> The only way I have found around it is to run the read-file code out of 
> the same directory that contains the X.py file
> 
> Even when I put statements into the code such as "from Y.X import X" ...
> the import statement works, but I am still unable to read the object

Is the program that reads the pickle file the same one that
was used to write it?

If not, what might be happening is that the writing program
has module X at the top level instead of inside package Y.
Then it will get pickled simply under the name "X" instead
of "Y.X", and the reading program will expect to find it at
the top level as well.

It's important that the reading and writing programs agree
about the location of the pickled classes in the package
namespace, unless you take steps to customise the pickling
process.

-- 
Greg

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web