Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #22532 > unrolled thread

why can't I pickle a class containing this dispatch dictionary?

Back to article view | Back to comp.lang.python

Contents

  why can't I pickle a class containing this dispatch dictionary? jkn <jkn_gg@nicorp.f9.co.uk> - 2012-04-02 04:17 -0700
    Re: why can't I pickle a class containing this dispatch dictionary? Michael Hrivnak <mhrivnak@hrivnak.org> - 2012-04-02 18:48 -0400
    Re: why can't I pickle a class containing this dispatch dictionary? Peter Otten <__peter__@web.de> - 2012-04-03 09:54 +0200
      Re: why can't I pickle a class containing this dispatch dictionary? jkn <jkn_gg@nicorp.f9.co.uk> - 2012-04-03 05:57 -0700
        Re: why can't I pickle a class containing this dispatch dictionary? Peter Otten <__peter__@web.de> - 2012-04-03 16:44 +0200
      Re: why can't I pickle a class containing this dispatch dictionary? 88888 Dihedral <dihedral88888@googlemail.com> - 2012-04-03 07:29 -0700
      Re: why can't I pickle a class containing this dispatch dictionary? 88888 Dihedral <dihedral88888@googlemail.com> - 2012-04-03 07:29 -0700

#22532 — why can't I pickle a class containing this dispatch dictionary?

Hi All
    I'm clearly not understanding the 'can't pickle instancemethod
objects' error; can someone help me to understand, & maybe suggest a
workaround, (apart from the obvious if ... elif...).

I'm running Python 2.6 on an embedded system.

== testpickle.py ==
import pickle

class Test(object):
    def __init__(self):
        self.myDict = {
            1: self.tag1,
            2: self.tag2
            }
    def dispatch(self, v):
        try:
            self.myDict[v]()
        except KeyError:
            print "No corresponding dictionary entry!"
        #
    def tag1(self):
        print "one"
    def tag2(self):
        print "two"


t = Test()
t.dispatch(1)
t.dispatch(2)
t.dispatch(0)

fd = open("pickle.out", "w")
pickle.dump(t, fd)
fd.close()
# EOF

$ python testpickle.py
one
two
No corresponding dictionary entry!
Traceback (most recent call last):
  File "ptest.py", line 29, in <module>
    pickle.dump(t, fd)
  File "/usr/lib/python2.6/pickle.py", line 1362, in dump
    Pickler(file, protocol).dump(obj)
  File "/usr/lib/python2.6/pickle.py", line 224, in dump
    self.save(obj)
  File "/usr/lib/python2.6/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.6/pickle.py", line 419, in save_reduce
    save(state)
  File "/usr/lib/python2.6/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.6/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.6/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/usr/lib/python2.6/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.6/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.6/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/usr/lib/python2.6/pickle.py", line 306, in save
    rv = reduce(self.proto)
  File "/usr/lib/python2.6/copy_reg.py", line 70, in _reduce_ex
    raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle instancemethod objects
$


    Thanks
    J^n

[toc] | [next] | [standalone]

#22553

Pickle cannot pickle a reference to an instance method.  So the
problem is that self.myDict has values which are references to
instance methods.

Without questioning what this is trying to do or why (I assume it's a
proof of concept), here is a way to make it picklable:
http://pastebin.com/1zqE52mD

Michael

On Mon, Apr 2, 2012 at 7:17 AM, jkn <jkn_gg@nicorp.f9.co.uk> wrote:
> Hi All
>    I'm clearly not understanding the 'can't pickle instancemethod
> objects' error; can someone help me to understand, & maybe suggest a
> workaround, (apart from the obvious if ... elif...).
>
> I'm running Python 2.6 on an embedded system.
>
> == testpickle.py ==
> import pickle
>
> class Test(object):
>    def __init__(self):
>        self.myDict = {
>            1: self.tag1,
>            2: self.tag2
>            }
>    def dispatch(self, v):
>        try:
>            self.myDict[v]()
>        except KeyError:
>            print "No corresponding dictionary entry!"
>        #
>    def tag1(self):
>        print "one"
>    def tag2(self):
>        print "two"
>
>
> t = Test()
> t.dispatch(1)
> t.dispatch(2)
> t.dispatch(0)
>
> fd = open("pickle.out", "w")
> pickle.dump(t, fd)
> fd.close()
> # EOF
>
> $ python testpickle.py
> one
> two
> No corresponding dictionary entry!
> Traceback (most recent call last):
>  File "ptest.py", line 29, in <module>
>    pickle.dump(t, fd)
>  File "/usr/lib/python2.6/pickle.py", line 1362, in dump
>    Pickler(file, protocol).dump(obj)
>  File "/usr/lib/python2.6/pickle.py", line 224, in dump
>    self.save(obj)
>  File "/usr/lib/python2.6/pickle.py", line 331, in save
>    self.save_reduce(obj=obj, *rv)
>  File "/usr/lib/python2.6/pickle.py", line 419, in save_reduce
>    save(state)
>  File "/usr/lib/python2.6/pickle.py", line 286, in save
>    f(self, obj) # Call unbound method with explicit self
>  File "/usr/lib/python2.6/pickle.py", line 649, in save_dict
>    self._batch_setitems(obj.iteritems())
>  File "/usr/lib/python2.6/pickle.py", line 663, in _batch_setitems
>    save(v)
>  File "/usr/lib/python2.6/pickle.py", line 286, in save
>    f(self, obj) # Call unbound method with explicit self
>  File "/usr/lib/python2.6/pickle.py", line 649, in save_dict
>    self._batch_setitems(obj.iteritems())
>  File "/usr/lib/python2.6/pickle.py", line 663, in _batch_setitems
>    save(v)
>  File "/usr/lib/python2.6/pickle.py", line 306, in save
>    rv = reduce(self.proto)
>  File "/usr/lib/python2.6/copy_reg.py", line 70, in _reduce_ex
>    raise TypeError, "can't pickle %s objects" % base.__name__
> TypeError: can't pickle instancemethod objects
> $
>
>
>    Thanks
>    J^n
>
> --
> http://mail.python.org/mailman/listinfo/python-list

[toc] | [prev] | [next] | [standalone]

#22579

jkn wrote:

>     I'm clearly not understanding the 'can't pickle instancemethod
> objects' error; can someone help me to understand, 

I think classes implemented in C need some extra work to make them 
picklable, and that hasn't been done for instance methods.

> & maybe suggest a
> workaround, (apart from the obvious if ... elif...).

You can implement pickling yourself:

import copy_reg 
import types

def pickle_instancemethod(m):
    return unpickle_instancemethod, (m.im_func.__name__, m.im_self, 
m.im_class) 
 
def unpickle_instancemethod(name, im_self, im_class):
    im_func = getattr(im_class, name)
    return im_func.__get__(im_self, im_class) 
 
copy_reg.pickle(types.MethodType, pickle_instancemethod) 

 
> I'm running Python 2.6 on an embedded system.
> 
> == testpickle.py ==
> import pickle
> 
> class Test(object):
>     def __init__(self):
>         self.myDict = {
>             1: self.tag1,
>             2: self.tag2
>             }
>     def dispatch(self, v):
>         try:
>             self.myDict[v]()
>         except KeyError:
>             print "No corresponding dictionary entry!"
>         #
>     def tag1(self):
>         print "one"
>     def tag2(self):
>         print "two"
> 
> 
> t = Test()
> t.dispatch(1)
> t.dispatch(2)
> t.dispatch(0)
> 
> fd = open("pickle.out", "w")
> pickle.dump(t, fd)
> fd.close()
> # EOF
> 
> $ python testpickle.py
> one
> two
> No corresponding dictionary entry!

> TypeError: can't pickle instancemethod objects
> $

[toc] | [prev] | [next] | [standalone]

#22591

Hi Peter

On Apr 3, 8:54 am, Peter Otten <__pete...@web.de> wrote:
> jkn wrote:
> >     I'm clearly not understanding the 'can't pickle instancemethod
> > objects' error; can someone help me to understand,
>
> I think classes implemented in C need some extra work to make them
> picklable, and that hasn't been done for instance methods.

by 'classes implemented in C', doyou mean new-style classes', or what,
please?


>
> > & maybe suggest a
> > workaround, (apart from the obvious if ... elif...).
>
> You can implement pickling yourself:
>
> [...]

Hmm - interesting, thanks. I'm more trying to understand the issue at
the moment, but it's always nice to learn...

    Thanks
    J^n

[toc] | [prev] | [next] | [standalone]

#22601

jkn wrote:

> Hi Peter
> 
> On Apr 3, 8:54 am, Peter Otten <__pete...@web.de> wrote:
>> jkn wrote:
>> > I'm clearly not understanding the 'can't pickle instancemethod
>> > objects' error; can someone help me to understand,
>>
>> I think classes implemented in C need some extra work to make them
>> picklable, and that hasn't been done for instance methods.
> 
> by 'classes implemented in C', doyou mean new-style classes', or what,
> please?

Given 

>>> class A(object):
...     def __init__(self, name):
...             self.name = name
...     def hello(self):
...             print "Hello,", self.name
... 
>>> a = A("Peter")
>>> hello = a.hello
>>> hello()
Hello, Peter

the object bound to the name 'hello' is an instance of the 'instancemethod' 
type:

>>> type(hello)
<type 'instancemethod'>

That type is implemented in C, see

http://hg.python.org/cpython/file/9599f091faa6/Objects/classobject.c

and doesn't support the pickle protocol while a similar class, 
functools.partial which is also written in C, see

http://hg.python.org/cpython/file/9599f091faa6/Modules/_functoolsmodule.c

does:

>>> from functools import partial
>>> import pickle
>>> def hello(obj):
...     print "Hi,", obj.name
... 
>>> hello2 = partial(hello, a)
>>> hello2()
Hi, Peter
>>> s = pickle.dumps(hello2)
>>> pickle.loads(s)()
Hi, Peter

[toc] | [prev] | [next] | [standalone]

#22596

Peter Otten於 2012年4月3日星期二UTC+8下午3時54分50秒寫道：
> jkn wrote:
> 
> >     I'm clearly not understanding the 'can't pickle instancemethod
> > objects' error; can someone help me to understand, 
> 
> I think classes implemented in C need some extra work to make them 
> picklable, and that hasn't been done for instance methods.
> 
> > & maybe suggest a
> > workaround, (apart from the obvious if ... elif...).
> 
> You can implement pickling yourself:
> 
> import copy_reg 
> import types
> 
> def pickle_instancemethod(m):
>     return unpickle_instancemethod, (m.im_func.__name__, m.im_self, 
> m.im_class) 
>  
> def unpickle_instancemethod(name, im_self, im_class):
>     im_func = getattr(im_class, name)
>     return im_func.__get__(im_self, im_class) 
>  
> copy_reg.pickle(types.MethodType, pickle_instancemethod) 
> 
>  
> > I'm running Python 2.6 on an embedded system.
> > 
> > == testpickle.py ==
> > import pickle
> > 
> > class Test(object):
> >     def __init__(self):
> >         self.myDict = {
> >             1: self.tag1,
> >             2: self.tag2
> >             }
> >     def dispatch(self, v):
> >         try:
> >             self.myDict[v]()
> >         except KeyError:
> >             print "No corresponding dictionary entry!"
> >         #
> >     def tag1(self):
> >         print "one"
> >     def tag2(self):
> >         print "two"
> > 
> > 
> > t = Test()
> > t.dispatch(1)
> > t.dispatch(2)
> > t.dispatch(0)
> > 
> > fd = open("pickle.out", "w")
> > pickle.dump(t, fd)
> > fd.close()
> > # EOF
> > 
> > $ python testpickle.py
> > one
> > two
> > No corresponding dictionary entry!
> 
> > TypeError: can't pickle instancemethod objects
> > $

Save your python files as a package in .pyd or .py  and use exec to get what you want. Of course you can use the data compression package to perform 
serialization operations, but that will increase start up time in loading your objects.

[toc] | [prev] | [next] | [standalone]

#22597

Peter Otten於 2012年4月3日星期二UTC+8下午3時54分50秒寫道：
> jkn wrote:
> 
> >     I'm clearly not understanding the 'can't pickle instancemethod
> > objects' error; can someone help me to understand, 
> 
> I think classes implemented in C need some extra work to make them 
> picklable, and that hasn't been done for instance methods.
> 
> > & maybe suggest a
> > workaround, (apart from the obvious if ... elif...).
> 
> You can implement pickling yourself:
> 
> import copy_reg 
> import types
> 
> def pickle_instancemethod(m):
>     return unpickle_instancemethod, (m.im_func.__name__, m.im_self, 
> m.im_class) 
>  
> def unpickle_instancemethod(name, im_self, im_class):
>     im_func = getattr(im_class, name)
>     return im_func.__get__(im_self, im_class) 
>  
> copy_reg.pickle(types.MethodType, pickle_instancemethod) 
> 
>  
> > I'm running Python 2.6 on an embedded system.
> > 
> > == testpickle.py ==
> > import pickle
> > 
> > class Test(object):
> >     def __init__(self):
> >         self.myDict = {
> >             1: self.tag1,
> >             2: self.tag2
> >             }
> >     def dispatch(self, v):
> >         try:
> >             self.myDict[v]()
> >         except KeyError:
> >             print "No corresponding dictionary entry!"
> >         #
> >     def tag1(self):
> >         print "one"
> >     def tag2(self):
> >         print "two"
> > 
> > 
> > t = Test()
> > t.dispatch(1)
> > t.dispatch(2)
> > t.dispatch(0)
> > 
> > fd = open("pickle.out", "w")
> > pickle.dump(t, fd)
> > fd.close()
> > # EOF
> > 
> > $ python testpickle.py
> > one
> > two
> > No corresponding dictionary entry!
> 
> > TypeError: can't pickle instancemethod objects
> > $

Save your python files as a package in .pyd or .py  and use exec to get what you want. Of course you can use the data compression package to perform 
serialization operations, but that will increase start up time in loading your objects.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web