Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #18164 > unrolled thread

pickling instances of metaclass generated classes

Back to article view | Back to comp.lang.python

Contents

  pickling instances of metaclass generated classes lars van gemerden <lars@rational-it.com> - 2011-12-29 01:55 -0800
    Re: pickling instances of metaclass generated classes Robert Kern <robert.kern@gmail.com> - 2011-12-29 11:08 +0000
    Re: pickling instances of metaclass generated classes Ian Kelly <ian.g.kelly@gmail.com> - 2011-12-29 12:55 -0700
      Re: pickling instances of metaclass generated classes lars van gemerden <lars@rational-it.com> - 2011-12-30 01:50 -0800
        Re: pickling instances of metaclass generated classes Peter Otten <__peter__@web.de> - 2011-12-30 12:41 +0100
        Re: pickling instances of metaclass generated classes Peter Otten <__peter__@web.de> - 2012-01-01 13:51 +0100
      Re: pickling instances of metaclass generated classes lars van gemerden <lars@rational-it.com> - 2011-12-30 03:16 -0800
        Re: pickling instances of metaclass generated classes lars van gemerden <lars@rational-it.com> - 2011-12-30 07:56 -0800
          Re: pickling instances of metaclass generated classes lars van gemerden <lars@rational-it.com> - 2011-12-30 08:51 -0800
            Re: pickling instances of metaclass generated classes Ian Kelly <ian.g.kelly@gmail.com> - 2011-12-30 10:09 -0700
    Re: pickling instances of metaclass generated classes lars van gemerden <lars@rational-it.com> - 2012-01-03 03:43 -0800

#18164 — pickling instances of metaclass generated classes

Hello,

Can someone help me with the following:

I am using metaclasses to make classes and these classes to make
instances. Now I want to use multiprocessing, which needs to pickle
these instances.

Pickle cannot find the class definitions of the instances. I am trying
to add a line to the __new__ of the metaclass to add the new class
under the right name in the right module/place, so pickle can find
it.

Is this the right approach? Can anyone explain to me where/how to add
these classes for pickle to find and maybe why?

Thanks in advance,

Lars

[toc] | [next] | [standalone]

#18165

On 12/29/11 9:55 AM, lars van gemerden wrote:
> Hello,
>
> Can someone help me with the following:
>
> I am using metaclasses to make classes and these classes to make
> instances. Now I want to use multiprocessing, which needs to pickle
> these instances.
>
> Pickle cannot find the class definitions of the instances. I am trying
> to add a line to the __new__ of the metaclass to add the new class
> under the right name in the right module/place, so pickle can find
> it.
>
> Is this the right approach? Can anyone explain to me where/how to add
> these classes for pickle to find and maybe why?

Can you post some code (preferably pared down to a minimal example that fails)? 
I'm not really clear on what you are doing. I would expect that a class defined 
by a class statement would usually work fine unless if the metaclass is doing 
something particularly weird to it.

In any case, you can probably just explicitly register a reduction function for 
each type using copy_reg.pickle():

http://docs.python.org/library/copy_reg

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco

[toc] | [prev] | [next] | [standalone]

#18190

On Thu, Dec 29, 2011 at 2:55 AM, lars van gemerden <lars@rational-it.com> wrote:
> Hello,
>
> Can someone help me with the following:
>
> I am using metaclasses to make classes and these classes to make
> instances. Now I want to use multiprocessing, which needs to pickle
> these instances.
>
> Pickle cannot find the class definitions of the instances. I am trying
> to add a line to the __new__ of the metaclass to add the new class
> under the right name in the right module/place, so pickle can find
> it.
>
> Is this the right approach? Can anyone explain to me where/how to add
> these classes for pickle to find and maybe why?

It sounds like you're trying to do something like this?

>>> class MetaClass(type):
...     pass
...
>>> instance = MetaClass('<Anonymous>', (object,), {})()
>>> instance
<__main__.<Anonymous> object at 0x00CC00F0>
>>> import pickle
>>> pickle.dumps(instance)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "c:\python27\lib\pickle.py", line 1374, in dumps
    Pickler(file, protocol).dump(obj)
  File "c:\python27\lib\pickle.py", line 224, in dump
    self.save(obj)
  File "c:\python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "c:\python27\lib\pickle.py", line 401, in save_reduce
    save(args)
  File "c:\python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "c:\python27\lib\pickle.py", line 562, in save_tuple
    save(element)
  File "c:\python27\lib\pickle.py", line 295, in save
    self.save_global(obj)
  File "c:\python27\lib\pickle.py", line 748, in save_global
    (obj, module, name))
pickle.PicklingError: Can't pickle <class '__main__.<Anonymous>'>:
it's not found as __main__.<Anonymous>


Yeah, pickle's not going to work with anonymous classes.  As you
suggest, you could dynamically add the classes to the module namespace
so that pickle.dumps will find them, but bear in mind that they will
also have to exist when calling pickle.loads, so you will need to be
able to reconstruct the same anonymous classes before unpickling later
on.

Cheers,
Ian

[toc] | [prev] | [next] | [standalone]

#18199

On Dec 29, 8:55 pm, Ian Kelly <ian.g.ke...@gmail.com> wrote:
> On Thu, Dec 29, 2011 at 2:55 AM, lars van gemerden <l...@rational-it.com> wrote:
>
> > Hello,
>
> > Can someone help me with the following:
>
> > I am using metaclasses to make classes and these classes to make
> > instances. Now I want to use multiprocessing, which needs to pickle
> > these instances.
>
> > Pickle cannot find the class definitions of the instances. I am trying
> > to add a line to the __new__ of the metaclass to add the new class
> > under the right name in the right module/place, so pickle can find
> > it.
>
> > Is this the right approach? Can anyone explain to me where/how to add
> > these classes for pickle to find and maybe why?
>
> It sounds like you're trying to do something like this?
>
> >>> class MetaClass(type):
>
> ...     pass
> ...>>> instance = MetaClass('<Anonymous>', (object,), {})()
> >>> instance
>
> <__main__.<Anonymous> object at 0x00CC00F0>>>> import pickle
> >>> pickle.dumps(instance)
>
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "c:\python27\lib\pickle.py", line 1374, in dumps
>     Pickler(file, protocol).dump(obj)
>   File "c:\python27\lib\pickle.py", line 224, in dump
>     self.save(obj)
>   File "c:\python27\lib\pickle.py", line 331, in save
>     self.save_reduce(obj=obj, *rv)
>   File "c:\python27\lib\pickle.py", line 401, in save_reduce
>     save(args)
>   File "c:\python27\lib\pickle.py", line 286, in save
>     f(self, obj) # Call unbound method with explicit self
>   File "c:\python27\lib\pickle.py", line 562, in save_tuple
>     save(element)
>   File "c:\python27\lib\pickle.py", line 295, in save
>     self.save_global(obj)
>   File "c:\python27\lib\pickle.py", line 748, in save_global
>     (obj, module, name))
> pickle.PicklingError: Can't pickle <class '__main__.<Anonymous>'>:
> it's not found as __main__.<Anonymous>
>
> Yeah, pickle's not going to work with anonymous classes.  As you
> suggest, you could dynamically add the classes to the module namespace
> so that pickle.dumps will find them, but bear in mind that they will
> also have to exist when calling pickle.loads, so you will need to be
> able to reconstruct the same anonymous classes before unpickling later
> on.
>
> Cheers,
> Ian

Thank you Ian for the minimal example. This is almost the case i was
trying to discribe, however the classes will be named at runtime and
the class definitions will be persisted in a database.

Can you help me with how to add the classes to the correct namespace?
The application spans multiple modules (or compared to the example,
the metaclass definition will be in another module then one where the
class and instance will be generated).

Cheers, Lars

[toc] | [prev] | [next] | [standalone]

#18205

lars van gemerden wrote:

> On Dec 29, 8:55 pm, Ian Kelly <ian.g.ke...@gmail.com> wrote:
>> On Thu, Dec 29, 2011 at 2:55 AM, lars van gemerden <l...@rational-it.com>
>> wrote:
>>
>> > Hello,
>>
>> > Can someone help me with the following:
>>
>> > I am using metaclasses to make classes and these classes to make
>> > instances. Now I want to use multiprocessing, which needs to pickle
>> > these instances.
>>
>> > Pickle cannot find the class definitions of the instances. I am trying
>> > to add a line to the __new__ of the metaclass to add the new class
>> > under the right name in the right module/place, so pickle can find
>> > it.
>>
>> > Is this the right approach? Can anyone explain to me where/how to add
>> > these classes for pickle to find and maybe why?
>>
>> It sounds like you're trying to do something like this?
>>
>> >>> class MetaClass(type):
>>
>> ...     pass
>> ...>>> instance = MetaClass('<Anonymous>', (object,), {})()
>> >>> instance
>>
>> <__main__.<Anonymous> object at 0x00CC00F0>>>> import pickle
>> >>> pickle.dumps(instance)
>>
>> Traceback (most recent call last):
>> File "<stdin>", line 1, in <module>
>> File "c:\python27\lib\pickle.py", line 1374, in dumps
>> Pickler(file, protocol).dump(obj)
>> File "c:\python27\lib\pickle.py", line 224, in dump
>> self.save(obj)
>> File "c:\python27\lib\pickle.py", line 331, in save
>> self.save_reduce(obj=obj, *rv)
>> File "c:\python27\lib\pickle.py", line 401, in save_reduce
>> save(args)
>> File "c:\python27\lib\pickle.py", line 286, in save
>> f(self, obj) # Call unbound method with explicit self
>> File "c:\python27\lib\pickle.py", line 562, in save_tuple
>> save(element)
>> File "c:\python27\lib\pickle.py", line 295, in save
>> self.save_global(obj)
>> File "c:\python27\lib\pickle.py", line 748, in save_global
>> (obj, module, name))
>> pickle.PicklingError: Can't pickle <class '__main__.<Anonymous>'>:
>> it's not found as __main__.<Anonymous>
>>
>> Yeah, pickle's not going to work with anonymous classes.  As you
>> suggest, you could dynamically add the classes to the module namespace
>> so that pickle.dumps will find them, but bear in mind that they will
>> also have to exist when calling pickle.loads, so you will need to be
>> able to reconstruct the same anonymous classes before unpickling later
>> on.
>>
>> Cheers,
>> Ian
> 
> Thank you Ian for the minimal example. This is almost the case i was
> trying to discribe, however the classes will be named at runtime and
> the class definitions will be persisted in a database.
> 
> Can you help me with how to add the classes to the correct namespace?
> The application spans multiple modules (or compared to the example,
> the metaclass definition will be in another module then one where the
> class and instance will be generated).

If the metaclass is global in whatever module you don't need to bother about 
that. The problem is that you cannot access the classes (metaclass 
instances) under a dotted name. One workaround is a pseudo-module that does 
whatever is needed to recreate the class:

import pickle
import sys

class MetaClass(type):
    pass

class M(object):
    def __init__(self, module):
        self.__module = module
    def __getattr__(self, name):
            print "creating class", name
            class_ = MetaClass(name, (), {"__module__": self.__module})
            setattr(self, name, class_)
            return class_

sys.modules["m"] = M("m")
import m
c = m.x
s = pickle.dumps(c)
print repr(s)
d = pickle.loads(s)

assert c is d

sys.modules["m"] = M("m")
e = pickle.loads(s)

assert c is not e

The official way is probably what Robert mentioned, via the copy_reg module, 
but I didn't get it to work.

[toc] | [prev] | [next] | [standalone]

#18294

lars van gemerden wrote:

>> import pickle
>> import sys
>>
>> class MetaClass(type):
>>     pass
>>
>> class M(object):
>>     def __init__(self, module):
>>         self.__module = module
>>     def __getattr__(self, name):
>>             print "creating class", name
>>             class_ = MetaClass(name, (), {"__module__": self.__module})
>>             setattr(self, name, class_)
>>             return class_
>>
>> sys.modules["m"] = M("m")
>> import m
>> c = m.x
>> s = pickle.dumps(c)
>> print repr(s)
>> d = pickle.loads(s)
>>
>> assert c is d
>>
>> sys.modules["m"] = M("m")
>> e = pickle.loads(s)
>>
>> assert c is not e
>>
>> The official way is probably what Robert mentioned, via the copy_reg
>> module, but I didn't get it to work.
> 
> I will look further into this. does "sys.modules["m"] = M("m")" create
> a new module?

Assigning to sys.modules[modulename] can put arbitrary objects into the 
module cache, in this case an M instance. To drive the point home:

>>> import sys
>>> sys.modules["x"] = 42
>>> import x
>>> x
42
>>> sys.modules["x"] = "spam"
>>> import x
>>> x
'spam'

> Cheers, Lars
> 
> PS: I get an error when posting this to the usenet group

Sorry, that seems to happen when I post via gmane and don't manually clear 
the follow-up that my newsreader helpfully (knode) inserts. I've not yet 
found a permanent fix, but if that was the problem you should be able to 
answer this post.

[toc] | [prev] | [next] | [standalone]

#18203

On Dec 29, 8:55 pm, Ian Kelly <ian.g.ke...@gmail.com> wrote:
> On Thu, Dec 29, 2011 at 2:55 AM, lars van gemerden <l...@rational-it.com> wrote:
>
> > Hello,
>
> > Can someone help me with the following:
>
> > I am using metaclasses to make classes and these classes to make
> > instances. Now I want to use multiprocessing, which needs to pickle
> > these instances.
>
> > Pickle cannot find the class definitions of the instances. I am trying
> > to add a line to the __new__ of the metaclass to add the new class
> > under the right name in the right module/place, so pickle can find
> > it.
>
> > Is this the right approach? Can anyone explain to me where/how to add
> > these classes for pickle to find and maybe why?
>
> It sounds like you're trying to do something like this?
>
> >>> class MetaClass(type):
>
> ...     pass
> ...>>> instance = MetaClass('<Anonymous>', (object,), {})()
> >>> instance
>
> <__main__.<Anonymous> object at 0x00CC00F0>>>> import pickle
> >>> pickle.dumps(instance)
>
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "c:\python27\lib\pickle.py", line 1374, in dumps
>     Pickler(file, protocol).dump(obj)
>   File "c:\python27\lib\pickle.py", line 224, in dump
>     self.save(obj)
>   File "c:\python27\lib\pickle.py", line 331, in save
>     self.save_reduce(obj=obj, *rv)
>   File "c:\python27\lib\pickle.py", line 401, in save_reduce
>     save(args)
>   File "c:\python27\lib\pickle.py", line 286, in save
>     f(self, obj) # Call unbound method with explicit self
>   File "c:\python27\lib\pickle.py", line 562, in save_tuple
>     save(element)
>   File "c:\python27\lib\pickle.py", line 295, in save
>     self.save_global(obj)
>   File "c:\python27\lib\pickle.py", line 748, in save_global
>     (obj, module, name))
> pickle.PicklingError: Can't pickle <class '__main__.<Anonymous>'>:
> it's not found as __main__.<Anonymous>
>
> Yeah, pickle's not going to work with anonymous classes.  As you
> suggest, you could dynamically add the classes to the module namespace
> so that pickle.dumps will find them, but bear in mind that they will
> also have to exist when calling pickle.loads, so you will need to be
> able to reconstruct the same anonymous classes before unpickling later
> on.
>
> Cheers,
> Ian

Ian also wrote:

'''
Actually, I was wrong, you probably don't need to do that.  I suggest
going with Robert Kern's suggestion to either register the class with
the copy_reg module, or (perhaps better since it won't leak
registrations) implement a __reduce__ method on the class.  For
example, this seems to work:

>>> def reconstructor(*metaclass_args):
...     cls = MetaClass.build_class(*metaclass_args)
...     self = cls.__new__(cls)
...     return self
...
>>> class MetaClass(type):
...     @classmethod
...     def build_class(mcs, arg1, arg2, arg3):
...         # Do something useful with the args...
...         class _AnonymousClass(object):
...             __metaclass__ = mcs
...             def __reduce__(self):
...                 return (reconstructor, ('foo', 'bar', 'baz'),
self.__dict__)
...         return _AnonymousClass
...
>>> instance = MetaClass.build_class('foo', 'bar', 'baz')()
>>> instance
<__main__._AnonymousClass object at 0x011DB410>
>>> instance.banana = 42
>>> import pickle
>>> s = pickle.dumps(instance)
>>> s
"c__main__\nreconstructor
\np0\n(S'foo'\np1\nS'bar'\np2\nS'baz'\np3\ntp4\nRp5\n(dp6\nS'banana'\np7\nI42\nsb."
>>> inst2 = pickle.loads(s)
>>> inst2
<__main__._AnonymousClass object at 0x011DBE90>
>>> inst2.banana
42
>>> inst2.__class__ is instance.__class__
False

Cheers,
Ian

'''

[toc] | [prev] | [next] | [standalone]

#18214

On Dec 30, 12:16 pm, lars van gemerden <l...@rational-it.com> wrote:
> On Dec 29, 8:55 pm, Ian Kelly <ian.g.ke...@gmail.com> wrote:
>
>
>
>
>
>
>
>
>
> > On Thu, Dec 29, 2011 at 2:55 AM, lars van gemerden <l...@rational-it.com> wrote:
>
> > > Hello,
>
> > > Can someone help me with the following:
>
> > > I am using metaclasses to make classes and these classes to make
> > > instances. Now I want to use multiprocessing, which needs to pickle
> > > these instances.
>
> > > Pickle cannot find the class definitions of the instances. I am trying
> > > to add a line to the __new__ of the metaclass to add the new class
> > > under the right name in the right module/place, so pickle can find
> > > it.
>
> > > Is this the right approach? Can anyone explain to me where/how to add
> > > these classes for pickle to find and maybe why?
>
> > It sounds like you're trying to do something like this?
>
> > >>> class MetaClass(type):
>
> > ...     pass
> > ...>>> instance = MetaClass('<Anonymous>', (object,), {})()
> > >>> instance
>
> > <__main__.<Anonymous> object at 0x00CC00F0>>>> import pickle
> > >>> pickle.dumps(instance)
>
> > Traceback (most recent call last):
> >   File "<stdin>", line 1, in <module>
> >   File "c:\python27\lib\pickle.py", line 1374, in dumps
> >     Pickler(file, protocol).dump(obj)
> >   File "c:\python27\lib\pickle.py", line 224, in dump
> >     self.save(obj)
> >   File "c:\python27\lib\pickle.py", line 331, in save
> >     self.save_reduce(obj=obj, *rv)
> >   File "c:\python27\lib\pickle.py", line 401, in save_reduce
> >     save(args)
> >   File "c:\python27\lib\pickle.py", line 286, in save
> >     f(self, obj) # Call unbound method with explicit self
> >   File "c:\python27\lib\pickle.py", line 562, in save_tuple
> >     save(element)
> >   File "c:\python27\lib\pickle.py", line 295, in save
> >     self.save_global(obj)
> >   File "c:\python27\lib\pickle.py", line 748, in save_global
> >     (obj, module, name))
> > pickle.PicklingError: Can't pickle <class '__main__.<Anonymous>'>:
> > it's not found as __main__.<Anonymous>
>
> > Yeah, pickle's not going to work with anonymous classes.  As you
> > suggest, you could dynamically add the classes to the module namespace
> > so that pickle.dumps will find them, but bear in mind that they will
> > also have to exist when calling pickle.loads, so you will need to be
> > able to reconstruct the same anonymous classes before unpickling later
> > on.
>
> > Cheers,
> > Ian
> Ian also wrote:
>
> '''
> Actually, I was wrong, you probably don't need to do that.  I suggest
> going with Robert Kern's suggestion to either register the class with
> the copy_reg module, or (perhaps better since it won't leak
> registrations) implement a __reduce__ method on the class.  For
> example, this seems to work:
>
> >>> def reconstructor(*metaclass_args):
>
> ...     cls = MetaClass.build_class(*metaclass_args)
> ...     self = cls.__new__(cls)
> ...     return self
> ...>>> class MetaClass(type):
>
> ...     @classmethod
> ...     def build_class(mcs, arg1, arg2, arg3):
> ...         # Do something useful with the args...
> ...         class _AnonymousClass(object):
> ...             __metaclass__ = mcs
> ...             def __reduce__(self):
> ...                 return (reconstructor, ('foo', 'bar', 'baz'),
> self.__dict__)
> ...         return _AnonymousClass
> ...>>> instance = MetaClass.build_class('foo', 'bar', 'baz')()
> >>> instance
>
> <__main__._AnonymousClass object at 0x011DB410>>>> instance.banana = 42
> >>> import pickle
> >>> s = pickle.dumps(instance)
> >>> s
>
> "c__main__\nreconstructor
> \np0\n(S'foo'\np1\nS'bar'\np2\nS'baz'\np3\ntp4\nRp5\n(dp6\nS'banana'\np7\nI 42\nsb.">>> inst2 = pickle.loads(s)
> >>> inst2
>
> <__main__._AnonymousClass object at 0x011DBE90>>>> inst2.banana
> 42
> >>> inst2.__class__ is instance.__class__
>
> False
>
> Cheers,
> Ian
>
> '''

Interesting, though I cannot say I completely understand this solution
(looked up __reduce__, but still). I am trying to adapt this example
to a situation where the metaclass generated classes are named at
runtime (not anonymous), but cannot figure it out.

Cheers, Lars

[toc] | [prev] | [next] | [standalone]

#18215

On Dec 30, 4:56 pm, lars van gemerden <l...@rational-it.com> wrote:
> On Dec 30, 12:16 pm, lars van gemerden <l...@rational-it.com> wrote:
>
>
>
>
>
>
>
>
>
> > On Dec 29, 8:55 pm, Ian Kelly <ian.g.ke...@gmail.com> wrote:
>
> > > On Thu, Dec 29, 2011 at 2:55 AM, lars van gemerden <l...@rational-it.com> wrote:
>
> > > > Hello,
>
> > > > Can someone help me with the following:
>
> > > > I am using metaclasses to make classes and these classes to make
> > > > instances. Now I want to use multiprocessing, which needs to pickle
> > > > these instances.
>
> > > > Pickle cannot find the class definitions of the instances. I am trying
> > > > to add a line to the __new__ of the metaclass to add the new class
> > > > under the right name in the right module/place, so pickle can find
> > > > it.
>
> > > > Is this the right approach? Can anyone explain to me where/how to add
> > > > these classes for pickle to find and maybe why?
>
> > > It sounds like you're trying to do something like this?
>
> > > >>> class MetaClass(type):
>
> > > ...     pass
> > > ...>>> instance = MetaClass('<Anonymous>', (object,), {})()
> > > >>> instance
>
> > > <__main__.<Anonymous> object at 0x00CC00F0>>>> import pickle
> > > >>> pickle.dumps(instance)
>
> > > Traceback (most recent call last):
> > >   File "<stdin>", line 1, in <module>
> > >   File "c:\python27\lib\pickle.py", line 1374, in dumps
> > >     Pickler(file, protocol).dump(obj)
> > >   File "c:\python27\lib\pickle.py", line 224, in dump
> > >     self.save(obj)
> > >   File "c:\python27\lib\pickle.py", line 331, in save
> > >     self.save_reduce(obj=obj, *rv)
> > >   File "c:\python27\lib\pickle.py", line 401, in save_reduce
> > >     save(args)
> > >   File "c:\python27\lib\pickle.py", line 286, in save
> > >     f(self, obj) # Call unbound method with explicit self
> > >   File "c:\python27\lib\pickle.py", line 562, in save_tuple
> > >     save(element)
> > >   File "c:\python27\lib\pickle.py", line 295, in save
> > >     self.save_global(obj)
> > >   File "c:\python27\lib\pickle.py", line 748, in save_global
> > >     (obj, module, name))
> > > pickle.PicklingError: Can't pickle <class '__main__.<Anonymous>'>:
> > > it's not found as __main__.<Anonymous>
>
> > > Yeah, pickle's not going to work with anonymous classes.  As you
> > > suggest, you could dynamically add the classes to the module namespace
> > > so that pickle.dumps will find them, but bear in mind that they will
> > > also have to exist when calling pickle.loads, so you will need to be
> > > able to reconstruct the same anonymous classes before unpickling later
> > > on.
>
> > > Cheers,
> > > Ian
> > Ian also wrote:
>
> > '''
> > Actually, I was wrong, you probably don't need to do that.  I suggest
> > going with Robert Kern's suggestion to either register the class with
> > the copy_reg module, or (perhaps better since it won't leak
> > registrations) implement a __reduce__ method on the class.  For
> > example, this seems to work:
>
> > >>> def reconstructor(*metaclass_args):
>
> > ...     cls = MetaClass.build_class(*metaclass_args)
> > ...     self = cls.__new__(cls)
> > ...     return self
> > ...>>> class MetaClass(type):
>
> > ...     @classmethod
> > ...     def build_class(mcs, arg1, arg2, arg3):
> > ...         # Do something useful with the args...
> > ...         class _AnonymousClass(object):
> > ...             __metaclass__ = mcs
> > ...             def __reduce__(self):
> > ...                 return (reconstructor, ('foo', 'bar', 'baz'),
> > self.__dict__)
> > ...         return _AnonymousClass
> > ...>>> instance = MetaClass.build_class('foo', 'bar', 'baz')()
> > >>> instance
>
> > <__main__._AnonymousClass object at 0x011DB410>>>> instance.banana = 42
> > >>> import pickle
> > >>> s = pickle.dumps(instance)
> > >>> s
>
> > "c__main__\nreconstructor
> > \np0\n(S'foo'\np1\nS'bar'\np2\nS'baz'\np3\ntp4\nRp5\n(dp6\nS'banana'\np7\nI 42\nsb.">>> inst2 = pickle.loads(s)
> > >>> inst2
>
> > <__main__._AnonymousClass object at 0x011DBE90>>>> inst2.banana
> > 42
> > >>> inst2.__class__ is instance.__class__
>
> > False
>
> > Cheers,
> > Ian
>
> > '''
>
> Interesting, though I cannot say I completely understand this solution
> (looked up __reduce__, but still). I am trying to adapt this example
> to a situation where the metaclass generated classes are named at
> runtime (not anonymous), but cannot figure it out.
>
> Cheers, Lars

Found a way to name the classes:

def reconstructor(*metaclass_args):
    cls = MetaClass2.build_class(*metaclass_args)
    self = cls.__new__(cls)
    return self

class MetaClass(type):
    @classmethod
    def build_class(mcs, name, arg1, arg2, arg3):
        return mcs(name, (object,), {"__reduce__": lambda e:
(reconstructor2, (name, arg1, arg2, arg3), e.__dict__)})

I still wonder whether it might be easier to add the class to the
namespace. Can anyone help me with that?

Regards, Lars

[toc] | [prev] | [next] | [standalone]

#18217

On Fri, Dec 30, 2011 at 9:51 AM, lars van gemerden <lars@rational-it.com> wrote:
> I still wonder whether it might be easier to add the class to the
> namespace. Can anyone help me with that?

from mypackage import mymodule

setattr(mymodule, myclass.__name__, myclass)

[toc] | [prev] | [next] | [standalone]

#18394

On Dec 29 2011, 10:55 am, lars van gemerden <l...@rational-it.com>
wrote:
> Hello,
>
> Can someone help me with the following:
>
> I am using metaclasses to make classes and these classes to make
> instances. Now I want to use multiprocessing, which needs to pickle
> these instances.
>
> Pickle cannot find the class definitions of the instances. I am trying
> to add a line to the __new__ of the metaclass to add the new class
> under the right name in the right module/place, so pickle can find
> it.
>
> Is this the right approach? Can anyone explain to me where/how to add
> these classes for pickle to find and maybe why?
>
> Thanks in advance,
>
> Lars

Ok,

After reading all posts (thanks a lot), I am considering to use the
following base metaclass for all metaclasses that must lead to
pickleable instances (not pickleable classes):


import sys

class Meta(type):
    def __new__(mcls, name, bases, attrs):
        cls = type.__new__(mcls, name, bases, attrs)
        setattr(sys.modules[__name__], name, cls)
        return cls


if __name__ == '__main__':
    instance = Meta("Klass", (str,),{})("apple")
    s = pickle.dumps(instance)
    delattr(sys.modules[__name__], "Klass")
    Meta("Klass", (str,),{})
    inst = pickle.loads(s)
    print instance
    print inst
    print type(instance) is type(inst)

Can anyone see any drawbacks to this approach? I've also tested the
case where the Meta metaclass is defined in another module.

Cheers, Lars

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web