Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #21437 > unrolled thread

stackoverflow question

Back to article view | Back to comp.lang.python

Contents

  stackoverflow question Ethan Furman <ethan@stoneleaf.us> - 2012-03-09 14:10 -0800
    Re: stackoverflow question Owen Jacobson <angrybaldguy@gmail.com> - 2012-03-10 15:03 -0500
      Re: stackoverflow question Ethan Furman <ethan@stoneleaf.us> - 2012-03-10 14:21 -0800
        Re: stackoverflow question Owen Jacobson <angrybaldguy@gmail.com> - 2012-03-11 00:48 -0500

#21437 — stackoverflow question

Hey all!

I posted a question/answer on SO earlier, but there seems to be some 
confusion around either the question or the answer (judging from the 
comments).

http://stackoverflow.com/q/9638921/208880

If anyone here is willing to take a look at it and let me know if I did 
not write it well, I would appreciate the feedback.


Here's the question text:
------------------------
I'm writing a metaclass to do some cool stuff, and part of its 
processing is to check that certain attributes exist when the class is 
created.  Some of these are mutable, and would normally be set in 
`__init__`, but since `__init__` isn't run until the instance is created 
the metaclass won't know that the attribute *will* be created, and 
raises an error.  I could do something like:

     class Test(meta=Meta):
         mutable = None
         def __init__(self):
             self.mutable = list()

But that isn't very elegant, and also violates DRY.

What I need is some way to have:

     class Test(metaclass=Meta):
         mutable = list()

     t1 = Test()
     t2 = Test()
     t1.mutable.append('one')
     t2.mutable.append('two')
     t1.mutable  # prints ['one']
     t2.mutable  # prints ['two']

Any ideas on how this can be accomplished?

Also, the metaclass doing the checking doesn't care what type of object 
the attribute is, only that it is there.
---------------------------

and the answer text:
-------------------
There are a couple ways to do this:

   1. Have the metaclass check all the attributes, and if they are one 
of the mutables (`list`, `dict`, `set`, etc.) replace the attribute with 
a descriptor that will activate on first access and update the instance 
with a fresh copy of the mutable.

   2. Provide the descriptor from (1) as a decorator to be used when 
writing the class.

I prefer (2) is it gives complete control to the class author, and 
simplifies those cases where the class-level mutable attribute *should* 
be shared amongst all the instances.

Here's the decorator-descriptor:

     class ReplaceMutable:
         def __init__(self, func):
             self.func = func
         def __call__(self):
             return self
         def __get__(self, instance, owner):
             if instance is None:
                 return self
             result = self.func()
             setattr(instance, self.func.__name__, result)
             return result

and the test class:

     class Test:
         @ReplaceMutable
         def mutable():
             return list()

     t1 = Test()
     t2 = Test()
     t1.mutable.append('one')
     t2.mutable.append('two')
     print(t1.mutable)
     print(t2.mutable)

How it works:

Just like `property`, `ReplaceMutable` is a descriptor object with the 
same name as the attribute it is replacing.  Unlike `property`, it does 
not define `__set__` nor `__delete__`, so when code tries to rebind the 
name (`mutable` in the test above) in the instance Python will allow it 
to do so.  This is the same idea behind caching descriptors.

`ReplaceMutable` is decorating a function (with the name of the desired 
attribute) that simply returns whatever the instance level attribute 
should be initialized with (an empty `list` in the example above).  So 
the first time the attribute is looked up on an instance it will not be 
found in the instance dictionary and Python will activate the 
descriptor; the descriptor then calls the function to retrieve the 
initial object/data/whatever, stores it in the instance, and then 
returns it.  The next time that attribute is accessed *on that instance* 
it will be in the instance dictionary, and that is what will be used.
-----------------------------------

Thanks,
~Ethan~

[toc] | [next] | [standalone]

#21465

On 2012-03-09 22:10:18 +0000, Ethan Furman said:

> Hey all!
> 
> I posted a question/answer on SO earlier, but there seems to be some 
> confusion around either the question or the answer (judging from the 
> comments).
> 
> http://stackoverflow.com/q/9638921/208880
> 
> If anyone here is willing to take a look at it and let me know if I did 
> not write it well, I would appreciate the feedback.
> 
> 
> Here's the question text:
> ------------------------
> I'm writing a metaclass to do some cool stuff, and part of its 
> processing is to check that certain attributes exist when the class is 
> created.  Some of these are mutable, and would normally be set in 
> `__init__`, but since `__init__` isn't run until the instance is 
> created the metaclass won't know that the attribute *will* be created, 
> and raises an error.  I could do something like:
> 
>      class Test(meta=Meta):
>          mutable = None
>          def __init__(self):
>              self.mutable = list()
> 
> But that isn't very elegant, and also violates DRY.
> 
> What I need is some way to have:
> 
>      class Test(metaclass=Meta):
>          mutable = list()
> 
>      t1 = Test()
>      t2 = Test()
>      t1.mutable.append('one')
>      t2.mutable.append('two')
>      t1.mutable  # prints ['one']
>      t2.mutable  # prints ['two']
> 
> Any ideas on how this can be accomplished?
> 
> Also, the metaclass doing the checking doesn't care what type of object 
> the attribute is, only that it is there.
> ---------------------------

Why check what you can ensure? The __init__ function your metaclass 
passes to type() doesn't have to be the __init__ method your metaclass 
received. Consider the following:

>>> import functools as f
>>> 
>>> def make_init(real_init):
>>>     """Define an __init__ method that ensures ``self.mutable`` is set. If the
>>>     passed ``real_init`` function later replaces ``self.mutable``, that value
>>>     is preserved; otherwise, ``self.mutable`` is set to a new, empty list.
>>> 
>>>     Arguments to the generated ``__init__`` method are passed to the original
>>>     ``real_init`` unchanged.
>>>     """
>>>     def __init__(self, *args, **kwargs):
>>>         self.mutable = list()
>>>         if real_init is not None:
>>>             return real_init(self, *args, **kwargs)
>>>     if real_init is not None:
>>>         f.update_wrapper(__init__, real_init)
>>>     return __init__
>>> 
>>> class Meta(type):
>>>     def __new__(meta, name, parents, attributes):
>>>         attributes['__init__'] = make_init(attributes.get('__init__', None))
>>>         return type.__new__(Meta, name, parents, attributes)
>>> 
>>> class C(object):
>>>     __metaclass__ = Meta
>>> 
>>> a, b = C(), C()
>>> 
>>> a.mutable.append(3)
>>> b.mutable.append(5)
>>> 
>>> a.mutable
[3]
>>> b.mutable
[5]

All instances of classes whose metaclass is Meta will, guaranteed, have 
an instance field named 'mutable'. Its value is a list created at 
instance creation time, unless the instance's __init__ provides a 
different value.

What've I missed?

-o

[toc] | [prev] | [next] | [standalone]

#21472

Owen Jacobson wrote:
> On 2012-03-09 22:10:18 +0000, Ethan Furman said:
> 
>> Hey all!
>>
>> I posted a question/answer on SO earlier, but there seems to be some 
>> confusion around either the question or the answer (judging from the 
>> comments).
>>
>> http://stackoverflow.com/q/9638921/208880
>>
>> If anyone here is willing to take a look at it and let me know if I 
>> did not write it well, I would appreciate the feedback.
>>
>>
>> Here's the question text:
>> ------------------------
>> I'm writing a metaclass to do some cool stuff, and part of its 
>> processing is to check that certain attributes exist when the class is 
>> created.  Some of these are mutable, and would normally be set in 
>> `__init__`, but since `__init__` isn't run until the instance is 
>> created the metaclass won't know that the attribute *will* be created, 
>> and raises an error.  I could do something like:
>>
>>      class Test(meta=Meta):
>>          mutable = None
>>          def __init__(self):
>>              self.mutable = list()
>>
>> But that isn't very elegant, and also violates DRY.
>>
>> What I need is some way to have:
>>
>>      class Test(metaclass=Meta):
>>          mutable = list()
>>
>>      t1 = Test()
>>      t2 = Test()
>>      t1.mutable.append('one')
>>      t2.mutable.append('two')
>>      t1.mutable  # prints ['one']
>>      t2.mutable  # prints ['two']
>>
>> Any ideas on how this can be accomplished?
>>
>> Also, the metaclass doing the checking doesn't care what type of 
>> object the attribute is, only that it is there.
>> ---------------------------
> 
> Why check what you can ensure? The __init__ function your metaclass 
> passes to type() doesn't have to be the __init__ method your metaclass 
> received. Consider the following:
> 
>>>> import functools as f
>>>>
>>>> def make_init(real_init):
>>>>     """Define an __init__ method that ensures ``self.mutable`` is 
>>>> set. If the
>>>>     passed ``real_init`` function later replaces ``self.mutable``, 
>>>> that value
>>>>     is preserved; otherwise, ``self.mutable`` is set to a new, empty 
>>>> list.
>>>>
>>>>     Arguments to the generated ``__init__`` method are passed to the 
>>>> original
>>>>     ``real_init`` unchanged.
>>>>     """
>>>>     def __init__(self, *args, **kwargs):
>>>>         self.mutable = list()
>>>>         if real_init is not None:
>>>>             return real_init(self, *args, **kwargs)
>>>>     if real_init is not None:
>>>>         f.update_wrapper(__init__, real_init)
>>>>     return __init__
>>>>
>>>> class Meta(type):
>>>>     def __new__(meta, name, parents, attributes):
>>>>         attributes['__init__'] = 
>>>> make_init(attributes.get('__init__', None))
>>>>         return type.__new__(Meta, name, parents, attributes)
>>>>
>>>> class C(object):
>>>>     __metaclass__ = Meta
>>>>
>>>> a, b = C(), C()
>>>>
>>>> a.mutable.append(3)
>>>> b.mutable.append(5)
>>>>
>>>> a.mutable
> [3]
>>>> b.mutable
> [5]
> 
> All instances of classes whose metaclass is Meta will, guaranteed, have 
> an instance field named 'mutable'. Its value is a list created at 
> instance creation time, unless the instance's __init__ provides a 
> different value.

The idea is good.  The devil is in the details, as usual.  How is the 
metaclass going to know:

   1) which attributes to replace
   2) what to replace them with?

~Ethan~

[toc] | [prev] | [next] | [standalone]

#21479

On 2012-03-10 22:21:55 +0000, Ethan Furman said:

> Owen Jacobson wrote:
>> On 2012-03-09 22:10:18 +0000, Ethan Furman said:
>> 
>>> Hey all!
>>> 
>>> I posted a question/answer on SO earlier, but there seems to be some 
>>> confusion around either the question or the answer (judging from the 
>>> comments).
>>> 
>>> http://stackoverflow.com/q/9638921/208880
>>> 
>>> If anyone here is willing to take a look at it and let me know if I did 
>>> not write it well, I would appreciate the feedback.
>>> 
>>> 
>>> Here's the question text:
>>> ------------------------
>>> I'm writing a metaclass to do some cool stuff, and part of its 
>>> processing is to check that certain attributes exist when the class is 
>>> created.  Some of these are mutable, and would normally be set in 
>>> `__init__`, but since `__init__` isn't run until the instance is 
>>> created the metaclass won't know that the attribute *will* be created, 
>>> and raises an error.  I could do something like:
>>> 
>>> class Test(meta=Meta):
>>> mutable = None
>>> def __init__(self):
>>> self.mutable = list()
>>> 
>>> But that isn't very elegant, and also violates DRY.
>>> 
>>> What I need is some way to have:
>>> 
>>> class Test(metaclass=Meta):
>>> mutable = list()
>>> 
>>> t1 = Test()
>>> t2 = Test()
>>> t1.mutable.append('one')
>>> t2.mutable.append('two')
>>> t1.mutable  # prints ['one']
>>> t2.mutable  # prints ['two']
>>> 
>>> Any ideas on how this can be accomplished?
>>> 
>>> Also, the metaclass doing the checking doesn't care what type of object 
>>> the attribute is, only that it is there.
>>> ---------------------------
>> 
>> Why check what you can ensure? The __init__ function your metaclass 
>> passes to type() doesn't have to be the __init__ method your metaclass 
>> received.

[… __init__-generation technique elided …]

>> All instances of classes whose metaclass is Meta will, guaranteed, have 
>> an instance field named 'mutable'. Its value is a list created at 
>> instance creation time, unless the instance's __init__ provides a 
>> different value.
> 
> The idea is good.  The devil is in the details, as usual.  How is the 
> metaclass going to know:
> 
>    1) which attributes to replace
>    2) what to replace them with?

I can think of at least three techniques; others are certainly possible:

1. As with the example code, the list is hard-coded in the metaclass's 
source code.
2. The list (or, rather, a dictionary) is drawn from a class attribute 
of the class being created:

	class Foo(object):
	    mandatory_fields = {'mutable': list, 'more_stuff': str}
	    __metaclass__ = IntrospectingMetaclass

3. A metaclass-returning factory produces new metaclasses on demand, 
each of which has a dict of mandatory fields baked into it. (This is a 
hybrid of the two approaches, and can easily have some messy side 
effects on your app's type ecology if used carelessly.)

Of course, you can also treat this the other way around: instead of 
enforcing that instances have specific fields, you could have users of 
those instances be aware that the field might not exist, and wrap 
access to the field in a try/except NameError block or use getattr to 
read the attribute.

What's appropriate really depends on how you plan to use this 
metaclass, and on the nature of the higher-level problem to which "I 
know, I'll use metaclasses" is your answer. How about telling us a 
slightly broader story about your problem?

-o

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web