Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #18004 > unrolled thread

confused about __new__

Back to article view | Back to comp.lang.python

Contents

  confused about __new__ "K. Richard Pixley" <rich@noir.com> - 2011-12-26 20:28 -0800
    Re: confused about __new__ Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-12-27 04:53 +0000
      Re: confused about __new__ K Richard Pixley <rich@noir.com> - 2011-12-26 21:27 -0800
    Re: confused about __new__ Fredrik Tolf <fredrik@dolda2000.com> - 2011-12-27 06:48 +0100
      Re: confused about __new__ K Richard Pixley <rich@noir.com> - 2011-12-27 09:41 -0800
        Re: confused about __new__ Ian Kelly <ian.g.kelly@gmail.com> - 2011-12-27 11:28 -0700
        Re: confused about __new__ Ian Kelly <ian.g.kelly@gmail.com> - 2011-12-27 13:34 -0700
          Re: confused about __new__ K Richard Pixley <rich@noir.com> - 2011-12-27 14:19 -0800
            Re: confused about __new__ Ian Kelly <ian.g.kelly@gmail.com> - 2011-12-27 15:43 -0700
        Re: confused about __new__ K Richard Pixley <rich@noir.com> - 2011-12-27 12:31 -0800
    Re: confused about __new__ Ian Kelly <ian.g.kelly@gmail.com> - 2011-12-27 00:28 -0700
    Re: confused about __new__ Lie Ryan <lie.1296@gmail.com> - 2011-12-27 18:47 +1100
    Re: confused about __new__ Fredrik Tolf <fredrik@dolda2000.com> - 2011-12-27 19:05 +0100

#18004 — confused about __new__

[Multipart message — attachments visible in raw view] — view raw

I'm confused about the following.  The idea here is that the set of 
instances of some class are small and finite, so I'd like to create them 
at class creation time, then hijack __new__ to simply return one of the 
preexisting classes instead of creating a new one each call.

This seems to work in python3, but fails in python2 with:

     Foo(23)
TypeError: unbound method __Foo__new__() must be called with Foo 
instance as first argument (got type instance instead)

I don't understand.  Can anyone explain?

--rich

[toc] | [next] | [standalone]

#18005

On Mon, 26 Dec 2011 20:28:26 -0800, K. Richard Pixley wrote:

> I'm confused about the following.  The idea here is that the set of
> instances of some class are small and finite, so I'd like to create them
> at class creation time, then hijack __new__ to simply return one of the
> preexisting classes instead of creating a new one each call.

I'm surprised it works in Python3. Try this in Python 2 instead:

Foo.__new__ = staticmethod(__Foo__new__)



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#18009

On 12/26/11 20:53 , Steven D'Aprano wrote:
> On Mon, 26 Dec 2011 20:28:26 -0800, K. Richard Pixley wrote:
>
>> I'm confused about the following.  The idea here is that the set of
>> instances of some class are small and finite, so I'd like to create them
>> at class creation time, then hijack __new__ to simply return one of the
>> preexisting classes instead of creating a new one each call.
>
> I'm surprised it works in Python3. Try this in Python 2 instead:
>
> Foo.__new__ = staticmethod(__Foo__new__)

Excellent.  Thank you.

--rich

[toc] | [prev] | [next] | [standalone]

#18010

On Mon, 26 Dec 2011, K. Richard Pixley wrote:
> I don't understand.  Can anyone explain?

I'm also a bit confused about __new__. I'd very much appreciate it if 
someone could explain the following aspects of it:

  * The manual (<http://docs.python.org/reference/datamodel.html>) says
    that __new__ is "a static method (special-cased so you need not declare
    it as such)". What does "special-cased" mean? Apparently, for
    instance, in OP's case,  Python did not automatically detect that it
    should not be bound as a method.

  * Is there any part of the manual that explains, holistically, the
    greater context of object instantiation into which __new__ fits? I can
    only find small parts of it spread around the documentation for __new__
    and __init__, but no complete explanation of it. There are several
    things I'm wondering about, like what it means to call a type object at
    all; how methods, properties and the like are bound; how pickle can
    instantiate a class without calling __init__; when and whether __dict__
    is created and a couple of other things. Is there a complete
    documentation of the process anywhere that I haven't managed to find?

--

Fredrik Tolf

[toc] | [prev] | [next] | [standalone]

#18031

On 12/26/11 21:48 , Fredrik Tolf wrote:
> On Mon, 26 Dec 2011, K. Richard Pixley wrote:
>> I don't understand. Can anyone explain?
>
> I'm also a bit confused about __new__. I'd very much appreciate it if
> someone could explain the following aspects of it:
>
> * The manual (<http://docs.python.org/reference/datamodel.html>) says
> that __new__ is "a static method (special-cased so you need not declare
> it as such)". What does "special-cased" mean? Apparently, for
> instance, in OP's case, Python did not automatically detect that it
> should not be bound as a method.
>
> * Is there any part of the manual that explains, holistically, the
> greater context of object instantiation into which __new__ fits? I can
> only find small parts of it spread around the documentation for __new__
> and __init__, but no complete explanation of it. There are several
> things I'm wondering about, like what it means to call a type object at
> all; how methods, properties and the like are bound; how pickle can
> instantiate a class without calling __init__; when and whether __dict__
> is created and a couple of other things. Is there a complete
> documentation of the process anywhere that I haven't managed to find?

Everything I know about this stuff I got by reading the manual.  Search 
for __init__ and you'll find a pretty complete description of how 
objects are created, how to perturb that process, how attributes are 
looked up, and how to perturb that process.  At least, you do in both 
python 2 and python 3 current manuals, I think.

The conceptual leap for me was in recognizing that a class is just an 
object.  The best way, (imo, so far), to create a singleton in python is 
to use the class itself as the singleton rather than ever instantiating 
it.  With a little work, you can even prevent it from ever being 
instantiated.

The concept of a "factory" gets a little weird then, as a factory might 
return an instantiation of some class but it might instead return a 
class proper - perhaps even an automatically constructed class - which 
is a somewhat different sort of factory.

Calling a type is a little bit different.  Calling a class is how you 
initiate an instantiation of that class.  Calling an instantiation leads 
to __call__, (which may or may not have much semantic meaning, depending 
on your class).  It's the difference between C() and C()().  (And the 
analogy holds for other builtin types, I think).

Super() is also fairly well documented.

But __metaclass__ could use a few more examples, imo.  I'm still not 
entirely clear on __metaclass__.  As I understand it, (which may well be 
completely wrong), __metaclass_ is the means of perturbing the results 
of "isinstance".  You can create, (or delete), inheritance arbitrarily, 
and without even requiring real, existing classes to do it.  I'm not 
clear on why you'd want to do this, nor why __metaclass__ is a better 
mechanism than, say, any of the implementations of "interface" which 
have been done in various tool kits or "mixins", which all seem to be 
somewhat overlapping in functionality.

--rich

[toc] | [prev] | [next] | [standalone]

#18043

On Tue, Dec 27, 2011 at 10:41 AM, K Richard Pixley <rich@noir.com> wrote:
> The conceptual leap for me was in recognizing that a class is just an
> object.  The best way, (imo, so far), to create a singleton in python is to
> use the class itself as the singleton rather than ever instantiating it.
>  With a little work, you can even prevent it from ever being instantiated.

I don't think that's actually possible:

Python 3.2 (r32:88445, Feb 20 2011, 21:29:02) [MSC v.1500 32 bit
(Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> class Foo:
...     def __new__(cls):
...         raise TypeError
...     def __init__(self):
...         raise TypeError
...
>>> type(object.__new__(Foo))
<class '__main__.Foo'>

I prefer the Borg pattern for singletons myself.  When you need a Borg
instance, you just instantiate the class normally, do whatever you
need with it, and then discard it normally.  It doesn't matter how
many instances there are (there can even be zero) because they all
share identical state.  It's better encapsulation too, since client
code does not even need to know that the class is a singleton variant;
that becomes an implementation detail.

http://www.aleax.it/Python/5ep.html
http://code.activestate.com/recipes/66531/


> But __metaclass__ could use a few more examples, imo.  I'm still not
> entirely clear on __metaclass__.  As I understand it, (which may well be
> completely wrong), __metaclass_ is the means of perturbing the results of
> "isinstance".  You can create, (or delete), inheritance arbitrarily, and
> without even requiring real, existing classes to do it.

A metaclass is the class of another class (which is normally `type` if
no metaclass is specified).  It's used to customize the way that the
class is created, in the same way that the class can be used to
customize the way instances are created (e.g. by overriding __new__).
You could use it to mess with the class's base classes, although I
think that usage is rare.  It's more common to use it to automatically
add or modify entries in the class dict.  For example, Django models
use a metaclass to collect all the fields and options declared for the
model class and to setup a default manager for the model class.

Note that many things that can be done with a metaclass can also be
done with a class decorator.  The main difference as I see it is that
the metaclass is inherited by subclasses, whereas a decorator would
need to be applied individually to each subclass.

[toc] | [prev] | [next] | [standalone]

#18050

On Tue, Dec 27, 2011 at 1:31 PM, K Richard Pixley <rich@noir.com> wrote:
> On 12/27/11 10:28 , Ian Kelly wrote:
>>
>> On Tue, Dec 27, 2011 at 10:41 AM, K Richard Pixley<rich@noir.com>  wrote:
>>>
>>> The conceptual leap for me was in recognizing that a class is just an
>>> object.  The best way, (imo, so far), to create a singleton in python is
>>> to
>>> use the class itself as the singleton rather than ever instantiating it.
>>>  With a little work, you can even prevent it from ever being
>>> instantiated.
>>
>> I don't think that's actually possible:
>>
>> Python 3.2 (r32:88445, Feb 20 2011, 21:29:02) [MSC v.1500 32 bit
>> (Intel)] on win32
>> Type "help", "copyright", "credits" or "license" for more information.
>>>>>
>>>>> class Foo:
>>
>> ...     def __new__(cls):
>> ...         raise TypeError
>> ...     def __init__(self):
>> ...         raise TypeError
>> ...
>>>>>
>>>>> type(object.__new__(Foo))
>>
>> <class '__main__.Foo'>
>
>
> Try:
>
> class Foo(object):
>    def __new__(cls):
>        return cls


Okay:

Python 3.2 (r32:88445, Feb 20 2011, 21:29:02) [MSC v.1500 32 bit
(Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> class Foo:
...     def __new__(cls):
...         return cls
...
>>> f1 = object.__new__(Foo)
>>> f2 = object.__new__(Foo)
>>> type(f1), type(f2)
(<class '__main__.Foo'>, <class '__main__.Foo'>)
>>> f1 is f2
False

[toc] | [prev] | [next] | [standalone]

#18065

On 12/27/11 12:34 , Ian Kelly wrote:
> On Tue, Dec 27, 2011 at 1:31 PM, K Richard Pixley<rich@noir.com>  wrote:
>> On 12/27/11 10:28 , Ian Kelly wrote:
>>>
>>> On Tue, Dec 27, 2011 at 10:41 AM, K Richard Pixley<rich@noir.com>    wrote:
>>>>
>>>> The conceptual leap for me was in recognizing that a class is just an
>>>> object.  The best way, (imo, so far), to create a singleton in python is
>>>> to
>>>> use the class itself as the singleton rather than ever instantiating it.
>>>>   With a little work, you can even prevent it from ever being
>>>> instantiated.
>>>
>>> I don't think that's actually possible:
>>>
>>> Python 3.2 (r32:88445, Feb 20 2011, 21:29:02) [MSC v.1500 32 bit
>>> (Intel)] on win32
>>> Type "help", "copyright", "credits" or "license" for more information.
>>>>>>
>>>>>> class Foo:
>>>
>>> ...     def __new__(cls):
>>> ...         raise TypeError
>>> ...     def __init__(self):
>>> ...         raise TypeError
>>> ...
>>>>>>
>>>>>> type(object.__new__(Foo))
>>>
>>> <class '__main__.Foo'>
>>
>>
>> Try:
>>
>> class Foo(object):
>>     def __new__(cls):
>>         return cls
>
>
> Okay:
>
> Python 3.2 (r32:88445, Feb 20 2011, 21:29:02) [MSC v.1500 32 bit
> (Intel)] on win32
> Type "help", "copyright", "credits" or "license" for more information.
>>>> class Foo:
> ...     def __new__(cls):
> ...         return cls
> ...
>>>> f1 = object.__new__(Foo)
>>>> f2 = object.__new__(Foo)
>>>> type(f1), type(f2)
> (<class '__main__.Foo'>,<class '__main__.Foo'>)
>>>> f1 is f2
> False

I'm not seeing why you're using "object.__new__".  With Foo(), it seems 
fine.

rich@fuji-land.noir.com> python
Python 2.7.2 (default, Dec 12 2011, 13:05:49)
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.1.00)] on 
darwin
Type "help", "copyright", "credits" or "license" for more information.
 >>> class Foo(object):
...     def __new__(cls):
...         return cls
...
 >>> f1 = Foo()
 >>> f2 = Foo()
 >>> id(Foo)
4298515984
 >>> id(f1)
4298515984
 >>> id(f2)
4298515984

rich@fuji-land.noir.com> python3
Python 3.2.2 (default, Dec 26 2011, 15:03:08)
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.1.00)] on 
darwin
Type "help", "copyright", "credits" or "license" for more information.
 >>> class Foo(object):
...     def __new__(cls):
...         return cls
...
 >>> f1 = Foo()
 >>> f2 = Foo()
 >>> id(f1)
4298841008
 >>> id(f2)
4298841008
 >>> f1.stuff = True
 >>> f2.stuff
True
 >>> id(Foo)
4298841008

Are you trying to demonstrate that I haven't prevented you from 
instantiating Foo?  If so, then I will cede that point.  I certainly 
don't know enough about python internals just now to even claim to be 
capable of protecting a class from a hostile user.  My guess is that 
short of a privileged/unprivileged split, or castrating the interpreter 
and locking you into it, such a protection would not be possible.

My point was that I can indeed intercept common and convenient usage to 
create a lovely singleton semantic.  I can't force you to use it. (Nor 
do I have any motivation to so do.)

--rich

[toc] | [prev] | [next] | [standalone]

#18069

On Tue, Dec 27, 2011 at 3:19 PM, K Richard Pixley <rich@noir.com> wrote:
> Are you trying to demonstrate that I haven't prevented you from
> instantiating Foo?  If so, then I will cede that point.  I certainly don't
> know enough about python internals just now to even claim to be capable of
> protecting a class from a hostile user.  My guess is that short of a
> privileged/unprivileged split, or castrating the interpreter and locking you
> into it, such a protection would not be possible.

Yes, that was the point.  I certainly don't think that anybody should
ever be creating class instances by directly invoking object.__new__
unless they really, really know what they are doing.  I was just
pointing out that I don't think it's actually possible to prevent
classes from being instantiated.

> My point was that I can indeed intercept common and convenient usage to
> create a lovely singleton semantic.  I can't force you to use it. (Nor do I
> have any motivation to so do.)

Sure, I don't disagree with that.  And the object.__new__ workaround
is a general point that doesn't apply only to singletons.  Any class
that customizes its instantiation process can be messed up by doing
that.

[toc] | [prev] | [next] | [standalone]

#18051

On 12/27/11 10:28 , Ian Kelly wrote:
> On Tue, Dec 27, 2011 at 10:41 AM, K Richard Pixley<rich@noir.com>  wrote:
>> The conceptual leap for me was in recognizing that a class is just an
>> object.  The best way, (imo, so far), to create a singleton in python is to
>> use the class itself as the singleton rather than ever instantiating it.
>>   With a little work, you can even prevent it from ever being instantiated.
> I don't think that's actually possible:
>
> Python 3.2 (r32:88445, Feb 20 2011, 21:29:02) [MSC v.1500 32 bit
> (Intel)] on win32
> Type "help", "copyright", "credits" or "license" for more information.
>>>> class Foo:
> ...     def __new__(cls):
> ...         raise TypeError
> ...     def __init__(self):
> ...         raise TypeError
> ...
>>>> type(object.__new__(Foo))
> <class '__main__.Foo'>

Try:

class Foo(object):
     def __new__(cls):
         return cls

--rich

[toc] | [prev] | [next] | [standalone]

#18015

On Mon, Dec 26, 2011 at 10:48 PM, Fredrik Tolf <fredrik@dolda2000.com> wrote:
> I'm also a bit confused about __new__. I'd very much appreciate it if
> someone could explain the following aspects of it:
>
>  * The manual (<http://docs.python.org/reference/datamodel.html>) says
>   that __new__ is "a static method (special-cased so you need not declare
>   it as such)". What does "special-cased" mean? Apparently, for
>   instance, in OP's case,  Python did not automatically detect that it
>   should not be bound as a method.

It apparently has to do with the class creation.  For the
special-casing to happen, the __new__ method has to be present when
the class is created.  If it is, then it automatically gets wrapped in
a staticmethod.  In the OP's case, he was adding the __new__ method
after class creation, so the wrapping did not happen automatically.
Compare:

>>> def my_new(cls): return object.__new__(cls)
...
>>> class Foo(object):
...     __new__ = my_new
...
>>> class Bar(object): pass
...
>>> Bar.__new__ = my_new
>>> Foo.__dict__['__new__']
<staticmethod object at 0x0237D6F0>
>>> Bar.__dict__['__new__']
<function my_new at 0x02381430>

>  * Is there any part of the manual that explains, holistically, the
>   greater context of object instantiation into which __new__ fits? I can
>   only find small parts of it spread around the documentation for __new__
>   and __init__, but no complete explanation of it. There are several
>   things I'm wondering about, like what it means to call a type object at
>   all;

I don't know of anything that organizes it that way specifically, but
I believe the Data Model reference pretty much covers what you're
looking for.  From the type hierarchy, under "Callable Types":
http://docs.python.org/reference/datamodel.html#the-standard-type-hierarchy
"""
Class Types
    Class types, or “new-style classes,” are callable. These objects
normally act as factories for new instances of themselves, but
variations are possible for class types that override __new__(). The
arguments of the call are passed to __new__() and, in the typical
case, to __init__() to initialize the new instance.
"""

AFAIK, that's pretty much it.  When a type is called, __new__ is
called to create the new instance, and then __init__ is called to
initialize it (if __new__ returned an instance of the type).

> how methods, properties and the like are bound;

When they're accessed, using the descriptor protocol, not as part of
the instantiation process.  See:
http://docs.python.org/reference/datamodel.html#invoking-descriptors

> how pickle can
>   instantiate a class without calling __init__;

By calling the __new__ method directly instead of calling the type:
http://docs.python.org/library/pickle.html#object.__getnewargs__

> when and whether __dict__
>   is created and a couple of other things.

Under the hood as part of the object creation process, unless the
class uses __slots__.

[toc] | [prev] | [next] | [standalone]

#18016

On 12/27/2011 04:48 PM, Fredrik Tolf wrote:
> On Mon, 26 Dec 2011, K. Richard Pixley wrote:
>> I don't understand. Can anyone explain?
>
> I'm also a bit confused about __new__. I'd very much appreciate it if
> someone could explain the following aspects of it:
>
> * The manual (<http://docs.python.org/reference/datamodel.html>) says
> that __new__ is "a static method (special-cased so you need not declare
> it as such)". What does "special-cased" mean? Apparently, for
> instance, in OP's case, Python did not automatically detect that it
> should not be bound as a method.

If you declare new in the regular way:

class Foo(object):
     def __new__(cls):
         ...

Python would create __new__ as a static method even without applying the 
staticmethod decorator; that python does not detect to special case 
__new__ when it is added dynamically is probably an oversight in Python 
2.x that was fixed in Python 3.x.

[toc] | [prev] | [next] | [standalone]

#18037

[Multipart message — attachments visible in raw view] — view raw

On Tue, 27 Dec 2011, Ian Kelly wrote:
> On Mon, Dec 26, 2011 at 10:48 PM, Fredrik Tolf <fredrik@dolda2000.com> wrote:
>> I'm also a bit confused about __new__. I'd very much appreciate it if
>> someone could explain the following aspects of it:
>>
>>  * The manual (<http://docs.python.org/reference/datamodel.html>) says
>>   that __new__ is "a static method (special-cased so you need not declare
>>   it as such)". What does "special-cased" mean? Apparently, for
>>   instance, in OP's case,  Python did not automatically detect that it
>>   should not be bound as a method.
>
> It apparently has to do with the class creation.  For the
> special-casing to happen, the __new__ method has to be present when
> the class is created.  If it is, then it automatically gets wrapped in
> a staticmethod.  In the OP's case, he was adding the __new__ method
> after class creation, so the wrapping did not happen automatically.

Hmm, thank you. After trying a couple of things, it appears that the 
reason OP's method worked in Python 3 is that __get__ on ordinary 
functions in Python 3 simply returns the function itself when called on 
type lookup, rather than Python 2's unbound method objects.

> http://docs.python.org/reference/datamodel.html#the-standard-type-hierarchy
> """
> Class Types
>    Class types, or “new-style classes,” are callable. These objects
> normally act as factories for new instances of themselves, but
> variations are possible for class types that override __new__(). The
> arguments of the call are passed to __new__() and, in the typical
> case, to __init__() to initialize the new instance.
> """
>
> AFAIK, that's pretty much it.  When a type is called, __new__ is
> called to create the new instance, and then __init__ is called to
> initialize it (if __new__ returned an instance of the type).

Since that description doesn't include when __dict__ is created, it isn't 
complete, however. I guess it's safe to assume that __dict__ is created 
inside object.__new__, but that also leaves some things unclear; see 
below.

>> how methods, properties and the like are bound;
>
> When they're accessed, using the descriptor protocol, not as part of
> the instantiation process.  See:
> http://docs.python.org/reference/datamodel.html#invoking-descriptors

Ah, sorry. I had read that, but I had mistakenly thought that the same 
instance method object was always returned, which would have implied that 
it had to be stored for the instance somewhere, at some point. I see now 
that that is not actually the case, however. That clears up a whole lot 
for me. Thanks!

>> when and whether __dict__
>>   is created and a couple of other things.
>
> Under the hood as part of the object creation process, unless the
> class uses __slots__.

And also unless the object created is of type `object' or any other 
built-in type, which leaves me wondering exactly under what circumstances 
__dict__ actually is created. Is it some kind of special case in 
object.__new__ for types created with the `class' statement? There also 
appear to be other special cases in object.__new__:

>>> object.__new__(dict)
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
TypeError: object.__new__(dict) is not safe, use dict.__new__()

I guess my question reduces into the following: What does object.__new__ 
actually do, and what other special cases does it implement?

--

Fredrik Tolf

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web