Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #6328 > unrolled thread

Python's super() considered super!

Back to article view | Back to comp.lang.python

Contents

  Python's super() considered super! Raymond Hettinger <python@rcn.com> - 2011-05-26 09:31 -0700
    Re: Python's super() considered super! Raymond Hettinger <python@rcn.com> - 2011-05-26 09:39 -0700
      Re: Python's super() considered super! Dotan Cohen <dotancohen@gmail.com> - 2011-05-26 21:13 +0300
      Re: Python's super() considered super! Ian Kelly <ian.g.kelly@gmail.com> - 2011-05-26 12:38 -0600
      Re: Python's super() considered super! Dotan Cohen <dotancohen@gmail.com> - 2011-05-26 21:56 +0300
      Re: Python's super() considered super! Terry Reedy <tjreedy@udel.edu> - 2011-05-26 16:15 -0400
    Re: Python's super() considered super! Ben Finney <ben+python@benfinney.id.au> - 2011-05-27 11:39 +1000
      Re: Python's super() considered super! Raymond Hettinger <python@rcn.com> - 2011-05-27 00:16 -0700
        Re: Python's super() considered super! Ben Finney <ben+python@benfinney.id.au> - 2011-05-27 18:49 +1000
          Re: Python's super() considered super! Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-05-27 10:37 +0000
            Re: Python's super() considered super! Duncan Booth <duncan.booth@invalid.invalid> - 2011-05-27 10:53 +0000
              Re: Python's super() considered super! Ethan Furman <ethan@stoneleaf.us> - 2011-05-27 10:42 -0700
                Re: Python's super() considered super! Duncan Booth <duncan.booth@invalid.invalid> - 2011-05-30 09:18 +0000
      RE: Python's super() considered super! "Prasad, Ramit" <ramit.prasad@jpmchase.com> - 2011-05-27 14:10 -0400
      Re: Python's super() considered super! Chris Angelico <rosuav@gmail.com> - 2011-05-28 04:40 +1000
    Re: Python's super() considered super! sturlamolden <sturlamolden@yahoo.no> - 2011-05-27 07:27 -0700
      Re: Python's super() considered super! Mel <mwilson@the-wire.com> - 2011-05-27 10:33 -0400
        Re: Python's super() considered super! Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-05-27 14:49 +0000
          Re: Python's super() considered super! harrismh777 <harrismh777@charter.net> - 2011-05-27 10:07 -0500
      Re: Python's super() considered super! Duncan Booth <duncan.booth@invalid.invalid> - 2011-05-27 15:05 +0000
        Re: Python's super() considered super! sturlamolden <sturlamolden@yahoo.no> - 2011-05-27 08:24 -0700
        Re: Python's super() considered super! sturlamolden <sturlamolden@yahoo.no> - 2011-05-27 08:31 -0700
          Re: Python's super() considered super! Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-05-27 16:06 +0000
            Re: Python's super() considered super! Stefan Behnel <stefan_ml@behnel.de> - 2011-05-27 23:49 +0200
              Re: Python's super() considered super! sturlamolden <sturlamolden@yahoo.no> - 2011-05-27 16:57 -0700
                Re: Python's super() considered super! Thomas Rachel <nutznetz-0c1b6768-bfa9-48d5-a470-7603bd3aa915@spamschutz.glglgl.de> - 2011-05-28 07:29 +0200
            Re: Python's super() considered super! sturlamolden <sturlamolden@yahoo.no> - 2011-05-27 17:04 -0700
        Re: Python's super() considered super! Ryan Kelly <ryan@rfk.id.au> - 2011-05-28 09:57 +1000
      Re: Python's super() considered super! sturlamolden <sturlamolden@yahoo.no> - 2011-05-27 08:11 -0700
    Re: Python's super() considered super! Ian Kelly <ian.g.kelly@gmail.com> - 2011-05-27 12:31 -0600
    Re: Python's super() considered super! Chris Angelico <rosuav@gmail.com> - 2011-05-28 04:46 +1000
      Re: Python's super() considered super! John Nagle <nagle@animats.com> - 2011-05-27 13:47 -0700
    Re: Python's super() considered super! Ethan Furman <ethan@stoneleaf.us> - 2011-05-27 15:24 -0700

Page 2 of 2 — ← Prev page 1 [2]

#6402

On 27 Mai, 17:05, Duncan Booth <duncan.bo...@invalid.invalid> wrote:

> >>> class C(B): pass
> >>> C().foo()
>
> ... infinite recursion follows ...

That's true :(

[toc] | [prev] | [next] | [standalone]

#6403

On 27 Mai, 17:05, Duncan Booth <duncan.bo...@invalid.invalid> wrote:

> Oops. There's a reason why Python 2 requires you to be explicit about
> the class; you simply cannot work it out automatically at run time.
> Python 3 fixes this by working it out at compile time, but for Python 2
> there is no way around it.

Then it should be a keyword, not a function.

Sturla

[toc] | [prev] | [next] | [standalone]

#6408

On Fri, 27 May 2011 08:31:40 -0700, sturlamolden wrote:

> On 27 Mai, 17:05, Duncan Booth <duncan.bo...@invalid.invalid> wrote:
> 
>> Oops. There's a reason why Python 2 requires you to be explicit about
>> the class; you simply cannot work it out automatically at run time.
>> Python 3 fixes this by working it out at compile time, but for Python 2
>> there is no way around it.
> 
> Then it should be a keyword, not a function.

Why? The fault is not that super is a function, or that you monkey-
patched it, or that you used a private function to do that monkey-
patching. The fault was that you made a common, but silly, mistake when 
reasoning about type(self) inside a class. 

I made the same mistake: assume that type(self) will always be the same 
class as that the method is defined in. But of course it won't be. With 
the luxury of hindsight, it is a silly mistake to make, but I promise you 
that you're not the first, and won't be the last, to make it.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#6434

Steven D'Aprano, 27.05.2011 18:06:
> On Fri, 27 May 2011 08:31:40 -0700, sturlamolden wrote:
>
>> On 27 Mai, 17:05, Duncan Booth<duncan.bo...@invalid.invalid>  wrote:
>>
>>> Oops. There's a reason why Python 2 requires you to be explicit about
>>> the class; you simply cannot work it out automatically at run time.
>>> Python 3 fixes this by working it out at compile time, but for Python 2
>>> there is no way around it.
>>
>> Then it should be a keyword, not a function.
>
> Why?

I think Sturla is referring to the "compile time" bit. CPython cannot know 
that the builtin super() will be called at runtime, even if it sees a 
"super()" function call.

CPython doesn't evaluate the super call at compile time, it only keeps a 
reference to the surrounding class in the code object of the method. So 
super() without arguments basically inherits the class argument from the 
context the method was found in at compile time. This has two quirks:

1) Copying a method from one class to another keeps the original context. 
So the class argument to super() is basically fixed at compile time, 
regardless of the class the method will be executed on at runtime.

2) The class is only kept as a reference when CPython sees a function call 
that looks like "super" at compile time, which isn't much more than a 
heuristic.

The PEP doesn't mention the first issue, but it is actually explicit about 
the second issue:

http://www.python.org/dev/peps/pep-3135/

"""
While super is not a reserved word, the parser recognizes the use of super 
in a method definition and only passes in the __class__ cell when this is 
found. Thus, calling a global alias of super without arguments will not 
necessarily work.
"""

And the prove:

   >>> _super = super
   >>> class T(object):
   ...   def test(self): print('TEST')
  ...
   >>> class B(T):
   ...   def meth(self): _super().test()
   ...
   >>> B().meth()
   Traceback (most recent call last):
   SystemError: super(): __class__ cell not found

I assume this is done in order to reduce the chance of accidentally keeping 
a class object alive for eternity, only because a method was originally 
defined therein that inherits the class reference in its code object. So 
it's a tradeoff between memory overhead and usability issues.

While I think that the tradeoff is generally ok, I agree with Sturla that a 
keyword would have been the correct solution, whereas this is a clear 
"works only in the common cases" approach.

Stefan

[toc] | [prev] | [next] | [standalone]

#6439

On 27 Mai, 23:49, Stefan Behnel <stefan...@behnel.de> wrote:

> I think Sturla is referring to the "compile time" bit. CPython cannot know
> that the builtin super() will be called at runtime, even if it sees a
> "super()" function call.

Yes. And opposite: CPython cannot know that builtin super() is not
called,
even if it does not see the name 'super'. I can easily make foo()
alias super().

In both cases, the cure is a keyword -- or make sure that __class__
is always defined.

If super is to be I keyword, we could argue that self and cls should
be
keywords as well, and methods should always be bound. That speaks in
favour
of a super() function. But then it should always be evaluated at run-
time,
without any magic from the parser.

Magic functions belong in Perl, not Python.

Sturla

[toc] | [prev] | [next] | [standalone]

#6450

Am 28.05.2011 01:57 schrieb sturlamolden:

> Yes. And opposite: CPython cannot know that builtin super() is not
> called,
> even if it does not see the name 'super'. I can easily make foo()
> alias super().

Another alternative would have been to make use of __xxx magic.

If every class had an "automatically available" attribute, e. g. 
__<classname>_classname which thus could be accessed via __classname 
from inside, keeping the 2.x syntax would have been the best, using 
super(__classname, self).


> In both cases, the cure is a keyword -- or make sure that __class__
> is always defined.
>
> If super is to be I keyword, we could argue that self and cls should
> be keywords as well, and methods should always be bound. That speaks in
> favour of a super() function. But then it should always be evaluated at run-
> time, without any magic from the parser.
>
> Magic functions belong in Perl, not Python.

ACK.


Thomas

[toc] | [prev] | [next] | [standalone]

#6441

On 27 Mai, 18:06, Steven D'Aprano <steve
+comp.lang.pyt...@pearwood.info> wrote:

> Why? The fault is not that super is a function, or that you monkey-
> patched it, or that you used a private function to do that monkey-
> patching. The fault was that you made a common, but silly, mistake when
> reasoning about type(self) inside a class.

That was indeed a silly mistake, but not what I am talking about.
See Stefan's reponse.


Sturla

[toc] | [prev] | [next] | [standalone]

#6440

[Multipart message — attachments visible in raw view] — view raw

On Fri, 2011-05-27 at 15:05 +0000, Duncan Booth wrote:
> sturlamolden <sturlamolden@yahoo.no> wrote:
> > I really don't like the Python 2 syntax of super, as it violates
> > the DRY principle: Why do I need to write super(type(self),self)
> > when super() will do? Assuming that 'self' will always be named
> > 'self' in my code, I tend to patch __builtins__.super like this:
> > 
> > import sys
> > def super():
> >     self = sys._getframe().f_back.f_locals['self']
> >     return __builtins__.super(type(self),self)
> > 
> > This way the nice Python 3.x syntax can be used in Python 2.x.
> > 
> > 
> Oh dear, you haven't thought this one through.
>
> ...snip...
>
> >>> C().foo()
> ... infinite recursion follows ...
> 
> Oops. There's a reason why Python 2 requires you to be explicit about 
> the class; you simply cannot work it out automatically at run time. 
> Python 3 fixes this by working it out at compile time, but for Python 2 
> there is no way around it.

Oh?  There's not much that can't be done at runtime if you're willing to
work hard enough.  Let me propose the following awful awful hack:


  import sys

  _builtin_super = __builtins__.super

  _sentinel = object()

  def _auto_super(typ=_sentinel,type_or_obj=_sentinel):
      """Automagically call correct super() at runtime"""
      #  Infer the correct call if used without arguments.
      if typ is _sentinel:
          # We'll need to do some frame hacking.
          f = sys._getframe(1)
          # Get the first positional argument of the function.
          type_or_obj = f.f_locals[f.f_code.co_varnames[0]]
          # Get the MRO for investigation
          try:
              mro = type_or_obj.__mro__
          except AttributeError:
              try:
                  mro = type_or_obj.__class__.__mro__
              except AttributeError:
                  raise RuntimeError("super() used with old-style class")
          #  Now, find the class owning the currently-executing method.
          for typ in mro:
              for meth in typ.__dict__.itervalues():
                  if not isinstance(meth,type(_auto_super)):
                      continue
                  if meth.func_code is f.f_code:
                      # Aha!  Found you.
                      break
              else:
                  continue
              break
          else:
              raise RuntimeError("super() called outside a method")
      #  Now just dispatch to builtin super.
      if type_or_obj is not _sentinel:
          return _builtin_super(typ,type_or_obj)
      return _builtin_super(typ)


Now, try is with the following:

    class Base(object):
        def hello(self,msg):
            print "hello", msg

    class Sub1(Base):
        def hello(self,msg):
            print "gunna say it"
            super().hello(msg)

    class Sub2(Base):
        def hello(self,msg):
            print "yes I am"
            super().hello(msg)

    class Diamond(Sub1,Sub2):
        def hello(self,msg):
            print "here we go..."
            super().hello(msg)

    d = Diamond()
    d.hello("autosuper!")


And you get the expected output:

    here we go...
    gunna say it
    yes I am
    hello autosuper!


There may well be some cases where this breaks down, but it seems to do
the right thing in simple cases.


Not that I'm recommending anyone use this, of course...




   Ryan


-- 
Ryan Kelly
http://www.rfk.id.au  |  This message is digitally signed. Please visit
ryan@rfk.id.au        |  http://www.rfk.id.au/ramblings/gpg/ for details

[toc] | [prev] | [next] | [standalone]

#6400

On 27 Mai, 16:27, sturlamolden <sturlamol...@yahoo.no> wrote:

> Assuming that 'self' will always be named
> 'self' in my code, I tend to patch __builtins__.super like this:
>
> import sys
> def super():
>     self = sys._getframe().f_back.f_locals['self']
>     return __builtins__.super(type(self),self)


A monkey-patch to __builtins__.super would probably also work.

Assuming the first argument to the callee is 'self' or 'cls':

import sys
_super = __builtins__.super
def monkeypatch(*args, **kwargs):
    if (args == ()) and (kwargs=={}):
        try:
            obj = sys._getframe().f_back.f_locals['self']
        except KeyError:
            obj = sys._getframe().f_back.f_locals['cls']
        return _super(type(obj),obj)
    else:
        return _super(*args, **kwargs)

class patcher(object):
   def __init__(self):
      __builtins__.super = monkeypatch
   def __del__(self):
      __builtins__.super = _super

_patch = patcher()



Sturla

[toc] | [prev] | [next] | [standalone]

#6422

On Thu, May 26, 2011 at 10:31 AM, Raymond Hettinger <python@rcn.com> wrote:
> I just posted a tutorial and how-to guide for making effective use of
> super().

I posted this already on the HackerNews thread but it seems to have
largely gone unnoticed, so I'm reposting it here.

It seems to me that the example of combining built-in dictionary
classes is naively optimistic. For starters, OrderedDict, as it
happens, does not use super! It calls the dict super-class methods
directly. Since dict luckily happens to be the next class in the MRO,
this doesn't really matter for the purpose of this example, but I can
envision a scenario where some plucky programmer inherits from both
OrderedCounter and some other dict subclass, and the result doesn't
work because OrderedDict does the wrong thing.

And OrderedDict isn't the only one. Maybe for some reason I would like
to have an OrderedCounter where all the counts default to 42. So I do
this:

  class DefaultOrderedCounter(defaultdict, OrderedCounter):
      pass
  doc = DefaultOrderedCounter(lambda: 42)
  doc.update('abracadabra')

Which results in:

  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "c:\python32\lib\collections.py", line 507, in update
      _count_elements(self, iterable)
    File "c:\python32\lib\collections.py", line 63, in __setitem__
      self.__map[key] = link = Link()
  AttributeError: 'DefaultOrderedCounter' object has no attribute
'_OrderedDict__map'

Whoops! Apparently defaultdict doesn't use super either. Of course a
better way to do this would be to subclass OrderedCounter and just
override the __missing__ method by hand, but that's not the point.

The article goes into "How to Incorporate a Non-cooperative Class",
which basically says "wrap it up in a proxy class". But that's not
really going to work here, since the result would be two separate
dicts, with the defaultdictwrapper methods operating on one dict, and
the other methods operating on the other.

[toc] | [prev] | [next] | [standalone]

#6426

On Sat, May 28, 2011 at 4:31 AM, Ian Kelly <ian.g.kelly@gmail.com> wrote:
> It seems to me that the example of combining built-in dictionary
> classes is naively optimistic.

So... Can anyone offer a non-trivial example of multiple inheritance
that *doesn't* have pitfalls? From what I've seen, MI always seems to
require cooperation from the authors of all involved classes. It may
be a useful tool when you control everything, but whenever you use
someone else's code, there seems to be this massive barrier of risk
(if that makes sense). For the DefaultOrderedCounter, I would be
strongly inclined to inherit singly, and then manually implement the
other half (whichever half is easier); in this case that happens to be
trivial (override __missing__), but even were it not, it would be a
means of being certain that things won't break.

Chris Angelico

[toc] | [prev] | [next] | [standalone]

#6431

On 5/27/2011 11:46 AM, Chris Angelico wrote:
> On Sat, May 28, 2011 at 4:31 AM, Ian Kelly<ian.g.kelly@gmail.com>  wrote:
>> It seems to me that the example of combining built-in dictionary
>> classes is naively optimistic.
>
> So... Can anyone offer a non-trivial example of multiple inheritance
> that *doesn't* have pitfalls?   From what I've seen, MI always seems > to require cooperation from 
the authors of all involved classes.

     Good point.

     Multiple inheritance is messy enough when the structure is just
a tree.  When the structure is allowed to be a directed acyclic
graph, the whole thing just gets too complicated.

     It doesn't even necessarily do what you want.  If, say, you
have two classes that need dictionaries, and were implemented
by inheriting from "dict", a class that imports both has one
"dict", not two - something that was not anticipated in the
design of the original classes.  That ought to be an error,
not a single "dict" shared by two unconnected classes.

     What causes this kind of mess is a determination not to
report anything as an error if it can be given some kind of
meaningful semantics, even if the semantics have marginal
value.  That's the kind of thinking which leads to

	[1,2] * 2

returning

	[1,2,1,2]



				John Nagle

[toc] | [prev] | [next] | [standalone]

#6435

I suspect the larger issue is that Multiple Inheritance is complex, but 
folks don't appreciate that.  Ask anyone about meta-classes and their 
eyes bug-out, but MI?  Simple!  NOT.

On the other hand, perhaps the docs should declare that the built-in 
objects are not designed for MI, so that that, at least, would be one 
less bug waiting to happen.

~Ethan~

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | comp.lang.python

csiph-web