Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder4.news.weretis.net!feeds.phibee-telecom.net!newsfeed.xs4all.nl!newsfeed3a.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 6.6 \(1510\))
Subject: Re: Design thought for callbacks
From: Cem Karan <cfkaran2@gmail.com>
In-Reply-To: <54e8c017$0$13008$c3e8da3$5496439d@news.astraweb.com>
Date: Sun, 22 Feb 2015 08:13:09 -0500
Content-Transfer-Encoding: quoted-printable
References: <33677AE8-B2FA-49F9-9304-C8D93784255D@gmail.com> <CAPTjJmpEHQV2PpaZvszuTGn0_GNg0sWV6aLenbQsZrSXCy2YKQ@mail.gmail.com> <39813568-6DB8-4341-A130-C256CFF352EE@gmail.com> <CAPTjJmrfMowbK7z+eN7xfMv2Ck5peJpBPyBRHvhXM4-ocxRR6w@mail.gmail.com> <mailman.18962.1424527666.18130.python-list@python.org> <54e8c017$0$13008$c3e8da3$5496439d@news.astraweb.com>
To: Steven D'Aprano <steve+comp.lang.python@pearwood.info>
Cc: python-list@python.org
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.19007.1424610799.18130.python-list@python.org>
Lines: 368
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:86110


On Feb 21, 2015, at 12:27 PM, Steven D'Aprano =
<steve+comp.lang.python@pearwood.info> wrote:

> Cem Karan wrote:
>=20
>>=20
>> On Feb 21, 2015, at 8:15 AM, Chris Angelico <rosuav@gmail.com> wrote:
>>=20
>>> On Sun, Feb 22, 2015 at 12:13 AM, Cem Karan <cfkaran2@gmail.com> =
wrote:
>>>> OK, so it would violate the principle of least surprise for you.=20
>>>> Interesting.  Is this a general pattern in python?  That is, =
callbacks
>>>> are owned by what they are registered with?
>>>>=20
>>>> In the end, I want to make a library that offers as few surprises =
to the
>>>> user as possible, and no matter how I think about callbacks, they =
are
>>>> surprising to me.  If callbacks are strongly-held, then calling =
'del
>>>> foo' on a callable object may not make it go away, which can lead =
to
>>>> weird and nasty situations.
>=20
> How?
>=20
> The whole point of callbacks is that you hand over responsibility to =
another
> piece of code, and then forget about your callback. The library will =
call
> it, when and if necessary, and when the library no longer needs your
> callback, it is free to throw it away. (If I wish the callback to =
survive
> beyond the lifetime of your library's use of it, I have to keep a =
reference
> to the function.)

Marko mentioned it earlier; if you think you've gotten rid of all =
references to some chunk of code, and it is still alive afterwards, that =
can be surprising.

>>>> Weakly-held callbacks mean that I (as the=20
>>>> programmer), know that objects will go away after the next garbage
>>>> collection (see Frank's earlier message), so I don't get 'dead'
>>>> callbacks coming back from the grave to haunt me.
>=20
> I'm afraid this makes no sense to me. Can you explain, or better still
> demonstrate, a scenario where "dead callbacks rise from the grave", so =
to
> speak?

"""
#! /usr/bin/env python

class Callback_object(object):
    def __init__(self, msg):
        self._msg =3D msg
    def callback(self, stuff):
        print("=46rom {0!s}: {1!s}".format(self._msg, stuff))

class Fake_library(object):
    def __init__(self):
        self._callbacks =3D list()
    def register_callback(self, callback):
        self._callbacks.append(callback)
    def execute_callbacks(self):
        for thing in self._callbacks:
            thing('Surprise!')

if __name__ =3D=3D "__main__":
    foo =3D Callback_object("Evil Zombie")
    lib =3D Fake_library()
    lib.register_callback(foo.callback)

    # Way later, after the user forgot all about the callback above
    foo =3D Callback_object("Your Significant Other")
    lib.register_callback(foo.callback)

    # And finally getting around to running all those callbacks.
    lib.execute_callbacks()
"""

Output:
=46rom Evil Zombie: Surprise!
=46rom Your Significant Other: Surprise!

In this case, the user made an error (just as Marko said in his earlier =
message), and forgot about the callback he registered with the library.  =
The callback isn't really rising from the dead; as you say, either its =
been garbage collected, or it hasn't been.  However, you may not be =
ready for a callback to be called at that moment in time, which means =
you're surprised by unexpected behavior.

>>>> So, what's the consensus on the list, strongly-held callbacks, or
>>>> weakly-held ones?
>>>=20
>>> I don't know about Python specifically, but it's certainly a general
>>> pattern in other languages. They most definitely are owned, and it's
>>> the only model that makes sense when you use closures (which won't
>>> have any other references anywhere).
>>=20
>> I agree about closures; its the only way they could work.
>=20
> *scratches head* There's nothing special about closures. You can =
assign them
> to a name like any other object.
>=20
> def make_closure():
>    x =3D 23
>    def closure():
>        return x + 1
>    return closure
>=20
> func =3D make_closure()
>=20
> Now you can register func as a callback, and de-register it when your =
done:
>=20
> register(func)
> unregister(func)
>=20
>=20
> Of course, if you thrown away your reference to func, you have no =
(easy) way
> of de-registering it. That's no different to any other object which is
> registered by identity. (Registering functions by name is a bad idea, =
since
> multiple functions can have the same name.)
>=20
> As an alternative, your callback registration function might return a =
ticket
> for the function:
>=20
> ticket =3D register(func)
> del func
> unregister(ticket)
>=20
> but that strikes me as over-kill. And of course, the simplest ticket =
is to
> return the function itself :-)

Agreed on all points; closures are just ordinary objects.  The only =
difference (in my opinion) is that they are 'fire and forget'; if you =
are registering or tracking them then you've kind of defeated the =
purpose.  THAT is what I meant about how you handle closures.

>=20
>> When I was=20
>> originally thinking about the library, I was trying to include all =
types
>> of callbacks, including closures and callable objects.  The callable
>> objects may pass themselves, or one of their methods to the library, =
or
>> may do something really weird.
>=20
> I don't think they can do anything too weird. They have to pass a =
callable
> object. Your library just calls that object. You shouldn't need to =
care
> whether it is a function, a method, a type, a callable instance, or
> something else. You just call it, and when you're done calling it =
forever,
> you just throw it away.

That doesn't quite solve the problem, but it comes close.  The headache =
(as shown in my earlier code) is that you think you've gotten rid of =
something before it is called, but it turns out you haven't.  I'm =
starting to think that there isn't a solution to this other than telling =
the programmer "Don't do that". =20

>> Although I just realized that closures may cause another problem.  In =
my
>> code, I expect that many different callbacks can be registered for =
the
>> same event.  Unregistering means you request to be unregistered for =
the
>> event. How do you do that with a closure?  Aren't they anonymous?
>=20
> Not unless you create them using lambda. Using the make_closure =
function
> above:
>=20
>=20
> py> func =3D make_closure()
> py> func.__name__
> 'closure'
>=20
> Of course, if you call make_closure twice, both functions will have =
the same
> internal name. You can set the function __name__ and __qualname__ to =
fix
> that. This is how the functools.wraps decorator works.
>=20
> But that's a red herring. Don't register functions by name! Not all =
callable
> objects have names, and those that do, you may have multiple =
*distinct*
> callbacks with the same name.
>=20
> There are two reasonable approaches: unregister by identity, or by =
returning
> a ticket which uniquely identifies the callback. The user is =
responsible
> for keeping track of their own ticket. If I lose it, I can't =
unregister my
> callback any more. So sad, sucks to be me.
>=20
>=20
> The simplest possible identity-based scheme would be something like =
this:
>=20
>=20
> # don't hate me for using a global variable
> CALLBACKS =3D []
>=20
> def register(func):
>    if func not in CALLBACKS:
>         CALLBACKS.append(func)
>=20
> def unregister(func):
>    try:
>        CALLBACKS.remove(func)
>    except ValueError:
>        pass
>=20
>=20
> That's probably a bit too simple, since it won't behave as expected =
with
> bound methods. The problem is that bound methods are generated on the =
fly,
> so this won't work:
>=20
> register(instance.spam)
> # later
> unregister(instance.spam)  # a different instance!
>=20
> I would have to do this:
>=20
> bound_method =3D instance.spam
> register(bound_method)
> unregister(bound_method)
>=20
>=20
> But a more sophisticated unregister function should work:
>=20
> # Untested
> def unregister(func):
>    for i, f in enumerate(CALLBACKS):
>        if (f is func) or (isinstance(f, types.MethodType)=20
>                and f.__wrapped__ is func):
>            del CALLBACKS[i]
>            return

Are you sure about that?  I just tested out the following code, and it =
appears to work correctly:

"""
#! /usr/bin/env python

class Callback_object(object):
    def __init__(self, msg):
        self._msg =3D msg
    def callback(self, stuff):
        print("=46rom {0!s}: {1!s}".format(self._msg, stuff))

class Fake_library(object):
    def __init__(self):
        self._callbacks =3D set()
    def register_callback(self, callback):
        self._callbacks.add(callback)
    def unregister_callback(self, callback):
        self._callbacks.discard(callback)
    def execute_callbacks(self):
        for thing in self._callbacks:
            thing('Surprise!')

if __name__ =3D=3D "__main__":
    foo =3D Callback_object("Evil Zombie")
    lib =3D Fake_library()
    lib.register_callback(foo.callback)
    lib.unregister_callback(foo.callback)
    lib.execute_callbacks()
"""

I'll admit though, I don't know if it worked because I got lucky, or if =
python guarantees it works...

> The simplest possible ticket-based system is probably something like =
this:
>=20
> CALLBACKS =3D {}
> NEXT_TICKET =3D 1
>=20
> def register(func):
>    global NEXT_TICKET
>    ticket =3D NEXT_TICKET
>    NEXT_TICKET +=3D 1
>    callbacks[ticket] =3D func
>    return ticket
>=20
> def unregister(ticket):
>    if ticket in CALLBACKS:
>        del CALLBACKS[ticket]
>=20

I'd probably go with something similar to this, except that I'd use =
UUIDs for the tickets.  I know me and my users, and somewhere along the =
line I'd use a ticket to unregister from the wrong callback dictionary!

>>> If you're expecting 'del foo' to destroy the object, then you have a
>>> bigger problem than callbacks, because that's simply not how Python
>>> works. You can't _ever_ assume that deleting something from your =
local
>>> namespace will destroy the object, because there can always be more
>>> references. So maybe you need a more clear way of saying "I'm done
>>> with this, get rid of it".
>>=20
>> Agreed about 'del', and I don't assume that the object goes away at =
the
>> point.  The problem is debugging and determining WHY your object is =
still
>> around.  I know a combination of logging and gc.get_referrers() will
>> probably help you figure out why something is still around, but I'm =
trying
>> to avoid that headache.
>=20
> Why do you care? Surely all your library should care about is whether =
or not
> they have a reference to the callback.If they do, they should call it =
(when
> appropriate). If they don't, they aren't responsible for it.

I care because I care about anyone using my code.  Telling them 'tough, =
its your problem' doesn't get you many friends.  Making a library that =
performs as expected and where it is easy to debug what went wrong makes =
everyone (including me!) happy.

>> I guess the real problem is how this creates cycles in the call =
graph.=20
>> User code effectively owns the library code, which via callbacks owns =
the
>> user code.  I have no idea what the best point the cycle is to break =
it,
>> and not surprise someone down the road.  The only idea I have is to
>> redesign the library a little, and make anything that accepts a =
callback
>> actually be a subclass of collections.abc.Container, or even
>> collections.abc.MutableSet.  That makes it very obvious that the =
object
>> owns the callback, and that you will need to remove your object to
>> unregister it.
>=20
> My brain hurts from the complexity of your solution. What is the =
actual
> problem you are trying to solve? I would like to see an example of an
> actual failure before trying to solve a purely theoretical failure =
mode.

If I'm making your head hurt, then my solution is a bad solution.  The =
whole reason I started this discussion was to figure out if an =
alternative method would make more sense to my potential endusers =
(fellow programmers).  If strongly-held callbacks cause fewer headaches, =
then that is what I'll go with.

Thanks,
Cem Karan=