Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 6.6 \(1510\))
Subject: Re: Design thought for callbacks
From: Cem Karan <cfkaran2@gmail.com>
In-Reply-To: <CALwzidmgj1y==3iv8Zub+aHyFR3P3okeLBRKu+R92914mDXfqQ@mail.gmail.com>
Date: Sun, 8 Mar 2015 14:30:32 -0400
Cc: Python <python-list@python.org>
Content-Transfer-Encoding: quoted-printable
References: <33677AE8-B2FA-49F9-9304-C8D93784255D@gmail.com> <CAPTjJmpEHQV2PpaZvszuTGn0_GNg0sWV6aLenbQsZrSXCy2YKQ@mail.gmail.com> <mailman.18957.1424524392.18130.python-list@python.org> <mcargl$h3k$1@reader1.panix.com> <mailman.19008.1424611304.18130.python-list@python.org> <cl2g0vFfiq9U1@mid.individual.net> <mailman.19117.1424755795.18130.python-list@python.org> <cl486mFu4l9U1@mid.individual.net> <mailman.19181.1424868605.18130.python-list@python.org> <cl7pmjF7595U1@mid.individual.net> <F521BFFE-506F-4B58-9A15-20D91A019BD0@gmail.com> <CALwzidm3AngMF9aG-gLpYur6CfaADM0YXGU=p0=dv-qqYXmmFA@mail.gmail.com> <8A3659BC-9100-4A3A-9117-47227B3D290B@gmail.com> <CALwzidmgj1y==3iv8Zub+aHyFR3P3okeLBRKu+R92914mDXfqQ@mail.gmail.com>
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.170.1425839439.21433.python-list@python.org>
Lines: 129
NNTP-Posting-Host: 2001:888:2000:d::a6
Path: csiph.com!usenet.pasdenom.info!bete-des-vosges.org!feed.ac-versailles.fr!nerim.net!novso.com!newsfeed.xs4all.nl!newsfeed1.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Xref: csiph.com comp.lang.python:87151

Hi all, I apologize for taking so long to reply, but neither my work =
schedule nor the weather have been kind in the past week.  That said, =
I've been thinking long and hard about what everyone has said, and have =
decided that it would be useful to write a wrap-up email that attempts =
to encapsulate everything everyone has said, as a record of sorts if =
nothing else.  As such, this email is fairly long and involved.

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Analysis of the problem
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

My original question was 'what is the least surprising/most pythonic way =
to write a callback API?'  Through reading what everyone has said, I =
realized that I wasn't being specific enough, simply because callback =
APIs can be quite different.  At the very least, the following questions =
need to be answered:

1) When a callback is registered, does it replace the prior callback?
2) If more than one callback can be registered, is there an ordering to =
them?
3) Can a callback be registered more than once?
4) When and how are callbacks deregistered?
5) Who is responsible for maintaining a strong reference to the =
callback?

As far as I know, there isn't a standard method to indicate to the =
caller that one callback replaces another one except via well-written =
documentation.  My personal feeling is that callbacks that replace other =
callbacks should be properties of the library.  By implementing a =
setter, getter, and deleter for each callback, the library makes it =
obvious that there is one and only one callback active at a time.  The =
only difficulty is making sure the user knows that the library retains =
the callback, but this is a documentation problem. =20

I realized that ordering could be a problem when I read through the =
documentation to asyncio.call_soon().  It promises that callbacks will =
be called in the order in which they were registered.  However, there =
are cases where the order doesn't matter.  Registration in both of these =
cases is fairly simple; the former appends the callback to a list, while =
the latter adds it to a set.  The list or set can be a property of the =
library, and registration is simply a matter of either inserting or =
adding.  But this brings up point 3; if a callback can be registered at =
most once and ordering matters, then we need something that is both a =
sequence and a set.  Subclassing either (or both) =
collections.abc.MutableSequence or collections.abc.MutableSet will lead =
to confusion due to unexpected violations of PEP 3119 =
(https://www.python.org/dev/peps/pep-3119/).  Once again, the only =
option appears to be careful documentation.

Registration is only half the problem.  The other half is determining =
when a callback should be unregistered.  Some callbacks are one-shots =
and are automatically unregistered as soon as they are called.  Others =
will be called each time an event occurs until they are explicitly =
unregistered from the library.  Which happens is another design choice =
that needs to be carefully documented.

Finally, we come to the part that started my original question; who =
retains the callback.  I had originally asked everyone if it would be =
surprising to store callbacks as weak references.  The idea was that =
unless someone else maintained a strong reference to the callback, it =
would be garbage collected, which would save users from 'surprising' =
results such as the following:

"""
#! /usr/bin/env python

class Callback_object(object):
    def __init__(self, msg):
        self._msg =3D msg
    def callback(self, stuff):
        print("=46rom {0!s}: {1!s}".format(self._msg, stuff))

class Fake_library(object):
    def __init__(self):
        self._callbacks =3D list()
    def register_callback(self, callback):
        self._callbacks.append(callback)
    def execute_callbacks(self):
        for thing in self._callbacks:
            thing('Surprise!')

if __name__ =3D=3D "__main__":
    cbo =3D Callback_object("Evil Zombie")
    lib =3D Fake_library()
    lib.register_callback(cbo.callback)

    # Way later, after the user forgot all about the callback above
    cbo =3D Callback_object("Your Significant Other")
    lib.register_callback(cbo.callback)

    # And finally getting around to running all those callbacks.
    lib.execute_callbacks()
"""

However, as others pointed out using a weak reference could actually =
increase confusion rather than decrease it.  The problem is that if =
there is a reference cycle elsewhere in the code, it is possible that =
the zombie object is still alive when it is supposed to be dead.  This =
will likely be difficult to debug. In addition, different types of =
callables have different requirements in order to correctly store weak =
references to them.  Both Ian Kelly and Fabio Zadrozny provided =
solutions to this, with Fabio providing a link to his code at =
http://pydev.blogspot.com.br/2015/02/design-for-client-side-applications-i=
n.html.

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Solution to my problem in particular
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

After considering all the comments above, I've decided to do the =
following for my API:

- All callbacks will be strongly retained (no weakrefs).
- Callbacks will be stored in a list, and the list will be exposed as a =
read-only property of the library.  This will let users reorder =
callbacks as necessary, add them multiple times in a row, etc.  I'm also =
hoping that by making it a list, it becomes obvious that the callback is =
strongly retained.
- Finally, callbacks are not one-shots.  This just happens to make sense =
for my code, but others may find other methods make more sense.


Thanks again to everyone for providing so many comments on my question, =
and I apologize again for taking so long to wrap things up.

Thanks,
Cem Karan=