Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #86028 > unrolled thread

Re: Design thought for callbacks

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Design thought for callbacks Cem Karan <cfkaran2@gmail.com> - 2015-02-21 08:13 -0500
    Re: Design thought for callbacks Grant Edwards <invalid@invalid.invalid> - 2015-02-21 20:57 +0000
      Re: Design thought for callbacks Marko Rauhamaa <marko@pacujo.net> - 2015-02-21 23:57 +0200
        Re: Design thought for callbacks Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-02-22 13:04 +1100
          Re: Design thought for callbacks Chris Angelico <rosuav@gmail.com> - 2015-02-22 13:11 +1100
            Re: Design thought for callbacks Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-02-22 15:38 +1100
              Re: Design thought for callbacks Chris Angelico <rosuav@gmail.com> - 2015-02-22 17:21 +1100
                Re: Design thought for callbacks Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-02-22 21:32 +1100
                  Re: Design thought for callbacks Chris Angelico <rosuav@gmail.com> - 2015-02-22 22:14 +1100
                    Re: Design thought for callbacks Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-02-23 00:45 +1100
                      Re: Design thought for callbacks Chris Angelico <rosuav@gmail.com> - 2015-02-23 01:01 +1100
          Re: Design thought for callbacks Marko Rauhamaa <marko@pacujo.net> - 2015-02-22 09:52 +0200
            Re: Design thought for callbacks Chris Angelico <rosuav@gmail.com> - 2015-02-22 19:17 +1100
              Re: Design thought for callbacks Marko Rauhamaa <marko@pacujo.net> - 2015-02-22 10:34 +0200
                Re: Design thought for callbacks Chris Angelico <rosuav@gmail.com> - 2015-02-22 19:58 +1100
                  Re: Design thought for callbacks Marko Rauhamaa <marko@pacujo.net> - 2015-02-22 11:14 +0200
                    Re: Design thought for callbacks Chris Angelico <rosuav@gmail.com> - 2015-02-22 21:21 +1100
                    Re: Design thought for callbacks Laura Creighton <lac@openend.se> - 2015-02-22 12:28 +0100
                    Re: Design thought for callbacks Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-02-23 00:57 +1100
                      Re: Design thought for callbacks Marko Rauhamaa <marko@pacujo.net> - 2015-02-22 16:06 +0200
      Re: Design thought for callbacks Ian Kelly <ian.g.kelly@gmail.com> - 2015-02-22 00:05 -0700
      Re: Design thought for callbacks Cem Karan <cfkaran2@gmail.com> - 2015-02-22 08:21 -0500
        Re: Design thought for callbacks Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2015-02-24 18:20 +1300
          Re: Design thought for callbacks random832@fastmail.us - 2015-02-24 00:29 -0500
            Re: Design thought for callbacks Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2015-02-25 10:19 +1300
              Re: Design thought for callbacks Cem Karan <cfkaran2@gmail.com> - 2015-02-25 07:50 -0500
                Re: Design thought for callbacks Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2015-02-26 18:36 +1300
                  Re: Design thought for callbacks Cem Karan <cfkaran2@gmail.com> - 2015-02-26 05:58 -0500
                  Re: Design thought for callbacks Ian Kelly <ian.g.kelly@gmail.com> - 2015-02-26 12:54 -0700
                  Re: Design thought for callbacks Ethan Furman <ethan@stoneleaf.us> - 2015-02-26 12:00 -0800
                  Re: Design thought for callbacks Cem Karan <cfkaran2@gmail.com> - 2015-03-02 06:06 -0500
                  Re: Design thought for callbacks Cem Karan <cfkaran2@gmail.com> - 2015-03-02 06:04 -0500
                  Re: Design thought for callbacks Ian Kelly <ian.g.kelly@gmail.com> - 2015-03-02 08:34 -0700
                  Re: Design thought for callbacks Cem Karan <cfkaran2@gmail.com> - 2015-03-08 14:30 -0400

Page 2 of 2 — ← Prev page 1 [2]

#86084

On Sat, Feb 21, 2015 at 1:57 PM, Grant Edwards <invalid@invalid.invalid> wrote:
> On 2015-02-21, Cem Karan <cfkaran2@gmail.com> wrote:
>>
>> On Feb 21, 2015, at 12:42 AM, Chris Angelico <rosuav@gmail.com> wrote:
>>
>>> On Sat, Feb 21, 2015 at 1:44 PM, Cem Karan <cfkaran2@gmail.com> wrote:
>>>> In order to inform users that certain bits of state have changed, I require them to register a callback with my code.  The problem is that when I store these callbacks, it naturally creates a strong reference to the objects, which means that if they are deleted without unregistering themselves first, my code will keep the callbacks alive.  Since this could lead to really weird and nasty situations, I would like to store all the callbacks in a WeakSet (https://docs.python.org/3/library/weakref.html#weakref.WeakSet).  That way, my code isn't the reason why the objects are kept alive, and if they are no longer alive, they are automatically removed from the WeakSet, preventing me from accidentally calling them when they are dead.  My question is simple; is this a good design?  If not, why not?  Are there any potential 'gotchas' I should be worried about?
>>>>
>>>
>>> No, it's not. I would advise using strong references - if the callback
>>> is a closure, for instance, you need to hang onto it, because there
>>> are unlikely to be any other references to it. If I register a
>>> callback with you, I expect it to be called; I expect, in fact, that
>>> that *will* keep my object alive.
>>
>> OK, so it would violate the principle of least surprise for you.
>
> And me as well.  I would expect to be able to pass a closure as a
> callback and not have to keep a reference to it.  Perhaps that just a
> leftover from working with other languages (javascript, scheme, etc.).
> It doesn't matter if it's a string, a float, a callback, a graphic or
> whatever: if I pass your function/library an object, I expect _you_ to
> keep track of it until you're done with it.
>
>> Interesting.  Is this a general pattern in python?  That is,
>> callbacks are owned by what they are registered with?
>
> I'm not sure what you mean by "owned" or why it matters that it's a
> callback: it's an object that was passed to you: you need to hold onto
> a reference to it until you're done with it, and the polite thing to
> do is to delete references to it when you're done with it.
>
>> So, what's the consensus on the list, strongly-held callbacks, or
>> weakly-held ones?

Count me in the weak-ref crowd. It may be a nuisance to keep a
reference around on the object registering the callback, but it's
preferable to the alternative of messing around with disposables in
order to ensure that the callback gets cleaned up and doesn't create a
memory leak. I would also rather have my code fail by losing a
callback reference, which should be relatively easy to spot and
diagnose, than to have said memory leak go unnoticed.

[toc] | [prev] | [next] | [standalone]

#86111

On Feb 21, 2015, at 3:57 PM, Grant Edwards <invalid@invalid.invalid> wrote:

> On 2015-02-21, Cem Karan <cfkaran2@gmail.com> wrote:
>> 
>> On Feb 21, 2015, at 12:42 AM, Chris Angelico <rosuav@gmail.com> wrote:
>> 
>>> On Sat, Feb 21, 2015 at 1:44 PM, Cem Karan <cfkaran2@gmail.com> wrote:
>>>> In order to inform users that certain bits of state have changed, I require them to register a callback with my code.  The problem is that when I store these callbacks, it naturally creates a strong reference to the objects, which means that if they are deleted without unregistering themselves first, my code will keep the callbacks alive.  Since this could lead to really weird and nasty situations, I would like to store all the callbacks in a WeakSet (https://docs.python.org/3/library/weakref.html#weakref.WeakSet).  That way, my code isn't the reason why the objects are kept alive, and if they are no longer alive, they are automatically removed from the WeakSet, preventing me from accidentally calling them when they are dead.  My question is simple; is this a good design?  If not, why not?  Are there any potential 'gotchas' I should be worried about?
>>>> 
>>> 
>>> No, it's not. I would advise using strong references - if the callback
>>> is a closure, for instance, you need to hang onto it, because there
>>> are unlikely to be any other references to it. If I register a
>>> callback with you, I expect it to be called; I expect, in fact, that
>>> that *will* keep my object alive.
>> 
>> OK, so it would violate the principle of least surprise for you.
> 
> And me as well.  I would expect to be able to pass a closure as a
> callback and not have to keep a reference to it.  Perhaps that just a
> leftover from working with other languages (javascript, scheme, etc.).
> It doesn't matter if it's a string, a float, a callback, a graphic or
> whatever: if I pass your function/library an object, I expect _you_ to
> keep track of it until you're done with it.
> 
>> Interesting.  Is this a general pattern in python?  That is,
>> callbacks are owned by what they are registered with?
> 
> I'm not sure what you mean by "owned" or why it matters that it's a
> callback: it's an object that was passed to you: you need to hold onto
> a reference to it until you're done with it, and the polite thing to
> do is to delete references to it when you're done with it.

I tend to structure my code as a tree or DAG of objects.  The owner refers to the owned object, but the owned object has no reference to its owner.  With callbacks, you get cycles, where the owned owns the owner.  As a result, if you forget where your object has been registered, it may be kept alive when you aren't expecting it.  My hope was that with WeakSets I could continue to preserve the DAG or tree while still having the benefits of callbacks.  However, it looks like that is too surprising to most people.

Thanks,
Cem Karan

[toc] | [prev] | [next] | [standalone]

#86293

Cem Karan wrote:
> I tend to structure my code as a tree or DAG of objects.  The owner refers to
> the owned object, but the owned object has no reference to its owner.  With
> callbacks, you get cycles, where the owned owns the owner.

This is why I suggested registering a listener object
plus a method name instead of a callback. It avoids that
reference cycle, because there is no long-lived callback
object keeping a reference to the listener.

-- 
Greg

[toc] | [prev] | [next] | [standalone]

#86295

On Tue, Feb 24, 2015, at 00:20, Gregory Ewing wrote:
> Cem Karan wrote:
> > I tend to structure my code as a tree or DAG of objects.  The owner refers to
> > the owned object, but the owned object has no reference to its owner.  With
> > callbacks, you get cycles, where the owned owns the owner.
> 
> This is why I suggested registering a listener object
> plus a method name instead of a callback. It avoids that
> reference cycle, because there is no long-lived callback
> object keeping a reference to the listener.

How does that help? Everywhere you would have had a reference to the
"callback object", you now have a reference to the listener object.
You're just shuffling deck chairs around: if B shouldn't reference A
because A owns B, then removing C from the B->C->A reference chain does
nothing to fix this.

[toc] | [prev] | [next] | [standalone]

#86344

random832@fastmail.us wrote:
> On Tue, Feb 24, 2015, at 00:20, Gregory Ewing wrote:
> 
>>This is why I suggested registering a listener object
>>plus a method name instead of a callback. It avoids that
>>reference cycle, because there is no long-lived callback
>>object keeping a reference to the listener.
> 
> How does that help? Everywhere you would have had a reference to the
> "callback object", you now have a reference to the listener object.

The point is that the library can keep a weak reference
to the listener object, whereas it can't reliably keep
a weak reference to a bound method.

-- 
Greg

[toc] | [prev] | [next] | [standalone]

#86397

On Feb 24, 2015, at 4:19 PM, Gregory Ewing <greg.ewing@canterbury.ac.nz> wrote:

> random832@fastmail.us wrote:
>> On Tue, Feb 24, 2015, at 00:20, Gregory Ewing wrote:
>>> This is why I suggested registering a listener object
>>> plus a method name instead of a callback. It avoids that
>>> reference cycle, because there is no long-lived callback
>>> object keeping a reference to the listener.
>> How does that help? Everywhere you would have had a reference to the
>> "callback object", you now have a reference to the listener object.
> 
> The point is that the library can keep a weak reference
> to the listener object, whereas it can't reliably keep
> a weak reference to a bound method.

I think I see what you're talking about now.  Does WeakMethod (https://docs.python.org/3/library/weakref.html#weakref.WeakMethod) solve this problem?  Note that I can force my users to use the latest stable version of python at all times, so WeakMethod IS available to me.

Thanks,
Cem Karan

[toc] | [prev] | [next] | [standalone]

#86475

Cem Karan wrote:
> I think I see what you're talking about now.  Does WeakMethod
> (https://docs.python.org/3/library/weakref.html#weakref.WeakMethod) solve
> this problem?

Yes, that looks like it would work.

-- 
Greg

[toc] | [prev] | [next] | [standalone]

#86492

On Feb 26, 2015, at 12:36 AM, Gregory Ewing <greg.ewing@canterbury.ac.nz> wrote:

> Cem Karan wrote:
>> I think I see what you're talking about now.  Does WeakMethod
>> (https://docs.python.org/3/library/weakref.html#weakref.WeakMethod) solve
>> this problem?
> 
> Yes, that looks like it would work.


Cool!  

Thanks,
Cem Karan

[toc] | [prev] | [next] | [standalone]

#86536

[Multipart message — attachments visible in raw view] — view raw

On Feb 26, 2015 4:00 AM, "Cem Karan" <cfkaran2@gmail.com> wrote:
>
>
> On Feb 26, 2015, at 12:36 AM, Gregory Ewing <greg.ewing@canterbury.ac.nz>
wrote:
>
> > Cem Karan wrote:
> >> I think I see what you're talking about now.  Does WeakMethod
> >> (https://docs.python.org/3/library/weakref.html#weakref.WeakMethod)
solve
> >> this problem?
> >
> > Yes, that looks like it would work.
>
>
> Cool!

Sometimes I wonder whether anybody reads my posts. I suggested a solution
involving WeakMethod four days ago that additionally extends the concept to
non-method callbacks (requiring a small amount of extra effort from the
client in those cases, but I think that is unavoidable. There is no way
that the framework can determine the appropriate lifetime for a
closure-based callback.)

[toc] | [prev] | [next] | [standalone]

#86538

[Multipart message — attachments visible in raw view] — view raw

On 02/26/2015 11:54 AM, Ian Kelly wrote:

> Sometimes I wonder whether anybody reads my posts.

It's entirely possible the OP wasn't ready to understand your solution four days ago, but two days later the OP was.

--
~Ethan~

[toc] | [prev] | [next] | [standalone]

#86717

On Feb 26, 2015, at 3:00 PM, Ethan Furman <ethan@stoneleaf.us> wrote:

> On 02/26/2015 11:54 AM, Ian Kelly wrote:
> 
>> Sometimes I wonder whether anybody reads my posts.
> 
> It's entirely possible the OP wasn't ready to understand your solution four days ago, but two days later the OP was.

Thank you Ethan, that was precisely my problem.

Thanks,
Cem Karan

[toc] | [prev] | [next] | [standalone]

#86719

On Feb 26, 2015, at 2:54 PM, Ian Kelly <ian.g.kelly@gmail.com> wrote:
> On Feb 26, 2015 4:00 AM, "Cem Karan" <cfkaran2@gmail.com> wrote:
> >
> >
> > On Feb 26, 2015, at 12:36 AM, Gregory Ewing <greg.ewing@canterbury.ac.nz> wrote:
> >
> > > Cem Karan wrote:
> > >> I think I see what you're talking about now.  Does WeakMethod
> > >> (https://docs.python.org/3/library/weakref.html#weakref.WeakMethod) solve
> > >> this problem?
> > >
> > > Yes, that looks like it would work.
> >
> >
> > Cool!
> 
> Sometimes I wonder whether anybody reads my posts. I suggested a solution involving WeakMethod four days ago that additionally extends the concept to non-method callbacks (requiring a small amount of extra effort from the client in those cases, but I think that is unavoidable. There is no way that the framework can determine the appropriate lifetime for a closure-based callback.)

I apologize about taking so long to reply to everyone's posts, but I've been busy at home.

Ian, it took me a while to do some research to understand WHY what you were suggesting was important; you're right about storing the object as well as the method/function separately, but I think that WeakMethod might solve that completely, correct?  Are there any cases where WeakMethod wouldn't work?

Thanks,
Cem Karan

[toc] | [prev] | [next] | [standalone]

#86750

On Mon, Mar 2, 2015 at 4:04 AM, Cem Karan <cfkaran2@gmail.com> wrote:
> On Feb 26, 2015, at 2:54 PM, Ian Kelly <ian.g.kelly@gmail.com> wrote:
>> On Feb 26, 2015 4:00 AM, "Cem Karan" <cfkaran2@gmail.com> wrote:
>> >
>> >
>> > On Feb 26, 2015, at 12:36 AM, Gregory Ewing <greg.ewing@canterbury.ac.nz> wrote:
>> >
>> > > Cem Karan wrote:
>> > >> I think I see what you're talking about now.  Does WeakMethod
>> > >> (https://docs.python.org/3/library/weakref.html#weakref.WeakMethod) solve
>> > >> this problem?
>> > >
>> > > Yes, that looks like it would work.
>> >
>> >
>> > Cool!
>>
>> Sometimes I wonder whether anybody reads my posts. I suggested a solution involving WeakMethod four days ago that additionally extends the concept to non-method callbacks (requiring a small amount of extra effort from the client in those cases, but I think that is unavoidable. There is no way that the framework can determine the appropriate lifetime for a closure-based callback.)
>
> I apologize about taking so long to reply to everyone's posts, but I've been busy at home.
>
> Ian, it took me a while to do some research to understand WHY what you were suggesting was important; you're right about storing the object as well as the method/function separately, but I think that WeakMethod might solve that completely, correct?  Are there any cases where WeakMethod wouldn't work?

WeakMethod only works for bound method objects. If you pass it a
non-method function, you'll get a TypeError:

>>> from weakref import WeakMethod
>>> WeakMethod(lambda: None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.4/weakref.py", line 49, in __new__
    .format(type(meth))) from None
TypeError: argument should be a bound method, not <class 'function'>

This check uses duck typing, so you could perhaps write a method-like
class with __self__ and __func__ attributes and pass that to the
WeakMethod constructor in the case of non-methods. There's a bigger
issue with this however, which is that WeakMethod works by keeping
weak references to *both* the object and the function, meaning that as
soon as the function has no other references, the WeakMethod expires
even if the object is still alive. This isn't a problem for methods
because it's the transience of the method object, not the underlying
function, that WeakMethod seeks to work around. But it doesn't by
itself do anything to solve the problem of closures or lambdas that
may themselves be transient.

Revisiting the implementation I suggested previously, I want to make a
correction. This would be better solved with a WeakValueDictionary:

class Listenable:
    def __init__(self):
        self._callbacks = weakref.WeakValueDictionary()

    def listen(self, callback, owner=None):
        if owner is None:
            if isinstance(callback, types.MethodType):
                owner = weakref.WeakMethod(callback)
            else:
                owner = callback
        # TODO: Should anything happen if the callback is already in the dict?
        self._callbacks[callback] = owner

    def do_callbacks(self, message):
        for callback in self._callbacks.keys():
            callback(message)

This approach has two benefits over the previous one: it simplifies
the callback management a bit, and it avoids making the assumption
that the owner is hashable (it assumes instead that the callback is
hashable, but I think that's reasonable).

[toc] | [prev] | [next] | [standalone]

#87151

Hi all, I apologize for taking so long to reply, but neither my work schedule nor the weather have been kind in the past week.  That said, I've been thinking long and hard about what everyone has said, and have decided that it would be useful to write a wrap-up email that attempts to encapsulate everything everyone has said, as a record of sorts if nothing else.  As such, this email is fairly long and involved.

=======================
Analysis of the problem
=======================

My original question was 'what is the least surprising/most pythonic way to write a callback API?'  Through reading what everyone has said, I realized that I wasn't being specific enough, simply because callback APIs can be quite different.  At the very least, the following questions need to be answered:

1) When a callback is registered, does it replace the prior callback?
2) If more than one callback can be registered, is there an ordering to them?
3) Can a callback be registered more than once?
4) When and how are callbacks deregistered?
5) Who is responsible for maintaining a strong reference to the callback?

As far as I know, there isn't a standard method to indicate to the caller that one callback replaces another one except via well-written documentation.  My personal feeling is that callbacks that replace other callbacks should be properties of the library.  By implementing a setter, getter, and deleter for each callback, the library makes it obvious that there is one and only one callback active at a time.  The only difficulty is making sure the user knows that the library retains the callback, but this is a documentation problem.  

I realized that ordering could be a problem when I read through the documentation to asyncio.call_soon().  It promises that callbacks will be called in the order in which they were registered.  However, there are cases where the order doesn't matter.  Registration in both of these cases is fairly simple; the former appends the callback to a list, while the latter adds it to a set.  The list or set can be a property of the library, and registration is simply a matter of either inserting or adding.  But this brings up point 3; if a callback can be registered at most once and ordering matters, then we need something that is both a sequence and a set.  Subclassing either (or both) collections.abc.MutableSequence or collections.abc.MutableSet will lead to confusion due to unexpected violations of PEP 3119 (https://www.python.org/dev/peps/pep-3119/).  Once again, the only option appears to be careful documentation.

Registration is only half the problem.  The other half is determining when a callback should be unregistered.  Some callbacks are one-shots and are automatically unregistered as soon as they are called.  Others will be called each time an event occurs until they are explicitly unregistered from the library.  Which happens is another design choice that needs to be carefully documented.

Finally, we come to the part that started my original question; who retains the callback.  I had originally asked everyone if it would be surprising to store callbacks as weak references.  The idea was that unless someone else maintained a strong reference to the callback, it would be garbage collected, which would save users from 'surprising' results such as the following:

"""
#! /usr/bin/env python

class Callback_object(object):
    def __init__(self, msg):
        self._msg = msg
    def callback(self, stuff):
        print("From {0!s}: {1!s}".format(self._msg, stuff))

class Fake_library(object):
    def __init__(self):
        self._callbacks = list()
    def register_callback(self, callback):
        self._callbacks.append(callback)
    def execute_callbacks(self):
        for thing in self._callbacks:
            thing('Surprise!')

if __name__ == "__main__":
    cbo = Callback_object("Evil Zombie")
    lib = Fake_library()
    lib.register_callback(cbo.callback)

    # Way later, after the user forgot all about the callback above
    cbo = Callback_object("Your Significant Other")
    lib.register_callback(cbo.callback)

    # And finally getting around to running all those callbacks.
    lib.execute_callbacks()
"""

However, as others pointed out using a weak reference could actually increase confusion rather than decrease it.  The problem is that if there is a reference cycle elsewhere in the code, it is possible that the zombie object is still alive when it is supposed to be dead.  This will likely be difficult to debug. In addition, different types of callables have different requirements in order to correctly store weak references to them.  Both Ian Kelly and Fabio Zadrozny provided solutions to this, with Fabio providing a link to his code at http://pydev.blogspot.com.br/2015/02/design-for-client-side-applications-in.html.

====================================
Solution to my problem in particular
====================================

After considering all the comments above, I've decided to do the following for my API:

- All callbacks will be strongly retained (no weakrefs).
- Callbacks will be stored in a list, and the list will be exposed as a read-only property of the library.  This will let users reorder callbacks as necessary, add them multiple times in a row, etc.  I'm also hoping that by making it a list, it becomes obvious that the callback is strongly retained.
- Finally, callbacks are not one-shots.  This just happens to make sense for my code, but others may find other methods make more sense.


Thanks again to everyone for providing so many comments on my question, and I apologize again for taking so long to wrap things up.

Thanks,
Cem Karan

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | comp.lang.python

csiph-web