Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #86015 > unrolled thread

Re: Design thought for callbacks

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Design thought for callbacks "Frank Millman" <frank@chagford.com> - 2015-02-21 07:41 +0200
    Re: Design thought for callbacks Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-02-22 03:15 +1100
      Re: Design thought for callbacks Marko Rauhamaa <marko@pacujo.net> - 2015-02-21 19:08 +0200
        Re: Design thought for callbacks Cem Karan <cfkaran2@gmail.com> - 2015-02-22 07:16 -0500
          Re: Design thought for callbacks Marko Rauhamaa <marko@pacujo.net> - 2015-02-22 14:46 +0200
            Re: Design thought for callbacks Cem Karan <cfkaran2@gmail.com> - 2015-02-22 09:17 -0500
        Re: Design thought for callbacks Laura Creighton <lac@openend.se> - 2015-02-22 13:52 +0100
        Re: Design thought for callbacks Cem Karan <cfkaran2@gmail.com> - 2015-02-22 09:10 -0500
      Re: Design thought for callbacks "Frank Millman" <frank@chagford.com> - 2015-02-22 08:44 +0200
        Re: Design thought for callbacks Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2015-02-22 23:15 +1300
          Re: Design thought for callbacks Cem Karan <cfkaran2@gmail.com> - 2015-02-22 09:22 -0500
            Re: Design thought for callbacks Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2015-02-24 18:45 +1300
              Re: Design thought for callbacks Cem Karan <cfkaran2@gmail.com> - 2015-02-24 06:06 -0500
          Re: Design thought for callbacks Ian Kelly <ian.g.kelly@gmail.com> - 2015-02-22 23:57 -0700

#86015 — Re: Design thought for callbacks

"Cem Karan" <cfkaran2@gmail.com> wrote in message 
news:33677AE8-B2FA-49F9-9304-C8D93784255D@gmail.com...
> Hi all, I'm working on a project that will involve the use of callbacks, 
> and I want to bounce an idea I had off of everyone to make sure I'm not 
> developing a bad idea.  Note that this is for python 3.4 code; I don't 
> need to worry about any version of python earlier than that.
>
> In order to inform users that certain bits of state have changed, I 
> require them to register a callback with my code.  The problem is that 
> when I store these callbacks, it naturally creates a strong reference to 
> the objects, which means that if they are deleted without unregistering 
> themselves first, my code will keep the callbacks alive.  Since this could 
> lead to really weird and nasty situations, I would like to store all the 
> callbacks in a WeakSet 
> (https://docs.python.org/3/library/weakref.html#weakref.WeakSet).  That 
> way, my code isn't the reason why the objects are kept alive, and if they 
> are no longer alive, they are automatically removed from the WeakSet, 
> preventing me from accidentally calling them when they are dead.  My 
> question is simple; is this a good design?  If not, why not?
>   Are there any potential 'gotchas' I should be worried about?
>

I tried something similar a while ago, and I did find a gotcha.

The problem lies in this phrase - "if they are no longer alive, they are 
automatically removed from the WeakSet, preventing me from accidentally 
calling them when they are dead."

I found that the reference was not removed immediately, but was waiting to 
be garbage collected. During that window, I could call the callback, which 
resulted in an error.

There may have been a simple workaround. Perhaps someone else can comment.

Frank Millman

[toc] | [next] | [standalone]

#86053

Frank Millman wrote:

> I tried something similar a while ago, and I did find a gotcha.
> 
> The problem lies in this phrase - "if they are no longer alive, they are
> automatically removed from the WeakSet, preventing me from accidentally
> calling them when they are dead."
> 
> I found that the reference was not removed immediately, but was waiting to
> be garbage collected. During that window, I could call the callback, which
> resulted in an error.

I don't understand how this could possibly work. (Or fail to work, as the
case may be.)

If the callback has been garbage collected, then you cannot call it, because
you don't have any references to it and so cannot refer to it in any way.

If the callback has *not* been garbage collected, then you can safely call
it. You have a reference to the callback, therefore it exists. (If Python
ever garbage collects an object that still has references to it, that would
be a critical bug, and you would likely get some sort of seg fault).

The only thing I can think of is you have a situation where your callback
refers to another object, B, via a weak reference. Once all the regular
strong references to the callback and B are gone, theoretically you could
have a race condition where the callback is waiting to be garbage collected
but B has already been garbage collected. If, in that window, you call the
callback, *and* if the callback fails to correctly check that the weak
reference to B still exists, then you could get a Python exception.

The solution is simple: anytime you have a weak reference, you must always
check it before you use it.

Other than that, I cannot see how calling a function which has *not* yet
been garbage collected can fail, just because the only reference still
existing is a weak reference.


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#86055

Steven D'Aprano <steve+comp.lang.python@pearwood.info>:

> Other than that, I cannot see how calling a function which has *not*
> yet been garbage collected can fail, just because the only reference
> still existing is a weak reference.

Maybe the logic of the receiving object isn't prepared for the callback
anymore after an intervening event.

The problem then, of course, is in the logic and not in the callbacks.


Marko

[toc] | [prev] | [next] | [standalone]

#86103

On Feb 21, 2015, at 12:08 PM, Marko Rauhamaa <marko@pacujo.net> wrote:

> Steven D'Aprano <steve+comp.lang.python@pearwood.info>:
> 
>> Other than that, I cannot see how calling a function which has *not*
>> yet been garbage collected can fail, just because the only reference
>> still existing is a weak reference.
> 
> Maybe the logic of the receiving object isn't prepared for the callback
> anymore after an intervening event.
> 
> The problem then, of course, is in the logic and not in the callbacks.

This was PRECISELY the situation I was thinking about.  My hope was to make the callback mechanism slightly less surprising by allowing the user to track them, releasing them when they aren't needed without having to figure out where the callbacks were registered.  However, it appears I'm making things more surprising rather than less.

Thanks,
Cem Karan

[toc] | [prev] | [next] | [standalone]

#86106

Cem Karan <cfkaran2@gmail.com>:

> On Feb 21, 2015, at 12:08 PM, Marko Rauhamaa <marko@pacujo.net> wrote:
>> Maybe the logic of the receiving object isn't prepared for the callback
>> anymore after an intervening event.
>> 
>> The problem then, of course, is in the logic and not in the callbacks.
>
> This was PRECISELY the situation I was thinking about. My hope was to
> make the callback mechanism slightly less surprising by allowing the
> user to track them, releasing them when they aren't needed without
> having to figure out where the callbacks were registered. However, it
> appears I'm making things more surprising rather than less.

When dealing with callbacks, my advice is to create your objects as
explicit finite state machines. Don't try to encode the object state
implicitly or indirectly. Rather, give each and every state a symbolic
name and log the state transitions for troubleshooting.

Your callbacks should then consider what to do in each state. There are
different ways to express this in Python, but it always boils down to a
state/transition matrix.

Callbacks sometimes cannot be canceled after they have been committed to
and have been shipped to the event pipeline. Then, the receiving object
must brace itself for the impending spurious callback.


Marko

[toc] | [prev] | [next] | [standalone]

#86123

On Feb 22, 2015, at 7:46 AM, Marko Rauhamaa <marko@pacujo.net> wrote:

> Cem Karan <cfkaran2@gmail.com>:
> 
>> On Feb 21, 2015, at 12:08 PM, Marko Rauhamaa <marko@pacujo.net> wrote:
>>> Maybe the logic of the receiving object isn't prepared for the callback
>>> anymore after an intervening event.
>>> 
>>> The problem then, of course, is in the logic and not in the callbacks.
>> 
>> This was PRECISELY the situation I was thinking about. My hope was to
>> make the callback mechanism slightly less surprising by allowing the
>> user to track them, releasing them when they aren't needed without
>> having to figure out where the callbacks were registered. However, it
>> appears I'm making things more surprising rather than less.
> 
> When dealing with callbacks, my advice is to create your objects as
> explicit finite state machines. Don't try to encode the object state
> implicitly or indirectly. Rather, give each and every state a symbolic
> name and log the state transitions for troubleshooting.
> 
> Your callbacks should then consider what to do in each state. There are
> different ways to express this in Python, but it always boils down to a
> state/transition matrix.
> 
> Callbacks sometimes cannot be canceled after they have been committed to
> and have been shipped to the event pipeline. Then, the receiving object
> must brace itself for the impending spurious callback.

Nononono, I'm NOT encoding anything implicitly!  As Frank mentioned earlier, this is more of a pub/sub problem.  E.g., 'USB dongle has gotten plugged in', or 'key has been pressed'.  The user code needs to decide what to do next, the library code provides a nice, clean interface to some potentially weird hardware.

Thanks,
Cem Karan

[toc] | [prev] | [next] | [standalone]

#86107

In a message of Sun, 22 Feb 2015 07:16:14 -0500, Cem Karan writes:

>This was PRECISELY the situation I was thinking about.  My hope was
>to make the callback mechanism slightly less surprising by allowing
>the user to track them, releasing them when they aren't needed
>without having to figure out where the callbacks were registered.
>However, it appears I'm making things more surprising rather than
>less.

You may be able to accomplish your goal by using a Queue with a
producer/consumer model.
see: http://stackoverflow.com/questions/9968592/turn-functions-with-a-callback-into-python-generators

especially the bottom of that.

I haven't run the code, but it looks mostly reasonable, except that
you do not want to rely on the Queue maxsize being 1 here, and
indeed, I almost always want a bigger Queue  in any case.  Use
Queue.task_done if blocking the producer features in your design.

The problem that you are up against is that callbacks are inherantly
confusing, even to programmers who are learning about them for the
first time.  They don't fit people's internal model of 'how code works'.
There isn't a whole lot one can do about that except to
try to make the magic do as little as possible, so that more of the
code works 'the way people expect'.

Laura

[toc] | [prev] | [next] | [standalone]

#86121

On Feb 22, 2015, at 7:52 AM, Laura Creighton <lac@openend.se> wrote:

> In a message of Sun, 22 Feb 2015 07:16:14 -0500, Cem Karan writes:
> 
>> This was PRECISELY the situation I was thinking about.  My hope was
>> to make the callback mechanism slightly less surprising by allowing
>> the user to track them, releasing them when they aren't needed
>> without having to figure out where the callbacks were registered.
>> However, it appears I'm making things more surprising rather than
>> less.
> 
> You may be able to accomplish your goal by using a Queue with a
> producer/consumer model.
> see: http://stackoverflow.com/questions/9968592/turn-functions-with-a-callback-into-python-generators
> 
> especially the bottom of that.
> 
> I haven't run the code, but it looks mostly reasonable, except that
> you do not want to rely on the Queue maxsize being 1 here, and
> indeed, I almost always want a bigger Queue  in any case.  Use
> Queue.task_done if blocking the producer features in your design.
> 
> The problem that you are up against is that callbacks are inherantly
> confusing, even to programmers who are learning about them for the
> first time.  They don't fit people's internal model of 'how code works'.
> There isn't a whole lot one can do about that except to
> try to make the magic do as little as possible, so that more of the
> code works 'the way people expect'.

I think what you're suggesting is that library users register a Queue instead of a callback, correct?  The problem is that I'll then have a strong reference to the Queue, which means I'll be pumping events into it after the user code has gone away.  I was hoping to solve the problem of forgotten registrations in the library.

Thanks,
Cem Karan

[toc] | [prev] | [next] | [standalone]

#86083

"Steven D'Aprano" <steve+comp.lang.python@pearwood.info> wrote in message 
news:54e8af1b$0$12976$c3e8da3$5496439d@news.astraweb.com...
> Frank Millman wrote:
>
>> I tried something similar a while ago, and I did find a gotcha.
>>
>> The problem lies in this phrase - "if they are no longer alive, they are
>> automatically removed from the WeakSet, preventing me from accidentally
>> calling them when they are dead."
>>
>> I found that the reference was not removed immediately, but was waiting 
>> to
>> be garbage collected. During that window, I could call the callback, 
>> which
>> resulted in an error.
>
> I don't understand how this could possibly work. (Or fail to work, as the
> case may be.)
>
> If the callback has been garbage collected, then you cannot call it, 
> because
> you don't have any references to it and so cannot refer to it in any way.
>
> If the callback has *not* been garbage collected, then you can safely call
> it. You have a reference to the callback, therefore it exists. (If Python
> ever garbage collects an object that still has references to it, that 
> would
> be a critical bug, and you would likely get some sort of seg fault).
>
> The only thing I can think of is you have a situation where your callback
> refers to another object, B, via a weak reference. Once all the regular
> strong references to the callback and B are gone, theoretically you could
> have a race condition where the callback is waiting to be garbage 
> collected
> but B has already been garbage collected. If, in that window, you call the
> callback, *and* if the callback fails to correctly check that the weak
> reference to B still exists, then you could get a Python exception.
>
> The solution is simple: anytime you have a weak reference, you must always
> check it before you use it.
>
> Other than that, I cannot see how calling a function which has *not* yet
> been garbage collected can fail, just because the only reference still
> existing is a weak reference.
>

You are right. I tried to reproduce the problem and I can't.

Before describing what I think was happening, I want to clarify something.

Most of this thread uses the word 'callback' in the sense of an 
'asynchronous' scenario - the caller wants something to happen some time in 
the future, and then forget about it, but it is important that it does 
actually happen eventually.

That is not what I was doing, and it is not what I thought the OP was asking 
for.

"In order to inform users that certain bits of state have changed, I require 
them to register a callback with my code."

This sounds to me like a pub/sub scenario. When a 'listener' object comes 
into existence it is passed a reference to a 'controller' object that holds 
state. It wants to be informed when the state changes, so it registers a 
callback function with the controller. When the controller detects a change 
in state, it calls all the callback functions, thereby notifying each 
listener. When the listener goes out of scope, it is important that it 
deregisters with the controller.

Now back to my scenario. You are right that so long as the controller 
maintains a reference to the callback function, the listener cannot be 
garbage collected, and therefore the callback will always succeed.

As far as I can remember, I had a situation where the listener used the 
information to pass the information to a gui on a client. When the listener 
was no longer required, a close() fiunction was called which cleaned up and 
closed connections. When the callback was called after the listener was 
closed, whatever it was trying to do failed (I forget the details).

Hope this makes sense.

Frank

[toc] | [prev] | [next] | [standalone]

#86092

Frank Millman wrote:
> "In order to inform users that certain bits of state have changed, I require 
> them to register a callback with my code."
> 
> This sounds to me like a pub/sub scenario. When a 'listener' object comes 
> into existence it is passed a reference to a 'controller' object that holds 
> state. It wants to be informed when the state changes, so it registers a 
> callback function with the controller.

Perhaps instead of registering a callback function, you
should be registering the listener object together with
a method name.

You can then keep a weak reference to the listener object,
since if it is no longer referenced elsewhere, it presumably
no longer needs to be notified of anything.

-- 
Greg

[toc] | [prev] | [next] | [standalone]

#86124

On Feb 22, 2015, at 5:15 AM, Gregory Ewing <greg.ewing@canterbury.ac.nz> wrote:

> Frank Millman wrote:
>> "In order to inform users that certain bits of state have changed, I require them to register a callback with my code."
>> This sounds to me like a pub/sub scenario. When a 'listener' object comes into existence it is passed a reference to a 'controller' object that holds state. It wants to be informed when the state changes, so it registers a callback function with the controller.
> 
> Perhaps instead of registering a callback function, you
> should be registering the listener object together with
> a method name.
> 
> You can then keep a weak reference to the listener object,
> since if it is no longer referenced elsewhere, it presumably
> no longer needs to be notified of anything.

I see what you're saying, but I don't think it gains us too much.  If I store an object and an unbound method of the object, or if I store the bound method directly, I suspect it will yield approximately the same results.

Thanks,
Cem Karan

[toc] | [prev] | [next] | [standalone]

#86296

Cem Karan wrote:
> On Feb 22, 2015, at 5:15 AM, Gregory Ewing <greg.ewing@canterbury.ac.nz>
> wrote:
> 
>> Perhaps instead of registering a callback function, you should be
>> registering the listener object together with a method name.
> 
> I see what you're saying, but I don't think it gains us too much.  If I store
> an object and an unbound method of the object, or if I store the bound method
> directly, I suspect it will yield approximately the same results.

It would be weird and unpythonic to have to register both
an object and an unbound method, and if you use a bound
method you can't keep a weak reference to it.

-- 
Greg

[toc] | [prev] | [next] | [standalone]

#86313

I'm combining two messages into one, 

On Feb 24, 2015, at 12:29 AM, random832@fastmail.us wrote:

> On Tue, Feb 24, 2015, at 00:20, Gregory Ewing wrote:
>> Cem Karan wrote:
>>> I tend to structure my code as a tree or DAG of objects.  The owner refers to
>>> the owned object, but the owned object has no reference to its owner.  With
>>> callbacks, you get cycles, where the owned owns the owner.
>> 
>> This is why I suggested registering a listener object
>> plus a method name instead of a callback. It avoids that
>> reference cycle, because there is no long-lived callback
>> object keeping a reference to the listener.
> 
> How does that help? Everywhere you would have had a reference to the
> "callback object", you now have a reference to the listener object.
> You're just shuffling deck chairs around: if B shouldn't reference A
> because A owns B, then removing C from the B->C->A reference chain does
> nothing to fix this.

On Feb 24, 2015, at 12:45 AM, Gregory Ewing <greg.ewing@canterbury.ac.nz> wrote:

> Cem Karan wrote:
>> On Feb 22, 2015, at 5:15 AM, Gregory Ewing <greg.ewing@canterbury.ac.nz>
>> wrote:
>>> Perhaps instead of registering a callback function, you should be
>>> registering the listener object together with a method name.
>> I see what you're saying, but I don't think it gains us too much.  If I store
>> an object and an unbound method of the object, or if I store the bound method
>> directly, I suspect it will yield approximately the same results.
> 
> It would be weird and unpythonic to have to register both
> an object and an unbound method, and if you use a bound
> method you can't keep a weak reference to it.


Greg, random832 said what I was thinking earlier, that you've only increased the diameter of your cycle without actually fixing it.  Can you give a code example where your method breaks the cycle entirely?

Thanks,
Cem Karan

[toc] | [prev] | [next] | [standalone]

#86198

On Sun, Feb 22, 2015 at 7:22 AM, Cem Karan <cfkaran2@gmail.com> wrote:
>
> On Feb 22, 2015, at 5:15 AM, Gregory Ewing <greg.ewing@canterbury.ac.nz> wrote:
>
>> Frank Millman wrote:
>>> "In order to inform users that certain bits of state have changed, I require them to register a callback with my code."
>>> This sounds to me like a pub/sub scenario. When a 'listener' object comes into existence it is passed a reference to a 'controller' object that holds state. It wants to be informed when the state changes, so it registers a callback function with the controller.
>>
>> Perhaps instead of registering a callback function, you
>> should be registering the listener object together with
>> a method name.
>>
>> You can then keep a weak reference to the listener object,
>> since if it is no longer referenced elsewhere, it presumably
>> no longer needs to be notified of anything.
>
> I see what you're saying, but I don't think it gains us too much.  If I store an object and an unbound method of the object, or if I store the bound method directly, I suspect it will yield approximately the same results.

Well, it ties the weak ref to the lifetime of the object owning the
callback rather than to the lifetime of the potentially unreferenced
callback itself. I'm not fond of the scheme though because it forces
the callback to be a method, and I'd prefer not to make that
assumption.

Also, I just noticed that Python 3.4 adds a weakref.WeakMethod class
that solves the problem for the bound method case. That still leaves
the closure and lambda cases, but here's a thought: add an optional
argument to the callback registration method that specifies what
object to tie the weak ref to. Something like:

class Listenable:
    def __init__(self):
        self._callbacks = weakref.WeakKeyDictionary()

    def listen(self, callback, owner=None):
        if owner is None:
            if isinstance(callback, types.MethodType):
                owner = weakref.WeakMethod(callback)
            else:
                owner = callback
        self._callbacks.setdefault(owner, []).append(callback)

    def do_callbacks(self, message):
        for callbacks in self._callbacks.values():
            for callback in callbacks:
                callback(message)

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web