Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #86035 > unrolled thread

Re: Design thought for callbacks

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Design thought for callbacks Cem Karan <cfkaran2@gmail.com> - 2015-02-21 09:07 -0500
    Re: Design thought for callbacks Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-02-22 04:27 +1100
      Re: Design thought for callbacks Cem Karan <cfkaran2@gmail.com> - 2015-02-22 08:13 -0500
        Re: Design thought for callbacks Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-02-24 00:31 +1100
      Re: Design thought for callbacks Ethan Furman <ethan@stoneleaf.us> - 2015-02-22 13:02 -0800
      Re: Design thought for callbacks Cem Karan <cfkaran2@gmail.com> - 2015-02-22 16:17 -0500
        Re: Design thought for callbacks Marko Rauhamaa <marko@pacujo.net> - 2015-02-22 23:34 +0200
          Re: Design thought for callbacks Cem Karan <cfkaran2@gmail.com> - 2015-02-22 17:09 -0500
          Re: Design thought for callbacks Laura Creighton <lac@openend.se> - 2015-02-22 23:29 +0100
          Re: Design thought for callbacks Chris Angelico <rosuav@gmail.com> - 2015-02-23 09:41 +1100
          Re: Design thought for callbacks Cem Karan <cfkaran2@gmail.com> - 2015-02-23 06:47 -0500
          Re: Design thought for callbacks "Frank Millman" <frank@chagford.com> - 2015-02-23 14:29 +0200
          Re: Design thought for callbacks Cem Karan <cfkaran2@gmail.com> - 2015-02-24 06:00 -0500

#86035 — Re: Design thought for callbacks

On Feb 21, 2015, at 8:15 AM, Chris Angelico <rosuav@gmail.com> wrote:

> On Sun, Feb 22, 2015 at 12:13 AM, Cem Karan <cfkaran2@gmail.com> wrote:
>> OK, so it would violate the principle of least surprise for you.  Interesting.  Is this a general pattern in python?  That is, callbacks are owned by what they are registered with?
>> 
>> In the end, I want to make a library that offers as few surprises to the user as possible, and no matter how I think about callbacks, they are surprising to me.  If callbacks are strongly-held, then calling 'del foo' on a callable object may not make it go away, which can lead to weird and nasty situations.  Weakly-held callbacks mean that I (as the programmer), know that objects will go away after the next garbage collection (see Frank's earlier message), so I don't get 'dead' callbacks coming back from the grave to haunt me.
>> 
>> So, what's the consensus on the list, strongly-held callbacks, or weakly-held ones?
> 
> I don't know about Python specifically, but it's certainly a general
> pattern in other languages. They most definitely are owned, and it's
> the only model that makes sense when you use closures (which won't
> have any other references anywhere).

I agree about closures; its the only way they could work.  When I was originally thinking about the library, I was trying to include all types of callbacks, including closures and callable objects.  The callable objects may pass themselves, or one of their methods to the library, or may do something really weird.  

Although I just realized that closures may cause another problem.  In my code, I expect that many different callbacks can be registered for the same event.  Unregistering means you request to be unregistered for the event. How do you do that with a closure?  Aren't they anonymous?

> If you're expecting 'del foo' to destroy the object, then you have a
> bigger problem than callbacks, because that's simply not how Python
> works. You can't _ever_ assume that deleting something from your local
> namespace will destroy the object, because there can always be more
> references. So maybe you need a more clear way of saying "I'm done
> with this, get rid of it".

Agreed about 'del', and I don't assume that the object goes away at the point.  The problem is debugging and determining WHY your object is still around.  I know a combination of logging and gc.get_referrers() will probably help you figure out why something is still around, but I'm trying to avoid that headache.  

I guess the real problem is how this creates cycles in the call graph.  User code effectively owns the library code, which via callbacks owns the user code.  I have no idea what the best point the cycle is to break it, and not surprise someone down the road.  The only idea I have is to redesign the library a little, and make anything that accepts a callback actually be a subclass of collections.abc.Container, or even collections.abc.MutableSet.  That makes it very obvious that the object owns the callback, and that you will need to remove your object to unregister it.  The only problem is how to handle closures; since they are anonymous, how do you decide which one to remove?

Thanks,
Cem Karan

[toc] | [next] | [standalone]

#86058

Cem Karan wrote:

> 
> On Feb 21, 2015, at 8:15 AM, Chris Angelico <rosuav@gmail.com> wrote:
> 
>> On Sun, Feb 22, 2015 at 12:13 AM, Cem Karan <cfkaran2@gmail.com> wrote:
>>> OK, so it would violate the principle of least surprise for you. 
>>> Interesting.  Is this a general pattern in python?  That is, callbacks
>>> are owned by what they are registered with?
>>> 
>>> In the end, I want to make a library that offers as few surprises to the
>>> user as possible, and no matter how I think about callbacks, they are
>>> surprising to me.  If callbacks are strongly-held, then calling 'del
>>> foo' on a callable object may not make it go away, which can lead to
>>> weird and nasty situations.

How?

The whole point of callbacks is that you hand over responsibility to another
piece of code, and then forget about your callback. The library will call
it, when and if necessary, and when the library no longer needs your
callback, it is free to throw it away. (If I wish the callback to survive
beyond the lifetime of your library's use of it, I have to keep a reference
to the function.)


>>> Weakly-held callbacks mean that I (as the 
>>> programmer), know that objects will go away after the next garbage
>>> collection (see Frank's earlier message), so I don't get 'dead'
>>> callbacks coming back from the grave to haunt me.

I'm afraid this makes no sense to me. Can you explain, or better still
demonstrate, a scenario where "dead callbacks rise from the grave", so to
speak?


>>> So, what's the consensus on the list, strongly-held callbacks, or
>>> weakly-held ones?
>> 
>> I don't know about Python specifically, but it's certainly a general
>> pattern in other languages. They most definitely are owned, and it's
>> the only model that makes sense when you use closures (which won't
>> have any other references anywhere).
> 
> I agree about closures; its the only way they could work.

*scratches head* There's nothing special about closures. You can assign them
to a name like any other object.

def make_closure():
    x = 23
    def closure():
        return x + 1
    return closure

func = make_closure()

Now you can register func as a callback, and de-register it when your done:

register(func)
unregister(func)


Of course, if you thrown away your reference to func, you have no (easy) way
of de-registering it. That's no different to any other object which is
registered by identity. (Registering functions by name is a bad idea, since
multiple functions can have the same name.)

As an alternative, your callback registration function might return a ticket
for the function:

ticket = register(func)
del func
unregister(ticket)

but that strikes me as over-kill. And of course, the simplest ticket is to
return the function itself :-)



> When I was 
> originally thinking about the library, I was trying to include all types
> of callbacks, including closures and callable objects.  The callable
> objects may pass themselves, or one of their methods to the library, or
> may do something really weird.

I don't think they can do anything too weird. They have to pass a callable
object. Your library just calls that object. You shouldn't need to care
whether it is a function, a method, a type, a callable instance, or
something else. You just call it, and when you're done calling it forever,
you just throw it away.


> Although I just realized that closures may cause another problem.  In my
> code, I expect that many different callbacks can be registered for the
> same event.  Unregistering means you request to be unregistered for the
> event. How do you do that with a closure?  Aren't they anonymous?

Not unless you create them using lambda. Using the make_closure function
above:


py> func = make_closure()
py> func.__name__
'closure'

Of course, if you call make_closure twice, both functions will have the same
internal name. You can set the function __name__ and __qualname__ to fix
that. This is how the functools.wraps decorator works.

But that's a red herring. Don't register functions by name! Not all callable
objects have names, and those that do, you may have multiple *distinct*
callbacks with the same name.

There are two reasonable approaches: unregister by identity, or by returning
a ticket which uniquely identifies the callback. The user is responsible
for keeping track of their own ticket. If I lose it, I can't unregister my
callback any more. So sad, sucks to be me.


The simplest possible identity-based scheme would be something like this:


# don't hate me for using a global variable
CALLBACKS = []

def register(func):
    if func not in CALLBACKS:
         CALLBACKS.append(func)

def unregister(func):
    try:
        CALLBACKS.remove(func)
    except ValueError:
        pass


That's probably a bit too simple, since it won't behave as expected with
bound methods. The problem is that bound methods are generated on the fly,
so this won't work:

register(instance.spam)
# later
unregister(instance.spam)  # a different instance!

I would have to do this:

bound_method = instance.spam
register(bound_method)
unregister(bound_method)


But a more sophisticated unregister function should work:

# Untested
def unregister(func):
    for i, f in enumerate(CALLBACKS):
        if (f is func) or (isinstance(f, types.MethodType) 
                and f.__wrapped__ is func):
            del CALLBACKS[i]
            return


The simplest possible ticket-based system is probably something like this:

CALLBACKS = {}
NEXT_TICKET = 1

def register(func):
    global NEXT_TICKET
    ticket = NEXT_TICKET
    NEXT_TICKET += 1
    callbacks[ticket] = func
    return ticket

def unregister(ticket):
    if ticket in CALLBACKS:
        del CALLBACKS[ticket]




>> If you're expecting 'del foo' to destroy the object, then you have a
>> bigger problem than callbacks, because that's simply not how Python
>> works. You can't _ever_ assume that deleting something from your local
>> namespace will destroy the object, because there can always be more
>> references. So maybe you need a more clear way of saying "I'm done
>> with this, get rid of it".
> 
> Agreed about 'del', and I don't assume that the object goes away at the
> point.  The problem is debugging and determining WHY your object is still
> around.  I know a combination of logging and gc.get_referrers() will
> probably help you figure out why something is still around, but I'm trying
> to avoid that headache.

Why do you care? Surely all your library should care about is whether or not
they have a reference to the callback.If they do, they should call it (when
appropriate). If they don't, they aren't responsible for it.


> I guess the real problem is how this creates cycles in the call graph. 
> User code effectively owns the library code, which via callbacks owns the
> user code.  I have no idea what the best point the cycle is to break it,
> and not surprise someone down the road.  The only idea I have is to
> redesign the library a little, and make anything that accepts a callback
> actually be a subclass of collections.abc.Container, or even
> collections.abc.MutableSet.  That makes it very obvious that the object
> owns the callback, and that you will need to remove your object to
> unregister it.

My brain hurts from the complexity of your solution. What is the actual
problem you are trying to solve? I would like to see an example of an
actual failure before trying to solve a purely theoretical failure mode.

If I register something as a callback, I expect that callback will stay
alive for as long as the callbacks are needed. If I might want to
unregister it, then I have to keep a reference to the function, otherwise
how will I know what I am unregistering?

# this makes no sense and cannot work
register(func)
del func
# later
unregister(func)

So if I do that, (1) it won't work; (2) I'll probably get an exception; (3)
the solution is "don't do that"; and (4) solving this problem is not YOUR
responsibility.

When your code is done with the callbacks, you can just remove them, no
questions asked. If I still have a reference to the callback, that
reference will still be valid no matter what you do. If I don't have a
reference to it, presumably that's because I don't need it any more. I
can't access the callback anyway.


> The only problem is how to handle closures; since they are 
> anonymous, how do you decide which one to remove?

You identify them by identity, or by a ticket, the same as for any other
object.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#86110

On Feb 21, 2015, at 12:27 PM, Steven D'Aprano <steve+comp.lang.python@pearwood.info> wrote:

> Cem Karan wrote:
> 
>> 
>> On Feb 21, 2015, at 8:15 AM, Chris Angelico <rosuav@gmail.com> wrote:
>> 
>>> On Sun, Feb 22, 2015 at 12:13 AM, Cem Karan <cfkaran2@gmail.com> wrote:
>>>> OK, so it would violate the principle of least surprise for you. 
>>>> Interesting.  Is this a general pattern in python?  That is, callbacks
>>>> are owned by what they are registered with?
>>>> 
>>>> In the end, I want to make a library that offers as few surprises to the
>>>> user as possible, and no matter how I think about callbacks, they are
>>>> surprising to me.  If callbacks are strongly-held, then calling 'del
>>>> foo' on a callable object may not make it go away, which can lead to
>>>> weird and nasty situations.
> 
> How?
> 
> The whole point of callbacks is that you hand over responsibility to another
> piece of code, and then forget about your callback. The library will call
> it, when and if necessary, and when the library no longer needs your
> callback, it is free to throw it away. (If I wish the callback to survive
> beyond the lifetime of your library's use of it, I have to keep a reference
> to the function.)

Marko mentioned it earlier; if you think you've gotten rid of all references to some chunk of code, and it is still alive afterwards, that can be surprising.

>>>> Weakly-held callbacks mean that I (as the 
>>>> programmer), know that objects will go away after the next garbage
>>>> collection (see Frank's earlier message), so I don't get 'dead'
>>>> callbacks coming back from the grave to haunt me.
> 
> I'm afraid this makes no sense to me. Can you explain, or better still
> demonstrate, a scenario where "dead callbacks rise from the grave", so to
> speak?

"""
#! /usr/bin/env python

class Callback_object(object):
    def __init__(self, msg):
        self._msg = msg
    def callback(self, stuff):
        print("From {0!s}: {1!s}".format(self._msg, stuff))

class Fake_library(object):
    def __init__(self):
        self._callbacks = list()
    def register_callback(self, callback):
        self._callbacks.append(callback)
    def execute_callbacks(self):
        for thing in self._callbacks:
            thing('Surprise!')

if __name__ == "__main__":
    foo = Callback_object("Evil Zombie")
    lib = Fake_library()
    lib.register_callback(foo.callback)

    # Way later, after the user forgot all about the callback above
    foo = Callback_object("Your Significant Other")
    lib.register_callback(foo.callback)

    # And finally getting around to running all those callbacks.
    lib.execute_callbacks()
"""

Output:
From Evil Zombie: Surprise!
From Your Significant Other: Surprise!

In this case, the user made an error (just as Marko said in his earlier message), and forgot about the callback he registered with the library.  The callback isn't really rising from the dead; as you say, either its been garbage collected, or it hasn't been.  However, you may not be ready for a callback to be called at that moment in time, which means you're surprised by unexpected behavior.

>>>> So, what's the consensus on the list, strongly-held callbacks, or
>>>> weakly-held ones?
>>> 
>>> I don't know about Python specifically, but it's certainly a general
>>> pattern in other languages. They most definitely are owned, and it's
>>> the only model that makes sense when you use closures (which won't
>>> have any other references anywhere).
>> 
>> I agree about closures; its the only way they could work.
> 
> *scratches head* There's nothing special about closures. You can assign them
> to a name like any other object.
> 
> def make_closure():
>    x = 23
>    def closure():
>        return x + 1
>    return closure
> 
> func = make_closure()
> 
> Now you can register func as a callback, and de-register it when your done:
> 
> register(func)
> unregister(func)
> 
> 
> Of course, if you thrown away your reference to func, you have no (easy) way
> of de-registering it. That's no different to any other object which is
> registered by identity. (Registering functions by name is a bad idea, since
> multiple functions can have the same name.)
> 
> As an alternative, your callback registration function might return a ticket
> for the function:
> 
> ticket = register(func)
> del func
> unregister(ticket)
> 
> but that strikes me as over-kill. And of course, the simplest ticket is to
> return the function itself :-)

Agreed on all points; closures are just ordinary objects.  The only difference (in my opinion) is that they are 'fire and forget'; if you are registering or tracking them then you've kind of defeated the purpose.  THAT is what I meant about how you handle closures.

> 
>> When I was 
>> originally thinking about the library, I was trying to include all types
>> of callbacks, including closures and callable objects.  The callable
>> objects may pass themselves, or one of their methods to the library, or
>> may do something really weird.
> 
> I don't think they can do anything too weird. They have to pass a callable
> object. Your library just calls that object. You shouldn't need to care
> whether it is a function, a method, a type, a callable instance, or
> something else. You just call it, and when you're done calling it forever,
> you just throw it away.

That doesn't quite solve the problem, but it comes close.  The headache (as shown in my earlier code) is that you think you've gotten rid of something before it is called, but it turns out you haven't.  I'm starting to think that there isn't a solution to this other than telling the programmer "Don't do that".  

>> Although I just realized that closures may cause another problem.  In my
>> code, I expect that many different callbacks can be registered for the
>> same event.  Unregistering means you request to be unregistered for the
>> event. How do you do that with a closure?  Aren't they anonymous?
> 
> Not unless you create them using lambda. Using the make_closure function
> above:
> 
> 
> py> func = make_closure()
> py> func.__name__
> 'closure'
> 
> Of course, if you call make_closure twice, both functions will have the same
> internal name. You can set the function __name__ and __qualname__ to fix
> that. This is how the functools.wraps decorator works.
> 
> But that's a red herring. Don't register functions by name! Not all callable
> objects have names, and those that do, you may have multiple *distinct*
> callbacks with the same name.
> 
> There are two reasonable approaches: unregister by identity, or by returning
> a ticket which uniquely identifies the callback. The user is responsible
> for keeping track of their own ticket. If I lose it, I can't unregister my
> callback any more. So sad, sucks to be me.
> 
> 
> The simplest possible identity-based scheme would be something like this:
> 
> 
> # don't hate me for using a global variable
> CALLBACKS = []
> 
> def register(func):
>    if func not in CALLBACKS:
>         CALLBACKS.append(func)
> 
> def unregister(func):
>    try:
>        CALLBACKS.remove(func)
>    except ValueError:
>        pass
> 
> 
> That's probably a bit too simple, since it won't behave as expected with
> bound methods. The problem is that bound methods are generated on the fly,
> so this won't work:
> 
> register(instance.spam)
> # later
> unregister(instance.spam)  # a different instance!
> 
> I would have to do this:
> 
> bound_method = instance.spam
> register(bound_method)
> unregister(bound_method)
> 
> 
> But a more sophisticated unregister function should work:
> 
> # Untested
> def unregister(func):
>    for i, f in enumerate(CALLBACKS):
>        if (f is func) or (isinstance(f, types.MethodType) 
>                and f.__wrapped__ is func):
>            del CALLBACKS[i]
>            return

Are you sure about that?  I just tested out the following code, and it appears to work correctly:

"""
#! /usr/bin/env python

class Callback_object(object):
    def __init__(self, msg):
        self._msg = msg
    def callback(self, stuff):
        print("From {0!s}: {1!s}".format(self._msg, stuff))

class Fake_library(object):
    def __init__(self):
        self._callbacks = set()
    def register_callback(self, callback):
        self._callbacks.add(callback)
    def unregister_callback(self, callback):
        self._callbacks.discard(callback)
    def execute_callbacks(self):
        for thing in self._callbacks:
            thing('Surprise!')

if __name__ == "__main__":
    foo = Callback_object("Evil Zombie")
    lib = Fake_library()
    lib.register_callback(foo.callback)
    lib.unregister_callback(foo.callback)
    lib.execute_callbacks()
"""

I'll admit though, I don't know if it worked because I got lucky, or if python guarantees it works...

> The simplest possible ticket-based system is probably something like this:
> 
> CALLBACKS = {}
> NEXT_TICKET = 1
> 
> def register(func):
>    global NEXT_TICKET
>    ticket = NEXT_TICKET
>    NEXT_TICKET += 1
>    callbacks[ticket] = func
>    return ticket
> 
> def unregister(ticket):
>    if ticket in CALLBACKS:
>        del CALLBACKS[ticket]
> 

I'd probably go with something similar to this, except that I'd use UUIDs for the tickets.  I know me and my users, and somewhere along the line I'd use a ticket to unregister from the wrong callback dictionary!

>>> If you're expecting 'del foo' to destroy the object, then you have a
>>> bigger problem than callbacks, because that's simply not how Python
>>> works. You can't _ever_ assume that deleting something from your local
>>> namespace will destroy the object, because there can always be more
>>> references. So maybe you need a more clear way of saying "I'm done
>>> with this, get rid of it".
>> 
>> Agreed about 'del', and I don't assume that the object goes away at the
>> point.  The problem is debugging and determining WHY your object is still
>> around.  I know a combination of logging and gc.get_referrers() will
>> probably help you figure out why something is still around, but I'm trying
>> to avoid that headache.
> 
> Why do you care? Surely all your library should care about is whether or not
> they have a reference to the callback.If they do, they should call it (when
> appropriate). If they don't, they aren't responsible for it.

I care because I care about anyone using my code.  Telling them 'tough, its your problem' doesn't get you many friends.  Making a library that performs as expected and where it is easy to debug what went wrong makes everyone (including me!) happy.

>> I guess the real problem is how this creates cycles in the call graph. 
>> User code effectively owns the library code, which via callbacks owns the
>> user code.  I have no idea what the best point the cycle is to break it,
>> and not surprise someone down the road.  The only idea I have is to
>> redesign the library a little, and make anything that accepts a callback
>> actually be a subclass of collections.abc.Container, or even
>> collections.abc.MutableSet.  That makes it very obvious that the object
>> owns the callback, and that you will need to remove your object to
>> unregister it.
> 
> My brain hurts from the complexity of your solution. What is the actual
> problem you are trying to solve? I would like to see an example of an
> actual failure before trying to solve a purely theoretical failure mode.

If I'm making your head hurt, then my solution is a bad solution.  The whole reason I started this discussion was to figure out if an alternative method would make more sense to my potential endusers (fellow programmers).  If strongly-held callbacks cause fewer headaches, then that is what I'll go with.

Thanks,
Cem Karan

[toc] | [prev] | [next] | [standalone]

#86217

Cem Karan wrote:

> 
> On Feb 21, 2015, at 12:27 PM, Steven D'Aprano
> <steve+comp.lang.python@pearwood.info> wrote:

>> The simplest possible identity-based scheme would be something like this:
>> 
>> 
>> # don't hate me for using a global variable
>> CALLBACKS = []
>> 
>> def register(func):
>>    if func not in CALLBACKS:
>>         CALLBACKS.append(func)
>> 
>> def unregister(func):
>>    try:
>>        CALLBACKS.remove(func)
>>    except ValueError:
>>        pass

Oops! That's not identity-based, that's *equality* based.

Both the `in` operator and the list `remove` method implicitly perform
equality checks, not identity checks. Which means that they will work with
methods, since method equality compares against the underlying function,
which is the same:


py> class Spam(object):
...     def method(self):
...             pass
...
py> s = Spam()
py> a = s.method
py> b = s.method
py> a is b
False
py> a == b
True
py> a.__func__ is b.__func__
True


So, when I say this:

>> That's probably a bit too simple, since it won't behave as expected with
>> bound methods. The problem is that bound methods are generated on the
>> fly, so this won't work:

I was mistaken.


> Are you sure about that?  I just tested out the following code, and it
> appears to work correctly:

You are correct.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#86154

[Multipart message — attachments visible in raw view] — view raw

On 02/22/2015 05:13 AM, Cem Karan wrote:

> Output:
> From Evil Zombie: Surprise!
> From Your Significant Other: Surprise!
> 
> In this case, the user made an error (just as Marko said in his earlier message),
> and forgot about the callback he registered with the library.  The callback isn't
> really rising from the dead; as you say, either its been garbage collected, or it
> hasn't been.  However, you may not be ready for a callback to be called at that
> moment in time, which means you're surprised by unexpected behavior.

But the unexpected behavior is not a problem with Python, nor with your library -- it's a bug in the fellow-programmer's
code, and you can't (or at least shouldn't) try to prevent those kinds of bugs from manifesting -- they'll just get
bitten somewhere else by the same bug.

--
~Ethan~

[toc] | [prev] | [next] | [standalone]

#86155

On Feb 22, 2015, at 4:02 PM, Ethan Furman <ethan@stoneleaf.us> wrote:

> On 02/22/2015 05:13 AM, Cem Karan wrote:
> 
>> Output:
>> From Evil Zombie: Surprise!
>> From Your Significant Other: Surprise!
>> 
>> In this case, the user made an error (just as Marko said in his earlier message),
>> and forgot about the callback he registered with the library.  The callback isn't
>> really rising from the dead; as you say, either its been garbage collected, or it
>> hasn't been.  However, you may not be ready for a callback to be called at that
>> moment in time, which means you're surprised by unexpected behavior.
> 
> But the unexpected behavior is not a problem with Python, nor with your library -- it's a bug in the fellow-programmer's
> code, and you can't (or at least shouldn't) try to prevent those kinds of bugs from manifesting -- they'll just get
> bitten somewhere else by the same bug.

I agree with you, but until a relatively new programmer has gotten used to what callbacks are and what they imply, I want to make things easy.  For example, if the API subclasses collections.abc.MutableSet, and the documentation states that you can only add callbacks to this particular type of set, then a new programmer will naturally decide that either a) they need to dispose of the set, and if that isn't possible, then b) they need to delete their callback from the set.  It won't occur to them that their live object will just magically 'go away'; its a member of a set!

My goal is to make things as pythonic (whatever that means in this case) and obvious as possible.  Ideally, a novice can more or less guess what will happen with my API without really having to read the documentation on it.  

Thanks,
Cem Karan

[toc] | [prev] | [next] | [standalone]

#86158

Cem Karan <cfkaran2@gmail.com>:

> My goal is to make things as pythonic (whatever that means in this
> case) and obvious as possible. Ideally, a novice can more or less
> guess what will happen with my API without really having to read the
> documentation on it.

If you try to shield your user from the complexities of asynchronous
programming, you will only cause confusion. You will definitely need to
document all nooks and crannies of the semantics of the callback API and
your user will have to pay attention to every detail of your spec.

Your user, whether novice or an expert, will thank you for your
unambiguous specification even if it is complicated.


Marko

[toc] | [prev] | [next] | [standalone]

#86162

On Feb 22, 2015, at 4:34 PM, Marko Rauhamaa <marko@pacujo.net> wrote:

> Cem Karan <cfkaran2@gmail.com>:
> 
>> My goal is to make things as pythonic (whatever that means in this
>> case) and obvious as possible. Ideally, a novice can more or less
>> guess what will happen with my API without really having to read the
>> documentation on it.
> 
> If you try to shield your user from the complexities of asynchronous
> programming, you will only cause confusion. You will definitely need to
> document all nooks and crannies of the semantics of the callback API and
> your user will have to pay attention to every detail of your spec.
> 
> Your user, whether novice or an expert, will thank you for your
> unambiguous specification even if it is complicated.

Documentation is a given; it MUST be there.  That said, documenting something, but still making it surprising, is a bad idea.  For example, several people have been strongly against using a WeakSet to hold callbacks because they expect a library to hold onto callbacks.  If I chose not to do that, and used a WeakSet, then even if I documented it, it would still end up surprising people (and from the sound of it, more people would be surprised than not).

Thanks,
Cem Karan

[toc] | [prev] | [next] | [standalone]

#86166

In a message of Sun, 22 Feb 2015 17:09:01 -0500, Cem Karan writes:

>Documentation is a given; it MUST be there.  That said, documenting
>something, but still making it surprising, is a bad idea.  For
>example, several people have been strongly against using a WeakSet to
>hold callbacks because they expect a library to hold onto callbacks.
>If I chose not to do that, and used a WeakSet, then even if I
>documented it, it would still end up surprising people (and from the
>sound of it, more people would be surprised than not).

>Thanks, Cem Karan

No matter what you do, alas, will surprise the hell out of people
because callbacks do not behave as people expect.  Among people who
have used callbacks, what you are polling is 'what are people
familiar with', and it seems for the people around here, now,
WeakSets are not what they are familiar with.

But that is not so surprising.  How many people use WeakSets for
_anything_?  I've never used them, aside from 'ooh! cool shiny
new language feature!  Let's kick it around the park!'  That people
aren't familiar with WeakSets doesn't mean all that much.

The question I have is does this architecture make things harder,
easier or about the same to debug?  To write tests for? to do Test
Driven Design with?

Laura

[toc] | [prev] | [next] | [standalone]

#86167

On Mon, Feb 23, 2015 at 9:29 AM, Laura Creighton <lac@openend.se> wrote:
> But that is not so surprising.  How many people use WeakSets for
> _anything_?  I've never used them, aside from 'ooh! cool shiny
> new language feature!  Let's kick it around the park!'  That people
> aren't familiar with WeakSets doesn't mean all that much.

I haven't used weak *sets*, but I've used weak *mappings* on occasion.
It's certainly not a common thing, but they have their uses. I have a
MUD which must guarantee that there be no more than one instance of
any given room (identified by a string that looks like a Unix path),
but which will, if it can, flush rooms out of memory when nothing
refers to them. So it has a mapping from the path strings to the
instances, but with weak refs for the instances; if anything else is
referring to that instance (eg a player character in the room), it'll
hang around, and any time anyone else needs that room, they'll get the
same instance back from the mapping; but any time the garbage
collector notices that a room can be disposed of, it will be.

Definitely not common though.

ChrisA

[toc] | [prev] | [next] | [standalone]

#86207

On Feb 22, 2015, at 5:29 PM, Laura Creighton <lac@openend.se> wrote:

> In a message of Sun, 22 Feb 2015 17:09:01 -0500, Cem Karan writes:
> 
>> Documentation is a given; it MUST be there.  That said, documenting
>> something, but still making it surprising, is a bad idea.  For
>> example, several people have been strongly against using a WeakSet to
>> hold callbacks because they expect a library to hold onto callbacks.
>> If I chose not to do that, and used a WeakSet, then even if I
>> documented it, it would still end up surprising people (and from the
>> sound of it, more people would be surprised than not).
> 
>> Thanks, Cem Karan
> 
> No matter what you do, alas, will surprise the hell out of people
> because callbacks do not behave as people expect.  Among people who
> have used callbacks, what you are polling is 'what are people
> familiar with', and it seems for the people around here, now,
> WeakSets are not what they are familiar with.

And that's fine.  I know that regardless of what I do, some people are going to be surprised.  I'm trying to develop APIs that reduce that surprise as far as possible.  That means I can spend more time coding and less time answering questions... :)

> But that is not so surprising.  How many people use WeakSets for
> _anything_?  I've never used them, aside from 'ooh! cool shiny
> new language feature!  Let's kick it around the park!'  That people
> aren't familiar with WeakSets doesn't mean all that much.

Actually, I use them when building caches of stuff, and I use weak references when I have trees of stuff so the child nodes know of, but don't hold onto, their parents.  But I agree with you, there aren't a huge number of use-cases.

> The question I have is does this architecture make things harder,
> easier or about the same to debug?  To write tests for? to do Test
> Driven Design with?

Good questions!  That was why I was asking about 'gotchas' with WeakSets originally.  Honestly, the only way to know for sure would be to write two APIs for doing similar things, and then see how people react to them.  The problem is, how do you set up such a study so it is statistically valid?

Cem

[toc] | [prev] | [next] | [standalone]

#86210

"Cem Karan" <cfkaran2@gmail.com> wrote in message 
news:A3C11A70-5846-4915-BB26-B23793B65670@gmail.com...
>
>
> Good questions!  That was why I was asking about 'gotchas' with WeakSets 
> originally.  Honestly, the only way to know for sure would be to write two 
> APIs for doing similar things, and then see how people react to them.  The 
> problem is, how do you set up such a study so it is statistically valid?
>

Just in case you missed Steven's comment on my 'gotcha', and my reply, it is 
worth repeating that what I reported as a gotcha was not what it seemed.

If you set up the callback as a weakref, and the listening object goes out 
of scope, it will wait to be garbage collected. However, as far as I can 
tell, the weakref is removed at the same time as the object is gc'd, so 
there is no 'window' where the weakref exists but the object it is 
referencing does not exist.

My problem was that I had performed a cleanup operation on the listening 
object before letting it go out of scope, and it was no longer in a valid 
state to deal with the callback, resulting in an error. If you do not have 
that situation, your original idea may well work.

Frank

[toc] | [prev] | [next] | [standalone]

#86312

On Feb 23, 2015, at 7:29 AM, "Frank Millman" <frank@chagford.com> wrote:

> 
> "Cem Karan" <cfkaran2@gmail.com> wrote in message 
> news:A3C11A70-5846-4915-BB26-B23793B65670@gmail.com...
>> 
>> 
>> Good questions!  That was why I was asking about 'gotchas' with WeakSets 
>> originally.  Honestly, the only way to know for sure would be to write two 
>> APIs for doing similar things, and then see how people react to them.  The 
>> problem is, how do you set up such a study so it is statistically valid?
>> 
> 
> Just in case you missed Steven's comment on my 'gotcha', and my reply, it is 
> worth repeating that what I reported as a gotcha was not what it seemed.
> 
> If you set up the callback as a weakref, and the listening object goes out 
> of scope, it will wait to be garbage collected. However, as far as I can 
> tell, the weakref is removed at the same time as the object is gc'd, so 
> there is no 'window' where the weakref exists but the object it is 
> referencing does not exist.
> 
> My problem was that I had performed a cleanup operation on the listening 
> object before letting it go out of scope, and it was no longer in a valid 
> state to deal with the callback, resulting in an error. If you do not have 
> that situation, your original idea may well work.

Thank you Frank, I did read Steve's comment to your reply earlier, but what you said in your original reply made sense to me.  I don't have control over user code.  That means that if someone wants to write code such that they perform some kind of cleanup and are no longer able to handle the callback, they are free to do so.  While I can't prevent this from happening, I can make it as obvious as possible in my code that before you perform any cleanup, you also need to unregister from the library.  That is my main goal in developing pythonic/obvious methods of registering callbacks.

Thanks,
Cem Karan

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web