Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #86028 > unrolled thread

Re: Design thought for callbacks

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Design thought for callbacks Cem Karan <cfkaran2@gmail.com> - 2015-02-21 08:13 -0500
    Re: Design thought for callbacks Grant Edwards <invalid@invalid.invalid> - 2015-02-21 20:57 +0000
      Re: Design thought for callbacks Marko Rauhamaa <marko@pacujo.net> - 2015-02-21 23:57 +0200
        Re: Design thought for callbacks Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-02-22 13:04 +1100
          Re: Design thought for callbacks Chris Angelico <rosuav@gmail.com> - 2015-02-22 13:11 +1100
            Re: Design thought for callbacks Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-02-22 15:38 +1100
              Re: Design thought for callbacks Chris Angelico <rosuav@gmail.com> - 2015-02-22 17:21 +1100
                Re: Design thought for callbacks Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-02-22 21:32 +1100
                  Re: Design thought for callbacks Chris Angelico <rosuav@gmail.com> - 2015-02-22 22:14 +1100
                    Re: Design thought for callbacks Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-02-23 00:45 +1100
                      Re: Design thought for callbacks Chris Angelico <rosuav@gmail.com> - 2015-02-23 01:01 +1100
          Re: Design thought for callbacks Marko Rauhamaa <marko@pacujo.net> - 2015-02-22 09:52 +0200
            Re: Design thought for callbacks Chris Angelico <rosuav@gmail.com> - 2015-02-22 19:17 +1100
              Re: Design thought for callbacks Marko Rauhamaa <marko@pacujo.net> - 2015-02-22 10:34 +0200
                Re: Design thought for callbacks Chris Angelico <rosuav@gmail.com> - 2015-02-22 19:58 +1100
                  Re: Design thought for callbacks Marko Rauhamaa <marko@pacujo.net> - 2015-02-22 11:14 +0200
                    Re: Design thought for callbacks Chris Angelico <rosuav@gmail.com> - 2015-02-22 21:21 +1100
                    Re: Design thought for callbacks Laura Creighton <lac@openend.se> - 2015-02-22 12:28 +0100
                    Re: Design thought for callbacks Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-02-23 00:57 +1100
                      Re: Design thought for callbacks Marko Rauhamaa <marko@pacujo.net> - 2015-02-22 16:06 +0200
      Re: Design thought for callbacks Ian Kelly <ian.g.kelly@gmail.com> - 2015-02-22 00:05 -0700
      Re: Design thought for callbacks Cem Karan <cfkaran2@gmail.com> - 2015-02-22 08:21 -0500
        Re: Design thought for callbacks Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2015-02-24 18:20 +1300
          Re: Design thought for callbacks random832@fastmail.us - 2015-02-24 00:29 -0500
            Re: Design thought for callbacks Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2015-02-25 10:19 +1300
              Re: Design thought for callbacks Cem Karan <cfkaran2@gmail.com> - 2015-02-25 07:50 -0500
                Re: Design thought for callbacks Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2015-02-26 18:36 +1300
                  Re: Design thought for callbacks Cem Karan <cfkaran2@gmail.com> - 2015-02-26 05:58 -0500
                  Re: Design thought for callbacks Ian Kelly <ian.g.kelly@gmail.com> - 2015-02-26 12:54 -0700
                  Re: Design thought for callbacks Ethan Furman <ethan@stoneleaf.us> - 2015-02-26 12:00 -0800
                  Re: Design thought for callbacks Cem Karan <cfkaran2@gmail.com> - 2015-03-02 06:06 -0500
                  Re: Design thought for callbacks Cem Karan <cfkaran2@gmail.com> - 2015-03-02 06:04 -0500
                  Re: Design thought for callbacks Ian Kelly <ian.g.kelly@gmail.com> - 2015-03-02 08:34 -0700
                  Re: Design thought for callbacks Cem Karan <cfkaran2@gmail.com> - 2015-03-08 14:30 -0400

Page 1 of 2  [1] 2  Next page →

#86028 — Re: Design thought for callbacks

On Feb 21, 2015, at 12:42 AM, Chris Angelico <rosuav@gmail.com> wrote:

> On Sat, Feb 21, 2015 at 1:44 PM, Cem Karan <cfkaran2@gmail.com> wrote:
>> In order to inform users that certain bits of state have changed, I require them to register a callback with my code.  The problem is that when I store these callbacks, it naturally creates a strong reference to the objects, which means that if they are deleted without unregistering themselves first, my code will keep the callbacks alive.  Since this could lead to really weird and nasty situations, I would like to store all the callbacks in a WeakSet (https://docs.python.org/3/library/weakref.html#weakref.WeakSet).  That way, my code isn't the reason why the objects are kept alive, and if they are no longer alive, they are automatically removed from the WeakSet, preventing me from accidentally calling them when they are dead.  My question is simple; is this a good design?  If not, why not?  Are there any potential 'gotchas' I should be worried about?
>> 
> 
> No, it's not. I would advise using strong references - if the callback
> is a closure, for instance, you need to hang onto it, because there
> are unlikely to be any other references to it. If I register a
> callback with you, I expect it to be called; I expect, in fact, that
> that *will* keep my object alive.

OK, so it would violate the principle of least surprise for you.  Interesting.  Is this a general pattern in python?  That is, callbacks are owned by what they are registered with?

In the end, I want to make a library that offers as few surprises to the user as possible, and no matter how I think about callbacks, they are surprising to me.  If callbacks are strongly-held, then calling 'del foo' on a callable object may not make it go away, which can lead to weird and nasty situations.  Weakly-held callbacks mean that I (as the programmer), know that objects will go away after the next garbage collection (see Frank's earlier message), so I don't get 'dead' callbacks coming back from the grave to haunt me.

So, what's the consensus on the list, strongly-held callbacks, or weakly-held ones?

Thanks,
Cem Karan

[toc] | [next] | [standalone]

#86069

On 2015-02-21, Cem Karan <cfkaran2@gmail.com> wrote:
>
> On Feb 21, 2015, at 12:42 AM, Chris Angelico <rosuav@gmail.com> wrote:
>
>> On Sat, Feb 21, 2015 at 1:44 PM, Cem Karan <cfkaran2@gmail.com> wrote:
>>> In order to inform users that certain bits of state have changed, I require them to register a callback with my code.  The problem is that when I store these callbacks, it naturally creates a strong reference to the objects, which means that if they are deleted without unregistering themselves first, my code will keep the callbacks alive.  Since this could lead to really weird and nasty situations, I would like to store all the callbacks in a WeakSet (https://docs.python.org/3/library/weakref.html#weakref.WeakSet).  That way, my code isn't the reason why the objects are kept alive, and if they are no longer alive, they are automatically removed from the WeakSet, preventing me from accidentally calling them when they are dead.  My question is simple; is this a good design?  If not, why not?  Are there any potential 'gotchas' I should be worried about?
>>> 
>> 
>> No, it's not. I would advise using strong references - if the callback
>> is a closure, for instance, you need to hang onto it, because there
>> are unlikely to be any other references to it. If I register a
>> callback with you, I expect it to be called; I expect, in fact, that
>> that *will* keep my object alive.
>
> OK, so it would violate the principle of least surprise for you.

And me as well.  I would expect to be able to pass a closure as a
callback and not have to keep a reference to it.  Perhaps that just a
leftover from working with other languages (javascript, scheme, etc.).
It doesn't matter if it's a string, a float, a callback, a graphic or
whatever: if I pass your function/library an object, I expect _you_ to
keep track of it until you're done with it.

> Interesting.  Is this a general pattern in python?  That is,
> callbacks are owned by what they are registered with?

I'm not sure what you mean by "owned" or why it matters that it's a
callback: it's an object that was passed to you: you need to hold onto
a reference to it until you're done with it, and the polite thing to
do is to delete references to it when you're done with it.

> So, what's the consensus on the list, strongly-held callbacks, or
> weakly-held ones?

--
Grant

[toc] | [prev] | [next] | [standalone]

#86070

Grant Edwards <invalid@invalid.invalid>:

> the polite thing to do is to delete references to it when you're done
> with it.

I disagree with that recommendation. You should do the natural thing and
not care who holds references to who.


Marko

[toc] | [prev] | [next] | [standalone]

#86074

Marko Rauhamaa wrote:

> Grant Edwards <invalid@invalid.invalid>:
> 
>> the polite thing to do is to delete references to it when you're done
>> with it.
> 
> I disagree with that recommendation. You should do the natural thing and
> not care who holds references to who.

I don't understand this. What is "the natural thing" if not to delete
references to an object when you are done with it? Normally you just let
things go out of scope, but if that won't happen, you have to take active
steps, such as calling del or setting the reference to None.


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#86075

On Sun, Feb 22, 2015 at 1:04 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> Marko Rauhamaa wrote:
>
>> Grant Edwards <invalid@invalid.invalid>:
>>
>>> the polite thing to do is to delete references to it when you're done
>>> with it.
>>
>> I disagree with that recommendation. You should do the natural thing and
>> not care who holds references to who.
>
> I don't understand this. What is "the natural thing" if not to delete
> references to an object when you are done with it? Normally you just let
> things go out of scope, but if that won't happen, you have to take active
> steps, such as calling del or setting the reference to None.

I think the disagreement here is over the interpretation of "done with
it". If you drop all references to a connected socket object, Python
can rightly assume that you're done with the file and want to close
it; but what if you drop all references to a listening socket that's
been configured to call a function whenever someone connects?

def client_connected(sock):
    sock.send("Hello!\r\n")
    # whatever

listener = socket(23, on_accept=client_connected)

What should happen if that main socket isn't bound to a name? In my
opinion, the fact that it's configured for callback mode should mean
that it's kept alive. But it's also understandable to want to treat it
as "done", that it can be disposed of. It seems weird to me that you
should have to have a name somewhere that you'll never use, though.

ChrisA

[toc] | [prev] | [next] | [standalone]

#86077

Chris Angelico wrote:

> On Sun, Feb 22, 2015 at 1:04 PM, Steven D'Aprano
> <steve+comp.lang.python@pearwood.info> wrote:
>> Marko Rauhamaa wrote:
>>
>>> Grant Edwards <invalid@invalid.invalid>:
>>>
>>>> the polite thing to do is to delete references to it when you're done
>>>> with it.
>>>
>>> I disagree with that recommendation. You should do the natural thing and
>>> not care who holds references to who.
>>
>> I don't understand this. What is "the natural thing" if not to delete
>> references to an object when you are done with it? Normally you just let
>> things go out of scope, but if that won't happen, you have to take active
>> steps, such as calling del or setting the reference to None.
> 
> I think the disagreement here is over the interpretation of "done with
> it". If you drop all references to a connected socket object, Python
> can rightly assume that you're done with the file and want to close
> it; but what if you drop all references to a listening socket that's
> been configured to call a function whenever someone connects?
> 
> def client_connected(sock):
>     sock.send("Hello!\r\n")
>     # whatever
> 
> listener = socket(23, on_accept=client_connected)
> 
> What should happen if that main socket isn't bound to a name? In my
> opinion, the fact that it's configured for callback mode should mean
> that it's kept alive. But it's also understandable to want to treat it
> as "done", that it can be disposed of. It seems weird to me that you
> should have to have a name somewhere that you'll never use, though.

But you are using it. You might not be using it by name, but you are using
it via the callback function. What did you expect, that Python should read
your mind and somehow intuit that you still care about this socket
listener, but not some other socket listener that you are done with?

If you have a servant that follows you around everywhere, throwing objects
away when you stop using them, then naturally if you stop using something
it will be thrown away. What did you expect?

You don't have to bind the listener to a name. Any reference will do. You
can dump it in a bucket:

bucket_of_stuff = []
bucket_of_stuff.append(some_function(a, b, c))
bucket_of_stuff.append(make_web_server())
bucket_of_stuff.append(socket(23, on_accept=client_connected))


So long as your bucket is still alive, the garbage collector won't collect
it or its contents.

Hypothetically we could have a system in place where you instruct the
garbage collector to not collect an object, then drop all references to it:

gc.never_collect(socket(23, on_accept=client_connected))

but that's a potential memory leak, because now you have no way of telling
the GC to collect it again once you've finished listening. If you never
finish listening, or at least not until your application shuts down, it
doesn't count as a leak. But in general, if listeners might come and go,
but you have no way for them to be garbage collected once they are done,
then that's a leak.

What's the easiest way for the GC to flag an object as "never collect this"?
It can keep a reference to it. Keeping a list of things you want to be kept
alive is simple, easy and obvious:

# hypothetically inside gc.py
_ALIVE = []
def never_collect(obj):
    _ALIVE.append(obj)


but that's so trivial that it's not worth putting it in the gc module.
Besides, that means any other code could reach into the gc and remove your
listener from the "keep alive list" without your knowledge or permission.

It's simpler and safer to just keep it alive yourself:

alive = []
alive.append(socket(...))

but of course that's just my bucket of stuff under a different name.

Using the idiom "keep objects alive by keeping a reference to them" (e.g.
bind them to a name, or stick them in a list which you keep) makes things
much simpler. You can trivially flag objects as "collect" or "don't
collect" as needed, without having to import the gc module or memorise some
obscure API or worry about implementation details ("if I flag an object
as 'never collect' *twice*, do I have to unflag it twice to undo it?"). You
just use the regular interface to the garbage collector: the existence of a
reference, any reference, keeps an object alive.




-- 
Steven

[toc] | [prev] | [next] | [standalone]

#86082

On Sun, Feb 22, 2015 at 3:38 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> But you are using it. You might not be using it by name, but you are using
> it via the callback function. What did you expect, that Python should read
> your mind and somehow intuit that you still care about this socket
> listener, but not some other socket listener that you are done with?
>
> You don't have to bind the listener to a name. Any reference will do. You
> can dump it in a bucket:
>
> bucket_of_stuff = []
> bucket_of_stuff.append(some_function(a, b, c))
> bucket_of_stuff.append(make_web_server())
> bucket_of_stuff.append(socket(23, on_accept=client_connected))

Sure, and whether it's a name or a list-element reference doesn't
matter: it seems wrong to have to stash a thing in a bucket in order
to keep its callbacks alive. I expect the callbacks _themselves_ to
keep it alive. But I can understand the opposite POV.

ChrisA

[toc] | [prev] | [next] | [standalone]

#86094

Chris Angelico wrote:

> On Sun, Feb 22, 2015 at 3:38 PM, Steven D'Aprano
> <steve+comp.lang.python@pearwood.info> wrote:
>> But you are using it. You might not be using it by name, but you are
>> using it via the callback function. What did you expect, that Python
>> should read your mind and somehow intuit that you still care about this
>> socket listener, but not some other socket listener that you are done
>> with?
>>
>> You don't have to bind the listener to a name. Any reference will do. You
>> can dump it in a bucket:
>>
>> bucket_of_stuff = []
>> bucket_of_stuff.append(some_function(a, b, c))
>> bucket_of_stuff.append(make_web_server())
>> bucket_of_stuff.append(socket(23, on_accept=client_connected))
> 
> Sure, and whether it's a name or a list-element reference doesn't
> matter: it seems wrong to have to stash a thing in a bucket in order
> to keep its callbacks alive. I expect the callbacks _themselves_ to
> keep it alive. 

Why? Do you expect that the Python garbage collector special cases callbacks
to keep them alive even when there are no references to them? How would it
distinguish a callback from some other function?

If I stuff a function in a list:

   [len]

would you expect the presence of the function to keep the list alive when
there are no references to the list?

Apart from "But I really, really, REALLY want a magical pony that feeds
itself and never poops and just disappears when I don't want it around!"
wishful-thinking, which I *totally* get, I don't see how you think this is
even possible. Maybe I'm missing something, but it seems to me that what
you're wishing for is impossible.

Perhaps if we had a special "callback" type which was treated as a special
case by the garbage collector. But that opens up a big can of worms: how do
you close/delete objects which are kept alive by the presence of callbacks
if you don't have a reference to either the object or the callback?



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#86097

On Sun, Feb 22, 2015 at 9:32 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> Why? Do you expect that the Python garbage collector special cases callbacks
> to keep them alive even when there are no references to them? How would it
> distinguish a callback from some other function?

No no no. It's the other way around. _Something_ has to be doing those
callbacks, and it's that _something_ that should be keeping them
alive. The fact that it's a registered callback should itself *be* a
reference (and not a weak reference), and should keep it alive.

ChrisA

[toc] | [prev] | [next] | [standalone]

#86114

Chris Angelico wrote:

> On Sun, Feb 22, 2015 at 9:32 PM, Steven D'Aprano
> <steve+comp.lang.python@pearwood.info> wrote:
>> Why? Do you expect that the Python garbage collector special cases
>> callbacks to keep them alive even when there are no references to them?
>> How would it distinguish a callback from some other function?
> 
> No no no. It's the other way around. _Something_ has to be doing those
> callbacks, and it's that _something_ that should be keeping them
> alive. The fact that it's a registered callback should itself *be* a
> reference (and not a weak reference), and should keep it alive.

That's much more reasonable than what you said earlier:

    it seems wrong to have to stash a thing in a bucket in order
    to keep its callbacks alive. I expect the callbacks themselves to
    keep it alive.


So yes. If I bind a callback to a button, say, or a listener, then the
button (or listener) keeps the callback alive, *not* the callback keeping
the button or listener alive.

But if there are no references to the button, or the listener, then it will
be garbage-collected, which will free the references to the callback and
allow it to be garbage-collected as well (if there are no further
references to it).



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#86118

On Mon, Feb 23, 2015 at 12:45 AM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
>> No no no. It's the other way around. _Something_ has to be doing those
>> callbacks, and it's that _something_ that should be keeping them
>> alive. The fact that it's a registered callback should itself *be* a
>> reference (and not a weak reference), and should keep it alive.
>
> That's much more reasonable than what you said earlier:
>
>     it seems wrong to have to stash a thing in a bucket in order
>     to keep its callbacks alive. I expect the callbacks themselves to
>     keep it alive.
>
>
> So yes. If I bind a callback to a button, say, or a listener, then the
> button (or listener) keeps the callback alive, *not* the callback keeping
> the button or listener alive.

I meant the same thing, but my terminology was poor. Yes, that's
correct; it's not any sort of magic about it being a callback, but
more that the one you register it with becomes the owner of something.
Hence, no weak references.

ChrisA

[toc] | [prev] | [next] | [standalone]

#86085

Steven D'Aprano <steve+comp.lang.python@pearwood.info>:

> Marko Rauhamaa wrote:
>> Grant Edwards <invalid@invalid.invalid>:
>>> the polite thing to do is to delete references to it when you're done
>>> with it.
>> 
>> I disagree with that recommendation. You should do the natural thing and
>> not care who holds references to who.
>
> I don't understand this. What is "the natural thing" if not to delete
> references to an object when you are done with it? Normally you just let
> things go out of scope, but if that won't happen, you have to take active
> steps, such as calling del or setting the reference to None.

Obviously, if you are hoarding objects in a collection, you're going to
land in trouble.

What I mean, though, is that you shouldn't think you need to create
object destructors where you routinely set all members to None.


Marko

[toc] | [prev] | [next] | [standalone]

#86087

On Sun, Feb 22, 2015 at 6:52 PM, Marko Rauhamaa <marko@pacujo.net> wrote:
> What I mean, though, is that you shouldn't think you need to create
> object destructors where you routinely set all members to None.

Sure, not *routinely*. It'd be a special case where it's not
specifically a destructor, and its job is to break a reference cycle.
For instance, you might have a close() method that clears out a bunch
of references, which will then allow everything to get cleaned up
promptly. Or (a very common case for me) a callback saying "remote end
is gone" (eg on a socket) might wipe out the callbacks, thus removing
their refloops.

ChrisA

[toc] | [prev] | [next] | [standalone]

#86088

Chris Angelico <rosuav@gmail.com>:

> Or (a very common case for me) a callback saying "remote end is gone"
> (eg on a socket) might wipe out the callbacks, thus removing their
> refloops.

Refloops are not to be worried about, let alone removed.


Marko

[toc] | [prev] | [next] | [standalone]

#86089

On Sun, Feb 22, 2015 at 7:34 PM, Marko Rauhamaa <marko@pacujo.net> wrote:
> Chris Angelico <rosuav@gmail.com>:
>
>> Or (a very common case for me) a callback saying "remote end is gone"
>> (eg on a socket) might wipe out the callbacks, thus removing their
>> refloops.
>
> Refloops are not to be worried about, let alone removed.

Why? They force the use of the much slower cycle-detecting GC, rather
than the quick and efficient CPython refcounter. I don't know how
other Pythons work, but mark-and-sweep has its own costs, and I don't
know of any system that's both prompt and able to detect refloops.
Helping it along means your program doesn't waste memory. Why such a
blanket statement?

ChrisA

[toc] | [prev] | [next] | [standalone]

#86091

Chris Angelico <rosuav@gmail.com>:

> On Sun, Feb 22, 2015 at 7:34 PM, Marko Rauhamaa <marko@pacujo.net> wrote:
>> Refloops are not to be worried about, let alone removed.
>
> Why?

Because the whole point of GC-languages is that you should stop worrying
about memory. Trying to mastermind and micromanage GC in the application
is, pardon my French, an antipattern.

> They force the use of the much slower cycle-detecting GC, rather than
> the quick and efficient CPython refcounter.

Java's Hotspot doesn't bother with refcounters but is much faster than
Python. CPython's refcounters are a historical accident that a Python
application developer shouldn't even be aware of.

> I don't know how other Pythons work, but mark-and-sweep has its own
> costs, and I don't know of any system that's both prompt and able to
> detect refloops.

It's exceedingly difficult (and pointless) to detect cycles in your
object structures. Python is going to have to do a GC occasionally
anyway. Yes, your worst-case response times are going to suffer, but
that's the cost of doing business.

> Helping it along means your program doesn't waste memory. Why such a
> blanket statement?

Because worrying Python programmers with evil spirits (reference loops)
leads to awkward coding practices and takes away one of the main
advantages of Python as a high-level programming language.


Marko

[toc] | [prev] | [next] | [standalone]

#86093

On Sun, Feb 22, 2015 at 8:14 PM, Marko Rauhamaa <marko@pacujo.net> wrote:
>> Helping it along means your program doesn't waste memory. Why such a
>> blanket statement?
>
> Because worrying Python programmers with evil spirits (reference loops)
> leads to awkward coding practices and takes away one of the main
> advantages of Python as a high-level programming language.

Right, and I suppose that, by extension, we should assume that the
Python interpreter can optimize this?

def fib(x):
    if x<2: return x
    return fib(x-2)+fib(x-1)

Just because a computer can, in theory, recognize that this is a pure
function, doesn't mean that we can and should depend on that. If you
want this to be optimized, you either fix your algorithm or explicitly
memoize the function - you don't assume that Python can do it for you.

Even when you write in a high level language, you need to understand
how computers work.

ChrisA

[toc] | [prev] | [next] | [standalone]

#86098

somebody, I got confused with the indent level wrote:

>> They force the use of the much slower cycle-detecting GC, rather than
>> the quick and efficient CPython refcounter.

Somebody has misunderstood something here.  When it comes to efficient
garbage collectors, refcounting is a turtle.  The CPython one is no
exception.  Ref counting, however, is fairly easy to write.  But when
the PyPy project first replaced its refcounting gc with its very first
and therefore not very efficient at all nursery gc ... that was the very
first time when a bunch of python programs ran faster on pypy than on
CPython.  This was before pypy had a JIT.

And today the pypy channel is full of people who want to link their
C extension into some Python code running on PyPy, and who find that
their C extension slows things down.  There are lots of reasons for
this, but one of the most common problems is 'this C extension is
faking refcounting.  All of this is wasted effort for PyPy and
usually makes the thing unJITable as well.'  Many of these people
rewrite their C extension as pure Python and find that then, with
PyPy, they get the speed improvements they were looking for.

So: two points.

One reason you might not want to rely on ref counting, because you expect
your code to run under PyPy one day.

and

If you are interested in manipulating garbage collection -- especially if
this is for your own pleasure and enjoyment, a worthy goal in my books --
you could do a lot worse than write your own gc in RPython for PyPy.
The gc code is not mixed in with all of the other VM stuff, so a gc is
small, and you don't have to worry about clobbering anything else while
you are working.  So it is great for experimenting, which was the whole
point.  Hacking gcs is fun! :)

Laura

[toc] | [prev] | [next] | [standalone]

#86117

Marko Rauhamaa wrote:

> Chris Angelico <rosuav@gmail.com>:
> 
>> On Sun, Feb 22, 2015 at 7:34 PM, Marko Rauhamaa <marko@pacujo.net> wrote:
>>> Refloops are not to be worried about, let alone removed.
>>
>> Why?
> 
> Because the whole point of GC-languages is that you should stop worrying
> about memory. Trying to mastermind and micromanage GC in the application
> is, pardon my French, an antipattern.

While it would be nice to be able to stop worrying about memory, try to
calculate 1000**1000**1000 and see how that works for you.

Garbage collection enables us to *mostly* automate the allocation and
deallocation of memory. If doesn't mean we can forget about it. GC is an
abstraction that frees us from most of the grunt work of allocating memory,
but it doesn't mean that there is never any need to think about memory. GC
is a leaky abstraction. Depending on the implementation, it may cause
distracting and annoying pauses in your application and/or resource leaks.
Even if there are no pauses, GC still carries a performance penalty. Good
programmers need to be aware of the limitations of their tools, and be
prepared to code accordingly.

When writing programs for educational purposes, we should try to code in the
simplest and most elegant way with no thought given to annoying practical
matters. At least at first. But when writing programs for actual use, we
should write for the implication we have, not the one we wish we had.


>> They force the use of the much slower cycle-detecting GC, rather than
>> the quick and efficient CPython refcounter.
> 
> Java's Hotspot doesn't bother with refcounters but is much faster than
> Python. CPython's refcounters are a historical accident that a Python
> application developer shouldn't even be aware of.

I don't know about Java's Hotspot, but I do know that CPython's ref counting
garbage collector has at least one advantage over the GC used by Jython and
IronPython: unlike them, open files are closed as soon as they are no
longer in use. Code like this may run out of operating system file handles
in Jython:

i = 0
while True:
    f = open('/tmp/x%d' % i)
    i += 1

while CPython will just keep going. I suppose it will *eventually* run out
of some resource, but probably not file handles.

Oh, a bit of trivia: Apple is abandoning their garbage collector and going
back to a reference counter:

https://developer.apple.com/news/?id=02202015a

Word on Reddit is that Apple is concerned about performance and battery
life.

P.S. A reminder that reference counting *is* a form of garbage collection.


>> I don't know how other Pythons work, but mark-and-sweep has its own
>> costs, and I don't know of any system that's both prompt and able to
>> detect refloops.
> 
> It's exceedingly difficult (and pointless) to detect cycles in your
> object structures. Python is going to have to do a GC occasionally
> anyway. Yes, your worst-case response times are going to suffer, but
> that's the cost of doing business.

In *general*, you're right. Who wants to spend all their time worrying about
cycles when the GC can do it for you? But if cycles are rare, and in known
parts of your code where it is simple to break them when you're done,
there's no disadvantage to doing so. Leave the GC for the hard cases.

It's like explicitly closing a file, either with file.close() or a context
manager. When using CPython, it doesn't really matter whether you close the
file or not, since the ref counter will normally close it automatically as
soon as the file goes out of scope. But it is cheap and easy to do so, so
why not do it? Then, when it otherwise would matter, say you are running
under Jython, it doesn't because you've closed the file.


>> Helping it along means your program doesn't waste memory. Why such a
>> blanket statement?
> 
> Because worrying Python programmers with evil spirits (reference loops)
> leads to awkward coding practices and takes away one of the main
> advantages of Python as a high-level programming language.

I think you exaggerate a tad. We're not trying to scare beginners, we're a
group of moderately experienced coders discussing "best practice" (or at
least "reasonable practice") when using callbacks.


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#86120

Steven D'Aprano <steve+comp.lang.python@pearwood.info>:

> I don't know about Java's Hotspot, but I do know that CPython's ref counting
> garbage collector has at least one advantage over the GC used by Jython and
> IronPython: unlike them, open files are closed as soon as they are no
> longer in use.

You can't depend on that kind of behavior. Dangling resources may or may
not be cleaned up, ever.

> Oh, a bit of trivia: Apple is abandoning their garbage collector and going
> back to a reference counter:
>
> https://developer.apple.com/news/?id=02202015a
>
> Word on Reddit is that Apple is concerned about performance and battery
> life.

That truly is a bit OT here.

> It's like explicitly closing a file, either with file.close() or a context
> manager.

Both methods are explicit. Closing files and other resources are not
directly related to GC.

Here's the thing: GC relieves your from dynamic memory management. You
are still on your own when it comes to other resources.

> We're not trying to scare beginners, we're a group of moderately
> experienced coders discussing "best practice" (or at least "reasonable
> practice") when using callbacks.

Who mentioned beginners? I'm abiding by the same best practices I'm
advocating.


Marko

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | comp.lang.python

csiph-web