Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #104180 > unrolled thread

__del__: when to use it? What happens when you SystemExit/NameError wrt del? Method vs function calls.

Back to article view | Back to comp.lang.python

Contents

  __del__: when to use it? What happens when you SystemExit/NameError wrt del? Method vs function calls. "Veek. M" <vek.m1234@gmail.com> - 2016-03-07 08:57 +0530
    Re: __del__: when to use it? What happens when you SystemExit/NameError wrt del? Method vs function calls. Chris Angelico <rosuav@gmail.com> - 2016-03-07 14:38 +1100
    Re: __del__: when to use it? What happens when you SystemExit/NameError wrt del? Method vs function calls. Ben Finney <ben+python@benfinney.id.au> - 2016-03-07 14:42 +1100
    Re: __del__: when to use it? What happens when you SystemExit/NameError wrt del? Method vs function calls. Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2016-03-07 16:56 +1100
    Re: __del__: when to use it? What happens when you SystemExit/NameError wrt del? Method vs function calls. "Veek. M" <vek.m1234@gmail.com> - 2016-03-07 11:43 +0530
      Re: __del__: when to use it? What happens when you SystemExit/NameError wrt del? Method vs function calls. Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2016-03-07 19:37 +1100
        Re: __del__: when to use it? What happens when you SystemExit/NameError wrt del? Method vs function calls. "Veek. M" <vek.m1234@gmail.com> - 2016-03-07 16:56 +0530

#104180 — __del__: when to use it? What happens when you SystemExit/NameError wrt del? Method vs function calls.

1. What are the rules for using __del__ besides: 'don't use it'.

2. What happens when I SystemExit? __del__ and gc are not invoked when I 
SystemExit and there's a circular reference - but why? The OS is going 
to reclaim the memory anyways so why be finicky about circular 
references - why can't we go ahead and call __dell_ and run gc?

3.
import foo
def __del__(self, foo=foo):
  foo.bar()

What happens here to prevent a NameError? Apparently without the foo=foo 
a NameError can occur? But why? import foo creates a reference to the 
object anyways so it's refcount will be above 0 anyways till __del__ is 
called.


4. also, are method calls more efficient than function calls?

[toc] | [next] | [standalone]

#104181

On Mon, Mar 7, 2016 at 2:27 PM, Veek. M <vek.m1234@gmail.com> wrote:
> 1. What are the rules for using __del__ besides: 'don't use it'.

Use it as a last-shot cleanup of resources you own. Don't depend on
it, but use it.

> 2. What happens when I SystemExit? __del__ and gc are not invoked when I
> SystemExit and there's a circular reference - but why? The OS is going
> to reclaim the memory anyways so why be finicky about circular
> references - why can't we go ahead and call __dell_ and run gc?

SystemExit is an exception, same as any other (albeit one that doesn't
inherit from Exception, so it's usually left uncaught); the assumption
is that all resources will be cleaned up on process termination
anyway, so there's no need to run the GC to clean up reference cycles.
It's a total waste of effort.

> 3.
> import foo
> def __del__(self, foo=foo):
>   foo.bar()
>
> What happens here to prevent a NameError? Apparently without the foo=foo
> a NameError can occur? But why? import foo creates a reference to the
> object anyways so it's refcount will be above 0 anyways till __del__ is
> called.

I'm not sure where you're trying to put that code. If you put that at
module level, __del__ won't get called, because there's nothing
special about that name in a module.

During interpreter shutdown, modules will get unloaded, and stuff in
modules will get unloaded. But I'm not even going to _start_
explaining exactly what happens, until I know where this is heading;
there's almost certainly a better way to do this. What is it you're
actually trying to accomplish?

> 4. also, are method calls more efficient than function calls?

No. They in fact *are* function calls, but with a slight bit of magic
so they get their first parameter passed in. You can consider them to
be identical to function calls.

ChrisA

[toc] | [prev] | [next] | [standalone]

#104182

"Veek. M" <vek.m1234@gmail.com> writes:

> 1. What are the rules for using __del__ besides: 'don't use it'.

What do you mean by “rules”?

If you want advice on using that method, I don't think a canonical
exhaustive “rules” set exists.

For example: Use ‘__del__’ to mark an object as no longer used; don't
expect that to result in the object actually going away at any
particular point in time.

> 2. What happens when I SystemExit? __del__ and gc are not invoked when
> I SystemExit and there's a circular reference - but why?

So that we can have a concrete example: Can you give a (very minimal and
simple) example demonstrating the behaviour, so we can run it too?

> 3.
> import foo
> def __del__(self, foo=foo):
>   foo.bar()

That appears to be a module-level function. It is not a method of any
class, so I am not clear on how it relates to garbage collection.

> 4. also, are method calls more efficient than function calls?

All method calls are function calls. This is because all methods are
functions.

-- 
 \     “Unix is an operating system, OS/2 is half an operating system, |
  `\    Windows is a shell, and DOS is a boot partition virus.” —Peter |
_o__)                                                        H. Coffin |
Ben Finney

[toc] | [prev] | [next] | [standalone]

#104188

On Monday 07 March 2016 14:27, Veek. M wrote:

> 1. What are the rules for using __del__ besides: 'don't use it'.

__del__ needs to be defined in a class to be called.

It will be called *at some point* when the instance is about to be garbage 
collected. There is no guarantee when that will be: it might be instantly, 
or a week later.

It may not be called at all, under some obscure circumstances which I don't 
remember.

The presence of a __del__ method may prevent the garbage collector from 
breaking reference cycles, e.g. if A refers to B, and B refers to A, under 
some circumstances if A or B have a __del__ method, the GC may not be able 
to break the cycle and neither object will be collected.

At interpreter shutdown, the __del__ method of objects may be called *after* 
the resources they are relying on have been deleted. See below.

All of these factors are version- and implementation-dependent. In 
particular, remember that IronPython and Jython are based on the .Net CLR 
and JVM, so will use completely different garbage collectors than the 
CPython garbage collector, and so behave slightly differently.


> 2. What happens when I SystemExit? __del__ and gc are not invoked when I
> SystemExit and there's a circular reference - but why? The OS is going
> to reclaim the memory anyways so why be finicky about circular
> references - why can't we go ahead and call __dell_ and run gc?

I'm sorry, I don't understand the GC well enough to answer that question.


> 3.
> import foo
> def __del__(self, foo=foo):
>   foo.bar()
> 
> What happens here to prevent a NameError? Apparently without the foo=foo
> a NameError can occur? But why? import foo creates a reference to the
> object anyways so it's refcount will be above 0 anyways till __del__ is
> called.

I assume that you intended the __del__ method to be inside a class.

Let us see what happens without the local reference:


import foo

class X:
    def __del__(self):
        foo.bar()

instance = X()
sys.exit()



The `import foo` line creates a module level reference to foo. But during 
interpreter shutdown, the module references may be deleted before the 
__del__ method is called. If that happens, then the code `foo.bar()` will do 
a global lookup for `foo`, and not find it, and you will get a NameError.

The way to fix this is to make the class hold onto a reference to foo. Then, 
even if the module references are deleted, the class reference won't be. 
Either of these ways should work:


class X:
    foo = foo  # make foo an attribute of the class itself
    def __del__(self):
        self.foo.bar()


class X:
    def __del__(self, foo=foo):  # make foo a local variable of the method
        foo.bar()



> 4. also, are method calls more efficient than function calls?

No. For technical reasons, methods are a thin wrapper around actual 
functions (the "descriptor protocol"). Although it is very fast to create 
that wrapper, it is not instantaneous, so methods are not faster than 
function calls. The difference is very small though, so it is very rare that 
you should need to care about the speed difference. (It is usually dwarfed 
by the function body itself.)

Also, remember that attribute lookups are resolved at runtime:

instance.attr.spam.ham.eggs.cheese.foo.bar.baz.foobar()


needs to do nine lookups at runtime. This includes method calls, so try to 
avoid long chains of "dots". If you are used to Java, you may need to change 
your style of programming:

http://dirtsimple.org/2004/12/python-is-not-java.html




-- 
Steve

[toc] | [prev] | [next] | [standalone]

#104189

Veek. M wrote:

> 1. What are the rules for using __del__ besides: 'don't use it'.
> 
> 2. What happens when I SystemExit? __del__ and gc are not invoked when
> I SystemExit and there's a circular reference - but why? The OS is
> going to reclaim the memory anyways so why be finicky about circular
> references - why can't we go ahead and call __dell_ and run gc?
> 
> 3.
> import foo
> def __del__(self, foo=foo):
>   foo.bar()
> 
> What happens here to prevent a NameError? Apparently without the
> foo=foo a NameError can occur? But why? import foo creates a reference
> to the object anyways so it's refcount will be above 0 anyways till
> __del__ is called.
> 
> 
> 4. also, are method calls more efficient than function calls?

1,2,3 are a result of reading Beazley - pg 179.

I'll post the relevant paras for 3 first because that's still open, then 
i'll paste the paras for 1,2 for completeness but Chris pretty much 
nailed that. ###'s are my comments.

------------------------
Text from Beazley for 3:

One final peculiarity about program termination is that the __del__ 
method for some objects may try to access global data or methods defined 
in other modules. Because these objects may already have been destroyed, 
a NameError exception occurs in __del__, and you may get an error such 
as the following:
Exception exceptions.NameError: 'c' in <method Bar.__del__
of Bar instance at c0310> ignored

If this occurs, it means that __del__ has aborted prematurely. It also 
implies that it may have failed in an attempt to perform an important 
operation (such as cleanly shutting down a server connection). If this 
is a concern, it’s probably a good idea to perform an explicit shutdown 
step in your code, rather than rely on the interpreter to destroy
objects cleanly at program termination.

The peculiar NameError exception can also be eliminated by declaring 
default arguments in the declaration of the __del__()
method:

import foo
class Bar(object):
def __del__(self, foo=foo):
    foo.bar()        # Use something in module foo

### Why the foo=foo? import foo, would increment the ref-count for 
object 'foo' so why go to all the effort of passing it in to every 
instance via the local stack (ref-to-object-foo)?
---------------------------

Text for 1,2 from Beazley:
It’s important to note that in some cases the __del__() method might not 
be invoked at program termination.This can occur if circular references 
exist between objects (in which case objects may be allocated but 
accessible from no known name-space).Although Python’s garbage collector 
can reclaim unused circular references dur-ing execution, it isn’t 
normally invoked on program termination.

### which begs the question, why not on program termination? How bad can 
it be since you are terminating anyway. __del__ seems like a total waste 
product method and given how finicky python is about junk i was 
wondering.. (after all print statement became print builtin so..)

[toc] | [prev] | [next] | [standalone]

#104194

On Monday 07 March 2016 17:13, Veek. M wrote:

> import foo
> class Bar(object):
> def __del__(self, foo=foo):
>     foo.bar()        # Use something in module foo
> 
> ### Why the foo=foo? import foo, would increment the ref-count for
> object 'foo' 

Yes. And then at shutdown, the module globals get deleted so it gets 
decremented again and then garbage collected.

You cannot tell what order objects will be deleted, so you cannot tell 
whether your instance will be deleted before or after foo. If it happens 
before foo, then __del__ will run and you will be fine. If it happens after 
foo, then foo is gone and your __del__ method will fail.


> so why go to all the effort of passing it in to every
> instance via the local stack (ref-to-object-foo)?

So that every instance has a guaranteed and reliable reference to foo that 
cannot be garbage collected before that instance.

Here is a sketch of what happens without the foo=foo:

import foo  # refcount = 1
x = Bar()  # make an instance
y = Bar()  # and another instance
...
z = Bar()  # another instance
...
...
del z  # z gets garbage collected, everything is fine
...
sys.exit()  # Shutdown starts, x and y still exist
# Python starts garbage collecting objects
# in some unknown order...
...
...
del x  # x gets garbage collected, foo still exists so everything is fine
del foo  # refcount = 0, so foo gets garbage collected
del y  # y gets garbage collected, but foo is gone!




Here is a sketch of what happens with the foo=foo:

import foo  # refcount = 1
x = Bar()  # make an instance, foo refcount = 2
y = Bar()  # and another instance, foo refcount = 3
...
z = Bar()  # another instance, foo refcount = 4
...
...
del z  # z gets garbage collected, foo refcount = 3
...
sys.exit()  # Shutdown starts, x and y still exist
# Python starts garbage collecting objects
# in some unknown order...
...
...
del x  # x gets garbage collected, foo refcount = 2
del y  # y gets garbage collected, foo refcount = 1
del foo  # refcount = 0, so foo gets garbage collected



This guarantees that so long as there are any instances yet to be garbage 
collected, foo cannot be be garbage collected. Only after the last of those 
instances are gone will foo be garbage collected.



> Text for 1,2 from Beazley:
> It’s important to note that in some cases the __del__() method might not
> be invoked at program termination.This can occur if circular references
> exist between objects (in which case objects may be allocated but
> accessible from no known name-space).Although Python’s garbage collector
> can reclaim unused circular references dur-ing execution, it isn’t
> normally invoked on program termination.
> 
> ### which begs the question, why not on program termination? 

I'm afraid I don't know. Perhaps read the source code? 


> How bad can
> it be since you are terminating anyway. __del__ seems like a total waste
> product method 

I won't say "total" waste product, but automatic destructor methods are only 
good for very simple use-cases where you don't care that they are called in 
a non-deterministic fashion.



-- 
Steve

[toc] | [prev] | [next] | [standalone]

#104208

Steven D'Aprano wrote:

> On Monday 07 March 2016 17:13, Veek. M wrote:
> 
>> import foo
>> class Bar(object):
>> def __del__(self, foo=foo):
>>     foo.bar()        # Use something in module foo
>> 
>> ### Why the foo=foo? import foo, would increment the ref-count for
>> object 'foo'
> 
> Yes. And then at shutdown, the module globals get deleted so it gets
> decremented again and then garbage collected.
> 
> You cannot tell what order objects will be deleted, so you cannot tell
> whether your instance will be deleted before or after foo. If it
> happens before foo, then __del__ will run and you will be fine. If it
> happens after foo, then foo is gone and your __del__ method will fail.
> 
> 
>> so why go to all the effort of passing it in to every
>> instance via the local stack (ref-to-object-foo)?
> 
> So that every instance has a guaranteed and reliable reference to foo
> that cannot be garbage collected before that instance.
> 
> Here is a sketch of what happens without the foo=foo:
> 
> import foo  # refcount = 1
> x = Bar()  # make an instance
> y = Bar()  # and another instance
> ...
> z = Bar()  # another instance
> ...
> ...
> del z  # z gets garbage collected, everything is fine
> ...
> sys.exit()  # Shutdown starts, x and y still exist
> # Python starts garbage collecting objects
> # in some unknown order...
> ...
> ...
> del x  # x gets garbage collected, foo still exists so everything is
> fine
> del foo  # refcount = 0, so foo gets garbage collected
> del y  # y gets garbage collected, but foo is gone!
> 
> 
> 
> 
> Here is a sketch of what happens with the foo=foo:
> 
> import foo  # refcount = 1
> x = Bar()  # make an instance, foo refcount = 2
> y = Bar()  # and another instance, foo refcount = 3
> ...
> z = Bar()  # another instance, foo refcount = 4
> ...
> ...
> del z  # z gets garbage collected, foo refcount = 3
> ...
> sys.exit()  # Shutdown starts, x and y still exist
> # Python starts garbage collecting objects
> # in some unknown order...
> ...
> ...
> del x  # x gets garbage collected, foo refcount = 2
> del y  # y gets garbage collected, foo refcount = 1
> del foo  # refcount = 0, so foo gets garbage collected
> 
> 
> 
> This guarantees that so long as there are any instances yet to be
> garbage collected, foo cannot be be garbage collected. Only after the
> last of those instances are gone will foo be garbage collected.
> 
> 
> 
>> Text for 1,2 from Beazley:
>> It’s important to note that in some cases the __del__() method might
>> not be invoked at program termination.This can occur if circular
>> references exist between objects (in which case objects may be
>> allocated but accessible from no known name-space).Although Python’s
>> garbage collector can reclaim unused circular references dur-ing
>> execution, it isn’t normally invoked on program termination.
>> 
>> ### which begs the question, why not on program termination?
> 
> I'm afraid I don't know. Perhaps read the source code?
> 
> 
>> How bad can
>> it be since you are terminating anyway. __del__ seems like a total
>> waste product method
> 
> I won't say "total" waste product, but automatic destructor methods
> are only good for very simple use-cases where you don't care that they
> are called in a non-deterministic fashion.
> 
> 
> 
Hi thanks! I kind of posted and your reply came through. Got it.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web