Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #47879 > unrolled thread

Debugging memory leaks

Back to article view | Back to comp.lang.python

Contents

  Debugging memory leaks writeson <doug.farrell@gmail.com> - 2013-06-12 18:24 -0700
    Re: Debugging memory leaks dieter <dieter@handshake.de> - 2013-06-13 08:29 +0200
      Re: Debugging memory leaks Giorgos Tzampanakis <giorgos.tzampanakis@gmail.com> - 2013-06-13 20:15 +0000
        Re: Debugging memory leaks Steve Simmons <square.steve@gmail.com> - 2013-06-13 22:45 +0100
        Re: Debugging memory leaks Chris Angelico <rosuav@gmail.com> - 2013-06-14 08:36 +1000
        Re: Debugging memory leaks Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-06-14 02:40 +0000
          Re: Debugging memory leaks Chris Angelico <rosuav@gmail.com> - 2013-06-14 19:30 +1000
          Re: Debugging memory leaks Giorgos Tzampanakis <giorgos.tzampanakis@gmail.com> - 2013-06-14 22:57 +0000
            Re: Debugging memory leaks Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-06-15 01:39 +0000
          Re: Debugging memory leaks dieter <dieter@handshake.de> - 2013-06-15 08:52 +0200
          Re: Debugging memory leaks Chris Angelico <rosuav@gmail.com> - 2013-06-15 17:21 +1000
          Re: Debugging memory leaks dieter <dieter@handshake.de> - 2013-06-16 08:18 +0200
        Re: Debugging memory leaks rusi <rustompmody@gmail.com> - 2013-06-14 06:53 -0700
          Re: Debugging memory leaks Chris Angelico <rosuav@gmail.com> - 2013-06-15 10:11 +1000
          Re: Debugging memory leaks Ben Finney <ben+python@benfinney.id.au> - 2013-06-15 10:16 +1000
            Re: Debugging memory leaks rusi <rustompmody@gmail.com> - 2013-06-14 20:16 -0700
              Re: Debugging memory leaks Ben Finney <ben+python@benfinney.id.au> - 2013-06-15 21:23 +1000
                Re: Debugging memory leaks rusi <rustompmody@gmail.com> - 2013-06-15 04:35 -0700
                  Re: Debugging memory leaks Chris Angelico <rosuav@gmail.com> - 2013-06-15 21:54 +1000
    Re: Debugging memory leaks writeson <doug.farrell@gmail.com> - 2013-06-13 11:07 -0700
      Re: Debugging memory leaks Dave Angel <davea@davea.name> - 2013-06-13 14:44 -0400
    Re: Debugging memory leaks rusi <rustompmody@gmail.com> - 2013-06-14 05:36 -0700
    Re: Debugging memory leaks "Frank Millman" <frank@chagford.com> - 2013-06-21 08:50 +0200

Page 2 of 2 — ← Prev page 1 [2]

#48008

On 06/13/2013 02:07 PM, writeson wrote:
> Dieter,
>
> Thanks for the response, and you're correct, debugging memory leaks is tough! So far I haven't had much luck other than determining I have a leak. I've used objgraph to see that objects are being created that don't seem to get cleaned up. What I can't figure out so far is why, they are local variable objects that "should" get cleaned up when they go out scope.
>

Pure python code shouldn't have any leaks, but instead can have what I 
call stagnation.  That's data that's no longer useful, but the program 
has fooled the system into thinking it should hang onto it.

A leak happens in C code, when all the pointers to a given hunk of 
memory have gone away, and there's no way to access it any longer.

In pure Python, you don't work with pointers, but with references, and 
they are ref-counted.  When the count goes to zero, the object is freed. 
  Periodically a gc sweep happens, which catches those circular 
references which never actually go to zero.


So post a fragment of code that seems to cause the problem, and maybe 
someone can explain why.

1) objects with a __del__() method
2) objects that are "cached" by some mechanism
3) objects that collectively represent a lot of data
4) objects that are exposed to buggy C code


-- 
DaveA

[toc] | [prev] | [next] | [standalone]

#48131

On Jun 13, 6:24 am, writeson <doug.farr...@gmail.com> wrote:
> Anyway, my real question is how to go about debugging memory leak problems in Python, particularly for a long running
> server process written with Twisted. I'm not sure how to use heapy or guppy, and objgraph doesn't tell me enough to
> locate the problem. If anyone as any suggestions or pointers it would be very much appreciated!

Can you explain in more detail what you get stuck with using heapy/
guppy?

[toc] | [prev] | [next] | [standalone]

#48846

"writeson" <doug.farrell@gmail.com> wrote in message 
news:09917103-b35e-4728-8fea-bcb4ce2bd1af@googlegroups.com...
> Hi all,
>
> I've written a program using Twisted that uses SqlAlchemy to access a 
> database using threads.deferToThread(...) and SqlAlchemy's 
> scoped_session(...). This program runs for a long time, but leaks memory 
> slowly to the point of needing to be restarted. I don't know that the 
> SqlAlchemy/threads thing is the problem, but thought I'd make you aware of 
> it.
>
> Anyway, my real question is how to go about debugging memory leak problems 
> in Python, particularly for a long running server process written with 
> Twisted. I'm not sure how to use heapy or guppy, and objgraph doesn't tell 
> me enough to locate the problem. If anyone as any suggestions or pointers 
> it would be very much appreciated!
>
> Thanks in advance,
> Doug

You have received lots of good advice, but there is one technique that I 
have found useful that has not been mentioned.

As you are probably aware, one of the main causes of a 'memory leak' in 
python is an object that is supposed to be garbage collected, but hangs 
around because there is still a reference pointing to it.

You cannot directly confirm that an object has been deleted, because 
invoking its '__del__' method causes side-effects which can prevent it from 
being deleted even if it is otherwise ok.

However, there is an indirect way of confirming it - a 'DelWatcher' class. I 
got this idea from a thread on a similar subject in this forum a long time 
ago. Here is how it works.

class DelWatcher:
    def __init__(self, obj):
        # do not store a reference to obj - that would create a circular 
reference
        # store some attribute that uniquely identifies the 'obj' instance
        self.name = obj.name
        print(self.name, 'created')
    def __del__(self):
        print(self.name, 'deleted')

class MyClass:
    def __init__(self, ...):
        [...]
        self._del = DelWatcher(self)

Now you can watch the objects as they are created, and then check that they 
are deleted when you expect them to be.

This can help to pinpoint where the memory leak is occurring.

HTH

Frank Millman

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | comp.lang.python

csiph-web