Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #48846

Re: Debugging memory leaks

Show all headers | View raw

"writeson" <doug.farrell@gmail.com> wrote in message 
news:09917103-b35e-4728-8fea-bcb4ce2bd1af@googlegroups.com...
> Hi all,
>
> I've written a program using Twisted that uses SqlAlchemy to access a 
> database using threads.deferToThread(...) and SqlAlchemy's 
> scoped_session(...). This program runs for a long time, but leaks memory 
> slowly to the point of needing to be restarted. I don't know that the 
> SqlAlchemy/threads thing is the problem, but thought I'd make you aware of 
> it.
>
> Anyway, my real question is how to go about debugging memory leak problems 
> in Python, particularly for a long running server process written with 
> Twisted. I'm not sure how to use heapy or guppy, and objgraph doesn't tell 
> me enough to locate the problem. If anyone as any suggestions or pointers 
> it would be very much appreciated!
>
> Thanks in advance,
> Doug

You have received lots of good advice, but there is one technique that I 
have found useful that has not been mentioned.

As you are probably aware, one of the main causes of a 'memory leak' in 
python is an object that is supposed to be garbage collected, but hangs 
around because there is still a reference pointing to it.

You cannot directly confirm that an object has been deleted, because 
invoking its '__del__' method causes side-effects which can prevent it from 
being deleted even if it is otherwise ok.

However, there is an indirect way of confirming it - a 'DelWatcher' class. I 
got this idea from a thread on a similar subject in this forum a long time 
ago. Here is how it works.

class DelWatcher:
    def __init__(self, obj):
        # do not store a reference to obj - that would create a circular 
reference
        # store some attribute that uniquely identifies the 'obj' instance
        self.name = obj.name
        print(self.name, 'created')
    def __del__(self):
        print(self.name, 'deleted')

class MyClass:
    def __init__(self, ...):
        [...]
        self._del = DelWatcher(self)

Now you can watch the objects as they are created, and then check that they 
are deleted when you expect them to be.

This can help to pinpoint where the memory leak is occurring.

HTH

Frank Millman

Back to comp.lang.python | Previous | Next — Previous in thread | Find similar | Unroll thread

Thread

Debugging memory leaks writeson <doug.farrell@gmail.com> - 2013-06-12 18:24 -0700
  Re: Debugging memory leaks dieter <dieter@handshake.de> - 2013-06-13 08:29 +0200
    Re: Debugging memory leaks Giorgos Tzampanakis <giorgos.tzampanakis@gmail.com> - 2013-06-13 20:15 +0000
      Re: Debugging memory leaks Steve Simmons <square.steve@gmail.com> - 2013-06-13 22:45 +0100
      Re: Debugging memory leaks Chris Angelico <rosuav@gmail.com> - 2013-06-14 08:36 +1000
      Re: Debugging memory leaks Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-06-14 02:40 +0000
        Re: Debugging memory leaks Chris Angelico <rosuav@gmail.com> - 2013-06-14 19:30 +1000
        Re: Debugging memory leaks Giorgos Tzampanakis <giorgos.tzampanakis@gmail.com> - 2013-06-14 22:57 +0000
          Re: Debugging memory leaks Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-06-15 01:39 +0000
        Re: Debugging memory leaks dieter <dieter@handshake.de> - 2013-06-15 08:52 +0200
        Re: Debugging memory leaks Chris Angelico <rosuav@gmail.com> - 2013-06-15 17:21 +1000
        Re: Debugging memory leaks dieter <dieter@handshake.de> - 2013-06-16 08:18 +0200
      Re: Debugging memory leaks rusi <rustompmody@gmail.com> - 2013-06-14 06:53 -0700
        Re: Debugging memory leaks Chris Angelico <rosuav@gmail.com> - 2013-06-15 10:11 +1000
        Re: Debugging memory leaks Ben Finney <ben+python@benfinney.id.au> - 2013-06-15 10:16 +1000
          Re: Debugging memory leaks rusi <rustompmody@gmail.com> - 2013-06-14 20:16 -0700
            Re: Debugging memory leaks Ben Finney <ben+python@benfinney.id.au> - 2013-06-15 21:23 +1000
              Re: Debugging memory leaks rusi <rustompmody@gmail.com> - 2013-06-15 04:35 -0700
                Re: Debugging memory leaks Chris Angelico <rosuav@gmail.com> - 2013-06-15 21:54 +1000
  Re: Debugging memory leaks writeson <doug.farrell@gmail.com> - 2013-06-13 11:07 -0700
    Re: Debugging memory leaks Dave Angel <davea@davea.name> - 2013-06-13 14:44 -0400
  Re: Debugging memory leaks rusi <rustompmody@gmail.com> - 2013-06-14 05:36 -0700
  Re: Debugging memory leaks "Frank Millman" <frank@chagford.com> - 2013-06-21 08:50 +0200

csiph-web