Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #47906

Re: Debugging memory leaks

Show all headers | View raw

writeson <doug.farrell@gmail.com> writes:

> ...
> Anyway, my real question is how to go about debugging memory leak problems in Python, particularly for a long running server process written with Twisted. I'm not sure how to use heapy or guppy, and objgraph doesn't tell me enough to locate the problem.

Analysing memory leaks is really difficult: huge amounts of data
is involved and usually, it is almost impossible to determine which
of the mentioned objects are leaked and which are rightfully in use.
In addition, long running Python processes usually have degrading
memory use - due to memory fragmentation. There is nothing you
can do against this.

Therefore: if the leak seems to be small, it may be much more
advicable to restart your process periodically (during times
where a restart does not hurt much) rather than try to find
(and fix) the leaks. Only when the leak is large enough that
it would force you to too frequent restarts, a deeper
analysis may be advicable (large leaks are easier to locate as well).


I have analysed memory leaks several times for Zope applications.

Zope helped me much by its "Top Refcount" functionality.
This uses the fact that a class/type instance (in many cases)
holds a reference to the corresponding class/type instance
(it seems not to work for all elementary types).
Thus looking at the refcount of the class/type gives you
an indication how many instances of this class/type are around.
Zope presents this information sorted by number.
Then you send requests against Zope and reexamine the information:
You get something like:

 Class                  June 13, 2013 8:18 am 	June 13, 2013 8:18 am 	Delta
...ApplicationManager 	15                      22                      +7
...DebugManager 	9                       12                      +3 

In the case above, my requests have created 7 additional
"ApplicationManager" and 3 additional "DebugManager" instances.

If Zope objects for which this functionality works leak,
then this is a powerful tool to detect those object classes.

You could implement something similar for your server.


As mentioned, the approach does not work for (many; all?) elementary
Python types. Thus, if the leak involves only those instances, it
cannot be detected this way.

Memory leaks are often introduced by C extensions - and do not
involve Python objects (but leak C level memory). Those, too,
cannot be analysed by Python level approaches.

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Debugging memory leaks writeson <doug.farrell@gmail.com> - 2013-06-12 18:24 -0700
  Re: Debugging memory leaks dieter <dieter@handshake.de> - 2013-06-13 08:29 +0200
    Re: Debugging memory leaks Giorgos Tzampanakis <giorgos.tzampanakis@gmail.com> - 2013-06-13 20:15 +0000
      Re: Debugging memory leaks Steve Simmons <square.steve@gmail.com> - 2013-06-13 22:45 +0100
      Re: Debugging memory leaks Chris Angelico <rosuav@gmail.com> - 2013-06-14 08:36 +1000
      Re: Debugging memory leaks Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-06-14 02:40 +0000
        Re: Debugging memory leaks Chris Angelico <rosuav@gmail.com> - 2013-06-14 19:30 +1000
        Re: Debugging memory leaks Giorgos Tzampanakis <giorgos.tzampanakis@gmail.com> - 2013-06-14 22:57 +0000
          Re: Debugging memory leaks Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-06-15 01:39 +0000
        Re: Debugging memory leaks dieter <dieter@handshake.de> - 2013-06-15 08:52 +0200
        Re: Debugging memory leaks Chris Angelico <rosuav@gmail.com> - 2013-06-15 17:21 +1000
        Re: Debugging memory leaks dieter <dieter@handshake.de> - 2013-06-16 08:18 +0200
      Re: Debugging memory leaks rusi <rustompmody@gmail.com> - 2013-06-14 06:53 -0700
        Re: Debugging memory leaks Chris Angelico <rosuav@gmail.com> - 2013-06-15 10:11 +1000
        Re: Debugging memory leaks Ben Finney <ben+python@benfinney.id.au> - 2013-06-15 10:16 +1000
          Re: Debugging memory leaks rusi <rustompmody@gmail.com> - 2013-06-14 20:16 -0700
            Re: Debugging memory leaks Ben Finney <ben+python@benfinney.id.au> - 2013-06-15 21:23 +1000
              Re: Debugging memory leaks rusi <rustompmody@gmail.com> - 2013-06-15 04:35 -0700
                Re: Debugging memory leaks Chris Angelico <rosuav@gmail.com> - 2013-06-15 21:54 +1000
  Re: Debugging memory leaks writeson <doug.farrell@gmail.com> - 2013-06-13 11:07 -0700
    Re: Debugging memory leaks Dave Angel <davea@davea.name> - 2013-06-13 14:44 -0400
  Re: Debugging memory leaks rusi <rustompmody@gmail.com> - 2013-06-14 05:36 -0700
  Re: Debugging memory leaks "Frank Millman" <frank@chagford.com> - 2013-06-21 08:50 +0200

csiph-web