Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #7928

Re: Improper creating of logger instances or a Memory Leak?

Show all headers | View raw

In article <ebafe7b6-aa93-4847-81d6-12d396a4ff3c@j28g2000vbp.googlegroups.com>
foobar  <wjshipman@gmail.com> wrote:
>I've run across a memory leak in a long running process which I can't
>determine if its my issue or if its the logger.

You do not say what version of python you are using, but on the
other hand I do not know how much the logger code has evolved
over time anyway. :-)

> Each application thread gets a logger instance in it's init() method
>via:
>
>        self.logger = logging.getLogger('ivr-'+str(self.rand))
>
>where self.rand is a suitably large random number to avoid collisions
>of the log file's name.

This instance will "live forever" (since the thread shares the
main logging manager with all other threads).
---------
class Manager:
    """
    There is [under normal circumstances] just one Manager instance, which
    holds the hierarchy of loggers.
    """
    def __init__(self, rootnode):
        """
        Initialize the manager with the root node of the logger hierarchy.
        """
        [snip]
        self.loggerDict = {}

    def getLogger(self, name):
        """
        Get a logger with the specified name (channel name), creating it
        if it doesn't yet exist. This name is a dot-separated hierarchical
        name, such as "a", "a.b", "a.b.c" or similar.

        If a PlaceHolder existed for the specified name [i.e. the logger
        didn't exist but a child of it did], replace it with the created
        logger and fix up the parent/child references which pointed to the
        placeholder to now point to the logger.
        """
        [snip]
                    self.loggerDict[name] = rv
        [snip]
[snip]
Logger.manager = Manager(Logger.root)
---------

So you will find all the various ivr-* loggers in
logging.Logger.manager.loggerDict[].

>finally the last statements in the run() method are:
>
>        filehandler.close()
>        self.logger.removeHandler(filehandler)
>        del self.logger #this was added to try and force a clean up of
>the logger instances.

There appears to be no __del__ handler and nothing that allows
removing a logger instance from the manager's loggerDict.  Of
course you could do this "manually", e.g.:

        ...
        self.logger.removeHandler(filehandler)
        del logging.Logger.manager.loggerDict[self.logger.name]
        del self.logger # optional

I am curious as to why you create a new logger for each thread.
The logging module has thread synchronization in it, so that you
can share one log (or several logs) amongst all threads, which is
more typically what one wants.
-- 
In-Real-Life: Chris Torek, Wind River Systems
Salt Lake City, UT, USA (40°39.22'N, 111°50.29'W)  +1 801 277 2603
email: gmail (figure it out)      http://web.torek.net/torek/index.html

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Improper creating of logger instances or a Memory Leak? foobar <wjshipman@gmail.com> - 2011-06-18 13:02 -0700
  Re: Improper creating of logger instances or a Memory Leak? Chris Torek <nospam@torek.net> - 2011-06-18 22:28 +0000
  Re: Improper creating of logger instances or a Memory Leak? Vinay Sajip <vinay_sajip@yahoo.co.uk> - 2011-06-19 14:29 +0000
  Re: Improper creating of logger instances or a Memory Leak? Vinay Sajip <vinay_sajip@yahoo.co.uk> - 2011-06-19 14:42 +0000
    Re: Improper creating of logger instances or a Memory Leak? foobar <wjshipman@gmail.com> - 2011-06-20 07:50 -0700
      Re: Improper creating of logger instances or a Memory Leak? Vinay Sajip <vinay_sajip@yahoo.co.uk> - 2011-06-20 17:19 -0700

csiph-web