Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder1.news.weretis.net!feeder.erje.net!eu.feeder.erje.net!xlned.com!feeder5.xlned.com!newsfeed.xs4all.nl!newsfeed2.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.002 X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'cpython': 0.05; 'detect': 0.07; 'referring': 0.07; '22,': 0.09; 'happen,': 0.09; 'objects,': 0.09; 'references,': 0.09; 'run,': 0.09; 'cc:addr:python-list': 0.11; 'python': 0.11; 'jan': 0.12; 'assume': 0.14; 'behave': 0.16; 'feasible': 0.16; 'flush': 0.16; 'from:addr:rosuav': 0.16; 'from:name:chris angelico': 0.16; 'garbage': 0.16; 'happily': 0.16; 'heap': 0.16; 'improper': 0.16; 'leakage': 0.16; 'subject:Self': 0.16; 'wrote:': 0.18; 'wed,': 0.18; 'slightly': 0.19; 'code,': 0.22; 'memory': 0.22; 'cc:addr:python.org': 0.22; 'candidates': 0.23; "aren't": 0.24; 'module,': 0.24; 'cc:2**0': 0.24; 'define': 0.26; 'extension': 0.26; 'references': 0.26; 'header:In-Reply-To:1': 0.27; 'correct': 0.29; 'generally': 0.29; 'message-id:@mail.gmail.com': 0.30; 'code': 0.31; 'that.': 0.31; 'usually': 0.31; '(maybe': 0.31; 'running': 0.33; 'entirely': 0.33; 'candidate': 0.34; 'problem': 0.35; 'objects': 0.35; 'but': 0.35; 'received:google.com': 0.35; 'there': 0.35; 'representing': 0.36; 'possible': 0.36; 'las': 0.37; 'mapping': 0.38; 'pm,': 0.38; 'track': 0.38; 'truly': 0.60; 'up,': 0.60; 'most': 0.60; 'simply': 0.61; "you're": 0.61; "you'll": 0.62; 'name': 0.63; 'map': 0.64; 'places': 0.64; 'different': 0.65; 'talking': 0.65; 'due': 0.66; 'to,': 0.72; 'programs,': 0.74; 'thing,': 0.91; 'to:none': 0.92 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:cc :content-type; bh=ERX6W0gq0nWv77ldDu2wcmNyYXY3t33YGM8Vwepc8GU=; b=toeIFSesS/6Tc0sdPEBboO84bTOJj667Kk+gbZlSNaG/y1bLciymLaYe7lKqL9iw5i 2qZF6JS1bzD0w4TgUa90JXlI7T94K7RpakyTmO/tuymo/SoOjgWxzLyFUdw5YampNVE+ sErn5LYvIjrRJvPRR3EPdcRqyifcJOTe4zZc/2GkUlD0MtvhOAKoVtIMMLLTgc9zpa44 HhIQpvux4xn1cx5w8bVn2uR2+w1OsjrFaobXNZgPYRx7x39Odm/ZMBMdXPsKJSPg3EP9 O7r500V+Xxr1UuDKjxM5fIxCukydJgzrsgKBnzOdrhePyGGZ8Kp0NgrE/qIbgPybgCtu gtYQ== MIME-Version: 1.0 X-Received: by 10.66.66.234 with SMTP id i10mr18144809pat.127.1390360105620; Tue, 21 Jan 2014 19:08:25 -0800 (PST) In-Reply-To: <0d1fc1a7-c585-45ba-8c1a-0cc468712a48@googlegroups.com> References: <0d1fc1a7-c585-45ba-8c1a-0cc468712a48@googlegroups.com> Date: Wed, 22 Jan 2014 14:08:25 +1100 Subject: Re: Self healthcheck From: Chris Angelico Cc: "python-list@python.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 27 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1390360114 news.xs4all.nl 2856 [2001:888:2000:d::a6]:53428 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:64478 On Wed, Jan 22, 2014 at 1:51 PM, Asaf Las wrote: > When designing long running background process > is it feasible to monitor object/memory leakage due > to improper programming? I assume you're talking about pure Python code, running under CPython. (If you're writing an extension module, say in C, there are completely different ways to detect reference leaks; and other Pythons will behave slightly differently.) There's no way to detect truly unreferenced objects, because they simply won't exist - not after a garbage collection run, and usually sooner than that. But if you want to find objects that you're somehow not using and yet still have live references to, you'll need to define "using" in a way that makes sense. Generally there aren't many ways that that can happen, so those few places are candidates for a weak reference system (maybe you map a name to the "master object" representing that thing, and you can recreate the master object from the disk, so when nothing else is referring to it, you can happily flush it out - that mapping is a good candidate for weak references). But for most programs, don't bother. CPython is pretty good at keeping track of its own references, so chances are you don't need to - and if you're seeing the process's memory usage going up, it's entirely possible you can neither detect nor correct the problem in Python code (eg heap fragmentation). ChrisA