Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!feeder.erje.net!eu.feeder.erje.net!newsfeed.fsmpi.rwth-aachen.de!feeds.phibee-telecom.net!newsfeed.xs4all.nl!newsfeed1a.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.003 X-Spam-Evidence: '*H*': 0.99; '*S*': 0.00; 'causing': 0.04; 'explicitly': 0.05; 'explicit': 0.07; 'subject:script': 0.09; 'try:': 0.09; 'cc:addr:python-list': 0.11; 'from:addr:rosuav': 0.16; 'from:name:chris angelico': 0.16; 'loop.': 0.16; 'stuff,': 0.16; 'prevent': 0.16; 'fix': 0.17; 'wrote:': 0.18; 'properly': 0.19; 'memory': 0.22; 'cc:addr:python.org': 0.22; 'cc:2**0': 0.24; 'query': 0.26; 'pass': 0.26; 'somewhere': 0.26; 'header:In-Reply- To:1': 0.27; 'am,': 0.29; 'especially': 0.30; 'message- id:@mail.gmail.com': 0.30; 'that.': 0.31; 'away.': 0.31; 'breaking': 0.31; 'larry': 0.31; 'them?': 0.31; 'this.': 0.32; 'figure': 0.32; 'subject:all': 0.32; 'url:python': 0.33; 'fri,': 0.33; 'something': 0.35; 'received:google.com': 0.35; 'returning': 0.36; 'next': 0.36; 'url:org': 0.36; 'url:library': 0.38; 'issue': 0.38; 'does': 0.39; 'how': 0.40; 'free': 0.61; 'skip:t 30': 0.61; 'mar': 0.68; 'to:none': 0.92 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:cc :content-type; bh=RHI1PMrtdqpanZgoh+yCH/Qk9ib9xS+ofD9kvBb5H0I=; b=dVaEp8ZoPHR1F3vlcW7YXOUB5kKIQfyWpvtwzAk+vx08YsmGIAUzcnUJxSaKFhBH3f fHhFRttgwuMJm36ROWPmr5TtNfbCXEUflMpAo7X7mG3VtIsDC5iJ0xRJTjxHaPQc/UW6 JHz/LQGZhEYN1P1CNcDMxuMOrzshB5KbcKT1ir0gBWaCv24ao+4IteOuliD3hF9mJQxX t90jrCPjfsdF3zyKS0YaGwV6lLwxbfJf9djg0xG79/Cugm7gG1bjvpaNLJOjvO0ZmO/s JpJE/zzTdC9KxweBPtAj+pSfbRnwllEWVBq1FTUacuIYEdZ33TGtrSCqqsffC00YDUPG 3nOQ== MIME-Version: 1.0 X-Received: by 10.66.118.71 with SMTP id kk7mr17619437pab.14.1394143898064; Thu, 06 Mar 2014 14:11:38 -0800 (PST) In-Reply-To: References: Date: Fri, 7 Mar 2014 09:11:37 +1100 Subject: Re: script uses up all memory From: Chris Angelico Cc: "python-list@python.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 24 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1394143906 news.xs4all.nl 2863 [2001:888:2000:d::a6]:47776 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:67955 On Fri, Mar 7, 2014 at 8:56 AM, Larry Martell wrote: > I figured out what is causing this. Each pass through the loop it does: > > self.tools = Tool.objects.filter(ip__isnull=False) > > And that is what is causing the memory consumption. If I move that > outside the loop and just do that once the memory issue goes away. Now > I need to figure out why this is happening and how to prevent it as > they do want to query the db each pass through the loop in case it has > been updated. Interesting. So the next thing to do is to look into the implementation of that. Does it allocate database resources and not free them? Does it have internal reference loops? Something to try: Put an explicit gc.collect() call into the loop. If that solves your problem, you have a refloop somewhere (and you can properly fix it by explicitly breaking the loop). If that keeps returning large numbers, and especially if it populates gc.garbage with a whole lot of stuff, then you definitely have refloops. http://docs.python.org/2/library/gc.html ChrisA