Path: csiph.com!newsfeed.hal-mli.net!feeder3.hal-mli.net!newsfeed.hal-mli.net!feeder1.hal-mli.net!newsfeed.xs4all.nl!newsfeed3.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.005 X-Spam-Evidence: '*H*': 0.99; '*S*': 0.00; 'causing': 0.04; 'explicitly': 0.05; 'explicit': 0.07; 'subject:script': 0.09; 'try:': 0.09; 'django': 0.11; 'dig': 0.16; 'loop.': 0.16; 'stuff,': 0.16; 'prevent': 0.16; 'fix': 0.17; 'wrote:': 0.18; 'code.': 0.18; 'properly': 0.19; 'thu,': 0.19; 'thanks.': 0.20; 'memory': 0.22; 'to:name:python-list@python.org': 0.22; 'query': 0.26; 'pass': 0.26; 'somewhere': 0.26; 'header:In-Reply-To:1': 0.27; 'chris': 0.29; 'am,': 0.29; 'especially': 0.30; 'message- id:@mail.gmail.com': 0.30; "i'm": 0.30; 'that.': 0.31; 'away.': 0.31; 'breaking': 0.31; 'larry': 0.31; 'them?': 0.31; 'this.': 0.32; 'figure': 0.32; 'subject:all': 0.32; 'url:python': 0.33; 'call.': 0.33; 'fri,': 0.33; 'something': 0.35; 'received:google.com': 0.35; 'returning': 0.36; 'next': 0.36; 'url:org': 0.36; 'url:library': 0.38; 'to:addr:python-list': 0.38; 'issue': 0.38; 'pm,': 0.38; 'does': 0.39; 'to:addr:python.org': 0.39; 'how': 0.40; 'solve': 0.60; 'free': 0.61; 'skip:t 30': 0.61; 'first': 0.61; 'mar': 0.68; 'victory': 0.84 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=tZG3HAwsX30d9JtICLmhPz5x4OOsd8Yg5zu5oH3EU9s=; b=lkfbSuHJjcFnlrDGHUSJ572tvenpJOuobto1W41nihpxb2Rs+5KM5ejXm/eloy0voK 4y7kGTf1oCBq9de5wX+lkEVlOYfabNmopLl1S26lWblACCftxTMQdRjlJ6T7G2GRB5Zc tbxOEnLoGmXUUvrDUQ0wUN140aNNe0Q/ArM5IwRtzk/ssp3gqQlPjTsHy3xmJhS7+WkE QIHpjYNDR+Z1SzJw4Jq/4crnDJsAQXxaGg/3HgNstqSIj6dTY/OBv+DblijUo0h47czq ZkaAHXPk52hra7az5Dcw+iLkGAQkmkFw6etPtg4gpcM4xfBO13hqTFrXIokFy0LH/Czj 3CHg== MIME-Version: 1.0 X-Received: by 10.194.60.37 with SMTP id e5mr13687461wjr.32.1394144486585; Thu, 06 Mar 2014 14:21:26 -0800 (PST) In-Reply-To: References: Date: Thu, 6 Mar 2014 17:21:26 -0500 Subject: Re: script uses up all memory From: Larry Martell To: "python-list@python.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 30 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1394144487 news.xs4all.nl 2835 [2001:888:2000:d::a6]:53371 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:67957 On Thu, Mar 6, 2014 at 5:11 PM, Chris Angelico wrote: > On Fri, Mar 7, 2014 at 8:56 AM, Larry Martell wrote: >> I figured out what is causing this. Each pass through the loop it does: >> >> self.tools = Tool.objects.filter(ip__isnull=False) >> >> And that is what is causing the memory consumption. If I move that >> outside the loop and just do that once the memory issue goes away. Now >> I need to figure out why this is happening and how to prevent it as >> they do want to query the db each pass through the loop in case it has >> been updated. > > Interesting. So the next thing to do is to look into the > implementation of that. Does it allocate database resources and not > free them? Does it have internal reference loops? > > Something to try: Put an explicit gc.collect() call into the loop. If > that solves your problem, you have a refloop somewhere (and you can > properly fix it by explicitly breaking the loop). If that keeps > returning large numbers, and especially if it populates gc.garbage > with a whole lot of stuff, then you definitely have refloops. > > http://docs.python.org/2/library/gc.html First I added del(self.tools) before the Django call. That did not stop the memory consumption. Then I added a call to gc.collect() after the del and that did solve it. gc.collect() returns 0 each time, so I'm going to declare victory and move on. No time to dig into the Django code. Thanks.