Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!feeder.erje.net!eu.feeder.erje.net!news.stack.nl!newsfeed.xs4all.nl!newsfeed4a.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Return-Path: <python-python-list@m.gmane.org>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=python.org; s=200901; t=1410097318; bh=3wZyX9SBa42CuSh+TPcJJTpilzzTWHyNoKd9hLw+cQ4=; h=To:From:Subject:Date:References:From; b=nOsm88bXn+CuX+6Blq96btq7gilHwszH4FuQWTeMNSOqidM9zt1IFKjbtI7yAc79z y7Hiv5zTgZMf/ADD6NfEAm1wVfLlF33fT1w7n8TDkPcvEBt/BUF8ipfcj6QDOc8J25 U0t3Wcm4ZMRnCLAFRkf5cQVkDR46gcvE/wAodQBQ=
X-Spam-Status: OK 0.008
X-Spam-Evidence: '*H*': 0.98; '*S*': 0.00; 'causing': 0.04; 'tool,': 0.07; 'received:80.91': 0.09; 'received:80.91.229': 0.09; 'received:gmane.org': 0.09; 'received:list': 0.09; 'python': 0.11; 'gui': 0.12; 'suggest': 0.14; 'attributed': 0.16; 'finer': 0.16; 'from:name:antoine pitrou': 0.16; 'message-id:@post.gmane.org': 0.16; 'received:213.41.240': 0.16; 'received:213.41.240.54': 0.16; 'received:80.91.229.3': 0.16; 'received:charlus.yi.org': 0.16; 'received:plane.gmane.org': 0.16; 'received:yi.org': 0.16; 'stable.': 0.16; 'subject:slow': 0.16; 'from:addr:python.org': 0.16; "python's": 0.19; 'memory': 0.22; 'separate': 0.22; 'header :User-Agent:1': 0.23; 'replace': 0.24; 'visible': 0.24; 'regardless': 0.24; 'header:X-Complaints-To:1': 0.27; 'consumption': 0.31; 'further?': 0.31; 'writes:': 0.31; 'url:python': 0.33; 'older': 0.33; 'core': 0.34; 'objects': 0.35; 'but': 0.35; 'there': 0.35; 'next': 0.36; 'charset:us-ascii': 0.36; 'url:org': 0.36; 'application': 0.37; 'growing': 0.38; 'tools,': 0.38; 'url:library': 0.38; 'to:addr:python-list': 0.38; 'to:addr:python.org': 0.39; 'received:org': 0.40; 'how': 0.40; 'even': 0.60; 'remove': 0.60; 'analysis,': 0.60; 'analyze': 0.60; 'url:3': 0.61; 'first': 0.61; 'such': 0.63; 'believe': 0.68; 'analysis': 0.75; 'grow': 0.77; '3.4': 0.84; 'antoine.': 0.84; 'execution.': 0.84; 'isolate': 0.84; 'factors': 0.97
X-Injected-Via-Gmane: http://gmane.org/
To: python-list@python.org
From: Antoine Pitrou <antoine@python.org>
Subject: Re: weakref, memory management and execution slow down in PyQt4
Date: Sun, 7 Sep 2014 13:28:56 +0000 (UTC)
References: <540BFE43.5030006@riseup.net>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
X-Gmane-NNTP-Posting-Host: sea.gmane.org
User-Agent: Loom/3.14 (http://gmane.org/)
X-Loom-IP: 213.41.240.54 (Mozilla/5.0 (X11; Linux x86_64; rv:32.0) Gecko/20100101 Firefox/32.0)
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups: comp.lang.python
Message-ID: <mailman.13848.1410097319.18130.python-list@python.org>
Lines: 29
NNTP-Posting-Host: 2001:888:2000:d::a6
X-Trace: 1410097319 news.xs4all.nl 2961 [2001:888:2000:d::a6]:57412
X-Complaints-To: abuse@xs4all.nl
Xref: csiph.com comp.lang.python:77671

kjs <bfb <at> riseup.net> writes:
> 
> I have come to believe that the growing number of weakrefs is slowing
> down execution. Is my analysis misguided? How can I introspect further?
> If the slowdown can be attributed to weakref escalation, what are some
> next steps?

The way to analyze this is to build some gradually smaller subsets of your
application until you can isolate what is causing the growth in number of
objects (if any). I would suggest first remove the GUI and replace it with
some dummy functions, to stress your core logic.

Note that "top" isn't a very reliable tool, as memory fragmentation and
other factors can cause your process' visible size to grow even though
Python's memory consumption may be stable. There are dedicated Python tools
for finer analysis, such as tracemalloc, which is standard on 3.4 and available
as a backport for older versions:

https://docs.python.org/3/library/tracemalloc.html
http://pytracemalloc.readthedocs.org/

But regardless of such tools, the approach above (try to decompose your
workload into separate parts until your find the culprit) is highly recommended.

Regards

Antoine.