Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder4.news.weretis.net!newsreader4.netcologne.de!news.netcologne.de!newsfeed.freenet.ag!news2.euro.net!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Return-Path: <rosuav@gmail.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.008
X-Spam-Evidence: '*H*': 0.98; '*S*': 0.00; 'memory.': 0.05; '21,': 0.07; 'subject:code': 0.07; 'subject:help': 0.07; 'dec': 0.15; '11:19': 0.16; 'from:addr:rosuav': 0.16; 'from:name:chris angelico': 0.16; 'run.': 0.16; 'subject:making': 0.16; 'wrote:': 0.17; 'typical': 0.17; 'elements': 0.23; 'header:In-Reply-To:1': 0.25; '(which': 0.26; 'am,': 0.27; 'message-id:@mail.gmail.com': 0.27; 'run': 0.28; 'e.g.': 0.30; 'fri,': 0.30; 'code': 0.31; 'could': 0.32; 'anyone': 0.33; 'to:addr:python-list': 0.33; 'received:google.com': 0.34; 'received:209.85': 0.35; 'but': 0.36; 'subject:with': 0.36; 'too': 0.36; 'enough': 0.36; 'execute': 0.37; 'received:209': 0.37; 'data': 0.37; 'subject:: ': 0.38; 'some': 0.38; 'to:addr:python.org': 0.39; 'takes': 0.39; 'header:Received:5': 0.40; 'email addr:gmail.com': 0.63; 'hours': 0.66; 'sounds': 0.71
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=0IG7p2MtGRIYwrEnETZMAhX/tu1KtDybRN67v/P3TzU=; b=lue1aFditIXtwjej+qFQ9ZvBW5xIeG25L40bAXN1IeClospPj+xbwRm4tGISIkvI1u Ffn2o/eKukq1MBsv3kYEIBYveGx+745xHx0KPfJrZ5/d9VRAOnAQ8Oz2az2sJN5NCJ66 KTS9PIraFDdLIsHUJLUI8PuhozPS8n/QxCNIEN5A8GSP41CfkYZxqjfKRsiaX02t3UGD oywGuqwPlumiWGT4+XU7jALyGtB3ekFV5j/4GIIVsyt+QVFhFWhgZfMTv3kP5nyGoiqk dDstcXAqORi7+raNkleWC/M2Vz4p7kCnkvvK/txj1yW6FaoN4YFSb4T97UspsoImABIX 5YRQ==
MIME-Version: 1.0
In-Reply-To: <bd5ae6b7-2440-42e4-a93c-eb877feebcfe@googlegroups.com>
References: <bd5ae6b7-2440-42e4-a93c-eb877feebcfe@googlegroups.com>
Date: Fri, 21 Dec 2012 11:38:03 +1100
Subject: Re: help with making my code more efficient
From: Chris Angelico <rosuav@gmail.com>
To: python-list@python.org
Content-Type: text/plain; charset=ISO-8859-1
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups: comp.lang.python
Message-ID: <mailman.1123.1356050292.29569.python-list@python.org>
Lines: 12
NNTP-Posting-Host: 2001:888:2000:d::a6
X-Trace: 1356050292 news.xs4all.nl 6853 [2001:888:2000:d::a6]:39992
X-Complaints-To: abuse@xs4all.nl
Xref: csiph.com comp.lang.python:35246

On Fri, Dec 21, 2012 at 11:19 AM, Larry.Martell@gmail.com
<Larry.Martell@gmail.com> wrote:
> This code works, but it takes way too long to run - e.g. when cdata has 600,000 elements (which is typical for my app) it takes 2 hours for this to run.
>
> Can anyone give me some suggestions on speeding this up?
>

It sounds like you may have enough data to want to not keep it all in
memory. Have you considered switching to a database? You could then
execute SQL queries against it.

ChrisA