Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder1.news.weretis.net!feeder.erje.net!eu.feeder.erje.net!newsfeed.freenet.ag!news2.euro.net!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.002 X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'python.': 0.02; 'diff': 0.05; '21,': 0.07; 'matches': 0.07; 'postgresql': 0.07; 'sql.': 0.07; 'subject:code': 0.07; 'subject:help': 0.07; 'sql,': 0.09; 'resulting': 0.13; 'dec': 0.15; 'decent': 0.16; 'from:addr:rosuav': 0.16; 'from:name:chris angelico': 0.16; 'subject:making': 0.16; 'wrote:': 0.17; 'originally': 0.23; 'header:In-Reply-To:1': 0.25; 'am,': 0.27; 'message- id:@mail.gmail.com': 0.27; 'index,': 0.29; 'selecting': 0.29; 'maybe': 0.29; 'that.': 0.30; 'fri,': 0.30; 'query': 0.30; 'figure': 0.30; 'helpful': 0.30; 'getting': 0.33; 'problem': 0.33; 'to:addr:python-list': 0.33; 'another': 0.33; "can't": 0.34; 'changed': 0.34; 'received:google.com': 0.34; 'doing': 0.35; 'received:209.85.220': 0.35; 'received:209.85': 0.35; 'but': 0.36; 'wanted': 0.36; 'subject:with': 0.36; 'too': 0.36; 'problems': 0.36; 'received:209': 0.37; 'data': 0.37; 'subject:: ': 0.38; 'speak': 0.38; 'to:addr:python.org': 0.39; 'list,': 0.39; 'header:Received:5': 0.40; 'your': 0.60; 'side': 0.61; 'solve': 0.62; 'email addr:gmail.com': 0.63; 'within': 0.64; 'taking': 0.65; 'wanted,': 0.84; 'engines,': 0.93 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; bh=TFHQmEOi9wr/QHgWx0HQUfqxcEjVVERAAg/1wmD0rvE=; b=JG9TofHN77N2sIOA+mCwtrUcXc958MZQf86O+cL1FqRs2TZBJoRxfeMsUGWYhcjNA4 kKWRnMxsqC5AnYQyjYw9lLdZraMEZG4Q9Tht4EDYb0LbYOQ5dT3m/orZ62AlhTSS+rMB v2DRiWUG73vs1jiQpyvDQA1RYhjs029ZPHWcxSTB70NN+gnKHE5RN0+GWqSodZ/WST/x 0aq9rbQAIMd7RbVQn36pjaq+D4UE0BcqkoU06il8I/Vxkp2A/yAqTNAzVPTVJnECyUPh libLvWB5rE1wXWppPzUj+PDdBW5UMGChb38L+6gYgDEDaC758Ie9KsIl2hlatMLAmTNS ms1Q== MIME-Version: 1.0 In-Reply-To: References: Date: Fri, 21 Dec 2012 13:56:52 +1100 Subject: Re: help with making my code more efficient From: Chris Angelico To: python-list@python.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 19 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1356058620 news.xs4all.nl 6940 [2001:888:2000:d::a6]:45692 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:35263 On Fri, Dec 21, 2012 at 11:43 AM, Larry.Martell@gmail.com wrote: > It came from a database. Originally I was getting just the data I wanted = using SQL, but that was taking too long also. I was selecting just the mess= ages I wanted, then for each one of those doing another query to get the da= ta within the time diff of each. That was resulting in tens of thousands of= queries. So I changed it to pull all the potential matches at once and the= n process it in python. Then the best thing to do is figure out how to solve your problem in SQL. Any decent database engine will be able to optimize that beautifully, and without multiple recursive searches. You may need to create an index, but maybe not even that. I can't speak for other engines, but PostgreSQL has an excellently helpful mailing list, if you have problems with that side of it. But have a shot at writing the SQL; chances are it'll work out easily. ChrisA