Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!news.mixmin.net!feeds.phibee-telecom.net!newsfeed.xs4all.nl!newsfeed4.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.016 X-Spam-Evidence: '*H*': 0.97; '*S*': 0.00; 'sized': 0.07; 'advance': 0.07; 'received:80.91': 0.09; 'received:80.91.229': 0.09; 'received:gmane.org': 0.09; 'received:list': 0.09; 'run,': 0.09; 'shaw': 0.09; 'comp': 0.16; 'determining': 0.16; 'finney': 0.16; 'hmm.': 0.16; 'in-place': 0.16; 'iterables': 0.16; 'reasonably': 0.16; 'received:80.91.229.3': 0.16; 'received:plane.gmane.org': 0.16; 'version.': 0.19; 'memory': 0.22; 'example': 0.22; 'coding': 0.22; 'header:User-Agent:1': 0.23; 'header:X-Complaints-To:1': 0.27; 'point': 0.28; 'quickly': 0.29; 'subject:list': 0.30; 'code': 0.31; 'constant': 0.31; "d'aprano": 0.31; 'fast.': 0.31; 'giant': 0.31; 'steven': 0.31; 'writes:': 0.31; 'could': 0.34; 'except': 0.35; 'but': 0.35; 'there': 0.35; 'version': 0.36; 'useful': 0.36; 'should': 0.36; 'list': 0.37; 'ben': 0.38; 'branch': 0.38; 'to:addr:python-list': 0.38; 'fact': 0.38; 'to:addr:python.org': 0.39; 'received:org': 0.40; 'trading': 0.61; 'times': 0.62; 'more': 0.64; 'determine': 0.67; '2-3': 0.68; 'judge': 0.68; 'apart': 0.72; 'million': 0.74; 'cut-off': 0.84; 'improvement,': 0.84; 'items,': 0.91; 'ratio': 0.91; '\xe2\x80\x9cthe': 0.91 X-Injected-Via-Gmane: http://gmane.org/ To: python-list@python.org From: Ben Finney Subject: Re: Optimizing list processing Date: Thu, 12 Dec 2013 12:18:30 +1100 References: <52a8fb2d$0$29992$c3e8da3$5496439d@news.astraweb.com> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Gmane-NNTP-Posting-Host: rasputin.madmonks.org X-Public-Key-ID: 0xAC128405 X-Public-Key-Fingerprint: 517C F14B B2F3 98B0 CB35 4855 B8B2 4C06 AC12 8405 X-Public-Key-URL: http://www.benfinney.id.au/contact/bfinney-gpg.asc X-Post-From: Ben Finney User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.4 (gnu/linux) Cancel-Lock: sha1:Ok5xGis0dDJ4O6vTnQJ1ZrNd9wo= X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 26 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1386811122 news.xs4all.nl 2908 [2001:888:2000:d::a6]:36484 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:61638 Steven D'Aprano writes: > For giant iterables (ten million items), this version is a big > improvement, about three times faster than the list comp version. […] > > Except that for more reasonably sized iterables, it's a pessimization. > With one million items, the ratio is the other way around: the list > comp version is 2-3 times faster than the in-place version. For > smaller lists, the ratio varies, but the list comp version is > typically around twice as fast. A good example of trading memory for > time. > Is there any way to determine which branch I should run, apart from > hard- coding some arbitrary and constant cut-off value? Hmm. The code isn't going to be able to accurately judge in advance the time it'll take to run. But perhaps it could quickly and accurately calculate the memory usage of your data structure? Is that useful for determining which branch to take? -- \ “The fact that a believer is happier than a skeptic is no more | `\ to the point than the fact that a drunken man is happier than a | _o__) sober one.” —George Bernard Shaw | Ben Finney