Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #61638

Re: Optimizing list processing

Path csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!news.mixmin.net!feeds.phibee-telecom.net!newsfeed.xs4all.nl!newsfeed4.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Return-Path <python-python-list@m.gmane.org>
X-Original-To python-list@python.org
Delivered-To python-list@mail.python.org
X-Spam-Status OK 0.016
X-Spam-Evidence '*H*': 0.97; '*S*': 0.00; 'sized': 0.07; 'advance': 0.07; 'received:80.91': 0.09; 'received:80.91.229': 0.09; 'received:gmane.org': 0.09; 'received:list': 0.09; 'run,': 0.09; 'shaw': 0.09; 'comp': 0.16; 'determining': 0.16; 'finney': 0.16; 'hmm.': 0.16; 'in-place': 0.16; 'iterables': 0.16; 'reasonably': 0.16; 'received:80.91.229.3': 0.16; 'received:plane.gmane.org': 0.16; 'version.': 0.19; 'memory': 0.22; 'example': 0.22; 'coding': 0.22; 'header:User-Agent:1': 0.23; 'header:X-Complaints-To:1': 0.27; 'point': 0.28; 'quickly': 0.29; 'subject:list': 0.30; 'code': 0.31; 'constant': 0.31; "d'aprano": 0.31; 'fast.': 0.31; 'giant': 0.31; 'steven': 0.31; 'writes:': 0.31; 'could': 0.34; 'except': 0.35; 'but': 0.35; 'there': 0.35; 'version': 0.36; 'useful': 0.36; 'should': 0.36; 'list': 0.37; 'ben': 0.38; 'branch': 0.38; 'to:addr:python-list': 0.38; 'fact': 0.38; 'to:addr:python.org': 0.39; 'received:org': 0.40; 'trading': 0.61; 'times': 0.62; 'more': 0.64; 'determine': 0.67; '2-3': 0.68; 'judge': 0.68; 'apart': 0.72; 'million': 0.74; 'cut-off': 0.84; 'improvement,': 0.84; 'items,': 0.91; 'ratio': 0.91; '\xe2\x80\x9cthe': 0.91
X-Injected-Via-Gmane http://gmane.org/
To python-list@python.org
From Ben Finney <ben+python@benfinney.id.au>
Subject Re: Optimizing list processing
Date Thu, 12 Dec 2013 12:18:30 +1100
References <52a8fb2d$0$29992$c3e8da3$5496439d@news.astraweb.com>
Mime-Version 1.0
Content-Type text/plain; charset=utf-8
Content-Transfer-Encoding 8bit
X-Gmane-NNTP-Posting-Host rasputin.madmonks.org
X-Public-Key-ID 0xAC128405
X-Public-Key-Fingerprint 517C F14B B2F3 98B0 CB35 4855 B8B2 4C06 AC12 8405
X-Public-Key-URL http://www.benfinney.id.au/contact/bfinney-gpg.asc
X-Post-From Ben Finney <bignose+hates-spam@benfinney.id.au>
User-Agent Gnus/5.13 (Gnus v5.13) Emacs/23.4 (gnu/linux)
Cancel-Lock sha1:Ok5xGis0dDJ4O6vTnQJ1ZrNd9wo=
X-BeenThere python-list@python.org
X-Mailman-Version 2.1.15
Precedence list
List-Id General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe <https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive <http://mail.python.org/pipermail/python-list/>
List-Post <mailto:python-list@python.org>
List-Help <mailto:python-list-request@python.org?subject=help>
List-Subscribe <https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups comp.lang.python
Message-ID <mailman.3947.1386811122.18130.python-list@python.org> (permalink)
Lines 26
NNTP-Posting-Host 2001:888:2000:d::a6
X-Trace 1386811122 news.xs4all.nl 2908 [2001:888:2000:d::a6]:36484
X-Complaints-To abuse@xs4all.nl
Xref csiph.com comp.lang.python:61638

Show key headers only | View raw

Steven D'Aprano <steve+comp.lang.python@pearwood.info> writes:

> For giant iterables (ten million items), this version is a big 
> improvement, about three times faster than the list comp version. […]
>
> Except that for more reasonably sized iterables, it's a pessimization.
> With one million items, the ratio is the other way around: the list
> comp version is 2-3 times faster than the in-place version. For
> smaller lists, the ratio varies, but the list comp version is
> typically around twice as fast. A good example of trading memory for
> time.

> Is there any way to determine which branch I should run, apart from
> hard- coding some arbitrary and constant cut-off value?

Hmm. The code isn't going to be able to accurately judge in advance the
time it'll take to run. But perhaps it could quickly and accurately
calculate the memory usage of your data structure? Is that useful for
determining which branch to take?

-- 
 \      “The fact that a believer is happier than a skeptic is no more |
  `\   to the point than the fact that a drunken man is happier than a |
_o__)                                 sober one.” —George Bernard Shaw |
Ben Finney

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Optimizing list processing Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-12-11 23:54 +0000
  Re: Optimizing list processing MRAB <python@mrabarnett.plus.com> - 2013-12-12 00:59 +0000
    Re: Optimizing list processing Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-12-12 01:43 +0000
      Re: Optimizing list processing MRAB <python@mrabarnett.plus.com> - 2013-12-12 02:09 +0000
  Re: Optimizing list processing duncan smith <buzzard@invalid.invalid> - 2013-12-12 01:02 +0000
  Re: Optimizing list processing Ben Finney <ben+python@benfinney.id.au> - 2013-12-12 12:18 +1100
  Re: Optimizing list processing Terry Reedy <tjreedy@udel.edu> - 2013-12-11 21:26 -0500
    Re: Optimizing list processing Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-12-12 12:08 +0000
      Re: Optimizing list processing Chris Angelico <rosuav@gmail.com> - 2013-12-12 23:25 +1100
      Re: Optimizing list processing MRAB <python@mrabarnett.plus.com> - 2013-12-12 13:32 +0000
      Re: Optimizing list processing Chris Angelico <rosuav@gmail.com> - 2013-12-13 01:06 +1100
      Re: Optimizing list processing Terry Reedy <tjreedy@udel.edu> - 2013-12-12 13:40 -0500
        Re: Optimizing list processing Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-12-13 00:14 +0000
          Re: Optimizing list processing Chris Angelico <rosuav@gmail.com> - 2013-12-13 12:01 +1100
  Re: Optimizing list processing Stefan Behnel <stefan_ml@behnel.de> - 2013-12-12 12:09 +0100
  Re: Optimizing list processing Peter Otten <__peter__@web.de> - 2013-12-12 16:08 +0100
    Re: Optimizing list processing Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-12-13 03:01 +0000
      Re: Optimizing list processing rusi <rustompmody@gmail.com> - 2013-12-12 21:35 -0800
  Re: Optimizing list processing Terry Reedy <tjreedy@udel.edu> - 2013-12-12 13:07 -0500

csiph-web