Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder1.news.weretis.net!feeder.erje.net!eu.feeder.erje.net!newsfeed.freenet.ag!news2.euro.net!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Return-Path: <ian.g.kelly@gmail.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.006
X-Spam-Evidence: '*H*': 0.99; '*S*': 0.00; 'method.': 0.05; 'correct.': 0.07; 'feasible.': 0.09; 'invalidate': 0.09; "object's": 0.09; 'skipped.': 0.09; 'typeerror:': 0.09; 'def': 0.10; 'itself.': 0.11; 'exhausted': 0.16; 'iterator': 0.16; 'iterators': 0.16; 'skips': 0.16; 'subject:optimization': 0.16; 'wrote:': 0.17; 'thu,': 0.17; '>>>': 0.18; 'all,': 0.21; '"",': 0.22; 'defined': 0.22; 'header:In-Reply-To:1': 0.25; '(most': 0.27; 'am,': 0.27; 'message-id:@mail.gmail.com': 0.27; 'subject:list': 0.28; 'methods.': 0.29; 'workaround': 0.29; 'skip:_ 10': 0.29; "skip:' 10": 0.30; 'file': 0.32; 'generally': 0.32; 'cases,': 0.33; 'point,': 0.33; 'traceback': 0.33; 'to:addr :python-list': 0.33; 'received:google.com': 0.34; 'self': 0.34; 'faster': 0.35; 'received:209.85': 0.35; 'but': 0.36; 'method': 0.36; 'should': 0.36; 'optimization': 0.37; 'does': 0.37; 'uses': 0.37; 'why': 0.37; 'received:209': 0.37; 'subject:: ': 0.38; 'mean': 0.38; 'object': 0.38; 'skip:l 20': 0.38; 'to:addr:python.org': 0.39; 'subject:-': 0.40; 'your': 0.60; 'most': 0.61; 'as:': 0.75; '2013': 0.84; 'costly': 0.84; 'hood': 0.84; 'partially': 0.84; 'to:name:python': 0.84
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:mime-version:in-reply-to:references:from:date:message-id :subject:to:content-type; bh=elB1cjEXH/bFP0LTkYkimTST7CB4z++bAhlkKVMsRQA=; b=llOJnNP9GiujnjjjZcT9GbY1ICufoA5QrKwk7gjZg6QCHxvY2JtGBMe7ZN1PpjVVA7 JZ1eejMqWWirGUtnPwfgZjIhnNuSBTISdsBn4XXvLZrsW4YMDT82fMczFB/PRMEtWXVp AcvHRCGnimL47w6twSzMQ/pObqFr1TQyapNKZLjhihuXgG64V3FrYCoIZsGCsoJ9pypW 1TQqsYFWbA5rmlsEAF8TvQ0JSsOxBwsTt3ujrC/AHjSF2HOgWW7hvIwDldb/O87R+mmx N0uBILreh1UtPGZk9QzQpm9nQm6QKk6k9r4Nq+WdwrWydCa25/CdRzhlRXQqKf6g5aLi kgJw==
X-Received: by 10.220.219.9 with SMTP id hs9mr13069200vcb.68.1362672075032; Thu, 07 Mar 2013 08:01:15 -0800 (PST)
MIME-Version: 1.0
In-Reply-To: <loom.20130307T121601-338@post.gmane.org>
References: <roy-572C99.22201106032013@70-1-84-166.pools.spcsdns.net> <20130306215735.649932ee@bigbox.christie.dr> <loom.20130307T121601-338@post.gmane.org>
From: Ian Kelly <ian.g.kelly@gmail.com>
Date: Thu, 7 Mar 2013 09:00:32 -0700
Subject: Re: Interesting list() un-optimization
To: Python <python-list@python.org>
Content-Type: text/plain; charset=ISO-8859-1
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups: comp.lang.python
Message-ID: <mailman.3035.1362672083.2939.python-list@python.org>
Lines: 34
NNTP-Posting-Host: 2001:888:2000:d::a6
X-Trace: 1362672083 news.xs4all.nl 6847 [2001:888:2000:d::a6]:53623
X-Complaints-To: abuse@xs4all.nl
Xref: csiph.com comp.lang.python:40793

On Thu, Mar 7, 2013 at 4:22 AM, Wolfgang Maier
<wolfgang.maier@biologie.uni-freiburg.de> wrote:
> Well, it skips the costly len() call because your iter(Foo()) returns
> iter(range()) under the hood and list() uses that object's __len__() method.

Iterators do not generally have __len__ methods.

>>> len(iter(range(10)))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: object of type 'range_iterator' has no len()

> In
> most cases, such a workaround will not be feasible. Why should iter(QuerySet())
> have a faster __len__() method defined than QuerySet() itself.

iter(QuerySet()) should not have any __len__ method defined at all,
which is why the optimization would be skipped.

> Most likely,
> iter(QuerySet()) just returns self anyway?

But on this point, you are correct.  The mongoengine QuerySet.__iter__
method is defined as:

    def __iter__(self):
        self.rewind()
        return self

This is unfortunate design.  Not only does it mean that the iterator's
__len__ method cannot be trusted (what should the __len__ of a
partially exhausted iterator return?), but it also means that requesting
an iterator over the QuerySet will also silently invalidate any
existing iterators.