Path: csiph.com!usenet.pasdenom.info!dedibox.gegeweb.org!gegeweb.eu!nntpfeed.proxad.net!proxad.net!feeder1-2.proxad.net!news.tele.dk!news.tele.dk!small.news.tele.dk!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
To: python-list@python.org
From: Wolfgang Maier <wolfgang.maier@biologie.uni-freiburg.de>
Subject: Re: Different behaviour in list comps and generator expressions
Date: Sat, 08 Nov 2014 23:42:29 +0100
References: <545d76fe$0$12980$c3e8da3$5496439d@news.astraweb.com> <m3m23c$n3s$1@ger.gmane.org>
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.2.0
In-Reply-To: <m3m23c$n3s$1@ger.gmane.org>
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.15589.1415486565.18130.python-list@python.org>
Lines: 95
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:80408

On 08.11.2014 22:31, Wolfgang Maier wrote:
> On 08.11.2014 02:50, Steven D'Aprano wrote:
>> The following list comprehension and generator expression are almost, but
>> not quite, the same:
>>
>> [expr for x in iterable]
>>
>> list(expr for x in iterable)
>>
>>
>> The difference is in the handling of StopIteration raised inside the
>> expr.
>> Generator expressions consume them and halt, while comprehensions allow
>> them to leak out.
>
> This is not the right description of what's happening. It is not the
> generator expression that consumes it, but the list constructor and, of
> course, list can't tell at which level in the inside code StopIteration
> got raised.
> So, yes this had me confused some times, too, but it is really not
> surprising.
>
>> A simple example:
>>
>> iterable = [iter([])]
>> list(next(x) for x in iterable)
>> => returns []
>>
>> But:
>>
>> [next(x) for x in iterable]
>> => raises StopIteration
>>
>
> Yes, but the equivalent to list(next(x) for x in iterable) is:
>
> l = []
> for item in (next(x) for x in iterable):
>      l.append(item)
>
> => returns [] because the for consumes the StopIteration
>
> this, on the other hand raises StopIteration
>
> l = []
> for item in [next(x) for x in iterable]:
>      l.append(item)
>
> because the error gets raised already when for causes iter() to be
> called on the list comprehension.
>
>>
>> Has anyone come across this difference in the wild? Was it a problem?
>> Do you
>> rely on that difference, or is it a nuisance? Has it caused difficulty in
>> debugging code?
>>
>> If you had to keep one behaviour, which would you keep?
>>
>
> The point is: there is no difference in the behavior of comprehensions
> and generator expressions, it is just how you access them:
> for anything that follows the iterator protocol, a comprehension (since
> it is evaluated immediately) raises during the iter() phase, while a
> generator expression (which gets evaluated lazily) raises during the
> next() phase where it gets swallowed.
>
> So in your example you would have to use the silly
>
> list([next(x) for x in iterable])
>
> if you want the error to get raised.
>
> I agree this is all rather non-intuitive, but how would you change it ?
>
>

Ah, I came across the related thread on python-ideas only now and from 
this I can see that, as I expected, you know everything I've written 
above already.

In light of the discussion on the other list:
I did find it annoying occasionally that raising StopIteration inside a 
generator expression conveys a different behavior than elsewhere. It did 
take me quite a while to understand why that is so, but after that it 
did not cause me much of a headache anymore.

I would say that Guido's suggestion of transforming StopIteration raised 
inside a generator into some other error to eliminate ambiguity would 
really help in some situations, but then I'm not sure whether that's 
worth breaking existing code.

Best,
Wolfgang