Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #94995 > unrolled thread

GOTCHA with list comprehension

Back to article view | Back to comp.lang.python

Contents

  GOTCHA with list comprehension Pavel S <pavel@schon.cz> - 2015-08-04 23:48 -0700
    Re: GOTCHA with list comprehension Pavel S <pavel@schon.cz> - 2015-08-05 00:03 -0700
      Re: GOTCHA with list comprehension Chris Angelico <rosuav@gmail.com> - 2015-08-05 17:20 +1000
    Re: GOTCHA with list comprehension Peter Otten <__peter__@web.de> - 2015-08-05 09:04 +0200
    Re: GOTCHA with list comprehension Chris Angelico <rosuav@gmail.com> - 2015-08-05 17:05 +1000
      Re: GOTCHA with list comprehension Pavel S <pavel@schon.cz> - 2015-08-05 00:10 -0700
        Re: GOTCHA with list comprehension Chris Angelico <rosuav@gmail.com> - 2015-08-05 17:21 +1000
          Re: GOTCHA with list comprehension Pavel S <pavel@schon.cz> - 2015-08-05 00:33 -0700
      Re: GOTCHA with list comprehension Marko Rauhamaa <marko@pacujo.net> - 2015-08-05 12:01 +0300
        Re: GOTCHA with list comprehension Chris Angelico <rosuav@gmail.com> - 2015-08-05 19:52 +1000
          Re: GOTCHA with list comprehension Marko Rauhamaa <marko@pacujo.net> - 2015-08-05 14:10 +0300
        Re: GOTCHA with list comprehension Saran Ahluwalia <ahlusar.ahluwalia@gmail.com> - 2015-08-07 09:08 -0400
    Re: GOTCHA with list comprehension Laura Creighton <lac@openend.se> - 2015-08-06 06:39 +0200
    Re: GOTCHA with list comprehension Chris Angelico <rosuav@gmail.com> - 2015-08-06 14:44 +1000

#94995 — GOTCHA with list comprehension

Hi,

I recently found interesting GOTCHA while doing list comprehension in python 2.6:

>>> values = ( True, False, 1, 2, 3, None )
>>> [ value for value in values if value if not None ]
[True, 1, 2, 3]

I was wondering why this list comprehension returns incorrect results and finally found a typo in the condition. The typo wasn't visible at the first look.

My intention was: if value is not None
But I wrote: if value if not None

Is that a language feature of list comprehension that it accepts conditions like: if A if B if C if D ...?

[toc] | [next] | [standalone]

#94997

It seems this is allowed by the grammar:

list_display        ::=  "[" [expression_list | list_comprehension] "]"
list_comprehension  ::=  expression list_for
list_for            ::=  "for" target_list "in" old_expression_list [list_iter]
old_expression_list ::=  old_expression [("," old_expression)+ [","]]
old_expression      ::=  or_test | old_lambda_expr
list_iter           ::=  list_for | list_if
list_if             ::=  "if" old_expression [list_iter]

So chaining multiple ifs is fine:

[ i for i in range(10) if True if True if True if True ]

Dne středa 5. srpna 2015 8:49:20 UTC+2 Pavel S napsal(a):
> Hi,
> 
> I recently found interesting GOTCHA while doing list comprehension in python 2.6:
> 
> >>> values = ( True, False, 1, 2, 3, None )
> >>> [ value for value in values if value if not None ]
> [True, 1, 2, 3]
> 
> I was wondering why this list comprehension returns incorrect results and finally found a typo in the condition. The typo wasn't visible at the first look.
> 
> My intention was: if value is not None
> But I wrote: if value if not None
> 
> Is that a language feature of list comprehension that it accepts conditions like: if A if B if C if D ...?

[toc] | [prev] | [next] | [standalone]

#95001

On Wed, Aug 5, 2015 at 5:03 PM, Pavel S <pavel@schon.cz> wrote:
> It seems this is allowed by the grammar:
>
> list_display        ::=  "[" [expression_list | list_comprehension] "]"
> list_comprehension  ::=  expression list_for
> list_for            ::=  "for" target_list "in" old_expression_list [list_iter]
> old_expression_list ::=  old_expression [("," old_expression)+ [","]]
> old_expression      ::=  or_test | old_lambda_expr
> list_iter           ::=  list_for | list_if
> list_if             ::=  "if" old_expression [list_iter]
>
> So chaining multiple ifs is fine:
>
> [ i for i in range(10) if True if True if True if True ]

Yep. A chain of 'if' clauses isn't materially different from a single
'if' with a bunch of 'and' checks, but if you alternate 'if' and 'for'
clauses, you get a comprehension that conditionally nests its
iterations.

ChrisA

[toc] | [prev] | [next] | [standalone]

#94998

Pavel S wrote:

> Hi,
> 
> I recently found interesting GOTCHA while doing list comprehension in
> python 2.6:
> 
>>>> values = ( True, False, 1, 2, 3, None )
>>>> [ value for value in values if value if not None ]
> [True, 1, 2, 3]
> 
> I was wondering why this list comprehension returns incorrect results and
> finally found a typo in the condition. The typo wasn't visible at the
> first look.
> 
> My intention was: if value is not None
> But I wrote: if value if not None
> 
> Is that a language feature of list comprehension that it accepts
> conditions like: if A if B if C if D ...?

I think it's just that a condition may be a constant expression. Python 
evaluates (not None) for every item in values. Other variants:

>>> if 42:
...     print("branch always taken")
... 
branch always taken
>>> always_yes = "yes" if True else "no"
>>> always_yes
'yes'
>>> [c for c in "foo" if "true in a boolean context"]
['f', 'o', 'o']

An optimizer might detect that (not None) is always True in a boolean 
context, but that would be an implementation detail.

[toc] | [prev] | [next] | [standalone]

#94999

On Wed, Aug 5, 2015 at 4:48 PM, Pavel S <pavel@schon.cz> wrote:
> Hi,
>
> I recently found interesting GOTCHA while doing list comprehension in python 2.6:
>
>>>> values = ( True, False, 1, 2, 3, None )
>>>> [ value for value in values if value if not None ]
> [True, 1, 2, 3]
>
> I was wondering why this list comprehension returns incorrect results and finally found a typo in the condition. The typo wasn't visible at the first look.
>
> My intention was: if value is not None
> But I wrote: if value if not None
>
> Is that a language feature of list comprehension that it accepts conditions like: if A if B if C if D ...?

It certainly is. You can chain 'for' and 'if' clauses as much as you
like, and they behave exactly the way you'd expect. You might possibly
get a warning from a linter with your code, though, as it has an
always-true condition ("if not None" can never be false), so it's
possible something might hint at what's going on; but other than that,
all you can do is test stuff and see if it's giving the right result.

Incidentally, why Python 2.6? Python 2.7 has been out for a pretty
long time now, and if you can't move to version 3.x, I would at least
recommend using 2.7. Since the release of 2.6.9 back before Frozen
came out, that branch has been completely unmaintained. Grab yourself
a 2.7 and take advantage of some neat new features (for old values of
"new"), and improved compatibility with 3.x.

ChrisA

[toc] | [prev] | [next] | [standalone]

#95000

$ cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 6.5 (Santiago)
$ python --version
Python 2.6.6

> Incidentally, why Python 2.6?
> 
> ChrisA

[toc] | [prev] | [next] | [standalone]

#95002

On Wed, Aug 5, 2015 at 5:10 PM, Pavel S <pavel@schon.cz> wrote:
> $ cat /etc/redhat-release
> Red Hat Enterprise Linux Server release 6.5 (Santiago)
> $ python --version
> Python 2.6.6
>
>> Incidentally, why Python 2.6?
>>

I guess that would be why :)

That's probably actually a patched 2.6.6 - from what I understand of
how Red Hat works, the version number is the number of the *oldest*
part of the code, so that quite possibly has a lot of backported
fixes. When I said 2.6 was out of support, I meant from python.org;
Red Hat supports stuff for a lot longer.

So, yeah, perfectly good reason for sticking with 2.6. For now. :)

ChrisA

[toc] | [prev] | [next] | [standalone]

#95003

Hi Chris, yeah, I have to stick on the software which my employer provides to me (we're enterprise company). I'm not root on that system. I'm happy with 2.6 now, two years ago we were on older RHEL with python 2.4 and it was a real pain :)

> > $ cat /etc/redhat-release
> > Red Hat Enterprise Linux Server release 6.5 (Santiago)
> > $ python --version
> > Python 2.6.6
> >
> >> Incidentally, why Python 2.6?
> >>
> 
> I guess that would be why :)
> 
> That's probably actually a patched 2.6.6 - from what I understand of
> how Red Hat works, the version number is the number of the *oldest*
> part of the code, so that quite possibly has a lot of backported
> fixes. When I said 2.6 was out of support, I meant from python.org;
> Red Hat supports stuff for a lot longer.
> 
> So, yeah, perfectly good reason for sticking with 2.6. For now. :)
> 
> ChrisA

[toc] | [prev] | [next] | [standalone]

#95005

Chris Angelico <rosuav@gmail.com>:

> You can chain 'for' and 'if' clauses as much as you like, and they
> behave exactly the way you'd expect.

How do you know what I'd expect?

I wouldn't know what to expect myself.


Marko

[toc] | [prev] | [next] | [standalone]

#95006

On Wed, Aug 5, 2015 at 7:01 PM, Marko Rauhamaa <marko@pacujo.net> wrote:
> Chris Angelico <rosuav@gmail.com>:
>
>> You can chain 'for' and 'if' clauses as much as you like, and they
>> behave exactly the way you'd expect.
>
> How do you know what I'd expect?
>
> I wouldn't know what to expect myself.

A list comprehension can always be unwound into statement form. [1] For example:

squares = [n*n for n in range(10)]

can be unwound into:

squares = []
for n in range(10):
    squares.append(n*n)

You simply take the expression at the beginning and put that into the
append() method call, and then the rest become statements. Here's a
filtered version:

odd_squares = [n*n for n in range(20) if n%2]

Which becomes:

odd_squares = []
for n in range(20):
    if n%2:
        odd_squares.append(n*n)

So what would you expect nested 'if' clauses to do? Well, they become
nested 'if' statements:

primes = [n for n in range(2,24) if n%2 if n%3]

primes = []
for n in range(2,24):
    if n%2:
        if n%3:
            primes.append(n)

What if we have multiple 'for' loops? Same thing!

ways_to_get_seven = [(a,b) for a in range(1,7) for b in range(1,7) if a+b==7]

ways_to_get_seven = []
for a in range(1,7):
    for b in range(1,7):
        if a+b==7:
            ways_to_get_seven.append((a,b))

No matter what combination of 'if' and 'for' you use, it can be
unwound like this, and it'll always behave the same way. Not sure what
to expect? Follow the simple rules of unwinding, and then read the
code that way.

Of course, if you don't know how to predict what the statement form
will do, then you won't understand what the comprehension will do. But
that's not the comprehension's fault.

peculiar = [a*x*x+b*x+c for a in range(1,10) for b in range(2,30) for
c in range(-3,5) if b*b-4*a*c>=0 if (-b+math.sqrt(b*b-4*a*c))/2/a>0
for x in (x*.01 for x in range(-500,500))]

I suppose you could graph that. Or something. But that's just bad
code, don't blame the list comp for that...

ChrisA

[1] To be technically correct, this unwinding should be done in a
nested function, which you then call. There are some other minor
technicalities too. But it's pretty much this.

[toc] | [prev] | [next] | [standalone]

#95013

Chris Angelico <rosuav@gmail.com>:

> On Wed, Aug 5, 2015 at 7:01 PM, Marko Rauhamaa <marko@pacujo.net> wrote:
>> Chris Angelico <rosuav@gmail.com>:
>>
>>> You can chain 'for' and 'if' clauses as much as you like, and they
>>> behave exactly the way you'd expect.
>>
>> How do you know what I'd expect?
>>
>> I wouldn't know what to expect myself.

> [...]

> So what would you expect nested 'if' clauses to do? Well, they become
> nested 'if' statements:
>
> primes = [n for n in range(2,24) if n%2 if n%3]
>
> primes = []
> for n in range(2,24):
>     if n%2:
>         if n%3:
>             primes.append(n)
>
> What if we have multiple 'for' loops? Same thing!

So no need to appeal to my expectations. Better just define it (as you
now did).


Marko

[toc] | [prev] | [next] | [standalone]

#95122

@ChrisA

You, my friend, certainly put the nail in the coffin!

Sent from my iPhone

> On Aug 5, 2015, at 5:52 AM, Chris Angelico <rosuav@gmail.com> wrote:
>
>> On Wed, Aug 5, 2015 at 7:01 PM, Marko Rauhamaa <marko@pacujo.net> wrote:
>> Chris Angelico <rosuav@gmail.com>:
>>
>>> You can chain 'for' and 'if' clauses as much as you like, and they
>>> behave exactly the way you'd expect.
>>
>> How do you know what I'd expect?
>>
>> I wouldn't know what to expect myself.
>
> A list comprehension can always be unwound into statement form. [1] For example:
>
> squares = [n*n for n in range(10)]
>
> can be unwound into:
>
> squares = []
> for n in range(10):
>    squares.append(n*n)
>
> You simply take the expression at the beginning and put that into the
> append() method call, and then the rest become statements. Here's a
> filtered version:
>
> odd_squares = [n*n for n in range(20) if n%2]
>
> Which becomes:
>
> odd_squares = []
> for n in range(20):
>    if n%2:
>        odd_squares.append(n*n)
>
> So what would you expect nested 'if' clauses to do? Well, they become
> nested 'if' statements:
>
> primes = [n for n in range(2,24) if n%2 if n%3]
>
> primes = []
> for n in range(2,24):
>    if n%2:
>        if n%3:
>            primes.append(n)
>
> What if we have multiple 'for' loops? Same thing!
>
> ways_to_get_seven = [(a,b) for a in range(1,7) for b in range(1,7) if a+b==7]
>
> ways_to_get_seven = []
> for a in range(1,7):
>    for b in range(1,7):
>        if a+b==7:
>            ways_to_get_seven.append((a,b))
>
> No matter what combination of 'if' and 'for' you use, it can be
> unwound like this, and it'll always behave the same way. Not sure what
> to expect? Follow the simple rules of unwinding, and then read the
> code that way.
>
> Of course, if you don't know how to predict what the statement form
> will do, then you won't understand what the comprehension will do. But
> that's not the comprehension's fault.
>
> peculiar = [a*x*x+b*x+c for a in range(1,10) for b in range(2,30) for
> c in range(-3,5) if b*b-4*a*c>=0 if (-b+math.sqrt(b*b-4*a*c))/2/a>0
> for x in (x*.01 for x in range(-500,500))]
>
> I suppose you could graph that. Or something. But that's just bad
> code, don't blame the list comp for that...
>
> ChrisA
>
> [1] To be technically correct, this unwinding should be done in a
> nested function, which you then call. There are some other minor
> technicalities too. But it's pretty much this.
> --
> https://mail.python.org/mailman/listinfo/python-list

[toc] | [prev] | [next] | [standalone]

#95063

In a message of Wed, 05 Aug 2015 17:05:49 +1000, Chris Angelico writes:
>Incidentally, why Python 2.6? Python 2.7 has been out for a pretty
>long time now, and if you can't move to version 3.x, I would at least
>recommend using 2.7. Since the release of 2.6.9 back before Frozen
>came out, that branch has been completely unmaintained. Grab yourself
>a 2.7 and take advantage of some neat new features (for old values of
>"new"), and improved compatibility with 3.x.
>
>ChrisA

Be careful suggesting that people upgrade, at least until you have
found out that they aren't running CentOS enterprise edition. The
CentOS packaging system is utterly dependent on having the Python
version it expects, and if you install a more recent version
(from source, say) the whole packaging system will stop working.
Recovering from this problem is also very, very difficult  (unless
you just restore from a recent full backup).

Been there, done that. :(

Laura

[toc] | [prev] | [next] | [standalone]

#95065

On Thu, Aug 6, 2015 at 2:39 PM, Laura Creighton <lac@openend.se> wrote:
> In a message of Wed, 05 Aug 2015 17:05:49 +1000, Chris Angelico writes:
>>Incidentally, why Python 2.6? Python 2.7 has been out for a pretty
>>long time now, and if you can't move to version 3.x, I would at least
>>recommend using 2.7. Since the release of 2.6.9 back before Frozen
>>came out, that branch has been completely unmaintained. Grab yourself
>>a 2.7 and take advantage of some neat new features (for old values of
>>"new"), and improved compatibility with 3.x.
>>
>>ChrisA
>
> Be careful suggesting that people upgrade, at least until you have
> found out that they aren't running CentOS enterprise edition. The
> CentOS packaging system is utterly dependent on having the Python
> version it expects, and if you install a more recent version
> (from source, say) the whole packaging system will stop working.
> Recovering from this problem is also very, very difficult  (unless
> you just restore from a recent full backup).
>
> Been there, done that. :(

Yeah. Fortunately there are ways to leave the system Python untouched
while adding in another Python for some other purpose... and, if
you're desperate enough, you can even leave the Apache-Python bridge
using an older build of Python while running key parts of your web
application in a newer version,  but that requires a certain measure
of insanity!

There will come a time, though, when every supported Linux distro is
shipping either 2.7 or 3.x. And it isn't far off. When that happens,
advising the upgrade will be simple.

ChrisA

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web