Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #99452 > unrolled thread

What does a list comprehension do (was: Late-binding of function defaults (was Re: What is a function parameter =[] for?))

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  What does a list comprehension do (was: Late-binding of function defaults (was Re: What is a function parameter =[] for?)) Antoon Pardon <antoon.pardon@rece.vub.ac.be> - 2015-11-25 14:51 +0100
    Re: What does a list comprehension do (was: Late-binding of function defaults (was Re: What is a function parameter =[] for?)) Nobody <nobody@nowhere.invalid> - 2015-11-26 11:13 +0000
      Re: What does a list comprehension do Antoon Pardon <antoon.pardon@rece.vub.ac.be> - 2015-11-26 12:52 +0100
        Re: What does a list comprehension do Marko Rauhamaa <marko@pacujo.net> - 2015-11-26 14:56 +0200
          Re: What does a list comprehension do Antoon Pardon <antoon.pardon@rece.vub.ac.be> - 2015-11-26 14:33 +0100
            Re: What does a list comprehension do Marko Rauhamaa <marko@pacujo.net> - 2015-11-26 15:56 +0200
              Re: What does a list comprehension do Antoon Pardon <antoon.pardon@rece.vub.ac.be> - 2015-11-26 15:26 +0100
                Re: What does a list comprehension do Marko Rauhamaa <marko@pacujo.net> - 2015-11-26 17:36 +0200
                  Re: What does a list comprehension do Antoon Pardon <antoon.pardon@rece.vub.ac.be> - 2015-11-26 20:18 +0100
                Re: What does a list comprehension do Jussi Piitulainen <harvest@is.invalid> - 2015-11-26 18:11 +0200

#99452 — What does a list comprehension do (was: Late-binding of function defaults (was Re: What is a function parameter =[] for?))

Op 20-11-15 om 08:49 schreef dieter:
> In addition, the last few days have had two discussions in this list
> demonstrating the conceptial difficulties of late binding -- one of them:
>
>       Why does "[lambda x: i * x for i in range(4)]" gives
>       a list of essentially the same functions?

Can you (or someone else) explain what a list comprehension is equivallent of.
Especially in python3.

Take this simple list comprhesion:

[x * x for x in range(10)]

what would this be equivallent of? Something like:

def lch1():
  ls = []
  for x in range(10):
    ls.append(x * x)
  return ls

Or more something like:

def lch2():
  def expr(x):
    return x * x

  ls = []
  for x in range(10):
    ls.append(expr(x))
  return ls

For this example it doesn't make a difference but for the example above
it would become important.

def lch3():
  ls = []
  for i in range(4):
    ls.append(lambda x: i * x)
  return ls

versus

def lch4():
  def expr(i):
    return lambda x: i * x

  ls = []
  for i in range(4)
    ls.append(expr(i))
  return ls

Now from the result we get I expect list comprehensions to work more
like lch1 and lch3 rather than lch2 and lch4. But I can understand
people who think of the expression as a function of the variable that
is iterated over.

Am I missing something? Would it be worthwile considering changing
this behaviour?

-- 
Antoon.

[toc] | [next] | [standalone]

#99555

On Wed, 25 Nov 2015 14:51:23 +0100, Antoon Pardon wrote:

> Am I missing something?

The issue is with lambdas rather than with list comprehensions per se.

Python's lambdas capture free variables by reference, not value.

	> x = 3
	> f = lambda y: x + y
	> f(0)
	3
	> x = 7
	> f(0)
	7

The same issue applies to nested functions:

	> def foo():
	=    x = 3
	=    def f(y):
	=       return x + y
	=    print f(0)
	=    x = 7
	=    print f(0)
	= 
	> foo()
	3
	7

And also to non-nested functions (but most people expect that):

	> x = 3
	> def f(y,x=x):
	=   return x + y
	=
	> print f(0)
	3
	> x=7
	> print f(0)
	3

If you want to capture a variable by value, add a parameter with a default
value using that variable:

	> def foo():
	=    x = 3
	=    def f(y, x=x):
	=       return x + y
	=    print f(0)
	=    x = 7
	=    print f(0)
	= 
	> foo()
	3
	3

This also works for lambdas:

	> x = 3
	> f = lambda y,x=x: x + y
	> f(0)
	3
	> x = 7
	> f(0)
	3

Returning to the original expression:

	> q = [lambda x: i * x for i in range(4)]
	> q[0](1), q[3](1)
	(3, 3)
	> q = [lambda x,i=i: i * x for i in range(4)]
	> q[0](1), q[3](1)
	(0, 3)

[toc] | [prev] | [next] | [standalone]

#99563 — Re: What does a list comprehension do

Op 26-11-15 om 12:13 schreef Nobody:
> Returning to the original expression:
>
> 	> q = [lambda x: i * x for i in range(4)]
> 	> q[0](1), q[3](1)
> 	(3, 3)
> 	> q = [lambda x,i=i: i * x for i in range(4)]
> 	> q[0](1), q[3](1)
> 	(0, 3)

Personnaly I would prefer:

>>> q = [(lambda i: lambda x: i * x)(i) for i in range(4)]
>>> q[0](1), q[3](1)
(0, 3)

And this is where I ask whether it would be worth the effort to change
the behaviour of python.

In general the following two seem equivallent:

   [<expression> for x in <iter>] and [(lambda x: <expression>)(x) for x in <iter>]

The only exceptions seems to be when <expression> is itself a lambda.

It also seems that people who try this for the first time are surprised
with what they get and seem to expect there list comprehension to
act as if they had written the second version.

So would it be advisable if python would translate the expression people
write into a lambda that gets called?

Are there issues that could come up?

[toc] | [prev] | [next] | [standalone]

#99571 — Re: What does a list comprehension do

Antoon Pardon <antoon.pardon@rece.vub.ac.be>:

> Personnaly I would prefer:
>
>>>> q = [(lambda i: lambda x: i * x)(i) for i in range(4)]
>>>> q[0](1), q[3](1)
> (0, 3)
>
> And this is where I ask whether it would be worth the effort to change
> the behaviour of python.

Don't go there.

Consider:

    q = []
    n = 0
    x = "hello"

    for i in range(4):
        def stepper():
            global n
            n += 1
            return i * x
        q.append(stepper)

    print(n)
    print(q[1]())
    print(n)
    x = "there"
    print(q[3]())
    print(n)

which prints:

    0
    hellohellohello
    1
    theretherethere
    2

after your change, you'd get:

    0
    hello
    0
    hellohellohello
    0

> It also seems that people who try this for the first time are
> surprised with what they get and seem to expect there list
> comprehension to act as if they had written the second version.

I might trip over that one, too. Still, nothing should be changed.


Marko

[toc] | [prev] | [next] | [standalone]

#99574 — Re: What does a list comprehension do

Op 26-11-15 om 13:56 schreef Marko Rauhamaa:
> Antoon Pardon <antoon.pardon@rece.vub.ac.be>:
>
>> Personnaly I would prefer:
>>
>>>>> q = [(lambda i: lambda x: i * x)(i) for i in range(4)]
>>>>> q[0](1), q[3](1)
>> (0, 3)
>>
>> And this is where I ask whether it would be worth the effort to change
>> the behaviour of python.
> Don't go there.
>
> Consider:
>
>     q = []
>     n = 0
>     x = "hello"
>
>     for i in range(4):
>         def stepper():
>             global n
>             n += 1
>             return i * x
>         q.append(stepper)
>
>     print(n)
>     print(q[1]())
>     print(n)
>     x = "there"
>     print(q[3]())
>     print(n)
>
> which prints:
>
>     0
>     hellohellohello
>     1
>     theretherethere
>     2
>
> after your change, you'd get:
>
>     0
>     hello
>     0
>     hellohellohello
>     0

I don't understand. What I propose would be a minor change in
how list comprehension works. I don't see how your example
can be turned into a list comprehension. I can of course
put the stepper function out of the for loop and then
convert the for loop into a list comprehension.

The results in that case is that my proposed change wouldn't
effect the results in python2 and that the code wouldn't run
in python3.

So could you clarify what list comprehension you are talking
about, that you think would cause trouble, should my proposal
be adopted.
-- 
Antoon.

[toc] | [prev] | [next] | [standalone]

#99576 — Re: What does a list comprehension do

Antoon Pardon <antoon.pardon@rece.vub.ac.be>:

> I don't understand. What I propose would be a minor change in
> how list comprehension works. I don't see how your example
> can be turned into a list comprehension.

The list comprehension is only a special case of the interaction between
closures and variables. If you dabble with list comprehensions and
lambdas, you'll need to make consistent changes in closure semantics.

That would lead to the trouble I mentioned. Closures need to have
dynamic access to the variables they refer to.

This whole issue is a consequence of Python's assignment semantics. If
Python didn't allow altering the values of variables, we wouldn't be
having this discussion.

BTW, all(!?) other languages from Java to Scheme share closure semantics
with Python so you would really be making a mess by changing Python.


Marko

[toc] | [prev] | [next] | [standalone]

#99578 — Re: What does a list comprehension do

Op 26-11-15 om 14:56 schreef Marko Rauhamaa:
> Antoon Pardon <antoon.pardon@rece.vub.ac.be>:
>
>> I don't understand. What I propose would be a minor change in
>> how list comprehension works. I don't see how your example
>> can be turned into a list comprehension.
> The list comprehension is only a special case of the interaction between
> closures and variables. If you dabble with list comprehensions and
> lambdas, you'll need to make consistent changes in closure semantics.

It would only dabble with the list comprehension not with the lambda.
The effect of the change would only be that a list comprehension like

    [ <expression> for <var> in <iter> ]

would implicitly be rewritten as follows:

    [ (lambda <var>: <expression>)(<var>) for <var> in <iter>]

There would no change on how lambdas work or functions or closures.

> BTW, all(!?) other languages from Java to Scheme share closure semantics
> with Python so you would really be making a mess by changing Python.

Not this proposal, which wouldn't touch closure semantics.

-- 
Antoon.

[toc] | [prev] | [next] | [standalone]

#99583 — Re: What does a list comprehension do

Antoon Pardon <antoon.pardon@rece.vub.ac.be>:

>     [ <expression> for <var> in <iter> ]
>
> would implicitly be rewritten as follows:
>
>     [ (lambda <var>: <expression>)(<var>) for <var> in <iter>]

Funny enough, that's how "list comprehensions" are created in Scheme:

   (map (lambda (i)
          (lambda (x) (* i x)))
        '(0 1 2 3))))

> There would no change on how lambdas work or functions or closures.

First of all, it's weird to spend any effort in trying to alter a very
special case. I don't recall having to generate a list of such
functions. In fact, I barely ever use lambda in Python; explicit def
statements are much more pleasing to the eye and are not restricted to
simple expressions.

Secondly, you'd lose the nice symmetry between for statements and
comprehensions/generators. For example:

   ( lambda x: i * x for i in range(4) )

corresponds to:

    for i in range(4):
        yield lambda x: i * x

Would you embed an extra lambda there, too?

How about:

    i = 0
    while i < 4:
        yield lambda x: i * x
        i += 1

or:

    for i in range(4):
        def f(x):
            return i * x
        yield f


Marko

[toc] | [prev] | [next] | [standalone]

#99593 — Re: What does a list comprehension do

Op 26-11-15 om 16:36 schreef Marko Rauhamaa:
> Antoon Pardon <antoon.pardon@rece.vub.ac.be>:
> 
>>     [ <expression> for <var> in <iter> ]
>>
>> would implicitly be rewritten as follows:
>>
>>     [ (lambda <var>: <expression>)(<var>) for <var> in <iter>]
> 
> Funny enough, that's how "list comprehensions" are created in Scheme:
> 
>    (map (lambda (i)
>           (lambda (x) (* i x)))
>         '(0 1 2 3))))
> 
>> There would no change on how lambdas work or functions or closures.
> 
> First of all, it's weird to spend any effort in trying to alter a very
> special case.

<shrug> Maybe it is small enough an effort but I wont loose any sleep over it
should the dev team have other priorities.

> I don't recall having to generate a list of such
> functions. In fact, I barely ever use lambda in Python; explicit def
> statements are much more pleasing to the eye and are not restricted to
> simple expressions.
> 
> Secondly, you'd lose the nice symmetry between for statements and
> comprehensions/generators. For example:
> 
>    ( lambda x: i * x for i in range(4) )
> 
> corresponds to:
> 
>     for i in range(4):
>         yield lambda x: i * x
> 
> Would you embed an extra lambda there, too?

No, not because that would be so troublesome in semantics
but because it would be annoying in python. Because in that
case you would embed a function (call) or introduce an
intermediate scope. Which would mean that if you want
to rebind a variable within the suite of the for loop
that is also used in other parts of the function/modules
you would have to declare that variable nonlocal/global at the
start of the suite. So it would be annoying plus that
it would be difficult to explain to people new to the
language.

Such an embedding would only be practical in a language where
you have to declare your variables.

-- 
Antoon.

[toc] | [prev] | [next] | [standalone]

#99586 — Re: What does a list comprehension do

Antoon Pardon writes:
> Op 26-11-15 om 14:56 schreef Marko Rauhamaa:
>> Antoon Pardon wrote:
>>
>>> I don't understand. What I propose would be a minor change in how
>>> list comprehension works. I don't see how your example can be turned
>>> into a list comprehension.
>>
>> The list comprehension is only a special case of the interaction
>> between closures and variables. If you dabble with list
>> comprehensions and lambdas, you'll need to make consistent changes in
>> closure semantics.
>
> It would only dabble with the list comprehension not with the lambda.
> The effect of the change would only be that a list comprehension like
>
>     [ <expression> for <var> in <iter> ]
>
> would implicitly be rewritten as follows:
>
>     [ (lambda <var>: <expression>)(<var>) for <var> in <iter>]
>
> There would no change on how lambdas work or functions or closures.
>
>> BTW, all(!?) other languages from Java to Scheme share closure semantics
>> with Python so you would really be making a mess by changing Python.
>
> Not this proposal, which wouldn't touch closure semantics.

It needs to take into account the possibility of more than one variable,
but that's a minor adjustment. I like it. It's simple enough.

The following code exercises different values from nested comprehension
loops, together with a shared global whose value actually changes before
each call to one of the listed thunks.

m = "good morning to you"

thoughts = [ (lambda n, u :
                  ((lambda : (u//2)*m),
                   (lambda : n)))
             (n, u)
             for n in range(10)
             if n % 2            # odd?
             for u in range(10)
             if u % 2 - 1        # less odd?
             if n < u  ]

for t, g in zip(thoughts, ((["hello"], ["eiku"], ["hej"]) +
                           tuple(20 * "."))):
    m = g
    message, number = t
    print(number(), *message())

# Prints:
1 hello
1 eiku eiku
1 hej hej hej
1 . . . .
3 . .
3 . . .
3 . . . .
5 . . .
5 . . . .
7 . . . .

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web