Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #25166 > unrolled thread

lambda in list comprehension acting funny

Back to article view | Back to comp.lang.python

Contents

  lambda in list comprehension acting funny Daniel Fetchinson <fetchinson@googlemail.com> - 2012-07-11 08:41 +0200
    Re: lambda in list comprehension acting funny "Colin J. Williams" <cjw@ncf.ca> - 2012-07-11 06:28 -0400
      Re: lambda in list comprehension acting funny Ian Kelly <ian.g.kelly@gmail.com> - 2012-07-11 10:34 -0600
        Re: lambda in list comprehension acting funny 88888 Dihedral <dihedral88888@googlemail.com> - 2012-07-11 20:39 -0700
          Re: lambda in list comprehension acting funny Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-07-12 03:51 +0000
            Re: lambda in list comprehension acting funny 88888 Dihedral <dihedral88888@googlemail.com> - 2012-07-11 22:04 -0700
              Re: lambda in list comprehension acting funny Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-07-12 06:18 +0000
                Re: lambda in list comprehension acting funny Mark Lawrence <breamoreboy@yahoo.co.uk> - 2012-07-12 08:40 +0100
        Re: lambda in list comprehension acting funny 88888 Dihedral <dihedral88888@googlemail.com> - 2012-07-11 20:39 -0700
    Re: lambda in list comprehension acting funny woooee <woooee@gmail.com> - 2012-07-11 11:38 -0700
      Re: lambda in list comprehension acting funny John Ladasky <john_ladasky@sbcglobal.net> - 2012-07-11 13:21 -0700
        Re: lambda in list comprehension acting funny Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-07-12 00:52 +0000
        Re: lambda in list comprehension acting funny Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2012-07-11 21:05 -0400
          Re: lambda in list comprehension acting funny Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-07-12 03:53 +0000
            Re: lambda in list comprehension acting funny Terry Reedy <tjreedy@udel.edu> - 2012-07-12 00:24 -0400
            Re: lambda in list comprehension acting funny Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2012-07-12 00:39 -0400
              Re: lambda in list comprehension acting funny Robert Miles <robertmiles@teranews.com> - 2012-08-15 19:26 -0500
        Re: lambda in list comprehension acting funny John O'Hagan <research@johnohagan.com> - 2012-07-12 15:29 +1000
        Re: lambda in list comprehension acting funny Robert Kern <robert.kern@gmail.com> - 2012-07-12 10:06 +0100
      Re: lambda in list comprehension acting funny Hans Mulder <hansmu@xs4all.nl> - 2012-07-12 00:22 +0200
      Re: lambda in list comprehension acting funny Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-07-11 23:47 +0000
        Re: lambda in list comprehension acting funny Daniel Fetchinson <fetchinson@googlemail.com> - 2012-07-12 04:54 +0200
    Re: lambda in list comprehension acting funny Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-07-12 03:59 +0000
      Re: lambda in list comprehension acting funny Ian Kelly <ian.g.kelly@gmail.com> - 2012-07-12 10:53 -0600
      Re: lambda in list comprehension acting funny Rotwang <sg552@hotmail.co.uk> - 2012-07-12 18:23 +0100
        Re: lambda in list comprehension acting funny Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-07-13 02:20 +0000
    Re: lambda in list comprehension acting funny rusi <rustompmody@gmail.com> - 2012-07-12 21:33 -0700
      Re: lambda in list comprehension acting funny Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-07-13 06:36 +0000
        Re: lambda in list comprehension acting funny rusi <rustompmody@gmail.com> - 2012-07-13 06:44 -0700
          Re: lambda in list comprehension acting funny rusi <rustompmody@gmail.com> - 2012-07-13 07:45 -0700
          RE: lambda in list comprehension acting funny "Prasad, Ramit" <ramit.prasad@jpmorgan.com> - 2012-07-13 16:12 +0000
            Re: lambda in list comprehension acting funny rusi <rustompmody@gmail.com> - 2012-07-13 09:46 -0700
              Re: lambda in list comprehension acting funny Chris Angelico <rosuav@gmail.com> - 2012-07-14 03:20 +1000
            Re: lambda in list comprehension acting funny Hans Mulder <hansmu@xs4all.nl> - 2012-07-13 19:53 +0200
              RE: lambda in list comprehension acting funny "Prasad, Ramit" <ramit.prasad@jpmorgan.com> - 2012-07-13 18:06 +0000
              Re: lambda in list comprehension acting funny Ian Kelly <ian.g.kelly@gmail.com> - 2012-07-13 12:54 -0600
                Re: lambda in list comprehension acting funny Hans Mulder <hansmu@xs4all.nl> - 2012-07-13 21:26 +0200
                Re: lambda in list comprehension acting funny Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-07-14 23:29 +0000
                  Re: lambda in list comprehension acting funny Chris Angelico <rosuav@gmail.com> - 2012-07-15 10:49 +1000
                    Re: lambda in list comprehension acting funny Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-07-15 08:32 +0000
                      Re: lambda in list comprehension acting funny Chris Angelico <rosuav@gmail.com> - 2012-07-15 18:44 +1000
                        Re: lambda in list comprehension acting funny Hans Mulder <hansmu@xs4all.nl> - 2012-07-15 21:25 +0200
                      Re: lambda in list comprehension acting funny Terry Reedy <tjreedy@udel.edu> - 2012-07-15 06:27 -0400
              Re: lambda in list comprehension acting funny rusi <rustompmody@gmail.com> - 2012-07-13 19:31 -0700
                Re: lambda in list comprehension acting funny Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-07-14 03:43 +0000
                  Re: lambda in list comprehension acting funny rusi <rustompmody@gmail.com> - 2012-07-13 21:53 -0700
                    Re: lambda in list comprehension acting funny Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-07-14 07:46 +0000
          Re: lambda in list comprehension acting funny Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2012-07-13 13:47 -0400

Page 3 of 3 — ← Prev page 1 2 [3]

#25342

On Sun, Jul 15, 2012 at 6:32 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> At compile time, Python parses the source code and turns it into byte-
> code. Class and function definitions are executed at run time, the same
> as any other statement.

Between the parse step and the 'def' execution, a code object is
created. When you call it, that code object already exists. Nothing
else really matters, unless there's a bug in the Python optimizer or
something weird like that. The nearest thing Python _has_ to a
"compile time" is the execution of def.

ChrisA

[toc] | [prev] | [next] | [standalone]

#25369

On 15/07/12 10:44:09, Chris Angelico wrote:
> On Sun, Jul 15, 2012 at 6:32 PM, Steven D'Aprano
> <steve+comp.lang.python@pearwood.info> wrote:
>> At compile time, Python parses the source code and turns it into byte-
>> code. Class and function definitions are executed at run time, the same
>> as any other statement.
> 
> Between the parse step and the 'def' execution, a code object is
> created. When you call it, that code object already exists. Nothing
> else really matters, unless there's a bug in the Python optimizer or
> something weird like that. The nearest thing Python _has_ to a
> "compile time" is the execution of def.
> 
> ChrisA

"Compile time" is the phase when your Python code is turned into byte
code, or a SyntaxError is raised.  In this phase, a "code object" is
created for the function.

"Function definition time" is when the "def" command is executed.
In this phase, default arguments are computed, and a "function object"
is created.  Among the attributes of the function object are the code
object, and "cell" objects containing the bindings for its non-local
variables.  These bindings are used to read the variable's current
value at the time the function uses the variable.


-- HansM

[toc] | [prev] | [next] | [standalone]

#25345

On 7/15/2012 4:32 AM, Steven D'Aprano wrote:
> On Sun, 15 Jul 2012 10:49:48 +1000, Chris Angelico wrote:
>
>> On Sun, Jul 15, 2012 at 9:29 AM, Steven D'Aprano
>> <steve+comp.lang.python@pearwood.info> wrote:
>>> Not necessarily *compile* time, but the distinction is between when the
>>> function is defined (which may at compile time, or it may be at run
>>> time) versus when the function is called.
>>
>> I'd treat the def/lambda statement as "compile time" and the () operator
>> as "run time".
>
> But function definitions occur at run time, not compile time -- they are
> executable statements, not instructions to the compiler to define a
> function.

The () operator is 'call time'. The main points are
a) the execution of def statements and lambda expressions and the 
execution of calls happen at different run times.
b) default arg objects are calculated at def/lambda time.
c) names in def bodies and lambda expressions are resolved at call time.
and
d) people more often forget c) when thinking about lambda expressions 
that for def statements.

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#25301

On Jul 13, 10:53 pm, Hans Mulder <han...@xs4all.nl> wrote:
> If you add `global VERBOSE` to `caller`, then there is only one
> variable named `VERBOSE` and what `function` does, depends on
> the most recent assignment to that variable.
>
> If you remove your `global VERBOSE`, then there are two
> variables by that name, one global and one local to `caller`.
> In that case, there is the question of which one `function`
> will use.

Good point. See below.

>
> The function `function` refers to a variable `VERBOSE` that
> isn't local.  In some programming langauages, the interpreter
> would then scan the call stack at run-time, looking for a scope
> where that name is defined.  It would find the local one in
> `caller`.  This is known as "dynamic binding".
>
> Other interpreters use the `VERBOSE` that was in scope at
> the point in the program text where `function` was defined.
> In this case, that would be the global one.  This is called
> "lexical binding".
>
> Some programming languages allow you to indicate on a per-
> variable basis whether you want dynamic or lexical binding.
>
> Python is firmly in the lexical camp.  Dynamic binding is not
> available in Python, and never will be.

Thats a good intention.  The question is whether it is uniformly
realized.

Consider the following

def foo(x):
    i = 100
    if x:
        j = [i for i in range(10)]
        return i
    else:
        return i

In python 2  two different 'i's could be returned. In python3 only one
i can be returned.
One could call it dynamic binding. Evidently Guido thinks it a bug
which is why he changed it.

The leakage of i in the OPs question is the same kind of bug.

tl;dr version:
This is 2012. Dynamic binding is considered a bug even by the lisp
community where it originated.
Lets at least call it a feature and a gotcha

[toc] | [prev] | [next] | [standalone]

#25303

On Fri, 13 Jul 2012 19:31:24 -0700, rusi wrote:

> Consider the following
> 
> def foo(x):
>     i = 100
>     if x:
>         j = [i for i in range(10)]
>         return i
>     else:
>         return i

A simpler example:

def foo():
    i = 100
    j = [i for i in range(10)]
    return i

In Python 3, foo() returns 100; in Python 2, it returns 9.


> In python 2  two different 'i's could be returned. In python3 only one i
> can be returned.
> One could call it dynamic binding. 

One could also call it a tractor, but it isn't either of those things.

The difference is whether or not list comprehensions create their own 
scope. In Python 2, they don't; in Python 3, they do. Personally I don't 
have an opinion as to which is better, but in neither case does this have 
any thing to do with lexical versus dynamic binding.


> Evidently Guido thinks it a bug which is why he changed it.

It was a case that either list comps be changed to *not* expose their 
loop variable, or generator expressions be changed *to* expose their loop 
variable. The Python 2 situation where list comps and gen expressions 
have opposite behaviour was unfortunate.


> The leakage of i in the OPs question is the same kind of bug.

Not at all. The OP was *deliberately* creating a closure using i -- under 
those circumstances, it would be a bug if i *didn't* leak.

The OP's gotcha was:

1) he expected the closure to use early binding, where i in each function 
was bound to the value of i in the enclosing scope at the moment the 
function was defined; 

2) but Python actually uses late binding for closures, where the value of 
i in each function is set to the value of i in the enclosing scope when 
the function is called.

As far as I can tell, Python always uses late binding for scopes; the 
only time it does early binding is for default values of function 
parameters.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#25304

On Jul 14, 8:43 am, Steven D'Aprano <steve
+comp.lang.pyt...@pearwood.info> wrote:
> On Fri, 13 Jul 2012 19:31:24 -0700, rusi wrote:
> > Consider the following
>
> > def foo(x):
> >     i = 100
> >     if x:
> >         j = [i for i in range(10)]
> >         return i
> >     else:
> >         return i
>
> A simpler example:
>
> def foo():
>     i = 100
>     j = [i for i in range(10)]
>     return i
>
> In Python 3, foo() returns 100; in Python 2, it returns 9.

You did not get the point.

Converting my example to your format:

def foo_steven(n):
    i = 100
    j = [i for i in range(n)]
    return i

$ python3
Python 3.2.3 (default, Jun 26 2012, 00:38:09)
[GCC 4.7.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> def foo_steven(n):
...     i = 100
...     j = [i for i in range(n)]
...     return i
...
>>> foo_steven(0)
100
>>> foo_steven(4)
100
>>>
$ python
Python 2.7.3rc2 (default, Apr 22 2012, 22:35:38)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> def foo_steven(n):
...     i = 100
...     j = [i for i in range(n)]
...     return i
...
>>> foo_steven(0)
100
>>> foo_steven(3)
2
>>>

Python 2:
When n>0 comprehension scope i is returned
When n=0 function scope i is returned

Python 3: The return statement is lexically outside the comprehension
and so that outside-scope's i is returned in all cases.

[toc] | [prev] | [next] | [standalone]

#25306

On Fri, 13 Jul 2012 21:53:10 -0700, rusi wrote:

> On Jul 14, 8:43 am, Steven D'Aprano <steve
> +comp.lang.pyt...@pearwood.info> wrote:
>> On Fri, 13 Jul 2012 19:31:24 -0700, rusi wrote:
>> > Consider the following
>>
>> > def foo(x):
>> >     i = 100
>> >     if x:
>> >         j = [i for i in range(10)]
>> >         return i
>> >     else:
>> >         return i
>>
>> A simpler example:
>>
>> def foo():
>>     i = 100
>>     j = [i for i in range(10)]
>>     return i
>>
>> In Python 3, foo() returns 100; in Python 2, it returns 9.
> 
> You did not get the point.

I got the point. I just thought it was unnecessarily complex and that it 
doesn't demonstrate what you think it does.


> Converting my example to your format:
> 
> def foo_steven(n):
>     i = 100
>     j = [i for i in range(n)]
>     return i
> 
> $ python3
> Python 3.2.3 (default, Jun 26 2012, 00:38:09) [GCC 4.7.1] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>>>> def foo_steven(n):
> ...     i = 100
> ...     j = [i for i in range(n)]
> ...     return i
> ...
>>>> foo_steven(0)
> 100
>>>> foo_steven(4)
> 100

Yes, we know that in Python 3, list comprehensions create their own 
scope, and the loop variable does not leak.


> $ python
> Python 2.7.3rc2 (default, Apr 22 2012, 22:35:38) [GCC 4.6.3] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>>>> def foo_steven(n):
> ...     i = 100
> ...     j = [i for i in range(n)]
> ...     return i
> ...
>>>> foo_steven(0)
> 100
>>>> foo_steven(3)
> 2

Yes, we know that in Python 2, list comprehensions don't create their own 
scope, and consequently the list variable does leak.


> Python 2:
> When n>0 comprehension scope i is returned 
> When n=0 function scope i is returned

Incorrect.

In Python 2, *there is no comprehension scope*. There is only local 
scope. In Python 2, regardless of the value of n, the local variable i is 
ALWAYS returned. It just happens that sometimes the local variable i is 
modified by the list comprehension, and sometimes it isn't. In Python 2, 
this is no more mysterious than this piece of code:

def example(n):
    i = 100
    for i in range(n):
        pass
    return i

py> example(0)
100
py> example(4)
3

If the loop doesn't execute, the loop variable isn't modified. In Python 
2, it doesn't matter whether you use a for-loop or a list comprehension, 
the loop variable is local to the function.


 
> Python 3: The return statement is lexically outside the comprehension
> and so that outside-scope's i is returned in all cases.

Correct. In Python 3, list comprehensions now match generator expressions 
and introduce their own scope which does not effect the local function 
scope.

Some history:

http://bugs.python.org/issue510384
http://bugs.python.org/issue1110705




-- 
Steven

[toc] | [prev] | [next] | [standalone]

#25277

On Fri, 13 Jul 2012 06:44:59 -0700 (PDT), rusi <rustompmody@gmail.com>
declaimed the following in gmane.comp.python.general:


> Heres a more appropriate analog
> 
> ------------------------------------------
> VERBOSE = True
> 
> def function(arg):
>     if VERBOSE:
>        print("calling function with arg %r" % arg)
>     process(arg)
> 
> def caller():
>     VERBOSE = False
>     function(1)
> 
> ---------------------------------------------
> Python semantics: function sees VERBOSE False

	Wrong... "function" still sees the module level VERBOSE with a value
of True, whereas "caller" created a local VERBOSE set to False. To get
your behavior needs:

	def caller():
		global VERBOSE
		VERBOSE = False
		function(1)


> Haskell semantics: function sees VERBOSE True
-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
        wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/

[toc] | [prev] | [standalone]

Page 3 of 3 — ← Prev page 1 2 [3]

Back to top | Article view | comp.lang.python

csiph-web