Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #21959 > unrolled thread

Re: List comprehension/genexp inconsistency.

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: List comprehension/genexp inconsistency. Ian Kelly <ian.g.kelly@gmail.com> - 2012-03-20 16:50 -0600
    Re: List comprehension/genexp inconsistency. Steve Howell <showell30@yahoo.com> - 2012-03-20 20:07 -0700

#21959 — Re: List comprehension/genexp inconsistency.

On Tue, Mar 20, 2012 at 3:16 PM, Dennis Lee Bieber
<wlfraed@ix.netcom.com> wrote:
> On Tue, 20 Mar 2012 16:23:22 -0400, "J. Cliff Dyer"
> <jcd@sdf.lonestar.org> declaimed the following in
> gmane.comp.python.general:
>
>>
>> When trying to create a class with a dual-loop generator expression in a
>> class definition, there is a strange scoping issue where the inner
>> variable is not found, (but the outer loop variable is found), while a
>> list comprehension has no problem finding both variables.
>>
>        Read http://www.python.org/dev/peps/pep-0289/ -- in particular, look
> for the word "leak"

No, this has nothing to do with the loop variable leaking.  It appears
to have to do with the fact that the variables and the generator
expression are inside a class block.  I think that it's related to the
reason that this doesn't work:

class Foo(object):
    x = 42
    def foo():
        print(x)
    foo()

In this case, x is not a local variable of foo, nor is it a global.
In order for foo to access x, it would have to be a closure -- but
Python can't make it a closure in this case, because the variable it
accesses is (or rather, will become) a class attribute, not a local
variable of a function that can be stored in a cell.  Instead, the
compiler just makes it a global reference in the hope that such a
global will actually be defined when the code is run.

For that reason, what surprises me about Cliff's example is that a
generator expression works at all in that context.  It seems to work
as long as it contains only one loop, but not if it contains two.  To
find out why, I tried disassembling one:

>>> class Foo(object):
...     x = 42
...     y = 12
...     g = (a+b for a in range(x) for b in range(y))
...
>>> dis.dis(Foo.g.gi_code)
  4           0 LOAD_FAST                0 (.0)
        >>    3 FOR_ITER                34 (to 40)
              6 STORE_FAST               1 (a)
              9 LOAD_GLOBAL              0 (range)
             12 LOAD_GLOBAL              1 (y)
             15 CALL_FUNCTION            1
             18 GET_ITER
        >>   19 FOR_ITER                15 (to 37)
             22 STORE_FAST               2 (b)
             25 LOAD_FAST                1 (a)
             28 LOAD_FAST                2 (b)
             31 BINARY_ADD
             32 YIELD_VALUE
             33 POP_TOP
             34 JUMP_ABSOLUTE           19
        >>   37 JUMP_ABSOLUTE            3
        >>   40 LOAD_CONST               0 (None)
             43 RETURN_VALUE

So that explains it.  Notice that "x" is never actually accessed in
that disassembly; only "y" is.  It turns out that the first iterator
[range(x)] is actually created before the generator ever starts
executing, and is stored as an anonymous local variable on the
generator's stack frame -- so it's created in the class scope, not in
the generator scope.  The second iterator, however, is recreated on
every iteration of the first iterator, so it can't be pre-built in
that manner.  It does get created in the generator scope, and when
that happens it blows up because it can't find the variable, just like
the function example above.

Cheers,
Ian

[toc] | [next] | [standalone]

#21967

On Mar 20, 3:50 pm, Ian Kelly <ian.g.ke...@gmail.com> wrote:
> On Tue, Mar 20, 2012 at 3:16 PM, Dennis Lee Bieber
>
> <wlfr...@ix.netcom.com> wrote:
> > On Tue, 20 Mar 2012 16:23:22 -0400, "J. Cliff Dyer"
> > <j...@sdf.lonestar.org> declaimed the following in
> > gmane.comp.python.general:
>
> >> When trying to create a class with a dual-loop generator expression in a
> >> class definition, there is a strange scoping issue where the inner
> >> variable is not found, (but the outer loop variable is found), while a
> >> list comprehension has no problem finding both variables.
>
> >        Readhttp://www.python.org/dev/peps/pep-0289/-- in particular, look
> > for the word "leak"
>
> No, this has nothing to do with the loop variable leaking.  It appears
> to have to do with the fact that the variables and the generator
> expression are inside a class block.

Interesting.

Just for completeness, the code does seem to work fine when you take
it out of the class:

  Python 2.6.1 (r261:67515, Jun 24 2010, 21:47:49)
  [GCC 4.2.1 (Apple Inc. build 5646)] on darwin
  Type "help", "copyright", "credits" or "license" for more
information.
  >>> foo, bar = 4, 4
  >>> g = (((x, y), x+y) for x in range(foo) for y in range(bar))
  >>> dict(g)
  {(0, 1): 1, (1, 2): 3, (3, 2): 5, (0, 0): 0, (3, 3): 6, (3, 0): 3,
(3, 1): 4, (2, 1): 3, (0, 2): 2, (2, 0): 2, (1, 3): 4, (2, 3): 5, (2,
2): 4, (1, 0): 1, (0, 3): 3, (1, 1): 2}
  >>> import dis
  >>> dis.dis(g.gi_code)
    1           0 SETUP_LOOP              57 (to 60)
                3 LOAD_FAST                0 (.0)
          >>    6 FOR_ITER                50 (to 59)
                9 STORE_FAST               1 (x)
               12 SETUP_LOOP              41 (to 56)
               15 LOAD_GLOBAL              0 (range)
               18 LOAD_GLOBAL              1 (bar)
               21 CALL_FUNCTION            1
               24 GET_ITER
          >>   25 FOR_ITER                27 (to 55)
               28 STORE_FAST               2 (y)
               31 LOAD_FAST                1 (x)
               34 LOAD_FAST                2 (y)
               37 BUILD_TUPLE              2
               40 LOAD_FAST                1 (x)
               43 LOAD_FAST                2 (y)
               46 BINARY_ADD
               47 BUILD_TUPLE              2
               50 YIELD_VALUE
               51 POP_TOP
               52 JUMP_ABSOLUTE           25
          >>   55 POP_BLOCK
          >>   56 JUMP_ABSOLUTE            6
          >>   59 POP_BLOCK
          >>   60 LOAD_CONST               0 (None)
               63 RETURN_VALUE

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web