Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #16672 > unrolled thread

Scope of variable inside list comprehensions?

Back to article view | Back to comp.lang.python

Contents

  Scope of variable inside list comprehensions? Roy Smith <roy@panix.com> - 2011-12-05 09:04 -0800
    Re: Scope of variable inside list comprehensions? Jussi Piitulainen <jpiitula@ling.helsinki.fi> - 2011-12-05 20:04 +0200
    Re: Scope of variable inside list comprehensions? Peter Otten <__peter__@web.de> - 2011-12-05 19:10 +0100
    Re: Scope of variable inside list comprehensions? Jean-Michel Pichavant <jeanmichel@sequans.com> - 2011-12-05 19:57 +0100
      Re: Scope of variable inside list comprehensions? Roy Smith <roy@panix.com> - 2011-12-05 11:15 -0800
      Re: Scope of variable inside list comprehensions? Roy Smith <roy@panix.com> - 2011-12-05 11:15 -0800
        Re: Scope of variable inside list comprehensions? Terry Reedy <tjreedy@udel.edu> - 2011-12-05 15:22 -0500
          Re: Scope of variable inside list comprehensions? Roy Smith <roy@panix.com> - 2011-12-05 14:36 -0800
          Re: Scope of variable inside list comprehensions? Roy Smith <roy@panix.com> - 2011-12-05 14:36 -0800
            Re: Scope of variable inside list comprehensions? Terry Reedy <tjreedy@udel.edu> - 2011-12-05 21:23 -0500
      Re: Scope of variable inside list comprehensions? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-12-05 22:35 +0000
        Re: Scope of variable inside list comprehensions? Roy Smith <roy@panix.com> - 2011-12-05 14:57 -0800
          Re: Scope of variable inside list comprehensions? Chris Angelico <rosuav@gmail.com> - 2011-12-06 13:35 +1100
        Re: Scope of variable inside list comprehensions? Jean-Michel Pichavant <jeanmichel@sequans.com> - 2011-12-06 11:38 +0100
    Re: Scope of variable inside list comprehensions? Rainer Grimm <r.grimm@science-computing.de> - 2011-12-05 22:42 -0800
      Re: Scope of variable inside list comprehensions? 88888 Dihedral <dihedral88888@googlemail.com> - 2011-12-06 02:16 -0800

#16672 — Scope of variable inside list comprehensions?

Consider the following django snippet.  Song(id) raises DoesNotExist if the id is unknown.

    try:
        songs = [Song(id) for id in song_ids]
    except Song.DoesNotExist:
        print "unknown song id (%d)" % id

Is id guaranteed to be in scope in the print statement?  I found one thread (http://mail.python.org/pipermail/python-bugs-list/2006-April/033235.html) which says yes, but hints that it might not always be in the future.  Now that we're in the future, is that still true?  And for Python 3 also?

The current docs, http://docs.python.org/tutorial/datastructures.html#list-comprehensions, are mute on this point.

[toc] | [next] | [standalone]

#16674

Roy Smith writes:

> Consider the following django snippet.  Song(id) raises DoesNotExist
> if the id is unknown.
> 
>     try:
>         songs = [Song(id) for id in song_ids]
>     except Song.DoesNotExist:
>         print "unknown song id (%d)" % id
> 
> Is id guaranteed to be in scope in the print statement?  I found one
> thread
> (http://mail.python.org/pipermail/python-bugs-list/2006-April/033235.html)
> which says yes, but hints that it might not always be in the future.
> Now that we're in the future, is that still true?  And for Python 3
> also?

Another id is in scope in this Python3 example (3.1.2, Ubuntu):

>>> try:
...    songs = [1/0 for id in [1]]
... except Exception:
...    print('Caught', id)
... 
Caught <built-in function id>

[toc] | [prev] | [next] | [standalone]

#16675

Roy Smith wrote:

> Consider the following django snippet.  Song(id) raises DoesNotExist if
> the id is unknown.
> 
>     try:
>         songs = [Song(id) for id in song_ids]
>     except Song.DoesNotExist:
>         print "unknown song id (%d)" % id
> 
> Is id guaranteed to be in scope in the print statement?  I found one
> thread
> (http://mail.python.org/pipermail/python-bugs-list/2006-April/033235.html)
> which says yes, but hints that it might not always be in the future.  Now
> that we're in the future, is that still true?  And for Python 3 also?
> 
> The current docs,
> http://docs.python.org/tutorial/datastructures.html#list-comprehensions,
> are mute on this point.

If you are using a generator expression id will already be out of scope in 
Python 2. In Python 3 list comprehensions have been changed to work the same 
way:

$ cat song.py
class DoesNotExist(Exception):
    pass

class Song:
    def __init__(self, id):
        if id == 2:
            raise DoesNotExist

ids = [1, 2]
try:
    songs = [Song(i) for i in ids]
except DoesNotExist as e:
    print("song #%d does not exist" % i)
$ python song.py
song #2 does not exist
$ python3 song.py
Traceback (most recent call last):
  File "song.py", line 11, in <module>
    songs = [Song(i) for i in ids]
  File "song.py", line 11, in <listcomp>
    songs = [Song(i) for i in ids]
  File "song.py", line 7, in __init__
    raise DoesNotExist
__main__.DoesNotExist

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "song.py", line 13, in <module>
    print("song #%d does not exist" % i)
NameError: name 'i' is not defined
$


$ cat song_gen.py
class DoesNotExist(Exception):
    pass

class Song:
    def __init__(self, id):
        if id == 2:
            raise DoesNotExist

ids = [1, 2]
try:
    songs = list(Song(i) for i in ids)
except DoesNotExist as e:
    print("song #%d does not exist" % i)
$ python  song_gen.py
Traceback (most recent call last):
  File "song_gen.py", line 13, in <module>
    print("song #%d does not exist" % i)
NameError: name 'i' is not defined

So you'd rather store the id in the exception.

[toc] | [prev] | [next] | [standalone]

#16679

Roy Smith wrote:
> Consider the following django snippet.  Song(id) raises DoesNotExist if the id is unknown.
>
>     try:
>         songs = [Song(id) for id in song_ids]
>     except Song.DoesNotExist:
>         print "unknown song id (%d)" % id
>
> Is id guaranteed to be in scope in the print statement?  I found one thread (http://mail.python.org/pipermail/python-bugs-list/2006-April/033235.html) which says yes, but hints that it might not always be in the future.  Now that we're in the future, is that still true?  And for Python 3 also?
>
> The current docs, http://docs.python.org/tutorial/datastructures.html#list-comprehensions, are mute on this point.
>   
For python 2, id will always be defined *as you meant it*. But you're 
doing something wrong : overiding the 'id' builtin function.
With python 3 you will probably print the 'id' builtin function 
representation, which is correct but not want you want to achieve.

The proper way to propagate information with exceptions is using the 
exception itself:

try:
    songs = [Song(_id) for _id in song_ids]
except Song.DoesNotExist, exc:
    print exc

class DoesNotExist(Exception):
    def __init__(self, songId):
        self.songId = songId
    def __str__(self):
        return "Unkown Song Id %s" % self.songId

class Song:
    def __init__(self, songId):
	if whatever:
             raise DoesNotExist(songId)
        self.id=songId

JM

[toc] | [prev] | [next] | [standalone]

#16680

Hmmm, the use of id was just a simplification for the sake of posting.  The real code is a bit more complicated and used a different variable name, but that's a good point.

As far as storing the value in the exception, unfortunately, DoesNotExist is not my exception; it comes from deep within django.  I'm just passing it along.

[toc] | [prev] | [next] | [standalone]

#16681

Hmmm, the use of id was just a simplification for the sake of posting.  The real code is a bit more complicated and used a different variable name, but that's a good point.

As far as storing the value in the exception, unfortunately, DoesNotExist is not my exception; it comes from deep within django.  I'm just passing it along.

[toc] | [prev] | [next] | [standalone]

#16684

On 12/5/2011 2:15 PM, Roy Smith wrote:
> Hmmm, the use of id was just a simplification for the sake of
> posting.  The real code is a bit more complicated and used a
> different variable name, but that's a good point.
>
> As far as storing the value in the exception, unfortunately,
> DoesNotExist is not my exception; it comes from deep within django.
> I'm just passing it along.

It is hard to sensibly answer a question when the questioner 
*significantly* changes the problem in the process of 'simplifying' it. 
Both of those are significant changes ;-)

Changing a name to a built-in name is a complexification, not a 
simplification, because it introduces new issues that were not in the 
original.

Changing the exception from one you do not control to one you apparently 
do also changes the appropriate answer. If you do not control the 
exception and you want guaranteed access to the loop variable with 
Python 3 (and the upgrade of django to work with Python 3 is more or 
less done), then use an explicit loop. If you want the loop to continue 
after an error, instead of stopping, you can put the try/except within 
the loop, instead of without. This possibility is one advantage of using 
an explicit loop.

songs = []
for song_id in song_ids:
   try:
     songs.append(Song(song_id))
   except django.error:
     print("unknown song id {}".format(song_id))

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#16691

Well, in my defense, I did ask a pretty narrow question, "Is id guaranteed to be in scope in the print statement?".  While I will admit that not knowing whether I could alter the exception, or whether id masked a builtin or not does complexify answering some questions, those are questions I didn't ask :-)

[toc] | [prev] | [next] | [standalone]

#16692

Well, in my defense, I did ask a pretty narrow question, "Is id guaranteed to be in scope in the print statement?".  While I will admit that not knowing whether I could alter the exception, or whether id masked a builtin or not does complexify answering some questions, those are questions I didn't ask :-)

[toc] | [prev] | [next] | [standalone]

#16696

On 12/5/2011 5:36 PM, Roy Smith wrote:
> Well, in my defense, I did ask a pretty narrow question, "Is id
> guaranteed to be in scope in the print statement?".

Yes for 2.x, guaranteed no for 3.x.

If you had simply asked "Is the loop variable of a list comprehension 
guaranteed to be in scope after the list comprehension?", without a 
distracting example, that is the answer you would have received.

I intend(ed) to inform, not attack, hence no 'defense' needed.

>  While I will
> admit that not knowing whether I could alter the exception, or
> whether id masked a builtin or not does complexify answering some
> questions, those are questions I didn't ask :-)

Except that it bears on the question you did ask because it means that 
the code will run in 3.x but with different results, whereas a random 
name will fail in 3.x with a NameError.

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#16690

On Mon, 05 Dec 2011 19:57:15 +0100, Jean-Michel Pichavant wrote:

> The proper way to propagate information with exceptions is using the
> exception itself:
> 
> try:
>     songs = [Song(_id) for _id in song_ids]
> except Song.DoesNotExist, exc:
>     print exc


I'm not entirely sure that this is the proper way to propagate the 
exception. I see far to many people catching exceptions to print them, or 
worse, to print a generic, useless message like "an error occurred".

The problem here is that having caught the exception, songs now does not 
exist, and will surely cause another, unexpected, exception in a moment 
or two when the code attempts to use it. Since the error is (apparently) 
unrecoverable, the right way as far as I can see is:

songs = [Song(_id) for _id in song_ids]

allowing any exception to be fatal and the traceback to be printed as 
normal.

If the error is recoverable, you will likely need to do more to recover 
from it than merely print the exception and continue.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#16694

Sigh.  I attempted to reduce this to a minimal example to focus the discussion on the question of list comprehension variable scope.  Instead I seem to have gotten people off on other tangents.  I suppose I should post more of the real code...

    song_ids = request.POST.getlist('song_id')
    try:
        songs = [Song.get(int(id)) for id in song_ids]
    except Song.DoesNotExist:
        return HttpResponseBadRequest("unknown song id (%d)" % id)

I may be in the minority here, but it doesn't bother me much that my use of 'id' shadows a built-in.  Especially in small scopes like this, I use whatever variable names make the the code easiest to read and don't worry about shadowing builtins.  I don't have an exhaustive list of builtins in my head, so even if I worried about common ones like file or id, I'm sure I'd miss some others.  So I don't sweat it.

If shadowing builtins was really evil, they'd be reserved keywords and then you wouldn't be able to do it.

[toc] | [prev] | [next] | [standalone]

#16697

On Tue, Dec 6, 2011 at 9:57 AM, Roy Smith <roy@panix.com> wrote:
> I may be in the minority here, but it doesn't bother me much that my use of 'id' shadows a built-in.  Especially in small scopes like this, I use whatever variable names make the the code easiest to read and don't worry about shadowing builtins.  I don't have an exhaustive list of builtins in my head, so even if I worried about common ones like file or id, I'm sure I'd miss some others.  So I don't sweat it.
>
> If shadowing builtins was really evil, they'd be reserved keywords and then you wouldn't be able to do it.

Agreed. The name 'id' is one that's shadowed benignly in a lot of
code. In the context of the original question, it's not an issue -
there'll be no confusion.

Python's scoping rules are "do what the programmer probably wants".
This works most of the time, but occasionally you get edge cases where
things are a bit weird, and C-style explicit scoping (with infinitely
nested block scope) begins to look better. But for probably 99% of
situations, Python's system "just works".

If you need more flexibility in the exception you throw, it may be
worth putting together a filter:

def HttpSongId(id):
   try:
       return Song.get(int(id))
   except Song.DoesNotExist:
       return HttpResponseBadRequest("unknown song id (%d)" % id)

   song_ids = request.POST.getlist('song_id')
   songs = [HttpSongId(id) for id in song_ids]

This encapsulates things in a somewhat weird way, but if there's any
other work to be done at the same time, you could probably come up
with a better name for the exception filter function.

ChrisA

[toc] | [prev] | [next] | [standalone]

#16716

Steven D'Aprano wrote:
> On Mon, 05 Dec 2011 19:57:15 +0100, Jean-Michel Pichavant wrote:
>
>   
>> The proper way to propagate information with exceptions is using the
>> exception itself:
>>
>> try:
>>     songs = [Song(_id) for _id in song_ids]
>> except Song.DoesNotExist, exc:
>>     print exc
>>     
>
>
> I'm not entirely sure that this is the proper way to propagate the 
> exception. I see far to many people catching exceptions to print them, or 
> worse, to print a generic, useless message like "an error occurred".
>   
[snip]

You misread me, I was referering to passing *information* with exception 
(in other words, use the exception attributes). In the example I gave, 
the exception has the songId value responsible for raising the error.
I totaly second your opinion on how poor the above handler is (hmm not 
sure about this grammar construct, it sounds like a Yoda sentence).

JM

[toc] | [prev] | [next] | [standalone]

#16704

Hello,

>     try:
>         songs = [Song(id) for id in song_ids]
>     except Song.DoesNotExist:
>         print "unknown song id (%d)" % id
that's is a bad programming style. So it will be forbidden with python 3. The reason is that list comprehension is a construct from the functional world. It's only syntactic sugar for the functions map and filter. So functions have to be pure functions. To say it in other words, they have to be side-effect free. But the python construct from above pollutes the namespace with name id.

Greetings from Rottenburg,
Rainer

[toc] | [prev] | [next] | [standalone]

#16715

On Tuesday, December 6, 2011 2:42:35 PM UTC+8, Rainer Grimm wrote:
> Hello,
> 
> >     try:
> >         songs = [Song(id) for id in song_ids]
> >     except Song.DoesNotExist:
> >         print "unknown song id (%d)" % id
> that's is a bad programming style. So it will be forbidden with python 3. The reason is that list comprehension is a construct from the functional world. It's only syntactic sugar for the functions map and filter. So functions have to be pure functions. To say it in other words, they have to be side-effect free. But the python construct from above pollutes the namespace with name id.
> 
> Greetings from Rottenburg,
> Rainer

The list might have to grow in a careless way that might lead to a crash
in the for inside a list that can't be trapped for errors directly. 

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web