Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #67091 > unrolled thread

exec and locals

Back to article view | Back to comp.lang.python

Contents

  exec and locals Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-02-26 13:15 +0000
    Re: exec and locals Chris Angelico <rosuav@gmail.com> - 2014-02-27 00:40 +1100
    Re: exec and locals Peter Otten <__peter__@web.de> - 2014-02-26 14:46 +0100
      Re: exec and locals Steven D'Aprano <steve@pearwood.info> - 2014-02-27 00:25 +0000
        Re: exec and locals Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2014-02-27 16:34 +1300
          Re: exec and locals Steven D'Aprano <steve@pearwood.info> - 2014-02-27 04:39 +0000
            Re: exec and locals Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2014-02-28 00:29 +1300
              Re: exec and locals Chris Angelico <rosuav@gmail.com> - 2014-02-27 22:41 +1100
              Re: exec and locals Steven D'Aprano <steve@pearwood.info> - 2014-02-28 01:49 +0000
        Re: exec and locals Dan Sommers <dan@tombstonezero.net> - 2014-02-27 03:47 +0000
        Re: exec and locals Dave Angel <davea@davea.name> - 2014-02-26 23:20 -0500
          Re: exec and locals Steven D'Aprano <steve@pearwood.info> - 2014-02-27 04:47 +0000
            Re: exec and locals Chris Angelico <rosuav@gmail.com> - 2014-02-27 16:05 +1100
    Re: exec and locals Peter Otten <__peter__@web.de> - 2014-02-26 14:55 +0100
    Re: exec and locals Alister <alister.ware@ntlworld.com> - 2014-02-26 14:00 +0000
      Re: exec and locals Steven D'Aprano <steve@pearwood.info> - 2014-02-27 00:31 +0000
        Re: exec and locals Alister <alister.ware@ntlworld.com> - 2014-02-27 09:59 +0000

#67091 — exec and locals

I have to dynamically generate some code inside a function using exec, 
but I'm not sure if it is working by accident or if I can rely on it.

Here is a trivial example:


py> def spam():
...     exec( """x = 23""" )
...     return x
...
py> spam()
23


(My real example is more complex than this.)

According to the documentation of exec, I don't think this should 
actually work, and yet it appears to. The documentation says:

    The default locals act as described for function locals() 
    below: modifications to the default locals dictionary should 
    not be attempted. Pass an explicit locals dictionary if you 
    need to see effects of the code on locals after function 
    exec() returns.

http://docs.python.org/3.4/library/functions.html#exec


I *think* this means that if I want to guarantee that a local variable x 
is created by exec, I need to do this instead:

py> def eggs():
...     mylocals = {}
...     exec( """x = 23""", globals(), mylocals)
...     x = mylocals['x']
...     return x
...
py> eggs()
23

The fact that it works in spam() above is perhaps an accident of 
implementation? Yes no maybe?



-- 
Steven

[toc] | [next] | [standalone]

#67092

On Thu, Feb 27, 2014 at 12:15 AM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> py> def spam():
> ...     exec( """x = 23""" )
> ...     return x
> ...
> py> spam()
> 23
>
>
> (My real example is more complex than this.)
>
> According to the documentation of exec, I don't think this should
> actually work, and yet it appears to.

Doesn't work for me, in IDLE in 3.4.0b2, nor in command-line Python on
3.4.0rc1+ (from hg a couple of weeks ago). But it did (happen to?)
work in 2.7. What version did you use?

ChrisA

[toc] | [prev] | [next] | [standalone]

#67094

Steven D'Aprano wrote:

> I have to dynamically generate some code inside a function using exec,
> but I'm not sure if it is working by accident or if I can rely on it.
> 
> Here is a trivial example:
> 
> 
> py> def spam():
> ...     exec( """x = 23""" )
> ...     return x
> ...
> py> spam()
> 23
> 
> 
> (My real example is more complex than this.)
> 
> According to the documentation of exec, I don't think this should
> actually work, and yet it appears to. The documentation says:
> 
>     The default locals act as described for function locals()
>     below: modifications to the default locals dictionary should
>     not be attempted. Pass an explicit locals dictionary if you
>     need to see effects of the code on locals after function
>     exec() returns.
> 
> http://docs.python.org/3.4/library/functions.html#exec
> 
> 
> I *think* this means that if I want to guarantee that a local variable x
> is created by exec, I need to do this instead:
> 
> py> def eggs():
> ...     mylocals = {}
> ...     exec( """x = 23""", globals(), mylocals)
> ...     x = mylocals['x']
> ...     return x
> ...
> py> eggs()
> 23
> 
> The fact that it works in spam() above is perhaps an accident of
> implementation? Yes no maybe?

eggs() should work in Python 2 and 3,
spam() should work in Python 2, but not in Python 3.

Fun fact: Python 2 tweaks the bytecode (LOAD_NAME instead of LOAD_GLOBAL) to 
make spam() work:

>>> def spam():
...     return x
... 
>>> dis.dis(spam)
  2           0 LOAD_GLOBAL              0 (x)
              3 RETURN_VALUE        
>>> def spam():
...     exec ""
...     return x
... 
>>> dis.dis(spam)
  2           0 LOAD_CONST               1 ('')
              3 LOAD_CONST               0 (None)
              6 DUP_TOP             
              7 EXEC_STMT           

  3           8 LOAD_NAME                0 (x)
             11 RETURN_VALUE        

[toc] | [prev] | [next] | [standalone]

#67106

On Wed, 26 Feb 2014 14:46:39 +0100, Peter Otten wrote:

> Steven D'Aprano wrote:
> 
>> I have to dynamically generate some code inside a function using exec,
>> but I'm not sure if it is working by accident or if I can rely on it.
>> 
>> Here is a trivial example:
>> 
>> 
>> py> def spam():
>> ...     exec( """x = 23""" )
>> ...     return x
>> ...
>> py> spam()
>> 23
>> 
>> 
>> (My real example is more complex than this.)
>> 
>> According to the documentation of exec, I don't think this should
>> actually work, and yet it appears to. The documentation says:
>> 
>>     The default locals act as described for function locals() below:
>>     modifications to the default locals dictionary should not be
>>     attempted. Pass an explicit locals dictionary if you need to see
>>     effects of the code on locals after function exec() returns.
>> 
>> http://docs.python.org/3.4/library/functions.html#exec
>> 
>> 
>> I *think* this means that if I want to guarantee that a local variable
>> x is created by exec, I need to do this instead:
>> 
>> py> def eggs():
>> ...     mylocals = {}
>> ...     exec( """x = 23""", globals(), mylocals) ...     x =
>> mylocals['x']
>> ...     return x
>> ...
>> py> eggs()
>> 23
>> 
>> The fact that it works in spam() above is perhaps an accident of
>> implementation? Yes no maybe?
> 
> eggs() should work in Python 2 and 3, spam() should work in Python 2,
> but not in Python 3.

Aha! That explains it -- I was reading the 3.x docs and testing in Python 
2.7.

Thanks everyone for answering.

By the way, if anyone cares what my actual use-case is, I have a function 
that needs to work under Python 2.4 through 3.4, and it uses a with 
statement. With statements are not available in 2.4 (or 2.5, unless you 
give a from __future__ import). So after messing about for a while with 
circular imports and dependency injections, I eventually settled on some 
code that works something like this:


def factory():
    blah blah blah
    try:
        exec("""def inner():
                    with something:
                        return something
                """, globals(), mylocals)
        inner = mylocals['inner']
    except SyntaxError:
        def inner():
            # manually operate the context manager
            call context manager __enter__
            try:
                try:
                    return something
                except:  # Yes, a bare except. Catch EVERYTHING.
                    blah blah blah
            finally:
                    call context manager __exit__
    blah blah blah
    return inner


(By the way, yes, I have to use a bare except, not just "except 
BaseException". Python 2.4 and 2.5 still have string exceptions.)


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#67109

Steven D'Aprano wrote:
>     except SyntaxError:
>         def inner():
>             # manually operate the context manager
>             call context manager __enter__
>             try:
>                 try:
>                     return something
>                 except:  # Yes, a bare except. Catch EVERYTHING.
>                     blah blah blah
>             finally:
>                     call context manager __exit__

Why not just use this version all the time? It should
work in both 2.x and 3.x.

-- 
Greg

[toc] | [prev] | [next] | [standalone]

#67114

On Thu, 27 Feb 2014 16:34:33 +1300, Gregory Ewing wrote:

> Steven D'Aprano wrote:
>>     except SyntaxError:
>>         def inner():
>>             # manually operate the context manager call context manager
>>             __enter__
>>             try:
>>                 try:
>>                     return something
>>                 except:  # Yes, a bare except. Catch EVERYTHING.
>>                     blah blah blah
>>             finally:
>>                     call context manager __exit__
> 
> Why not just use this version all the time? It should work in both 2.x
> and 3.x.

Because that's yucky. It's an aesthetic thing: when supported, I want the 
Python interpreter to manage the context manager.

The exec part is only half a dozen lines, only three lines of source 
code. It's no burden to keep it for the cases where it works (that is, at 
least 2.6 onwards).


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#67137

Steven D'Aprano wrote:
> On Thu, 27 Feb 2014 16:34:33 +1300, Gregory Ewing wrote:
> 
>>Why not just use this version all the time? It should work in both 2.x
>>and 3.x.
> 
> Because that's yucky. It's an aesthetic thing: when supported, I want the 
> Python interpreter to manage the context manager.

More yucky than wrapping the Py3 version in an
exec? To my way of thinking, that cancels out any
elegance that might have been gained from using
a with-statement.

Do you really need to use the context manager
at all? Could you just write the try-statement
that you would have written in Py2 if you
didn't have a context manager?

-- 
Greg

[toc] | [prev] | [next] | [standalone]

#67138

On Thu, Feb 27, 2014 at 10:29 PM, Gregory Ewing
<greg.ewing@canterbury.ac.nz> wrote:
> Steven D'Aprano wrote:
>>
>> On Thu, 27 Feb 2014 16:34:33 +1300, Gregory Ewing wrote:
>>
>>> Why not just use this version all the time? It should work in both 2.x
>>> and 3.x.
>>
>>
>> Because that's yucky. It's an aesthetic thing: when supported, I want the
>> Python interpreter to manage the context manager.
>
>
> More yucky than wrapping the Py3 version in an
> exec? To my way of thinking, that cancels out any
> elegance that might have been gained from using
> a with-statement.
>
> Do you really need to use the context manager
> at all? Could you just write the try-statement
> that you would have written in Py2 if you
> didn't have a context manager?

If I have to support two vastly different versions, I would prefer
(when possible) to write the code so that dropping the old version's
support is simply a matter of deleting stuff. Write the code for the
new version, then warp it as little as possible to support the old
version as well, and keep it clear which bits are for the old. Writing
code that avoids 'with' altogether goes against that.

ChrisA

[toc] | [prev] | [next] | [standalone]

#67182

On Fri, 28 Feb 2014 00:29:35 +1300, Gregory Ewing wrote:

> Steven D'Aprano wrote:
>> On Thu, 27 Feb 2014 16:34:33 +1300, Gregory Ewing wrote:
>> 
>>>Why not just use this version all the time? It should work in both 2.x
>>>and 3.x.
>> 
>> Because that's yucky. It's an aesthetic thing: when supported, I want
>> the Python interpreter to manage the context manager.
> 
> More yucky than wrapping the Py3 version in an exec? To my way of
> thinking, that cancels out any elegance that might have been gained from
> using a with-statement.
> 
> Do you really need to use the context manager at all? Could you just
> write the try-statement that you would have written in Py2 if you didn't
> have a context manager?

I don't *have* to do it any particular way, but the way it works now, the 
version of the inner function (which does eventually get exposed to the 
caller) is simple with statement wrapping a function call. So for seven 
of the eight versions of Python supported (2.5 through 3.4) the function 
is the simplest it can be. Even if the code creating that function is a 
tiny bit more complex, since it is wrapped inside an exec. For the other 
two versions (2.4 and 2.5), I have to fall back on a more complex chunk 
of code. (And yes, it's deliberate that 2.5 gets counted in both groups.)

I'm not saying that I have objective reasons for preferring this way over 
the manual alternative, or at least not *strong* objective reasons. It's 
mostly subjective. I don't expect to convince you my way is better, and I 
doubt that you'll convince me your way is better. But if I were to try, 
I'd put it this way:

At a cost of six (by memory) extra lines of code, including one call to 
exec, I have an inner function which is *guaranteed* to use the exact 
same semantics and efficiency of a with-statement when possible, because 
it *is* a with-statement. Any differences between with-statement and my 
manual handling will only affect 2.4 and sometimes 2.5, rather than 
everything. The only differences that I know of are insignificant -- the 
byte-code will be different, there may be trivial performance differences 
-- but if it turns out to be some meaningful difference, I have three 
choices:

  1) deny that the difference is meaningful;

  2) accept that the difference is meaningful, and fix it; or

  3) accept that the difference is meaningful, but say that it 
     won't be fixed for 2.4 and 2.5.


If I use the same manual handling for all versions, then I don't have 
that last option, no matter how unlikely it is that I will need it.

But really, it's a subjective choice of what feels right to me in this 
instance. If the body of the with-statement was bigger, or if the feature 
in question was something else, I might choose a different approach.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#67110

On Thu, 27 Feb 2014 00:25:45 +0000, Steven D'Aprano wrote:

> By the way, if anyone cares what my actual use-case is, I have a
> function that needs to work under Python 2.4 through 3.4, and it uses
> a with statement. With statements are not available in 2.4 (or 2.5,
> unless you give a from __future__ import). So after messing about for
> a while with circular imports and dependency injections, I eventually
> settled on some code that works something like this:

> def factory():
>     blah blah blah
>     try:
>         exec("""def inner():
>                     with something:
>                         return something
>                 """, globals(), mylocals)
>         inner = mylocals['inner']
>     except SyntaxError:
>         def inner():
>             # manually operate the context manager
>             call context manager __enter__
>             try:
>                 try:
>                     return something
>                 except:  # Yes, a bare except. Catch EVERYTHING.
>                     blah blah blah
>             finally:
>                     call context manager __exit__
>     blah blah blah
>     return inner

So why not something simpler?

def factory():

    def inner():
        '''Manually operate the context manager in order to maintain
            compatibility with Python 2.4 through 3.4.'''
        call context manager __enter__
        try:
            try:
                return something
            except:  # Yes, a bare except. Catch EVERYTHING.
                blah blah blah
        finally:
                call context manager __exit__
    blah blah blah
    return inner

I claim that the less unnecessary code you write, the fewer bugs you
will have.

Does my code misbehave under any of your target versions?

[toc] | [prev] | [next] | [standalone]

#67112

 Steven D'Aprano <steve@pearwood.info> Wrote in message:
> On Wed, 26 Feb 2014 14:46:39 +0100, Peter Otten wrote:
> 
>> Steven D'Aprano wrote:
>> 
>>> I have to dynamically generate some code inside a function using exec,
>>> but I'm not sure if it is working by accident or if I can rely on it.
>>> 
>> I eventually settled on some 
> code that works something like this:
> 
> 
> def factory():
>     blah blah blah
>     try:
>         exec("""def inner():
>                     

Before I would use exec,  I'd look hard at either generating a
 source file to import,  or using a preprocessor.  And if this
 code was to be installed,  make the version choice or the
 preprocess step happen at install time.

I once implemented a system that generated 20k lines of C++ header
 and sources. And the generated code was properly indented and
 fairly well commented. 



-- 
DaveA

[toc] | [prev] | [next] | [standalone]

#67115

On Wed, 26 Feb 2014 23:20:10 -0500, Dave Angel wrote:

> Before I would use exec,  I'd look hard at either generating a
>  source file to import,  

Yes, I went through the process of pulling out the code into a separate 
module, but that just made more complexity and was pretty nasty. If the 
function was stand-alone, it might have worked, but it needed access to 
other code in the module, so there were circular dependencies.


> or using a preprocessor. 

I don't think that it's easier/better to write a custom Python 
preprocessor and run the entire module through it, just to avoid a three-
line call to exec.

Guys, I know that exec is kinda dangerous and newbies should be 
discouraged from throwing every string they see at it, but this isn't my 
second day Python programming, and it's not an accident that Python 
supports the dynamic compilation and execution of source code at runtime. 
It's a deliberate language feature. We're allowed to use it :-)


> And if this code was
>  to be installed,  make the version choice or the preprocess step happen
>  at install time.

Completely inappropriate in my case. This is a module which can be called 
from multiple versions of Python from a single installation.


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#67116

On Thu, Feb 27, 2014 at 3:47 PM, Steven D'Aprano <steve@pearwood.info> wrote:
> Guys, I know that exec is kinda dangerous and newbies should be
> discouraged from throwing every string they see at it, but this isn't my
> second day Python programming, and it's not an accident that Python
> supports the dynamic compilation and execution of source code at runtime.
> It's a deliberate language feature. We're allowed to use it :-)

Code smell means "look at this". It doesn't mean "don't use this
feature ever". :) Steven's looked into this thoroughly, I'm sure, and
exec is important.

ChrisA

[toc] | [prev] | [next] | [standalone]

#67095

Peter Otten wrote:

> Steven D'Aprano wrote:
> 
>> I have to dynamically generate some code inside a function using exec,
>> but I'm not sure if it is working by accident or if I can rely on it.
>> 
>> Here is a trivial example:
>> 
>> 
>> py> def spam():
>> ...     exec( """x = 23""" )
>> ...     return x
>> ...
>> py> spam()
>> 23
>> 
>> 
>> (My real example is more complex than this.)
>> 
>> According to the documentation of exec, I don't think this should
>> actually work, and yet it appears to. The documentation says:
>> 
>>     The default locals act as described for function locals()
>>     below: modifications to the default locals dictionary should
>>     not be attempted. Pass an explicit locals dictionary if you
>>     need to see effects of the code on locals after function
>>     exec() returns.
>> 
>> http://docs.python.org/3.4/library/functions.html#exec
>> 
>> 
>> I *think* this means that if I want to guarantee that a local variable x
>> is created by exec, I need to do this instead:
>> 
>> py> def eggs():
>> ...     mylocals = {}
>> ...     exec( """x = 23""", globals(), mylocals)
>> ...     x = mylocals['x']
>> ...     return x
>> ...
>> py> eggs()
>> 23
>> 
>> The fact that it works in spam() above is perhaps an accident of
>> implementation? Yes no maybe?
> 
> eggs() should work in Python 2 and 3,
> spam() should work in Python 2, but not in Python 3.
> 
> Fun fact: Python 2 tweaks the bytecode (LOAD_NAME instead of LOAD_GLOBAL)
> to make spam() work:
> 
>>>> def spam():
> ...     return x
> ...
>>>> dis.dis(spam)
>   2           0 LOAD_GLOBAL              0 (x)
>               3 RETURN_VALUE
>>>> def spam():
> ...     exec ""
> ...     return x
> ...
>>>> dis.dis(spam)
>   2           0 LOAD_CONST               1 ('')
>               3 LOAD_CONST               0 (None)
>               6 DUP_TOP
>               7 EXEC_STMT
> 
>   3           8 LOAD_NAME                0 (x)
>              11 RETURN_VALUE

Some more bytcode fun, because it just occured to me that you can optimize 
away the code that triggered the modification:

>>> def spam():
...     return x
...     if 0: exec ""
... 
>>> dis.dis(spam)
  2           0 LOAD_NAME                0 (x)
              3 RETURN_VALUE        

[toc] | [prev] | [next] | [standalone]

#67096

On Wed, 26 Feb 2014 13:15:25 +0000, Steven D'Aprano wrote:

> I have to dynamically generate some code inside a function using exec,
> but I'm not sure if it is working by accident or if I can rely on it.
> 
> Here is a trivial example:
> 
> 
> py> def spam():
> ...     exec( """x = 23""" )
> ...     return x ...
> py> spam()
> 23
> 
> 
> (My real example is more complex than this.)
> 
> According to the documentation of exec, I don't think this should
> actually work, and yet it appears to. The documentation says:
> 
>     The default locals act as described for function locals() below:
>     modifications to the default locals dictionary should not be
>     attempted. Pass an explicit locals dictionary if you need to see
>     effects of the code on locals after function exec() returns.
> 
> http://docs.python.org/3.4/library/functions.html#exec
> 
> 
> I *think* this means that if I want to guarantee that a local variable x
> is created by exec, I need to do this instead:
> 
> py> def eggs():
> ...     mylocals = {}
> ...     exec( """x = 23""", globals(), mylocals)
> ...     x = mylocals['x']
> ...     return x ...
> py> eggs()
> 23
> 
> The fact that it works in spam() above is perhaps an accident of
> implementation? Yes no maybe?

I have no idea but as exec is generally considered to be a bad idea are 
you absolutely sure this is the correct way to achieve your end goal?

perhaps if you detailed your requirement someone may be able to suggest a 
safer solution.


-- 
"Regardless of the legal speed limit, your Buick must be operated at
speeds faster than 85 MPH (140kph)."
-- 1987 Buick Grand National owners manual.

[toc] | [prev] | [next] | [standalone]

#67107

On Wed, 26 Feb 2014 14:00:59 +0000, Alister wrote:

> On Wed, 26 Feb 2014 13:15:25 +0000, Steven D'Aprano wrote:
> 
>> I have to dynamically generate some code inside a function using exec,
>> but I'm not sure if it is working by accident or if I can rely on it.
[...]

> I have no idea but as exec is generally considered to be a bad idea are
> you absolutely sure this is the correct way to achieve your end goal?
> 
> perhaps if you detailed your requirement someone may be able to suggest
> a safer solution.

Thanks for your concern, but what I'm doing is perfectly safe. The string 
being exec'ed is a string literal known at compile-time and written by me 
(see my previous email for details) and the only reason I'm running it 
with exec at runtime rather than treating it as normal source code is 
that it relies on a feature that may not be available (with statement).

The joys of writing code that has to run under multiple incompatible 
versions of Python.


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#67129

On Thu, 27 Feb 2014 00:31:56 +0000, Steven D'Aprano wrote:

> On Wed, 26 Feb 2014 14:00:59 +0000, Alister wrote:
> 
>> On Wed, 26 Feb 2014 13:15:25 +0000, Steven D'Aprano wrote:
>> 
>>> I have to dynamically generate some code inside a function using exec,
>>> but I'm not sure if it is working by accident or if I can rely on it.
> [...]
> 
>> I have no idea but as exec is generally considered to be a bad idea are
>> you absolutely sure this is the correct way to achieve your end goal?
>> 
>> perhaps if you detailed your requirement someone may be able to suggest
>> a safer solution.
> 
> Thanks for your concern, but what I'm doing is perfectly safe. The
> string being exec'ed is a string literal known at compile-time and
> written by me (see my previous email for details) and the only reason
> I'm running it with exec at runtime rather than treating it as normal
> source code is that it relies on a feature that may not be available
> (with statement).
> 
> The joys of writing code that has to run under multiple incompatible
> versions of Python.

I hadn't noticed who had made the original post or a I wouldn't have 
bothered, you know Python & what is safe far better than I ever will




-- 
The world is not octal despite DEC.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web