Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #16783 > unrolled thread

Dynamic variable creation from string

Back to article view | Back to comp.lang.python

Contents

  Dynamic variable creation from string Massi <massi_srb@msn.com> - 2011-12-07 09:09 -0800
    Re: Dynamic variable creation from string John Gordon <gordon@panix.com> - 2011-12-07 17:45 +0000
      Re: Dynamic variable creation from string MRAB <python@mrabarnett.plus.com> - 2011-12-07 18:07 +0000
    Re: Dynamic variable creation from string "Waldek M." <wm@localhost.localdomain> - 2011-12-07 18:46 +0100
    Re: Dynamic variable creation from string Chris Angelico <rosuav@gmail.com> - 2011-12-08 04:52 +1100
    Re: Dynamic variable creation from string Terry Reedy <tjreedy@udel.edu> - 2011-12-07 17:59 -0500
    Re: Dynamic variable creation from string Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-12-08 00:03 +0000
      Re: Dynamic variable creation from string Terry Reedy <tjreedy@udel.edu> - 2011-12-07 19:27 -0500
        Re: Dynamic variable creation from string Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-12-08 01:59 +0000
        Re: Dynamic variable creation from string Jussi Piitulainen <jpiitula@ling.helsinki.fi> - 2011-12-08 10:54 +0200
          Re: Dynamic variable creation from string Massi <massi_srb@msn.com> - 2011-12-09 01:55 -0800
            Re: Dynamic variable creation from string Peter Otten <__peter__@web.de> - 2011-12-09 12:27 +0100
            Re: Dynamic variable creation from string Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-12-09 11:59 +0000
              Re: Dynamic variable creation from string Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-12-09 12:03 +0000
              Re: Dynamic variable creation from string Chris Angelico <rosuav@gmail.com> - 2011-12-09 23:08 +1100
            Re: Dynamic variable creation from string Ethan Furman <ethan@stoneleaf.us> - 2011-12-09 12:01 -0800
            Re: Dynamic variable creation from string Nobody <nobody@nowhere.com> - 2011-12-11 06:42 +0000
              Re: Dynamic variable creation from string alex23 <wuwei23@gmail.com> - 2011-12-11 19:31 -0800
                Re: Dynamic variable creation from string Ethan Furman <ethan@stoneleaf.us> - 2011-12-11 21:00 -0800
                  Re: Dynamic variable creation from string alex23 <wuwei23@gmail.com> - 2011-12-11 22:50 -0800
      Re: Dynamic variable creation from string Chris Angelico <rosuav@gmail.com> - 2011-12-08 14:13 +1100
    Re: Dynamic variable creation from string alex23 <wuwei23@gmail.com> - 2011-12-11 23:11 -0800
      Re: Dynamic variable creation from string 88888 Dihedral <dihedral88888@googlemail.com> - 2011-12-12 04:43 -0800
      Re: Dynamic variable creation from string 88888 Dihedral <dihedral88888@googlemail.com> - 2011-12-12 04:49 -0800
        Re: Dynamic variable creation from string alex23 <wuwei23@gmail.com> - 2011-12-12 15:45 -0800
          Re: Dynamic variable creation from string Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-12-13 01:35 +0000
            Re: Dynamic variable creation from string 88888 Dihedral <dihedral88888@googlemail.com> - 2011-12-13 06:21 -0800

Page 1 of 2  [1] 2  Next page →

#16783 — Dynamic variable creation from string

Hi everyone,

in my script I have a dictionary whose items are couples in the form
(string, integer values), say

D = {'a':1, 'b':2, 'c':3}

This dictionary is passed to a function as a parameter, e.g. :

def Sum(D) :
    return D['a']+D['b']+D['c']

Is there a way to create three variables dynamically inside Sum in
order to re write the function like this?

def Sum(D) :
    # Here some magic to create a,b,c from D
    return a+b+c

It is really important that the scope of a,b,c is limited to the Sum
function, they must not exisit outside it or inside any other nested
functions.
Thanks in advance for your help!

[toc] | [next] | [standalone]

#16784

In <b078a04b-024b-48dc-b24a-8f4ce75fa238@13g2000vbu.googlegroups.com> Massi <massi_srb@msn.com> writes:

> in my script I have a dictionary whose items are couples in the form
> (string, integer values), say

> D = {'a':1, 'b':2, 'c':3}

> This dictionary is passed to a function as a parameter, e.g. :

> def Sum(D) :
>     return D['a']+D['b']+D['c']

> Is there a way to create three variables dynamically inside Sum in
> order to re write the function like this?

Do you want to sum all the values in D?  If so, that's easy:

  def Sum(D):
      my_sum = 0
      for item in D:
          my_sum += D[item]
      return my_sum

-- 
John Gordon                   A is for Amy, who fell down the stairs
gordon@panix.com              B is for Basil, assaulted by bears
                                -- Edward Gorey, "The Gashlycrumb Tinies"

[toc] | [prev] | [next] | [standalone]

#16788

On 07/12/2011 17:45, John Gordon wrote:
> In<b078a04b-024b-48dc-b24a-8f4ce75fa238@13g2000vbu.googlegroups.com>  Massi<massi_srb@msn.com>  writes:
>
>> in my script I have a dictionary whose items are couples in the form
>> (string, integer values), say
>
>> D = {'a':1, 'b':2, 'c':3}
>
>> This dictionary is passed to a function as a parameter, e.g. :
>
>> def Sum(D) :
>>      return D['a']+D['b']+D['c']
>
>> Is there a way to create three variables dynamically inside Sum in
>> order to re write the function like this?
>
> Do you want to sum all the values in D?  If so, that's easy:
>
>    def Sum(D):
>        my_sum = 0
>        for item in D:
>            my_sum += D[item]
>        return my_sum
>
Or even:

def Sum(D):
     return sum(D.values())

[toc] | [prev] | [next] | [standalone]

#16785

On Wed, 7 Dec 2011 09:09:16 -0800 (PST), Massi wrote:
> def Sum(D) :
>     return D['a']+D['b']+D['c']
> 
> Is there a way to create three variables dynamically inside Sum in
> order to re write the function like this?
> 
> def Sum(D) :
>     # Here some magic to create a,b,c from D
>     return a+b+c

Hello,

> It is really important that the scope of a,b,c is limited to the Sum
> function, they must not exisit outside it or inside any other nested
> functions.
> Thanks in advance for your help!

Can you clarify a bit? I'm not sure why do you need to define any
additional variables at all. You do not return variables
from a function - you can return *a value* which may be
a simple type or complex type.

Or maybe you mean something like creating a global name
from inside of a function? Well, that doesn't sound like a good idea,
though...

>>> def foo():
	global a
	a = 5
	
>>> foo()
>>> print("The variable is: ", a)
The variable is:  5
>>> 



Best regards,
Waldek

[toc] | [prev] | [next] | [standalone]

#16787

On Thu, Dec 8, 2011 at 4:09 AM, Massi <massi_srb@msn.com> wrote:
> def Sum(D) :
>    # Here some magic to create a,b,c from D
>    return a+b+c

Welcome to TMTOWTDI land! We do magic here... several different ways.

You _may_ be able to do this, which is roughly equivalent to the
extract() function in PHP:

locals().update(D)

However, this is NOT guaranteed to work. It's more likely to work with
globals(), but that wouldn't restrict the scope the way you asked.

One handy trick that I found on the internet while researching this:
Use (or abuse!) function keyword parameters to do the dictionary
extraction. You have to explicitly name your desired keys this way,
but it may be suitable. (The function doesn't have to be defined
inside the other, incidentally.)

>>> def Sum(D):
       def inner_sum(a,b,c,**kwargs):
               # Real logic goes here.
               return a+b+c
       return inner_sum(**D)

>>> a={"a":5,"b":10,"c":20}
>>> Sum(a)
35

Alternatively, the exec() and eval() functions can be given
dictionaries which will serve as their variable scopes, so you could
possibly use that. Definitely looking like ugly code though.

For something as trivial as Sum(), you'd do best to simply type
D['a']+D['b']+D['c'] and be done with it. For something where you're
going to use them a lot, probably easiest to be explicit:

a,b,c = D['a'],D['b'],D['c']

However, this violates DRY principle, and risks hard-to-trace mismatch
bugs. I'd be inclined to simply be explicit all the time.

Depending on what D is, you may actually want to consider rewriting
this as a class with a member function.

class whatever:
 def Sum(self):
   return self.a+self.b+self.c

There should be one obvious way to do it, says the zen of Python.
Frankly, I'm not sure what that one obvious way is, here, although I'm
pretty certain that several of the options posited would be very bad
for your code!

Still, down's very nice... They ARE alternative possibilities.

ChrisA

[toc] | [prev] | [next] | [standalone]

#16793

On 12/7/2011 12:09 PM, Massi wrote:

> in my script I have a dictionary whose items are couples in the form
> (string, integer values), say
>
> D = {'a':1, 'b':2, 'c':3}
>
> This dictionary is passed to a function as a parameter, e.g. :
>
> def Sum(D) :
>      return D['a']+D['b']+D['c']
>
> Is there a way to create three variables dynamically inside Sum in
> order to re write the function like this?
>
> def Sum(D) :
>      # Here some magic to create a,b,c from D
>      return a+b+c

No. The set of local names for a function is determined when the 
definition is executed and the body is compiled. Python 2 had an 
exception -- 'from x import *' -- that required an alternate 
complilation pathway. That possibility was eliminated in Python 3 and is 
now a syntax error.

Within functions, 'locals().update(D)' is more or less guaranteed to 
*not* add local names to the local namespace, even if it does add keys 
to the locals() dict that shadows the local namespace.

 > It is really important that the scope of a,b,c is limited to the Sum
 > function, they must not exisit outside it or inside any other nested
 > functions.

Local names, by definition, are lexically scoped to the function 
definition and are not visible without. Since nested definitions are 
part of that lexical scope, local names are always visible within nested 
definitions. You cannot stop that. The association of local names is 
usually dynamically limited to one function call. The two exceptions are 
enclosure by a nested function that survives the function call and 
generators in a paused state.

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#16794

On Wed, 07 Dec 2011 09:09:16 -0800, Massi wrote:

> Is there a way to create three variables dynamically inside Sum in order
> to re write the function like this?
> 
> def Sum(D) :
>     # Here some magic to create a,b,c from D 
>     return a+b+c

No magic is needed.

a, b, c = D['a'], D['b'], D['c']

However, there is no way to create variables from an *arbitrary* set of 
keys. And a good thing too, because how could you use them?

def Sum(D):
   # Magic happens here
   return a + b + z + foo + spam + fred + ... ???

Since you don't know what variable names will be created, you don't know 
which variables you need to add. Dynamic creation of variables is ALMOST 
ALWAYS the wrong approach.

In this specific case, what you should do to add up an arbitrary number 
of values from a dict is:

sum(D.values())


> It is really important that the scope of a,b,c is limited to the Sum
> function, they must not exisit outside it or inside any other nested
> functions.

The first part is trivially easy; since you are assigning to local 
variables, by definition they will be local to the Sum function and will 
not exist outside it.

The second part is impossible, because that is not how Python works. 
Nested functions in Python can always see variables in their enclosing 
scope. If you don't want that behaviour, use another language, or don't 
use nested functions.


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#16795

On 12/7/2011 7:03 PM, Steven D'Aprano wrote:
> On Wed, 07 Dec 2011 09:09:16 -0800, Massi wrote:
>
>> Is there a way to create three variables dynamically inside Sum in order
>> to re write the function like this?

I should have mentioned in my earlier response that 'variable' is a bit 
vague and misleading. Python has names bound to objects.

>> def Sum(D) :
>>      # Here some magic to create a,b,c from D
>>      return a+b+c
>
> No magic is needed.
>
> a, b, c = D['a'], D['b'], D['c']

This is not what most people mean by 'dynamically created variables'. 
The names are static, in the code, before the code is executed.
In 2.x, 'from x import *' dynamically creates local names that are not 
in the code that contains the import. Dynamically creating objects is 
what Python code does all the time.

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#16796

On Wed, 07 Dec 2011 19:27:43 -0500, Terry Reedy wrote:

> On 12/7/2011 7:03 PM, Steven D'Aprano wrote:
>> On Wed, 07 Dec 2011 09:09:16 -0800, Massi wrote:
>>
>>> Is there a way to create three variables dynamically inside Sum in
>>> order to re write the function like this?
> 
> I should have mentioned in my earlier response that 'variable' is a bit
> vague and misleading. Python has names bound to objects.
> 
>>> def Sum(D) :
>>>      # Here some magic to create a,b,c from D return a+b+c
>>
>> No magic is needed.
>>
>> a, b, c = D['a'], D['b'], D['c']
> 
> This is not what most people mean by 'dynamically created variables'.

I know that. I'm just pointing out that the OP can solve his *stated* 
problem of creating a *fixed* number of variables with *known* names 
without any magic.

I went on to discuss the case of an unknown number of unknown names, and 
suggested that the OP not do that, because it is much less useful than 
people think.


> The names are static, in the code, before the code is executed. In 2.x,
> 'from x import *' dynamically creates local names that are not in the
> code that contains the import. Dynamically creating objects is what
> Python code does all the time.

Of course. And even dynamically creating names: names don't exist until 
they are created at runtime. Name deletion is also possible. But what 
isn't normally done is "dynamic variable creation" in the sense of 
creating arbitrary variables (names bound to objects) based on names 
known only at runtime. That's extremely rare, and for good reason.


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#16811

Terry Reedy writes:

> On 12/7/2011 7:03 PM, Steven D'Aprano wrote:
> > On Wed, 07 Dec 2011 09:09:16 -0800, Massi wrote:
> >
> >> Is there a way to create three variables dynamically inside Sum
> >> in order to re write the function like this?
> 
> I should have mentioned in my earlier response that 'variable' is a
> bit vague and misleading. Python has names bound to objects.

The language reference at python.org uses both terms - name and
variable - freely. Here is one instance.

<http://docs.python.org/py3k/reference/executionmodel.html#naming-and-binding>

# If a name is bound in a block, it is a local variable of that block,
# unless declared as nonlocal. If a name is bound at the module level,
# it is a global variable. (The variables of the module code block are
# local and global.) If a variable is used in a code block but not
# defined there, it is a free variable.

Perhaps you could consider accepting this. Or trying to change it. Or
starting to tell newcomers that the official documentation of the
language is, say, vague and misleading. (It seems good to me.)

[toc] | [prev] | [next] | [standalone]

#16906

Thank you all for your replies, first of all my Sum function was an
example simplifying what I have to do in my real funciton. In general
the D dictionary is complex, with a lot of keys, so I was searching
for a quick method to access all the variables in it without doing the
explicit creation:

a, b, c = D['a'], D['b'], D['c']

and without using directly the D dictionary (boring...).
When I talked about nested function I meant both cases Chris, but this
is not a really tighten bound.
I tried to follow the hints of Chris together with some help by google
and used the following code:

for k in D : exec "%s = D[k]" %k

That seems to do the trick, but someone speaks about "dirty code", can
anyone point me out which problems this can generate?
Again, thank you for your help!

[toc] | [prev] | [next] | [standalone]

#16908

Massi wrote:

> for k in D : exec "%s = D[k]" %k
> 
> That seems to do the trick, but someone speaks about "dirty code", can
> anyone point me out which problems this can generate?

exec can run arbitrary code, so everybody reading the above has to go back 
to the definition of D to verify that it can only contain "safe" keys. 
Filling D with user-input is right out because a malicious user could do 
anything he likes. Here's a harmless demo that creates a file:

>>> d = {"x = 42\nwith open('tmp.txt', 'w') as f:\n f.write('whatever')\nx": 
123}
>>> for k in d: exec "%s = d[k]" % k
...
>>> x
123
>>> open("tmp.txt").read()
'whatever'

[toc] | [prev] | [next] | [standalone]

#16909

On Fri, 09 Dec 2011 01:55:28 -0800, Massi wrote:

> for k in D : exec "%s = D[k]" %k
> 
> That seems to do the trick, but someone speaks about "dirty code", can
> anyone point me out which problems this can generate? Again, thank you
> for your help!

Just the second-most common source of viruses, malware and security 
vulnerabilities (behind buffer overflows): code injection attacks.

Code injection attacks make up at least three of the top 25 security 
vulnerabilities on the CWE/SANS list:

http://cwe.mitre.org/top25/index.html

including the top 2 most dangerous threats (beating even our old friend, 
the buffer overflow): SQL injection and OS command injection. Your use of 
exec is vulnerable to attack if a hostile user can fool you into using a 
dict like this one:

D = {'a': '42', 
     'import os;'\
     ' os.system("""echo "ha ha i ownz ur system rm-rf/" """); b': '23',
    }
for k in D : exec "%s = D[k]" % k


You might think you're safe from such attacks, but (1) it is MUCH harder 
to protect against them than you might think; and (2) code has a habit of 
being re-used. Today your application might only be used by you; next 
week your code might find itself embedded in a web-application where 
hostile script kiddies can destroy your server with a single upload.

My advice is:

(1) If you need to ask why exec is dangerous, you shouldn't touch it.
(2) If you're sure you can protect against code injection, you can't.
(3) If you think you need exec, you probably don't.
(4) If you think you can make exec safe with a prohibited list of 
dangerous strings, you probably can't.


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#16910

On Fri, 09 Dec 2011 11:59:16 +0000, Steven D'Aprano wrote:

> Just the second-most common source of viruses, malware and security
> vulnerabilities (behind buffer overflows): code injection attacks.

Oops, I forgot to go back and revise this sentence. Code injection 
attacks are now the most common, not second-most common, source of 
security vulnerabilities.

http://cwe.mitre.org/top25/index.html



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#16912

On Fri, Dec 9, 2011 at 10:59 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> (4) If you think you can make exec safe with a prohibited list of
> dangerous strings, you probably can't.

If you think that it's even _possible_ to make exec safe with a
blacklist, I have a nice padded cell for you over here.

Security is NEVER achieved with blacklists, ONLY whitelists.

ChrisA

[toc] | [prev] | [next] | [standalone]

#16931

Massi wrote:
> Thank you all for your replies, first of all my Sum function was an
> example simplifying what I have to do in my real funciton. In general
> the D dictionary is complex, with a lot of keys, so I was searching
> for a quick method to access all the variables in it without doing the
> explicit creation:
> 
> a, b, c = D['a'], D['b'], D['c']
> 
> and without using directly the D dictionary (boring...).
> When I talked about nested function I meant both cases Chris, but this
> is not a really tighten bound.
> I tried to follow the hints of Chris together with some help by google
> and used the following code:
> 
> for k in D : exec "%s = D[k]" %k
> 
> That seems to do the trick, but someone speaks about "dirty code", can
> anyone point me out which problems this can generate?
> Again, thank you for your help!

Besides the serious security issues, this method won't make the problem 
any better in Python 3.

~Ethan~

[toc] | [prev] | [next] | [standalone]

#16977

On Fri, 09 Dec 2011 01:55:28 -0800, Massi wrote:

> Thank you all for your replies, first of all my Sum function was an
> example simplifying what I have to do in my real funciton. In general
> the D dictionary is complex, with a lot of keys, so I was searching
> for a quick method to access all the variables in it without doing the
> explicit creation:
> 
> a, b, c = D['a'], D['b'], D['c']
> 
> and without using directly the D dictionary (boring...).

If just you're trying to avoid getting a repetitive strain injury in your
right-hand little finger from typing all the [''], you could turn
the keys into object attributes, e.g.:

	class DictObject:
	    def __init__(self, d):
	        for key, value in d.iteritems():
	            setattr(self, key, value)
	...
	o = DictObject(D)
	# use o.a, o.b, etc

[toc] | [prev] | [next] | [standalone]

#17004

On Dec 11, 4:42 pm, Nobody <nob...@nowhere.com> wrote:
> If just you're trying to avoid getting a repetitive strain injury in your
> right-hand little finger from typing all the [''], you could turn
> the keys into object attributes, e.g.:
>
>         class DictObject:
>             def __init__(self, d):
>                 for key, value in d.iteritems():
>                     setattr(self, key, value)
>         ...
>         o = DictObject(D)
>         # use o.a, o.b, etc

I hate this kind of laziness. I'd spend at least 5 minutes trying to
work out _why_ someone felt this was necessary and then get annoyed
that it was just to avoid typing.

[toc] | [prev] | [next] | [standalone]

#17009

alex23 wrote:
> On Dec 11, 4:42 pm, Nobody <nob...@nowhere.com> wrote:
>> If just you're trying to avoid getting a repetitive strain injury in your
>> right-hand little finger from typing all the [''], you could turn
>> the keys into object attributes, e.g.:
>>
>>         class DictObject:
>>             def __init__(self, d):
>>                 for key, value in d.iteritems():
>>                     setattr(self, key, value)
>>         ...
>>         o = DictObject(D)
>>         # use o.a, o.b, etc
> 
> I hate this kind of laziness. I'd spend at least 5 minutes trying to
> work out _why_ someone felt this was necessary and then get annoyed
> that it was just to avoid typing.

For me, half of it is to avoid the typing, the other half to avoid the 
reading.  ;)

~Ethan~

[toc] | [prev] | [next] | [standalone]

#17017

On Dec 12, 3:00 pm, Ethan Furman <et...@stoneleaf.us> wrote:
> For me, half of it is to avoid the typing, the other half to avoid the
> reading.  ;)

Believe me, I get that, I've been more than guilty of it in the
past :)

But the reading here is important. If I see a dictionary lookup, I
expect it to be accessing a dictionary; if I see an attribute
reference, it might be wrapped in a descriptor. (And yes, I'm very
much aware that ultimately an attribute lookup _is_ a dictionary
lookup, but it's everything else that potentially happens after that
point that is important here.) Changing the visual short circuits any
understandings; then again, I tend to hate pesudo/mini-DSLs for the
same reason... I can't just rely on my understanding of the language,
I need to understand a specific implementation and mentally flag
whether it's relevant or not.

I also don't understand the original desire to create dynamic
variables either. Either you know the names in advance, in which case
just set up the necessary scaffolding to inject them where you need
them, or you _don't_ and there's no value in creating them anyway
because you don't know how to refer to them in your code.

So if you know the names, use dictionary unpacking:

    userData = dict(a = 1, b = 2, c = 3)

    def func(a,b,c):
        return a + b + c

    >>> func(**userData)
    6

If the dictionary has more values than you need, catch & discard the
ones you don't want in **kwargs.

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | comp.lang.python

csiph-web