Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #110723 > unrolled thread

Sharing package data across files

Back to article view | Back to comp.lang.python

Contents

  Sharing package data across files scottpakin1@gmail.com - 2016-06-28 12:17 -0700
    Re: Sharing package data across files zackbaker@gmail.com - 2016-06-28 12:55 -0700
      Re: Sharing package data across files John Pote <johnpote@jptechnical.co.uk> - 2016-06-28 23:07 +0100
        Re: Sharing package data across files scottpakin1@gmail.com - 2016-06-28 17:02 -0700
        Re: Sharing package data across files Steven D'Aprano <steve@pearwood.info> - 2016-06-29 10:27 +1000
    Re: Sharing package data across files Michael Selik <michael.selik@gmail.com> - 2016-06-28 22:37 +0000
      Re: Sharing package data across files scottpakin1@gmail.com - 2016-06-28 16:51 -0700
        Re: Sharing package data across files Steven D'Aprano <steve@pearwood.info> - 2016-06-29 10:50 +1000
          Re: Sharing package data across files scottpakin1@gmail.com - 2016-06-28 18:24 -0700

#110723 — Sharing package data across files

I'm trying to create a package in which the constituent files share some state.  Apparently, I don't understand scopes, namespaces, and package semantics as well as I thought I did.  Here's the directory structure for a simplified example:

   example/
     __init__.py
     vars.py
     funcs.py

vars.py defines a single variable:

    foo = 123

funcs.py defines a function that reads and writes that variable:

    def bar():
        global foo
        foo += 1
        return foo

__init__.py exposes both of those to the caller:

    from vars import foo
    from funcs import bar

Alas, it seems the bar function does not reside in the same global scope as the foo variable:

    >>> from example import foo, bar
    >>> foo
    123
    >>> bar()
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "example/funcs.py", line 3, in bar
        foo += 1
    NameError: global name 'foo' is not defined

How can I make the example package work like one integrated module even though in reality it's split across multiple files?

Thanks,
-- Scott

[toc] | [next] | [standalone]

#110725

On Tuesday, June 28, 2016 at 1:17:23 PM UTC-6, scott...@gmail.com wrote:
> I'm trying to create a package in which the constituent files share some state.  Apparently, I don't understand scopes, namespaces, and package semantics as well as I thought I did.  Here's the directory structure for a simplified example:
> 
>    example/
>      __init__.py
>      vars.py
>      funcs.py
> 
> vars.py defines a single variable:
> 
>     foo = 123
> 
> funcs.py defines a function that reads and writes that variable:
> 
>     def bar():
>         global foo
>         foo += 1
>         return foo
> 
> __init__.py exposes both of those to the caller:
> 
>     from vars import foo
>     from funcs import bar
> 
> Alas, it seems the bar function does not reside in the same global scope as the foo variable:
> 
>     >>> from example import foo, bar
>     >>> foo
>     123
>     >>> bar()
>     Traceback (most recent call last):
>       File "<stdin>", line 1, in <module>
>       File "example/funcs.py", line 3, in bar
>         foo += 1
>     NameError: global name 'foo' is not defined
> 
> How can I make the example package work like one integrated module even though in reality it's split across multiple files?
> 
> Thanks,
> -- Scott

This problem of references is addressed in:
http://stackoverflow.com/questions/710551/import-module-or-from-module-import

From Michael Ray Lovett: 
For example, if I do this in module a:

from foo import bar
bar = "oranges"

No code outside of a will see bar as "oranges" because my setting of bar merely affected the name "bar" inside module a, it did not "reach into" the foo module object and update its "bar".

[toc] | [prev] | [next] | [standalone]

#110728

On 28/06/2016 20:55, zackbaker@gmail.com wrote:
> On Tuesday, June 28, 2016 at 1:17:23 PM UTC-6, scott...@gmail.com wrote:
>> I'm trying to create a package in which the constituent files share some state.  Apparently, I don't understand scopes, namespaces, and package semantics as well as I thought I did.  Here's the directory structure for a simplified example:
>>
>>     example/
>>       __init__.py
>>       vars.py
>>       funcs.py
>>
>> vars.py defines a single variable:
>>
>>      foo = 123
>>
>> funcs.py defines a function that reads and writes that variable:
>>
>>      def bar():
>>          global foo
>>          foo += 1
>>          return foo
>>
>> __init__.py exposes both of those to the caller:
>>
>>      from vars import foo
>>      from funcs import bar
>>
>> Alas, it seems the bar function does not reside in the same global scope as the foo variable:
>>
>>      >>> from example import foo, bar
>>      >>> foo
>>      123
>>      >>> bar()
>>      Traceback (most recent call last):
>>        File "<stdin>", line 1, in <module>
>>        File "example/funcs.py", line 3, in bar
>>          foo += 1
>>      NameError: global name 'foo' is not defined
>>
>> How can I make the example package work like one integrated module even though in reality it's split across multiple files?
>>
>> Thanks,
>> -- Scott
> This problem of references is addressed in:
> http://stackoverflow.com/questions/710551/import-module-or-from-module-import
>
> >From Michael Ray Lovett:
> For example, if I do this in module a:
>
> from foo import bar
> bar = "oranges"
>
> No code outside of a will see bar as "oranges" because my setting of bar merely affected the name "bar" inside module a, it did not "reach into" the foo module object and update its "bar".
Correct me if I'm wrong but is not the above only true if bar has been 
assigned to and thus references an imutable object? In your example the 
string "oranges".
If bar has been assigned to a mutable object in module foo then every 
module importing via "from foo import bar" will all import the name bar 
pointing to the same mutable object. If this mutable obj is changed via 
bar in one module then every other module importing bar will also see 
the change.
eg
In module foo:
     bar = ["apples","bananas","grapes"]

In module bar1
     from foo import bar
     bar[0] = "oranges"

In module barx at some later time
     from foo import bar
     ...
     print bar    #prints ["oranges","bananas","grapes"]

If my understanding here is correct then this would be a good case for 
never directly writing to a globle. Use getter()s and setter()s to make 
it obvious that any use of the setter() will be seen by all future calls 
to the getter().

Regards all,
John

[toc] | [prev] | [next] | [standalone]

#110733

On Tuesday, June 28, 2016 at 5:20:16 PM UTC-6, John Pote wrote:
> Correct me if I'm wrong but is not the above only true if bar has been 
> assigned to and thus references an imutable object? In your example the 
> string "oranges".
> If bar has been assigned to a mutable object in module foo then every 
> module importing via "from foo import bar" will all import the name bar 
> pointing to the same mutable object. If this mutable obj is changed via 
> bar in one module then every other module importing bar will also see 
> the change.
> eg
> In module foo:
>      bar = ["apples","bananas","grapes"]
> 
> In module bar1
>      from foo import bar
>      bar[0] = "oranges"
> 
> In module barx at some later time
>      from foo import bar
>      ...
>      print bar    #prints ["oranges","bananas","grapes"]

That does seem to be the case.  Making my example look like the above, with foo being assigned a list and bar mutating the first element of the list, indeed results in foo being updated:

    >>> from example import foo, bar
    >>> foo
    ['apples', 'bananas', 'grapes']
    >>> bar()
    ['oranges', 'bananas', 'grapes']
    >>> foo
    ['oranges', 'bananas', 'grapes']

> If my understanding here is correct then this would be a good case for 
> never directly writing to a globle. Use getter()s and setter()s to make 
> it obvious that any use of the setter() will be seen by all future calls 
> to the getter().

Yeah, it looks like I'll have to do more to my original (big) code than simply partition it into appropriate-sized chunks and drop it into a package directory.  Oh, well.

Thanks,
-- Scott

[toc] | [prev] | [next] | [standalone]

#110735

On Wed, 29 Jun 2016 08:07 am, John Pote wrote:

>> from foo import bar
>> bar = "oranges"
>>
>> No code outside of a will see bar as "oranges" because my setting of bar
>> merely affected the name "bar" inside module a, it did not "reach into"
>> the foo module object and update its "bar".

zackbaker@gmail.com's description above is correct.

> Correct me if I'm wrong but is not the above only true if bar has been
> assigned to and thus references an imutable object? In your example the
> string "oranges".

No, assignment and importing doesn't know the difference between immutable
and mutable objects. Importing works the same way for strings and it does
for lists.

However, this part is correct:

> If bar has been assigned to a mutable object in module foo then every
> module importing via "from foo import bar" will all import the name bar
> pointing to the same mutable object. If this mutable obj is changed via
> bar in one module then every other module importing bar will also see
> the change.

That's right. Think of it this way:

- every module has their own unique NAME "bar", so when a module assigns to
bar using `bar = something`, it is only changing the name in its own
namespace, it's not reaching into any other namespace;

- but there's only one OBJECT being imported, so if that object is mutable,
and you mutate it, all modules see the same change in state.

In other words, importing does not copy objects, it just creates a new name,
in the current namespace, that is bound to that object.

If it helps you to think about the underlying C implementation, you can
think about pointers to objects:

In foo, you have bar pointing to some object:

    bar ---> OBJ


"from foo import bar" creates a new local variable which likewise points to
the same OBJ (no copy is made):

    foo.bar ---> OBJ
                  ^
                  |
    a.bar --------+


Assignment changes the pointer, not the thing pointed to:

    foo.bar ---> OBJ
    a.bar --------------> "oranges"




-- 
Steven
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

[toc] | [prev] | [next] | [standalone]

#110727

On Tue, Jun 28, 2016 at 3:21 PM <scottpakin1@gmail.com> wrote:

> I'm trying to create a package in which the constituent files share some
> state.  Apparently, I don't understand scopes, namespaces, and package
> semantics as well as I thought I did.  Here's the directory structure for a
> simplified example:
>
>    example/
>      __init__.py
>      vars.py
>      funcs.py
>
> How can I make the example package work like one integrated module even
> though in reality it's split across multiple files?
>

Why do you want to?
Isn't easier to have the ``funcs`` module import the ``vars`` module?
Even easier, paste all the code into a single file.

[toc] | [prev] | [next] | [standalone]

#110731

On Tuesday, June 28, 2016 at 4:37:45 PM UTC-6, Michael Selik wrote:
> Why do you want to?

I have a standalone script that grew and grew until reaching an unmaintainable size.  I was hoping to refactor it into a relatively small top-level script plus a package containing a bunch of relatively small files.

> Isn't easier to have the ``funcs`` module import the ``vars`` module?

Easier, yes.  Correct, no:

    from vars import foo

    def bar():
        global foo
        foo += 1
        return foo

which surprisingly (to me, anyway) changes a _copy_ of foo, not the foo I'd think of as belonging to the example package:

    >>> from example import foo, bar
    >>> foo
    123
    >>> bar()
    124
    >>> foo
    123

> Even easier, paste all the code into a single file.

That kind of defeats my goal of splitting a single file into more maintainable chunks.

Thanks,
-- Scott

[toc] | [prev] | [next] | [standalone]

#110737

On Wed, 29 Jun 2016 09:51 am, scottpakin1@gmail.com wrote:

> On Tuesday, June 28, 2016 at 4:37:45 PM UTC-6, Michael Selik wrote:
>> Why do you want to?
> 
> I have a standalone script that grew and grew until reaching an
> unmaintainable size.  I was hoping to refactor it into a relatively small
> top-level script plus a package containing a bunch of relatively small
> files.
> 
>> Isn't easier to have the ``funcs`` module import the ``vars`` module?
> 
> Easier, yes.  Correct, no:

No, you are misinterpreting what you are seeing.

The correct answer (for some definition of correct) is to import vars and
operate on the attributes of it. Another solution would be to collect
related bits of functionality and their variables into classes.

Aside: you should reconsider the name "vars", as that shadows the builtin
function vars().

This code doesn't work:

>     from vars import foo
> 
>     def bar():
>         global foo
>         foo += 1
>         return foo


but not for the reason you think. However this will work:

    import vars
    def bar():
        vars.foo += 1
        return vars.foo

and that's probably the easiest way to refactor what you have now.

But honestly, if your standalone script contains *so many global variables*
that you need to do this, that's a code smell. You should rethink your use
of global variables, and consider refactoring your code into functions and
classes. Just a suggestion.

 
> which surprisingly (to me, anyway) changes a _copy_ of foo, not the foo
> I'd think of as belonging to the example package:

No, this explanation is incorrect. The object foo is not copied. Python
doesn't copy objects unless you explicitly tell it to, usually with the
copy module.

When you import something from another module, it creates a name in the
importer's local namespace:

# module vars.py
foo = 999


# module spam.py
from vars import foo


Now there are TWO names "foo". Remember that names are string keys in a
dict. We call that a namespace, and there are two of them:

the "vars" namespace is the dict vars.__dict__

the "spam" namespace is the dict spam.__dict__


and they BOTH have a key "foo". Obviously they must be different, as
otherwise any module that happens to use the same variable names as your
module would stomp all over your variables and change their values.

So at this point, you have

    vars.foo = 999
    spam.foo = 999  # the SAME object, not a copy


Inside the "vars" module, you don't give the fully-qualified
name "vars.foo", you just say "foo". And likewise in the "spam" module, you
don't give the fully-qualified name "spam.foo" but just say "foo". But
don't be fooled, they are *different* foos that just happen to both refer
to the same value.

When you say

    spam.foo = 100  # equivalent to spam.foo += 1

obviously spam's foo will change, but vars' foo doesn't. It's in a different
namespace, and assignment doesn't reach into different namespaces unless
you give the fully-qualified name. (That would be bad.)

Inside spam, you don't need to say "spam.foo" (in fact, you can't, unless
you first import spam from inside itself, which is weird thing to do). You
just say "foo = 100". But conceptually, it is the same.





-- 
Steven
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

[toc] | [prev] | [next] | [standalone]

#110738

Steven,

Very helpful!  Thanks for the thorough explanation.

-- Scott

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web