Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #49801 > unrolled thread

Default scope of variables

Back to article view | Back to comp.lang.python

Contents

  Default scope of variables Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-07-04 03:27 +0000
    Re: Default scope of variables Chris Angelico <rosuav@gmail.com> - 2013-07-04 14:07 +1000
      Re: Default scope of variables Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-07-04 05:32 +0000
        Re: Default scope of variables Chris Angelico <rosuav@gmail.com> - 2013-07-04 15:47 +1000
          Re: Default scope of variables Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-07-04 16:38 +0000
            Re: Default scope of variables Chris Angelico <rosuav@gmail.com> - 2013-07-05 02:58 +1000
            Re: Default scope of variables Wayne Werner <wayne@waynewerner.com> - 2013-07-07 08:13 -0500
            Re: Default scope of variables Chris Angelico <rosuav@gmail.com> - 2013-07-07 23:43 +1000
              Re: Default scope of variables Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-07-07 16:03 +0000
                Re: Default scope of variables Chris Angelico <rosuav@gmail.com> - 2013-07-08 10:46 +1000
              Re: Default scope of variables alex23 <wuwei23@gmail.com> - 2013-07-09 14:52 +1000
                Re: Default scope of variables Chris Angelico <rosuav@gmail.com> - 2013-07-09 15:07 +1000
                  Re: Default scope of variables alex23 <wuwei23@gmail.com> - 2013-07-09 16:08 +1000
                    Re: Default scope of variables Chris Angelico <rosuav@gmail.com> - 2013-07-09 16:13 +1000
                    Re: Default scope of variables "Frank Millman" <frank@chagford.com> - 2013-07-09 09:35 +0200
                    Re: Default scope of variables Chris Angelico <rosuav@gmail.com> - 2013-07-09 17:45 +1000
                    Re: Default scope of variables Ian Kelly <ian.g.kelly@gmail.com> - 2013-07-09 01:56 -0600
                    Re: Default scope of variables "Frank Millman" <frank@chagford.com> - 2013-07-09 10:22 +0200
                    Re: Default scope of variables "Frank Millman" <frank@chagford.com> - 2013-07-09 10:38 +0200
                    Re: Default scope of variables Ethan Furman <ethan@stoneleaf.us> - 2013-07-09 09:07 -0700
                    Re: Default scope of variables Ian Kelly <ian.g.kelly@gmail.com> - 2013-07-09 10:44 -0600
                    Re: Default scope of variables Ethan Furman <ethan@stoneleaf.us> - 2013-07-09 10:23 -0700
                    Re: Default scope of variables Ian Kelly <ian.g.kelly@gmail.com> - 2013-07-09 12:41 -0600
                    Re: Default scope of variables Ethan Furman <ethan@stoneleaf.us> - 2013-07-09 12:19 -0700
                    Re: Default scope of variables "Frank Millman" <frank@chagford.com> - 2013-07-10 07:54 +0200
                    Re: Default scope of variables Ian Kelly <ian.g.kelly@gmail.com> - 2013-07-10 09:42 -0600
                    Re: Default scope of variables Ethan Furman <ethan@stoneleaf.us> - 2013-07-10 08:29 -0700
                    Re: Default scope of variables "Frank Millman" <frank@chagford.com> - 2013-07-11 07:52 +0200
            Re: Default scope of variables Ethan Furman <ethan@stoneleaf.us> - 2013-07-07 09:52 -0700
            Re: Default scope of variables Ethan Furman <ethan@stoneleaf.us> - 2013-07-07 11:59 -0700
            Re: Default scope of variables Chris Angelico <rosuav@gmail.com> - 2013-07-08 10:48 +1000
              Re: Default scope of variables Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-07-08 02:23 +0000
                Re: Default scope of variables Chris Angelico <rosuav@gmail.com> - 2013-07-08 13:11 +1000
                  Re: Default scope of variables Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-07-08 05:00 +0000
                    Re: Default scope of variables Chris Angelico <rosuav@gmail.com> - 2013-07-08 15:14 +1000
        Re: Default scope of variables Peter Otten <__peter__@web.de> - 2013-07-04 08:48 +0200
        Re: Default scope of variables Ian Kelly <ian.g.kelly@gmail.com> - 2013-07-04 01:12 -0600
        Re: Default scope of variables Dave Angel <davea@davea.name> - 2013-07-04 03:06 -0400
          Re: Default scope of variables Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-07-04 15:52 +0000
          Re: Default scope of variables Rotwang <sg552@hotmail.co.uk> - 2013-07-04 17:54 +0100
            Re: Default scope of variables Peter Otten <__peter__@web.de> - 2013-07-04 20:36 +0200
            Re: Default scope of variables Joshua Landau <joshua.landau.ws@gmail.com> - 2013-07-05 01:04 +0100
            Re: Default scope of variables Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-07-05 01:24 +0000
              Re: Default scope of variables Dave Angel <davea@davea.name> - 2013-07-04 22:03 -0400
              Re: Default scope of variables Joshua Landau <joshua.landau.ws@gmail.com> - 2013-07-05 03:29 +0100
              Re: Default scope of variables Joshua Landau <joshua.landau.ws@gmail.com> - 2013-07-05 03:27 +0100
              Re: Default scope of variables Rotwang <sg552@hotmail.co.uk> - 2013-07-05 07:28 +0100
          Re: Default scope of variables Neil Cerutti <neilc@norwich.edu> - 2013-07-05 13:24 +0000
            Re: Default scope of variables Chris Angelico <rosuav@gmail.com> - 2013-07-05 23:43 +1000
              Re: Default scope of variables Neil Cerutti <neilc@norwich.edu> - 2013-07-05 15:36 +0000
            Re: Default scope of variables Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-07-07 16:08 +0000
              Re: Default scope of variables Neil Cerutti <neilc@norwich.edu> - 2013-07-08 11:54 +0000
                Re: Default scope of variables Joshua Landau <joshua.landau.ws@gmail.com> - 2013-07-08 14:14 +0100
        Re: Default scope of variables Lele Gaifax <lele@metapensiero.it> - 2013-07-04 14:43 +0200
        Re: Default scope of variables Wayne Werner <wayne@waynewerner.com> - 2013-07-04 10:45 -0500
    Re: Default scope of variables Joshua Landau <joshua.landau.ws@gmail.com> - 2013-07-04 05:30 +0100
      Re: Default scope of variables Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-07-04 05:45 +0000
    Re: Default scope of variables Chris Angelico <rosuav@gmail.com> - 2013-07-04 14:36 +1000
    Re: Default scope of variables Joshua Landau <joshua.landau.ws@gmail.com> - 2013-07-04 06:09 +0100
    Re: Default scope of variables Joshua Landau <joshua.landau.ws@gmail.com> - 2013-07-08 17:58 +0100

Page 1 of 3  [1] 2 3  Next page →

#49801 — Default scope of variables

Recently, there was a thread where people discussed variable 
declarations, with a couple people stating that they wished that Python 
required you to declare local variables, instead of globals.

I'm sure they have their (foolish, pathetic) *wink* reasons for this, but 
I thought I'd explain why I think Python makes the right decision to 
require declarations for globals but not locals.

I was reading some Javascript today, and wading through masses of code 
that looked something like this:

function update_results() {
    var n1 = foo;
    var n2 = bar;
    var m1 = foobar;
    var m2 = foobarbaz;
    var fe = n1 * m1;
    var fi = n2 * m2;
    var fo = n1 * m2;
    var fum = n2 * m1;
    ... 
    }

and so on for about a page and a half. Looking at that page in my editor, 
with that solid column of bold "var" keywords, it struck me just how 
redundant that enormous column of "var"s was. Of course they were 
variables, what else would they be?

Larry Wall, the creator of Perl, is fond of discussing Huffman coding as 
it relates to programming syntax. Common things should be short, rare 
things can be longer. Wall is not concerned about saving a few bytes in 
your source files, but about programmer effort: typing effort, reading 
effort, mental processing effort. Which do you prefer?

total = subtotal + extra

set total to subtotal plus extra

Even though the second is only 8 more characters, I bet you prefer the 
first version.

With respect to the Huffman coding of declarations, Javascript gets it 
backwards. Locals ought to be more common, but they require more typing. 
Locals are safer, better, more desirable than globals, and so it should 
be easier to use locals than globals, not the other way around. Having to 
declare "give me the safe kind of variable", while getting the harmful[1] 
kind of variable for free, strikes me as arse-backwards. Lazy, naive or 
careless coders get globals[2] by default or accident. That's bad.

Not just theoretically bad. Here's a real-world case where a single 
missed "var" lead to something much, much worse than a crash: code that 
kept going, but doing the wrong thing.

http://blog.safeshepherd.com/23/how-one-missing-var-ruined-our-launch/


The two situations:

1) Accidentally scope an intended local as global;
2) Accidentally scope an intended global as local;

are not symmetrical. In the first case, you get multiple invocations of 
your function overwriting each other's data. Confusion reigns, but the 
function calls will likely continue, pumping out garbage results instead 
of crashing. The likely result is typically fail-unsafe rather than fail-
safe. [Aside: fail-safe does not mean "safe from failing", but "fails in 
a safe manner".]

In the second case, any failure is far more likely to result in the 
function call failing hard with an exception (fail-safe) rather than 
churning out bad results, since each call of the function gets its own 
set of locals rather than using those from some other call.

So in Javascript, it's easy to get unsafe globals by accident; in Python, 
it's hard to get unsafe globals by accident. In my opinion, Python gets 
it right.




[1] As in, "Global variables considered harmful", one of the classic 
papers of computer science: 
http://c2.com/cgi/wiki?GlobalVariablesConsideredHarmful

[2] Actually, Javascript gives you something a little closer to Python's 
"nonlocal" by default: each enclosing function is searched for a matching 
variable, terminating at the global scope.


-- 
Steven

[toc] | [next] | [standalone]

#49804

On Thu, Jul 4, 2013 at 1:27 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> With respect to the Huffman coding of declarations, Javascript gets it
> backwards. Locals ought to be more common, but they require more typing.
> Locals are safer, better, more desirable than globals, and so it should
> be easier to use locals than globals, not the other way around. Having to
> declare "give me the safe kind of variable", while getting the harmful[1]
> kind of variable for free, strikes me as arse-backwards. Lazy, naive or
> careless coders get globals[2] by default or accident. That's bad.

I agree that Javascript has it wrong, but not quite for the reason you
say. The advantage of declaring locals is a flexibility: you can have
multiple unique variables with the same name in the same function.
Python lets you do that across but not within functions.

But Javascript/ECMAScript/whatever doesn't give you that. A var
declaration makes it function-local, no matter where the declaration
is. That's pointless. C++, on the other hand, lets you do this:

void somefunc() {
    for (int i=0;i<10;++i) {
        // do something with outer i
        for (int i=0;i<4;++i) {
            // do something with inner i
        }
        // outer i is visible again
    }
    // neither i is visible
}

Also, C++ overlays the "this is local" declaration with the "this
contains this type" declaration, which neither Python nor Javascript
bothers with; that makes the declaration feel less redundant.

Granted, this flexibility is mostly of value when writing huge
functions with complex nesting, but it is something that I find
convenient.

In terms of Huffman coding, every C++ variable must be declared,
somewhere. It's not a matter of declaring globals or declaring locals
- you just declare variables. If you declare them at file scope,
they're globals; if at function scope, they're locals. There's really
no difference. Everything's visible at its own level and those further
in, and not those further out.

I do see the convenience of the Python system, and I do like it; but
someone needs to speak up for the foolish and pathetic :)

ChrisA

[toc] | [prev] | [next] | [standalone]

#49813

On Thu, 04 Jul 2013 14:07:55 +1000, Chris Angelico wrote:

> On Thu, Jul 4, 2013 at 1:27 PM, Steven D'Aprano
> <steve+comp.lang.python@pearwood.info> wrote:
>> With respect to the Huffman coding of declarations, Javascript gets it
>> backwards. Locals ought to be more common, but they require more
>> typing. Locals are safer, better, more desirable than globals, and so
>> it should be easier to use locals than globals, not the other way
>> around. Having to declare "give me the safe kind of variable", while
>> getting the harmful[1] kind of variable for free, strikes me as
>> arse-backwards. Lazy, naive or careless coders get globals[2] by
>> default or accident. That's bad.
> 
> I agree that Javascript has it wrong, but not quite for the reason you
> say. The advantage of declaring locals is a flexibility: you can have
> multiple unique variables with the same name in the same function.

Well, if I ever have more than 63,000,000 variables[1] in a function, 
I'll keep that in mind. Until then, I'm pretty sure you can trivially 
avoid name clashes with globals that you wish to avoid clashing with.

Accidental shadowing can be a problem, but I've never heard of anyone 
saying that they were *forced* to shadow a global they needed access to. 
Just pick a different name.


> Python lets you do that across but not within functions.
> 
> But Javascript/ECMAScript/whatever doesn't give you that. A var
> declaration makes it function-local, no matter where the declaration is.
> That's pointless. C++, on the other hand, lets you do this:
> 
> void somefunc() {
>     for (int i=0;i<10;++i) {
>         // do something with outer i
>         for (int i=0;i<4;++i) {
>             // do something with inner i
>         }
>         // outer i is visible again
>     }
>     // neither i is visible
> }

That's truly horrible. If the cost of this "flexibility" is that I'll 
have to read, and debug, other people's code with this sort of thing, I'm 
happy to be less flexible. For what possible reason other than "because I 
can" would you want to use the same loop variable name in two nested 
loops?

I'm not suggesting that C++ should prohibit it. But just because a 
language allows something doesn't make it a *feature*. I can write this 
in Python:

a = a = a = a = a = 1

and it works, but the ability to do so is hardly a feature. It's just a 
side effect of how Python works.

I believe that the function is the right level for scope changes, not to 
large, not to small. I'm not even sure I like the fact that generator 
expressions (in Python 2 & 3) and list comprehensions (in Python 3) 
introduce their own scope.


 
[...]
> Granted, this flexibility is mostly of value when writing huge functions
> with complex nesting, but it is something that I find convenient.

"This feature is mostly of value when poking myself, and any developers 
who have to maintain my code after I'm gone, in the eye with a red-hot 
poker."




[1] Based on empirical evidence that Python supports names with length at 
least up to one million characters long, and assuming that each character 
can be an ASCII letter, digit or underscore.

-- 
Steven

[toc] | [prev] | [next] | [standalone]

#49817

On Thu, Jul 4, 2013 at 3:32 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> Accidental shadowing can be a problem, but I've never heard of anyone
> saying that they were *forced* to shadow a global they needed access to.
> Just pick a different name.

Here's one example of shadowing that comes from a C++ project at work.
I have a class that represents a database transaction (constructing it
begins a transaction, it has methods for doing queries, and its
destructor rolls back). There's also a class for a subtransation (same
thing, but it uses savepoints within the transaction). So to bracket a
piece of code in a subtransaction, I want to declare a new
subtransaction object with the same name as the outer transaction
object, and then dispose of it and "reveal" the original. There will
always be an object called "trans", and it will always be the
appropriate transaction to do queries on, but it'll change what it is.

ChrisA

[toc] | [prev] | [next] | [standalone]

#49885

On Thu, 04 Jul 2013 15:47:57 +1000, Chris Angelico wrote:

> On Thu, Jul 4, 2013 at 3:32 PM, Steven D'Aprano
> <steve+comp.lang.python@pearwood.info> wrote:
>> Accidental shadowing can be a problem, but I've never heard of anyone
>> saying that they were *forced* to shadow a global they needed access
>> to. Just pick a different name.
> 
> Here's one example of shadowing that comes from a C++ project at work. I
> have a class that represents a database transaction (constructing it
> begins a transaction, it has methods for doing queries, and its
> destructor rolls back). 

When the object finally gets garbage collected, doesn't that mean the 
last transaction will be rolled back?

> There's also a class for a subtransation (same
> thing, but it uses savepoints within the transaction). So to bracket a
> piece of code in a subtransaction, I want to declare a new
> subtransaction object with the same name as the outer transaction
> object, and then dispose of it and "reveal" the original. There will
> always be an object called "trans", and it will always be the
> appropriate transaction to do queries on, but it'll change what it is.

You don't need to introduce such scoping rules for variables for that use-
case. We have namespaces (classes) for that sort of thing :-) 

Python 3.3's ChainMap is probably a better solution, but here's another 
way to get the same sort of behaviour:


def function():
    class Namespace:
        # Set a class attribute.
        trans = Transaction()
    ns = Namespace()
    do_stuff_with(ns.trans)
    # Enter a subtransaction.
    ns.trans = Subtransaction()
    do_stuff_with(ns.trans)
    del ns.trans
    do_stuff_with(ns.trans)


Yes, it looks weird to see ns.trans used immediately after deleting it, 
but that's because it is a weird (or at least unusual) use-case.


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#49887

On Fri, Jul 5, 2013 at 2:38 AM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> On Thu, 04 Jul 2013 15:47:57 +1000, Chris Angelico wrote:
>> Here's one example of shadowing that comes from a C++ project at work. I
>> have a class that represents a database transaction (constructing it
>> begins a transaction, it has methods for doing queries, and its
>> destructor rolls back).
>
> When the object finally gets garbage collected, doesn't that mean the
> last transaction will be rolled back?

Oh. Uhm... ahh... it would have helped to mention that it also has a
commit() method! But yes, that's correct; if the object expires (this
is C++, so it's guaranteed to call the destructor at that close brace
- none of the Python vagueness about when __del__ is called) without
commit() being called, then the transaction will be rolled back. And
since this is PostgreSQL we use, the same applies if the process is
SIGKILLed or the power fails. If commit() doesn't happen, neither does
the transaction. (There are a few actions the program can take that
are deliberately non-transactional - log entries of various sorts,
mainly - but everything else is guarded in this way.)

ChrisA

[toc] | [prev] | [next] | [standalone]

#50101

On Fri, 5 Jul 2013, Chris Angelico wrote:

> Oh. Uhm... ahh... it would have helped to mention that it also has a
> commit() method! But yes, that's correct; if the object expires (this
> is C++, so it's guaranteed to call the destructor at that close brace
> - none of the Python vagueness about when __del__ is called) without
> commit() being called, then the transaction will be rolled back.

If one wants to duplicate this kind of behavior in Python, that's what 
context managers are for combined with a `with` block, which does 
guarantee that the __exit__ method will be called - in this case it could 
be something as simple as:

from contextlib import contextmanager

@contextmanager
def new_transaction(conn):
     tran = conn.begin_transaction()
     yield tran
     if not tran.committed:
         tran.rollback()



Which you would then use like:


conn = create_conn()
with new_transaction(conn) as tran:
      rows_affected = do_query_stuff(tran)
      if rows_affected == 42:
           tran.commit()



And then you get the desired constructor/destructor behavior of having 
guaranteed that code will be executed at the start and at the end. You can 
wrap things in try/catch for some error handling, or write your own 
context manager class.

HTH,
Wayne

[toc] | [prev] | [next] | [standalone]

#50102

On Sun, Jul 7, 2013 at 11:13 PM, Wayne Werner <wayne@waynewerner.com> wrote:
> Which you would then use like:
>
>
> conn = create_conn()
> with new_transaction(conn) as tran:
>      rows_affected = do_query_stuff(tran)
>      if rows_affected == 42:
>           tran.commit()

Yep. There's a problem, though, when you bring in subtransactions. The
logic wants to be like this:

with new_transaction(conn) as tran:
    tran.query("blah")
    with tran.subtransaction() as tran:
        tran.query("blah")
        with tran.subtransaction() as tran:
            tran.query("blah")
            # roll this subtransaction back
        tran.query("blah")
        tran.commit()
    tran.query("blah")
    tran.commit()

The 'with' statement doesn't allow this. I would need to use some kind
of magic to rebind the old transaction to the name, or else use a list
that gets magically populated:

with new_transaction(conn) as tran:
    tran[-1].query("blah")
    with subtransaction(tran):
        tran[-1].query("blah")
        with subtransaction(tran):
            tran[-1].query("blah")
            # roll this subtransaction back
        tran[-1].query("blah")
        tran[-1].commit()
    tran[-1].query("blah")
    tran[-1].commit()

I don't like the look of this. It might work, but it's hardly ideal.
This is why I like to be able to nest usages of the same name.

ChrisA

[toc] | [prev] | [next] | [standalone]

#50103

On Sun, 07 Jul 2013 23:43:24 +1000, Chris Angelico wrote:

> On Sun, Jul 7, 2013 at 11:13 PM, Wayne Werner <wayne@waynewerner.com>
> wrote:
>> Which you would then use like:
>>
>>
>> conn = create_conn()
>> with new_transaction(conn) as tran:
>>      rows_affected = do_query_stuff(tran)
>>      if rows_affected == 42:
>>           tran.commit()
> 
> Yep. There's a problem, though, when you bring in subtransactions. The
> logic wants to be like this:
[snip hideous code]
> I don't like the look of this. It might work, but it's hardly ideal.
> This is why I like to be able to nest usages of the same name.

Yes, and the obvious way to nest usages of the same name is to use a 
instance with a class attribute and instance attribute of the same name:

class Example:
    attr = 23

x = Example()
x.attr = 42
print(x.attr)
del x.attr
print(x.attr)


If you need more than two levels, you probably ought to re-design your 
code to be less confusing, otherwise you may be able to use ChainMap to 
emulate any number of nested scopes.

One interesting trick is you can use a ChainMap as function globals. 
Here's a sketch for what you can do in Python 3.3:


from types import FunctionType
from collections import ChainMap

class _ChainedDict(ChainMap, dict):
    # Function dicts must be instances of dict :-(
    pass


def chained_function(func, *dicts):
    """Return a new function, copied from func, using a 
    ChainMap as dict.
    """
    dicts = dicts + (func.__globals__, builtins.__dict__)
    d = _ChainedDict(*dicts)
    name = func.__name__
    newfunc = FunctionType(
            func.__code__, d, name, closure=func.__closure__)
    newfunc.__dict__.update(func.__dict__)
    newfunc.__defaults__ = func.__defaults__
    return newfunc



And in use:

py> f = chained_function(lambda x: x+y, {'y': 100})
py> f(1)
101
py> f.__globals__.maps.insert(0, {'y': 200})
py> f(1)
201
py> del f.__globals__.maps[0]['y']
py> f(1)
101


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#50111

On Mon, Jul 8, 2013 at 2:03 AM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> If you need more than two levels, you probably ought to re-design your
> code to be less confusing, otherwise you may be able to use ChainMap to
> emulate any number of nested scopes.

The subtransactions are primarily to represent the database equivalent
of a try/except block, so they need to be able to be nested
arbitrarily.

ChrisA

[toc] | [prev] | [next] | [standalone]

#50194

On 7/07/2013 11:43 PM, Chris Angelico wrote:
> Yep. There's a problem, though, when you bring in subtransactions. The
> logic wants to be like this:
>
> with new_transaction(conn) as tran:
>      tran.query("blah")
>      with tran.subtransaction() as tran:
>          tran.query("blah")
>          with tran.subtransaction() as tran:
>              tran.query("blah")
>              # roll this subtransaction back
>          tran.query("blah")
>          tran.commit()
>      tran.query("blah")
>      tran.commit()
>
> The 'with' statement doesn't allow this.

I'd take the 'explicit is better than implicit' approach to naming the
context managers here and rather than use 'tran' choose names that
reflect for what the transaction is to be used ie summarising what's in
the "blah" of each query.

with new_transaction(conn) as folder_tran:
     folder_tran.query("blah")
     with folder_tran.subtransaction() as file_tran:
         file_tran.query("blah")
         with file_tran.subtransaction() as type_tran:
             type_tran.query("blah")

(for want of a better heirachical example...)

[toc] | [prev] | [next] | [standalone]

#50202

On Tue, Jul 9, 2013 at 2:52 PM, alex23 <wuwei23@gmail.com> wrote:
> with new_transaction(conn) as folder_tran:
>     folder_tran.query("blah")
>     with folder_tran.subtransaction() as file_tran:
>         file_tran.query("blah")
>         with file_tran.subtransaction() as type_tran:
>             type_tran.query("blah")

Warp my code around a language limitation? Only if I absolutely have to.

The subtransactions are NOT concepted as separate transactions. They
are effectively the database equivalent of a try/except block. Would
you want to have to use a different name for a builtin just because
you're inside a try block?

a = int("123")
try:
    b = int1(user_input)
except ValueError:
    b = 0
c = int("234")

No. That assignment to b should be int(), same as a and c.

ChrisA

[toc] | [prev] | [next] | [standalone]

#50209

On 9/07/2013 3:07 PM, Chris Angelico wrote:
> The subtransactions are NOT concepted as separate transactions. They
> are effectively the database equivalent of a try/except block.

Sorry, I assumed each nested query was somehow related to the prior
one. In which case, I'd probably go with Ethan's suggestion of a
top-level transaction context manager with its own substransaction
method.

[toc] | [prev] | [next] | [standalone]

#50210

On Tue, Jul 9, 2013 at 4:08 PM, alex23 <wuwei23@gmail.com> wrote:
> On 9/07/2013 3:07 PM, Chris Angelico wrote:
>>
>> The subtransactions are NOT concepted as separate transactions. They
>> are effectively the database equivalent of a try/except block.
>
>
> Sorry, I assumed each nested query was somehow related to the prior
> one. In which case, I'd probably go with Ethan's suggestion of a
> top-level transaction context manager with its own substransaction
> method.

Yeah, that would probably be the best option in this particular
instance. Though I do still like the ability to have variables shadow
each other, even if there's a way around one particular piece of code
that uses the technique.

ChrisA

[toc] | [prev] | [next] | [standalone]

#50221

"Chris Angelico" <rosuav@gmail.com> wrote in message 
news:CAPTjJmqkmFd4-Jpugr-vubuB6riBV6K_Mwnxc_U3CVaBr_Wgbg@mail.gmail.com...
> On Tue, Jul 9, 2013 at 4:08 PM, alex23 <wuwei23@gmail.com> wrote:
>> On 9/07/2013 3:07 PM, Chris Angelico wrote:
>>>
>>> The subtransactions are NOT concepted as separate transactions. They
>>> are effectively the database equivalent of a try/except block.
>>
>>
>> Sorry, I assumed each nested query was somehow related to the prior
>> one. In which case, I'd probably go with Ethan's suggestion of a
>> top-level transaction context manager with its own substransaction
>> method.
>
> Yeah, that would probably be the best option in this particular
> instance. Though I do still like the ability to have variables shadow
> each other, even if there's a way around one particular piece of code
> that uses the technique.
>

I have been following this sub-thread with interest, as it resonates with 
what I am doing in my project.

In my case, one update can trigger another, which can trigger another, etc. 
It is important that they are treated as a single transaction. Each object 
has its own 'save' method, so there is not one place where all updates are 
executed, and I found it tricky to control.

I came up with the following context manager -

class DbSession:
    """
    A context manager to handle database activity.
    """

    def __init__(self):
        self.conn = None
        self.no_connections = 0
        self.transaction_active = False

    def __enter__(self):
        if self.conn is None:
            self.conn = _get_connection()  # get connection from pool
            self.conn.cur = self.conn.cursor()
            # all updates in same transaction use same timestamp
            self.conn.timestamp = datetime.now()
        self.no_connections += 1
        return self.conn

    def __exit__(self, type, exc, tb):
        if type is not None:  # an exception occurred
            if self.transaction_active:
                self.conn.rollback()
                self.transaction_active = False
            self.conn.release()  # return connection to pool
            self.conn = None
            return  # will reraise exception
        self.no_connections -= 1
        if not self.no_connections:
            if self.transaction_active:
                self.conn.commit()
                self.transaction_active = False
            self.conn.cur.close()
            self.conn.release()  # return connection to pool
            self.conn = None

All objects created within a session share a common DbSession instance.

When any of them need any database access, whether for reading or for 
updating, they execute the following -

    with db_session as conn:
        conn.transaction_active = True  # this line must be added if 
updating
        conn.cur.execute(__whatever__)

Now it 'just works'. I don't have the need for save-points - either all 
updates happen, or none of them do.

Frank Millman

[toc] | [prev] | [next] | [standalone]

#50222

On Tue, Jul 9, 2013 at 5:35 PM, Frank Millman <frank@chagford.com> wrote:
> I have been following this sub-thread with interest, as it resonates with
> what I am doing in my project.

Just FYI, none of my own code will help you as it's all using libpqxx,
but the docs for the library itself are around if you want them (it's
one of the standard ways for C++ programs to use PostgreSQL).

> I came up with the following context manager -
>
> class DbSession:
>     def __exit__(self, type, exc, tb):
>             if self.transaction_active:
>                 self.conn.commit()
>                 self.transaction_active = False

Hmm. So you automatically commit. I'd actually be inclined to _not_ do
this; make it really explicit in your code that you now will commit
this transaction (which might throw an exception if you have open
subtransactions). The way the code with libpqxx works is that any
transaction (including a subtransaction) must be explicitly committed,
or it will be rolled back. So there's one possible code path that
results in persistent changes to the database, and anything else
won't:

* If the object expires without being committed, it's rolled back.
* If an exception is thrown and unwinds the stack, roll back.
* If a Unix signal is sent that terminates the program, roll back.
* If the process gets killed -9, definitely roll back.
* If the computer the program's running on spontaneously combusts, roll back.
* If the hard drive is physically ripped from the server during
processing, roll back.

(Note though that most of these guarantees are from PostgreSQL, not
from libpqxx. I'm talking about the whole ecosystem here, not
something one single library can handle.)

I have absolute 100% confidence that nothing can possibly affect the
database unless I explicitly commit it (aside from a few
non-transaction actions like advancing a sequence pointer, which
specifically don't matter (rolled back transactions can result in gaps
in a sequence of record IDs, nothing more)). It's an extremely
comfortable work environment - I can write whatever code I like, and
if I'm not sure if it'll work or not, I just comment out the commit
line and run. *Nothing* can get past that and quietly commit it behind
my back.

ChrisA

[toc] | [prev] | [next] | [standalone]

#50223

On Tue, Jul 9, 2013 at 1:35 AM, Frank Millman <frank@chagford.com> wrote:
> When any of them need any database access, whether for reading or for
> updating, they execute the following -
>
>     with db_session as conn:
>         conn.transaction_active = True  # this line must be added if
> updating
>         conn.cur.execute(__whatever__)

I'd probably factor out the transaction_active line into a separate
DbSession method.

    @contextmanager
    def updating(self):
        with self as conn:
            conn.transaction_active = True
            yield conn

Then you can do "with db_session" if you're merely reading, or "with
db_session.updating()" if you're writing, and you don't need to repeat
the transaction_active line all over the place.

I would also probably make db_session a factory function instead of a global.

[toc] | [prev] | [next] | [standalone]

#50224

"Chris Angelico" <rosuav@gmail.com> wrote in message 
news:CAPTjJmr4Mr0qCGWqXwyvDCZ55NuaV79VbTT8BJNDSDvhrKq15w@mail.gmail.com...
> On Tue, Jul 9, 2013 at 5:35 PM, Frank Millman <frank@chagford.com> wrote:
>> I have been following this sub-thread with interest, as it resonates with
>> what I am doing in my project.
>
> Just FYI, none of my own code will help you as it's all using libpqxx,
> but the docs for the library itself are around if you want them (it's
> one of the standard ways for C++ programs to use PostgreSQL).
>

I support multiple databases (PostgreSQL, MS SQL Server, sqlite3 at this 
stage) so I use generic Python as much as possible.

>> I came up with the following context manager -
>>
>> class DbSession:
>>     def __exit__(self, type, exc, tb):
>>             if self.transaction_active:
>>                 self.conn.commit()
>>                 self.transaction_active = False
>
> Hmm. So you automatically commit. I'd actually be inclined to _not_ do
> this; make it really explicit in your code that you now will commit
> this transaction (which might throw an exception if you have open
> subtransactions).
>

I endeavour to keep all my database activity to the shortest time possible - 
get a connection, execute a command, release the connection. So setting 
'transaction_active = True' is my way of saying 'execute this command and 
commit it straight away'. That is explicit enough for me. If there are 
nested updates they all follow the same philosophy, so the transaction 
should complete quickly.

Frank

[toc] | [prev] | [next] | [standalone]

#50226

"Ian Kelly" <ian.g.kelly@gmail.com> wrote in message 
news:CALwzid=FzgjPebifx1stDBkh8iwLtWggwwPTPhZ1ykYg+05wEg@mail.gmail.com...
> On Tue, Jul 9, 2013 at 1:35 AM, Frank Millman <frank@chagford.com> wrote:
>> When any of them need any database access, whether for reading or for
>> updating, they execute the following -
>>
>>     with db_session as conn:
>>         conn.transaction_active = True  # this line must be added if
>> updating
>>         conn.cur.execute(__whatever__)
>
> I'd probably factor out the transaction_active line into a separate
> DbSession method.
>
>    @contextmanager
>    def updating(self):
>        with self as conn:
>            conn.transaction_active = True
>            yield conn
>
> Then you can do "with db_session" if you're merely reading, or "with
> db_session.updating()" if you're writing, and you don't need to repeat
> the transaction_active line all over the place.
>

I'll bear it in mind, but I will have to expend some mental energy to 
understand it first <g>, so it will have to wait until I can find some time.

> I would also probably make db_session a factory function instead of a 
> global.

It is not actually a global. When I create a new session, I create a 
db_session instance and store it as a session attribute. Whenever I create a 
database object during the session, I pass in the instance as an argument, 
so they all share the same one.

Frank

[toc] | [prev] | [next] | [standalone]

#50260

On 07/09/2013 01:38 AM, Frank Millman wrote:
>
> "Ian Kelly" <ian.g.kelly@gmail.com> wrote in message
> news:CALwzid=FzgjPebifx1stDBkh8iwLtWggwwPTPhZ1ykYg+05wEg@mail.gmail.com...
>> On Tue, Jul 9, 2013 at 1:35 AM, Frank Millman <frank@chagford.com> wrote:
>>> When any of them need any database access, whether for reading or for
>>> updating, they execute the following -
>>>
>>>      with db_session as conn:
>>>          conn.transaction_active = True  # this line must be added if
>>> updating
>>>          conn.cur.execute(__whatever__)
>>
>> I'd probably factor out the transaction_active line into a separate
>> DbSession method.
>>
>>     @contextmanager
>>     def updating(self):
>>         with self as conn:
>>             conn.transaction_active = True
>>             yield conn
>>
>> Then you can do "with db_session" if you're merely reading, or "with
>> db_session.updating()" if you're writing, and you don't need to repeat
>> the transaction_active line all over the place.
>>
>
> I'll bear it in mind, but I will have to expend some mental energy to
> understand it first <g>, so it will have to wait until I can find some time.

You could also do it like this:

     def updating(self):
         self.transaction_active = True
         return self

and a sample object:

     class Tester(object):
         def __init__(self):
             self.transaction_active = False
             print 'initializied'
         def __enter__(self, *args):
             print '__enter__: transaction_active =', self.transaction_active
             return self
         def __exit__(self, *args):
             self.transaction_active = False
             print '__exit__: transaction_active =', self.transaction_active
             return
         def updating(self):
             self.transaction_active = True
             print 'updating: self.transaction_active =', self.transaction_active
             return self

     with Tester() as conn:
         print 'in first with block'

     print '-' * 50

     with Tester().updating() as conn:
         print 'in second with block'

with it's test run:

     ethan@hydra:~$ python test_cm.py
     initialized
     __enter__: transaction_active = False
     in first with block
     __exit__: transaction_active = False
     --------------------------------------------------
     initialized
     updating: self.transaction_active = True
     __enter__: transaction_active = True
     in second with block
     __exit__: transaction_active = False

--
~Ethan~

[toc] | [prev] | [next] | [standalone]

Page 1 of 3  [1] 2 3  Next page →

Back to top | Article view | comp.lang.python

csiph-web