Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #18834 > unrolled thread

contextlib.contextmanager and try/finally

Back to article view | Back to comp.lang.python

Contents

  contextlib.contextmanager and try/finally johannh@gmail.com - 2012-01-11 07:45 -0800
    Re: contextlib.contextmanager and try/finally Neil Cerutti <neilc@norwich.edu> - 2012-01-11 16:46 +0000
    Re: contextlib.contextmanager and try/finally Robert Kern <robert.kern@gmail.com> - 2012-01-11 17:14 +0000
    Re: contextlib.contextmanager and try/finally Ian Kelly <ian.g.kelly@gmail.com> - 2012-01-11 10:20 -0700
      Re: contextlib.contextmanager and try/finally johannh@gmail.com - 2012-01-11 09:30 -0800
      Re: contextlib.contextmanager and try/finally johannh@gmail.com - 2012-01-11 09:30 -0800
    RE: contextlib.contextmanager and try/finally "Prasad, Ramit" <ramit.prasad@jpmorgan.com> - 2012-01-31 21:07 +0000
    RE: contextlib.contextmanager and try/finally Peter Otten <__peter__@web.de> - 2012-01-31 23:02 +0100
    Re: contextlib.contextmanager and try/finally Ian Kelly <ian.g.kelly@gmail.com> - 2012-01-31 15:09 -0700
      Re: contextlib.contextmanager and try/finally Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-02-01 02:15 +0000

#18834 — contextlib.contextmanager and try/finally

I'm trying to write a context manager to handle database connections, under the principle that I should not rely on CPython's reference-counting semantics to clean up scarce resources, like connections.

I wrote:

@contexlib.contextmanager
def ensure_connection(con=None):
    con_created = False
    if con is None:
        con_created, con = True, make_connection()
    try:
        yield con
    finally:
        if con_created:
            con.close()

However, then I read the following paragraph from PEP-343:

    Note that we're not guaranteeing that the finally-clause is
    executed immediately after the generator object becomes unused,
    even though this is how it will work in CPython.  This is similar
    to auto-closing files: while a reference-counting implementation
    like CPython deallocates an object as soon as the last reference
    to it goes away, implementations that use other GC algorithms do
    not make the same guarantee.  This applies to Jython, IronPython,
    and probably to Python running on Parrot.

That suggests that I cannot rely on the contextlib.contextmanager decorator to ensure that the connection is closed and would have to write my own object with __enter__ and __exit__ methods to guarantee this.

Is this understanding accurate?  If so, could someone illustrate why this is so? 

Thanks,
Johann

[toc] | [next] | [standalone]

#18835

On 2012-01-11, johannh@gmail.com <johannh@gmail.com> wrote:
> That suggests that I cannot rely on the
> contextlib.contextmanager decorator to ensure that the
> connection is closed and would have to write my own object with
> __enter__ and __exit__ methods to guarantee this.

contextmanager wraps your generator in an object with __enter__
and __exit__ methods for you.

-- 
Neil Cerutti

[toc] | [prev] | [next] | [standalone]

#18837

On 1/11/12 3:45 PM, johannh@gmail.com wrote:
> I'm trying to write a context manager to handle database connections, under the principle that I should not rely on CPython's reference-counting semantics to clean up scarce resources, like connections.
>
> I wrote:
>
> @contexlib.contextmanager
> def ensure_connection(con=None):
>      con_created = False
>      if con is None:
>          con_created, con = True, make_connection()
>      try:
>          yield con
>      finally:
>          if con_created:
>              con.close()
>
> However, then I read the following paragraph from PEP-343:
>
>      Note that we're not guaranteeing that the finally-clause is
>      executed immediately after the generator object becomes unused,
>      even though this is how it will work in CPython.  This is similar
>      to auto-closing files: while a reference-counting implementation
>      like CPython deallocates an object as soon as the last reference
>      to it goes away, implementations that use other GC algorithms do
>      not make the same guarantee.  This applies to Jython, IronPython,
>      and probably to Python running on Parrot.
>
> That suggests that I cannot rely on the contextlib.contextmanager decorator to ensure that the connection is closed and would have to write my own object with __enter__ and __exit__ methods to guarantee this.
>
> Is this understanding accurate?  If so, could someone illustrate why this is so?

Looking at the paragraph before this one, it appears that the PEP is talking 
about the .close() method on generators, which is really just a general purpose 
API for closing generators that might not be exhausted yet. It's not really 
related to the context manager stuff except that it came up during the design 
process of the context manager along with the related .send() and .throw() methods.

__enter__() will call .next() once to execute the code up to the yield 
statement. Then __exit__() will call .next() once again to execute the code 
after the yield statement, including the finally: clause. That's the only thing 
you need to rely on. Your connection-closing code will be called if __exit__() 
gets called. That will exhaust your generator, so the .close() method will not 
really do anything helpful or hurtful in such a case.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco

[toc] | [prev] | [next] | [standalone]

#18839

On Wed, Jan 11, 2012 at 8:45 AM,  <johannh@gmail.com> wrote:
> However, then I read the following paragraph from PEP-343:
>
>    Note that we're not guaranteeing that the finally-clause is
>    executed immediately after the generator object becomes unused,
>    even though this is how it will work in CPython.  This is similar
>    to auto-closing files: while a reference-counting implementation
>    like CPython deallocates an object as soon as the last reference
>    to it goes away, implementations that use other GC algorithms do
>    not make the same guarantee.  This applies to Jython, IronPython,
>    and probably to Python running on Parrot.
>
> That suggests that I cannot rely on the contextlib.contextmanager decorator to ensure that the connection is closed and would have to write my own object with __enter__ and __exit__ methods to guarantee this.
>
> Is this understanding accurate?  If so, could someone illustrate why this is so?

First, this is just a timing issue.  There is no guarantee that the
finally clause will be executed immediately, but it is guaranteed to
be executed at some point.  In the other implementations, it would
happen whenever the GC algorithm runs.

Second, I believe that passage is not referring to the contextmanager
decorator specifically, but more generally to the changes that were
made to allow generators to yield from within a try-finally construct
(previously this would have been illegal syntax, since there was no
way to guarantee the finally block would be performed).

Like Neil mentioned, a contextmanager generator is wrapped with an
__exit__ method that is guaranteed to be called and that explicitly
resumes or closes the generator.  So as long as your contextmanager
generator is properly written (i.e. it yields exactly once), the
finally block will execute in a timely fashion.

[toc] | [prev] | [next] | [standalone]

#18840

On Wednesday, January 11, 2012 11:20:19 AM UTC-6, Ian wrote:
> 
> Second, I believe that passage is not referring to the contextmanager
> decorator specifically, but more generally to the changes that were
> made to allow generators to yield from within a try-finally construct
> (previously this would have been illegal syntax, since there was no
> way to guarantee the finally block would be performed).

Thanks.  You and Robert have shown me the light.  That paragraph was talking about the finalization that happens to a suspended generator when it is GC'd.  Meanwhile, the contextmanager decorator ensures that its generator is run all the way through, so it would never be left in a suspended state, so its GC/finalization semantics are irrelevant. (Please correct, if wrong.)

(Neil: I understood that it created a context manager object, but I had been reading the quoted paragraph to mean that by going through a generator the decorator was re-introducing a dependency on GC time.  But I was wrong.)

Regards,
Johann

[toc] | [prev] | [next] | [standalone]

#18841

On Wednesday, January 11, 2012 11:20:19 AM UTC-6, Ian wrote:
> 
> Second, I believe that passage is not referring to the contextmanager
> decorator specifically, but more generally to the changes that were
> made to allow generators to yield from within a try-finally construct
> (previously this would have been illegal syntax, since there was no
> way to guarantee the finally block would be performed).

Thanks.  You and Robert have shown me the light.  That paragraph was talking about the finalization that happens to a suspended generator when it is GC'd.  Meanwhile, the contextmanager decorator ensures that its generator is run all the way through, so it would never be left in a suspended state, so its GC/finalization semantics are irrelevant. (Please correct, if wrong.)

(Neil: I understood that it created a context manager object, but I had been reading the quoted paragraph to mean that by going through a generator the decorator was re-introducing a dependency on GC time.  But I was wrong.)

Regards,
Johann

[toc] | [prev] | [next] | [standalone]

#19658

>Like Neil mentioned, a contextmanager generator is wrapped with an
>__exit__ method that is guaranteed to be called and that explicitly
>resumes or closes the generator.  So as long as your contextmanager
>generator is properly written (i.e. it yields exactly once), the
>finally block will execute in a timely fashion.

Is that true even in the face of something like sys.exit()?
What happens if 1) sys.exit is called while in the same thread
2) sys.exit is called from another thread but while this thread
is in context manager?

Ramit

Ramit Prasad | JPMorgan Chase Investment Bank | Currencies Technology
712 Main Street | Houston, TX 77002
work phone: 713 - 216 - 5423

--

This email is confidential and subject to important disclaimers and
conditions including on offers for the purchase or sale of
securities, accuracy and completeness of information, viruses,
confidentiality, legal privilege, and legal entity disclaimers,
available at http://www.jpmorgan.com/pages/disclosures/email.  

[toc] | [prev] | [next] | [standalone]

#19659

Prasad, Ramit wrote:

>>Like Neil mentioned, a contextmanager generator is wrapped with an
>>__exit__ method that is guaranteed to be called and that explicitly
>>resumes or closes the generator.  So as long as your contextmanager
>>generator is properly written (i.e. it yields exactly once), the
>>finally block will execute in a timely fashion.
> 
> Is that true even in the face of something like sys.exit()?
> What happens if 1) sys.exit is called while in the same thread
> 2) sys.exit is called from another thread but while this thread
> is in context manager?

sys.exit() uses the normal exception mechanism to terminate a program:

>>> import sys
>>> try:
...     sys.exit()
... except SystemExit:
...     print "I'd rather not"
...
I'd rather not
>>>

It won't derail a context manager:

>>> from contextlib import contextmanager
>>> @contextmanager
... def f():
...     try:
...             yield
...     finally: print "bye"
...
>>> import sys
>>> with f():
...     sys.exit()
...
bye
$

[toc] | [prev] | [next] | [standalone]

#19661

On Tue, Jan 31, 2012 at 2:07 PM, Prasad, Ramit
<ramit.prasad@jpmorgan.com> wrote:
>>Like Neil mentioned, a contextmanager generator is wrapped with an
>>__exit__ method that is guaranteed to be called and that explicitly
>>resumes or closes the generator.  So as long as your contextmanager
>>generator is properly written (i.e. it yields exactly once), the
>>finally block will execute in a timely fashion.
>
> Is that true even in the face of something like sys.exit()?

Yes.

> What happens if 1) sys.exit is called while in the same thread

Why don't you try it and find out?  To answer the question, though,
sys.exit() raises a SystemExit exception, which propagates out of the
with block and calls the __exit__ method, which then throws the
exception to the generator, which executes its finally clause and
exits.  The __exit__ method returns false, so the SystemExit exception
continues to propagate, and if it is not caught, then the process
exits.

> 2) sys.exit is called from another thread but while this thread
> is in context manager?

Then the other thread raises SystemExit, and the current thread is
unaffected.  sys.exit only affects the thread it is called in.

You can certainly come up with scenarios in which the finally clause
does not execute, e.g. killing the interpreter with "kill -9" or
yanking out the power cord.  Within the confines of the Python
interpreter, though, it is guaranteed that the finally block will
execute.

[toc] | [prev] | [next] | [standalone]

#19678

On Tue, 31 Jan 2012 15:09:34 -0700, Ian Kelly wrote:

> On Tue, Jan 31, 2012 at 2:07 PM, Prasad, Ramit
> <ramit.prasad@jpmorgan.com> wrote:
>>>Like Neil mentioned, a contextmanager generator is wrapped with an
>>>__exit__ method that is guaranteed to be called and that explicitly
>>>resumes or closes the generator.  So as long as your contextmanager
>>>generator is properly written (i.e. it yields exactly once), the
>>>finally block will execute in a timely fashion.
>>
>> Is that true even in the face of something like sys.exit()?
> 
> Yes.
[...]
> You can certainly come up with scenarios in which the finally clause
> does not execute, e.g. killing the interpreter with "kill -9" or yanking
> out the power cord.  Within the confines of the Python interpreter,
> though, it is guaranteed that the finally block will execute.

Almost, but not quite.

The finally block is not guaranteed to execute if the try block never 
exits -- infinite loops are still infinite loops.

Also, unlike sys.exit, os._exit doesn't work through the exception 
mechanism, can't be caught, and simply exits immediately.


>>> import os
>>> try:
...     os._exit(1)
... finally:
...     print "exiting"
... 
steve@runes:~$ 




-- 
Steven

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web