Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #103959 > unrolled thread

yield in try/finally case

Back to article view | Back to comp.lang.python

Contents

  yield in try/finally case 刘琦帆 <lqf.txx@gmail.com> - 2016-03-03 03:52 -0800
    Re: yield in try/finally case Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2016-03-03 12:13 +0000
      Re: yield in try/finally case 刘琦帆 <lqf.txx@gmail.com> - 2016-03-03 04:52 -0800
        Re: yield in try/finally case Peter Otten <__peter__@web.de> - 2016-03-03 14:47 +0100
        Re: yield in try/finally case Random832 <random832@fastmail.com> - 2016-03-03 10:12 -0500
        Re: yield in try/finally case Peter Otten <__peter__@web.de> - 2016-03-03 17:20 +0100
        Re: yield in try/finally case Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2016-03-03 16:24 +0000
    Re: yield in try/finally case Random832 <random832@fastmail.com> - 2016-03-03 10:00 -0500
      Re: yield in try/finally case Steven D'Aprano <steve@pearwood.info> - 2016-03-04 02:57 +1100

#103959 — yield in try/finally case

I have just saw PEP 255, and it says that 

"A yield statement is not allowed in the try clause of a try/finally construct.  The difficulty is that there's no guarantee the generator will ever be resumed, hence no guarantee that the finally block will ever get executed; that's too much a violation of finally's purpose to bear." from https://www.python.org/dev/peps/pep-0255/

But, meanwhile, the code showed on that page use yield in a try/finally case.
It really puzzles me. Is there anything wrong?

[toc] | [next] | [standalone]

#103960

On 3 March 2016 at 11:52, 刘琦帆 <lqf.txx@gmail.com> wrote:
>
> "A yield statement is not allowed in the try clause of a try/finally construct.  The difficulty is that there's no guarantee the generator will ever be resumed, hence no guarantee that the finally block will ever get executed; that's too much a violation of finally's purpose to bear." from https://www.python.org/dev/peps/pep-0255/
>
> But, meanwhile, the code showed on that page use yield in a try/finally case.
> It really puzzles me. Is there anything wrong?

I think what it means is that you can put a yield in the finally block
but not the try block so:

# Not allowed
def  f():
    try:
        yield 1
    finally:
        pass

# Allowed
def f():
    try:
        pass
    finally:
        yield 1

However that information is out of date. The restriction was removed
in some later Python version. Actually the construct is quite common
when using generator functions to implement context managers:

@contextlib.contextmanager
def replace_stdin(newstdin):
    oldstdin = sys.stdin
    try:
        sys.stdin = newstdin
        yield
    finally:
        sys.stdin = oldstdin

Although the restriction was removed the problem itself still remains.
There's no guarantee that a finally block will execute if there is a
yield in the try block. The same happens if you use a context manager
around a yield statement: the __exit__ method is not guaranteed to be
called. One implication of this is that in the following code it is
not guaranteed that the file will be closed:

def upperfile(filename):
    with open(filename) as fin:
        for line in fin:
            yield line.upper()

--
Oscar

[toc] | [prev] | [next] | [standalone]

#103961

在 2016年3月3日星期四 UTC+8下午8:14:29，Oscar Benjamin写道：
> On 3 March 2016 at 11:52, 刘琦帆 <lqf.txx@gmail.com> wrote:
> >
> > "A yield statement is not allowed in the try clause of a try/finally construct.  The difficulty is that there's no guarantee the generator will ever be resumed, hence no guarantee that the finally block will ever get executed; that's too much a violation of finally's purpose to bear." from https://www.python.org/dev/peps/pep-0255/
> >
> > But, meanwhile, the code showed on that page use yield in a try/finally case.
> > It really puzzles me. Is there anything wrong?
> 
> I think what it means is that you can put a yield in the finally block
> but not the try block so:
> 
> # Not allowed
> def  f():
>     try:
>         yield 1
>     finally:
>         pass
> 
> # Allowed
> def f():
>     try:
>         pass
>     finally:
>         yield 1
> 
> However that information is out of date. The restriction was removed
> in some later Python version. Actually the construct is quite common
> when using generator functions to implement context managers:
> 
> @contextlib.contextmanager
> def replace_stdin(newstdin):
>     oldstdin = sys.stdin
>     try:
>         sys.stdin = newstdin
>         yield
>     finally:
>         sys.stdin = oldstdin
> 
> Although the restriction was removed the problem itself still remains.
> There's no guarantee that a finally block will execute if there is a
> yield in the try block. The same happens if you use a context manager
> around a yield statement: the __exit__ method is not guaranteed to be
> called. One implication of this is that in the following code it is
> not guaranteed that the file will be closed:
> 
> def upperfile(filename):
>     with open(filename) as fin:
>         for line in fin:
>             yield line.upper()
> 
> --
> Oscar


It really nice of you to answer the question. But I am still confused with your last example, is there any case that the file with not be closed? I just run the code and no exception occur.

[toc] | [prev] | [next] | [standalone]

#103964

刘琦帆 wrote:

> 在 2016年3月3日星期四 UTC+8下午8:14:29，Oscar Benjamin写道：
>> On 3 March 2016 at 11:52, 刘琦帆 <lqf.txx@gmail.com> wrote:
>> >
>> > "A yield statement is not allowed in the try clause of a try/finally
>> > construct.  The difficulty is that there's no guarantee the generator
>> > will ever be resumed, hence no guarantee that the finally block will
>> > ever get executed; that's too much a violation of finally's purpose to
>> > bear." from https://www.python.org/dev/peps/pep-0255/
>> >
>> > But, meanwhile, the code showed on that page use yield in a try/finally
>> > case. It really puzzles me. Is there anything wrong?
>> 
>> I think what it means is that you can put a yield in the finally block
>> but not the try block so:
>> 
>> # Not allowed
>> def  f():
>>     try:
>>         yield 1
>>     finally:
>>         pass
>> 
>> # Allowed
>> def f():
>>     try:
>>         pass
>>     finally:
>>         yield 1
>> 
>> However that information is out of date. The restriction was removed
>> in some later Python version. Actually the construct is quite common
>> when using generator functions to implement context managers:
>> 
>> @contextlib.contextmanager
>> def replace_stdin(newstdin):
>>     oldstdin = sys.stdin
>>     try:
>>         sys.stdin = newstdin
>>         yield
>>     finally:
>>         sys.stdin = oldstdin
>> 
>> Although the restriction was removed the problem itself still remains.
>> There's no guarantee that a finally block will execute if there is a
>> yield in the try block. The same happens if you use a context manager
>> around a yield statement: the __exit__ method is not guaranteed to be
>> called. One implication of this is that in the following code it is
>> not guaranteed that the file will be closed:
>> 
>> def upperfile(filename):
>>     with open(filename) as fin:
>>         for line in fin:
>>             yield line.upper()
>> 
>> --
>> Oscar
> 
> 
> It really nice of you to answer the question. But I am still confused with
> your last example, is there any case that the file with not be closed? I
> just run the code and no exception occur.

It doesn't happen easily, you have to defeat CPython's garbage collection. 
Consider the follwing script:

$ cat upper1.py
import sys

_open = open
files = []

def open(*args, **kw):
    """Use a modified open() which keeps track of opened files.

    This allows us to check whether the files are properly closed and
    also to defeat garbage collection.
    """
    f = _open(*args, **kw)
    files.append(f)
    return f


for filename in sys.argv[1:]:
    with open(filename) as f:
        for line in f:
            print(line.upper(), end="")
            break

assert all(f.closed for f in files)

$ echo -e 'foo\nbar\nbaz' > tmp1.txt
$ echo -e 'hams\nspam\njam' > tmp2.txt
$ python3 upper1.py *.txt
FOO
HAMS

As expected it prints the first lines of the files provided as commandline 
args, in upper case. Now let's refactor:

$ cat upper2.py
import sys

_open = open
files = []

def open(*args, **kw):
    """Use a modified open() which keeps track of opened files.

    This allows us to check whether the files are properly closed (and
    also defeats garbage collection).
    """
    f = _open(*args, **kw)
    files.append(f)
    return f

def upperfile(filename):
    with open(filename) as f:
        for line in f:
            yield line.upper()

for uf in map(upperfile, sys.argv[1:]):
    for line in uf:
        print(line, end="")
        break

assert all(f.closed for f in files)

The change looks harmless, we moved the with statement and the conversion 
into the generater. But when whe run it:

$ python3 upper2.py *.txt
FOO
HAMS
Traceback (most recent call last):
  File "upper2.py", line 26, in <module>
    assert all(f.closed for f in files)
AssertionError

This is because the last generator uf = upperfile(...) is not garbage-
collected and wasn't explicitly closed either. If you do care here's one 
possible fix:

from contextlib import closing
...
for uf in map(upperfile, sys.argv[1:]):
    with closing(uf):
        for line in uf:
            print(line, end="")
            break

[toc] | [prev] | [next] | [standalone]

#103967

On Thu, Mar 3, 2016, at 08:47, Peter Otten wrote:
> This is because the last generator uf = upperfile(...) is not garbage-
> collected and wasn't explicitly closed either.

But the program hasn't ended yet when you run your assertion.

import sys

_open = open
files = []

def myclose(self):
    print("--- closed " + self.name)
    self._close()

def open(*args, **kw):
    f = _open(*args, **kw)
    f._close = f.close
    f.close = lambda: myclose(f)
    files.append(f)
    return f

def upperfile(filename):
    with open(filename) as f:
        for line in f:
            yield line.upper()

for uf in map(upperfile, sys.argv[1:]):
    for line in uf:
        print(line, end="")
        break

print("--- end of program")

====

FOO
--- closed tmp1.txt
HAMS
--- end of program
--- closed tmp2.txt

[toc] | [prev] | [next] | [standalone]

#103974

Random832 wrote:

> On Thu, Mar 3, 2016, at 08:47, Peter Otten wrote:
>> This is because the last generator uf = upperfile(...) is not garbage-
>> collected and wasn't explicitly closed either.
> 
> But the program hasn't ended yet when you run your assertion.

If your expectations are in line with Python's actual behaviour -- then 
fine. Normally someone who writes

with acquire_resource() as r:
    use(r)
assert r was released

wants the resource to be released when the with suite is left.

When the with-statement is moved into a generator

def gen_resource():
    with acquire_resource() as r:
        yield r

for r in gen_resource():
    use(r)
    break # use(r) triggering an exception would have the same effect
assert r was released # may fail

you are at the mercy of the Python interpreter's garbage collection 
strategy. 

Of course you are exiting the for-suite, not the with-suite. Nevertheless 
this surprised me when Oscar pointed it out the first time.

[toc] | [prev] | [next] | [standalone]

#103975

On 3 March 2016 at 15:12, Random832 <random832@fastmail.com> wrote:
> On Thu, Mar 3, 2016, at 08:47, Peter Otten wrote:
>> This is because the last generator uf = upperfile(...) is not garbage-
>> collected and wasn't explicitly closed either.
>
> But the program hasn't ended yet when you run your assertion.
>
> import sys
>
> _open = open
> files = []
>
> def myclose(self):
>     print("--- closed " + self.name)
>     self._close()
>
> def open(*args, **kw):
>     f = _open(*args, **kw)
>     f._close = f.close
>     f.close = lambda: myclose(f)
>     files.append(f)
>     return f
>
> def upperfile(filename):
>     with open(filename) as f:
>         for line in f:
>             yield line.upper()
>
> for uf in map(upperfile, sys.argv[1:]):
>     for line in uf:
>         print(line, end="")
>         break
>
> print("--- end of program")
>
> ====
>
> FOO
> --- closed tmp1.txt
> HAMS
> --- end of program
> --- closed tmp2.txt

If you're happy letting __del__ close the file then why bother with
the context manager in the first place? By the reasoning above we
don't even need to close the file in __del__ since all open files get
closed at process exit anyway. The point is that finally/__exit__ are
not achieving what they are supposed to achieve: the guarantee that
the file is immediately closed when you're done with it.

Try your code under pypy and you'll probably find that your context
manager doesn't fire even at process exit.

--
Oscar

[toc] | [prev] | [next] | [standalone]

#103966

On Thu, Mar 3, 2016, at 06:52, 刘琦帆 wrote:
> I have just saw PEP 255, and it says that 
> 
> "A yield statement is not allowed in the try clause of a try/finally
> construct.  The difficulty is that there's no guarantee the generator
> will ever be resumed, hence no guarantee that the finally block will ever
> get executed; that's too much a violation of finally's purpose to bear."
> from https://www.python.org/dev/peps/pep-0255/

I'm not sure I understand this reasoning. Why not simply execute it in
__del__ if it hasn't been reached until then? AIUI that is what C# does.

[toc] | [prev] | [next] | [standalone]

#103969

On Fri, 4 Mar 2016 02:00 am, Random832 wrote:

> On Thu, Mar 3, 2016, at 06:52, 刘琦帆 wrote:
>> I have just saw PEP 255, and it says that
>> 
>> "A yield statement is not allowed in the try clause of a try/finally
>> construct.  The difficulty is that there's no guarantee the generator
>> will ever be resumed, hence no guarantee that the finally block will ever
>> get executed; that's too much a violation of finally's purpose to bear."
>> from https://www.python.org/dev/peps/pep-0255/
> 
> I'm not sure I understand this reasoning. Why not simply execute it in
> __del__ if it hasn't been reached until then? AIUI that is what C# does.

I believe that under certain circumstances, __del__ may never be executed at
all; it may be executed under rather perilous circumstances where the
interpreter is already shutting down; and even if it is executed, it is not
guaranteed to be executed at any particular time or in any particular
order. In other words, cleaning up in __del__ may be non-deterministic,
while the whole point of try...finally is that the finally block is
executed in a deterministic fashion.

In any case, PEP 255 is obsolete: that is no longer a limitation of yield.


# Python 2.7

py> def gen():
...     try:
...             yield 1
...     finally:
...             yield 2
...
py> it = gen()
py> next(it)
1
py> next(it)
2





-- 
Steven

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web