Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #103869 > unrolled thread

urlopen, six, and py2

Back to article view | Back to comp.lang.python

Contents

  urlopen, six, and py2 Fabien <fabien.maussion@gmail.com> - 2016-03-02 15:05 +0100
    Re: urlopen, six, and py2 Matt Wheeler <m@funkyhat.org> - 2016-03-02 14:35 +0000
      Re: urlopen, six, and py2 Fabien <fabien.maussion@gmail.com> - 2016-03-02 16:36 +0100
        Re: urlopen, six, and py2 Chris Angelico <rosuav@gmail.com> - 2016-03-03 03:17 +1100
    Re: urlopen, six, and py2 Chris Angelico <rosuav@gmail.com> - 2016-03-03 01:53 +1100

#103869 — urlopen, six, and py2

Hi,

it seems that urlopen had no context manager for versions < 3. The 
following code therefore will crash on py2 but not on py3.

from six.moves.urllib.request import urlopen
with urlopen('http://www.google.com') as resp:
     _ = resp.read()

Error:
AttributeError: addinfourl instance has no attribute '__exit__'

I actually wonder if this is not something that the six library should 
take care of upstream, but in the meantime I could simply do what is 
suggested on this stackoverflow post:

http://stackoverflow.com/questions/30627937/tracebaclk-attributeerroraddinfourl-instance-has-no-attribute-exit

My question is: why does the python3 version need a "with" block while 
the python2 version doesn't? Can I skip the "with" entirely, or should I 
rather do the following:

from six.moves.urllib.request import urlopen

try:
     with urlopen('http://www.google.com') as resp:
         _ = resp.read()
except AttributeError:
     # python 2
     resp = urlopen('http://www.google.com')
     _ = resp.read()

(which is quite ugly).

Thanks!

Fabien

[toc] | [next] | [standalone]

#103872

On 2 March 2016 at 14:05, Fabien <fabien.maussion@gmail.com> wrote:
> [snip]
> My question is: why does the python3 version need a "with" block while the
> python2 version doesn't? Can I skip the "with" entirely, or should I rather
> do the following:

It's not a case of "need", using the "with" construction is an added
feature, not a burden!

> from six.moves.urllib.request import urlopen
>
> try:
>     with urlopen('http://www.google.com') as resp:
>         _ = resp.read()
> except AttributeError:
>     # python 2
>     resp = urlopen('http://www.google.com')
>     _ = resp.read()

This is poor practise as you aren't closing "resp".
This leaves state lying around that you don't need anymore, which is
the whole purpose of the context manager that 3 provides.
It will *usually* be cleaned up when leaving the current scope, but
won't if there is an exception thrown. Using the context manager
ensures resp is *always* cleaned up properly even if an exception is
thrown.

I agree that six should probably handle this, but in the meantime you
could use contextlib.closing ([1]) rather than rolling your own. It
even uses urlopen as an example.

If you absolutely want to roll your own then don't bother with the
context manager provided by 3 at all, just use:

resp = urlopen('http://www.google.com')
try:
    _ = resp.read()
finally:
    resp.close()

But as this is wordier and more fragile than using a context manager
to clean up for you, I expect you won't bother :).


[1] https://docs.python.org/2/library/contextlib.html#contextlib.closing

-- 
Matt Wheeler
http://funkyh.at

[toc] | [prev] | [next] | [standalone]

#103878

On 03/02/2016 03:35 PM, Matt Wheeler wrote:
> I agree that six should probably handle this,

Thank you Matt and Chris for your answers. Do you think I should open an 
issue on six? It sounds unlikely that I am the first one having this 
problem...

(until this difference with urlopen I have found six to be extremely 
good at helping not caring about python versions at all)

[toc] | [prev] | [next] | [standalone]

#103880

On Thu, Mar 3, 2016 at 2:36 AM, Fabien <fabien.maussion@gmail.com> wrote:
> On 03/02/2016 03:35 PM, Matt Wheeler wrote:
>>
>> I agree that six should probably handle this,
>
>
> Thank you Matt and Chris for your answers. Do you think I should open an
> issue on six? It sounds unlikely that I am the first one having this
> problem...
>
> (until this difference with urlopen I have found six to be extremely good at
> helping not caring about python versions at all)

What happens if you use 'requests' rather than urlopen? My guess is
that requests will already have dealt with this.

ChrisA

[toc] | [prev] | [next] | [standalone]

#103873

On Thu, Mar 3, 2016 at 1:35 AM, Matt Wheeler <m@funkyhat.org> wrote:
>> from six.moves.urllib.request import urlopen
>>
>> try:
>>     with urlopen('http://www.google.com') as resp:
>>         _ = resp.read()
>> except AttributeError:
>>     # python 2
>>     resp = urlopen('http://www.google.com')
>>     _ = resp.read()
>
> This is poor practise as you aren't closing "resp".
> This leaves state lying around that you don't need anymore, which is
> the whole purpose of the context manager that 3 provides.
> It will *usually* be cleaned up when leaving the current scope, but
> won't if there is an exception thrown. Using the context manager
> ensures resp is *always* cleaned up properly even if an exception is
> thrown.

Not sure why you say it won't if there's an exception thrown. This
code will do the same thing on both versions:

def get_data():
    resp = urlopen('http://www.google.com')
    return resp.read()

Absent the context manager, this depends on object disposal for its
cleanup. But whether this function returns normally or raises an
exception, the response object goes out of scope at the same time.
There's no guarantee that it'll be cleaned up immediately when the
function exits, but it's the same for the exceptional and
non-exceptional cases.

Agreed that try/finally is the best way to do cross-platform code here:

def get_data():
    resp = urlopen('http://www.google.com')
    try:
        return resp.read()
    finally:
        resp.close()

It's reasonably compact, and fairly clear. And it has the exact
guarantee that the context manager has (before the function exits, the
'finally' block will be performed, and resources will be properly
released).

ChrisA

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web