Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #98114 > unrolled thread

How to handle exceptions properly in a pythonic way?

Back to article view | Back to comp.lang.python

Contents

  How to handle exceptions properly in a pythonic way? zljubisic@gmail.com - 2015-11-02 11:24 -0800
    Re: How to handle exceptions properly in a pythonic way? John Gordon <gordon@panix.com> - 2015-11-02 20:05 +0000
      Re: How to handle exceptions properly in a pythonic way? zljubisic@gmail.com - 2015-11-04 19:41 -0800
        Re: How to handle exceptions properly in a pythonic way? Chris Angelico <rosuav@gmail.com> - 2015-11-05 14:58 +1100
          Re: How to handle exceptions properly in a pythonic way? zljubisic@gmail.com - 2015-11-04 20:18 -0800
            Re: How to handle exceptions properly in a pythonic way? Chris Angelico <rosuav@gmail.com> - 2015-11-05 15:51 +1100
              Re: How to handle exceptions properly in a pythonic way? zljubisic@gmail.com - 2015-11-09 13:43 -0800
            Re: How to handle exceptions properly in a pythonic way? tian.su.yale@gmail.com - 2015-11-04 21:02 -0800
    Re: How to handle exceptions properly in a pythonic way? Ian Kelly <ian.g.kelly@gmail.com> - 2015-11-02 13:58 -0700
      Re: How to handle exceptions properly in a pythonic way? zljubisic@gmail.com - 2015-11-04 20:11 -0800
    Re: How to handle exceptions properly in a pythonic way? tian.su.yale@gmail.com - 2015-11-04 21:15 -0800
      Re: How to handle exceptions properly in a pythonic way? zljubisic@gmail.com - 2015-11-09 13:36 -0800

#98114 — How to handle exceptions properly in a pythonic way?

Let's say that I have the following simple function:

def get_html(url):

    wpage = requests.get(url)
    
    return wpage.text

How to handle exceptions properly that can arise during execution of the requests.get(url)?

If I call this function with

try:
    html = get_html('www.abc.com/index.html')
except ConnectionError:
        service_exception
else:
    continue_with_the_code

and ConnectionError exception happens I can handle the exception with service_exception part of the code.

I could also put some try/except block in get_html function, but the only purpose why should I do it,  is to log the url and re rise the error. Is there any other reason why should I do this?
 
def get_html(url):

    try:
        wpage = requests.get(url)
    except ConnectionError:
        flog('warning', 'ConnectionError exception', log_except=True)
        raise

    return 

Anyway, whenever I am calling the get_html function I should put it in the try/except block and handle all exceptions.
Now which exceptions? Can I know them in advance or I should wait for one to happened and then put it in caller except block?

Let's say that I have called get_html function hundred times in try/except block with ConnectionError part, and now I found a new exception that can happen during execution of the requests.get(url) command. What should I do now?
Should I put the same except block that handles that new exception in every and each of these hundred calls? 

I could also handle all exceptions in get_html function, and in case that I am not able to get the html, I can return None. 

def get_html(url):

    try:
        wpage = requests.get(url)
    except ConnectionError:
        flog('warning', 'ConnectionError exception', log_except=True)
        raise
    except Exception as e:
        flog("warning", "{0}   [wpage = requests.get(url) {1}]".format(e, url), log_except=True)
        raise
    else:

        if wpage.status_code == requests.codes.ok:
            flog('debug', 'wpage.status_code = %s' % wpage.status_code)
            html = wpage.text
        else:
            flog('debug', 'url = %s' % url)
            html = None

    return html


Now caller should only check if function has returned something other than None. Caller doesn't need try/except block anymore. Something like this:

if get_html('www.abc.com/index.html') is not None:
    I_am_sure_that_html_has_been_downloaded_correctly


Now if I want to catch a new exception, I can catch it in get_html function, which is the only change in the program.

I have read some articles that returning None is not a good approach, so I am confused.

How to handle exceptions properly in a pythonic way?

Regards.

[toc] | [next] | [standalone]

#98116

In <4b303213-62e2-42d4-b2f6-4fc1f6025944@googlegroups.com> zljubisic@gmail.com writes:

> Let's say that I have the following simple function:

> def get_html(url):

>     wpage = requests.get(url)
>     
>     return wpage.text

> How to handle exceptions properly that can arise during execution of the
> requests.get(url)?

The best way is probably to do nothing at all, and let the caller handle
any exceptions.

-- 
John Gordon                   A is for Amy, who fell down the stairs
gordon@panix.com              B is for Basil, assaulted by bears
                                -- Edward Gorey, "The Gashlycrumb Tinies"

[toc] | [prev] | [next] | [standalone]

#98271

> The best way is probably to do nothing at all, and let the caller handle
> any exceptions.

In that case every call of the get_html function has to be in the try/except block with many exceptions.
Sometimes, it is enough just to know whether I managed to get the html or not.
In that case, I could for example, in get_html have try/except block which will as result raise let's say NoHtml custom exception or return some special value.

Raising an exception forces me to put every call of the get_html function in try/except block.
If I am returning a special value, than I can call get_html and then test the value.

I am not sure which approach is better.

[toc] | [prev] | [next] | [standalone]

#98272

On Thu, Nov 5, 2015 at 2:41 PM,  <zljubisic@gmail.com> wrote:
> Raising an exception forces me to put every call of the get_html function in try/except block.
> If I am returning a special value, than I can call get_html and then test the value.
>
> I am not sure which approach is better.

Raising an exception means that if you fail to put it in a try/except,
you get a traceback that walks you through the exact location of the
problem.

Returning a special value means that if you fail to test the value,
you either get an exception further on (if you get back None and try
to treat it as a string, for instance), or get incorrect results, with
no indication of the actual cause of the problem.

Which would you prefer?

ChrisA

[toc] | [prev] | [next] | [standalone]

#98274

> Which would you prefer?

So if I am just checking for the ConnectionError in get_html and a new exception arises, I will have traceback to the get_html function showing that unhandled exception has happened.
Now I have to put additional exception block for managing the new exception in the get_html function and I am covered.

Is that what you wanted to say?

[toc] | [prev] | [next] | [standalone]

#98275

On Thu, Nov 5, 2015 at 3:18 PM,  <zljubisic@gmail.com> wrote:
>> Which would you prefer?
>
> So if I am just checking for the ConnectionError in get_html and a new exception arises, I will have traceback to the get_html function showing that unhandled exception has happened.
> Now I have to put additional exception block for managing the new exception in the get_html function and I am covered.
>
> Is that what you wanted to say?

Yep. I would definitely recommend having get_html raise an exception
on any problem; here's the structure I'd use:

def get_html(...):
    try:
        ... actually go get the info
        return info
    except (ConnectionError, OSError, SocketError) as e:
        raise ContentNotFoundError from e

This means there are three possibilities:

1) Everything works correctly, and get_html returns something useful.

2) An exception occurs of a type that you expect - the server didn't
respond, or the OS threw back a failure, or whatever. (You'd decide
what set of exceptions this represents.) In this case, you get a
single exception that wraps that up and summarizes the problem as
"Content Not Found". The original exception is retained in the
__cause__ of the ContentNotFoundError, so no information is lost.

3) An unexpected exception occurs - your script runs out of memory, or
there's a bug in your code, or anything else. The exception will
simply bubble up, and be dealt with somewhere else.

To use this code, what you'd do is:

try:
    data = get_html(...)
except ContentNotFoundError:
    # deal with the case of not having anything to work with

Omitting the try/except is a perfectly reasonable way of working, too.
Just call get_html, and if ever it can't get a result, the exception
continues to propagate. It's easy, it's straight-forward, and it's
unambiguous - any problem you don't expect becomes an exception.

ChrisA

[toc] | [prev] | [next] | [standalone]

#98553

Hi,

> def get_html(...):
>     try:
>         ... actually go get the info
>         return info
>     except (ConnectionError, OSError, SocketError) as e:
>         raise ContentNotFoundError from e

Personally, I never liked "early returns". I would rather used a variable and the last line in the function would be return variable.
Anyway, I got the point.

> To use this code, what you'd do is:
> 
> try:
>     data = get_html(...)
> except ContentNotFoundError:
>     # deal with the case of not having anything to work with

I got a picture. Thanks.

I have to rethink now about two approaches. One would be as you have suggested in this mail - raising ContentNotFoundError, and another one would be not to mask root exceptions and deal with them instead of ContentNotFoundError.

You have clarified to me the concept. Thanks.

Regards. 

[toc] | [prev] | [next] | [standalone]

#98276

在 2015年11月4日星期三 UTC-6下午10:18:33，zlju...@gmail.com写道：
> > Which would you prefer?
> 
> So if I am just checking for the ConnectionError in get_html and a new exception arises, I will have traceback to the get_html function showing that unhandled exception has happened.
> Now I have to put additional exception block for managing the new exception in the get_html function and I am covered.
> 
> Is that what you wanted to say?

Hi,
If I may, I feel you are trying to address a few separate questions, although they do relate to each other.
1. Coding Design: with the try/except block inside or outside the function
2. Exception Handling: What to do with new un-anticipated exceptions
3. How to record the exception for reference: logging or special value

My feeling is:
1. Personally prefer to put try/except block outside the function, to keep the code clean and easy to follow.
2. I would suggest follow the Test Driven Development (TDD) approach. You are not able to anticipate all types of possible exceptions. However, I'm sure you have a pretty good idea about what exceptions are more likely to happen and cause you problem. In that case, just develop your code to pass these tests, and refactor it in the future if really needed. Not necessary to push your code to be perfect, able to handle every possible exception.
3. Kind of personal choice here, since no matter which way you go, you do need to follow it up and deal with them. Probably I would log it just for record keeping purpose. As long as you follow up and trace back to the root cause, they seem to serve the same goal.
Hope this helps...
All the best,
Tian

[toc] | [prev] | [next] | [standalone]

#98117

On Mon, Nov 2, 2015 at 12:24 PM,  <zljubisic@gmail.com> wrote:
> I have read some articles that returning None is not a good approach, so I am confused.
>
> How to handle exceptions properly in a pythonic way?

I'm having a hard time understanding what question you're asking. You
have a lot of discussion about where to handle exceptions, but the
examples that you show are of logging the the exception and re-raising
it, which is a good practice but distinct from handling the exception.

Then you seem to be asking about the practice of returning None as an
alternative to propagating the exception, but the example you show
only returns None in a case where no exception was raised in the first
place.

None is arguably reasonable to return when it makes sense as a
non-exceptional value in the context. For example, if you're looking
up a policy that may or may not exist, then your get_policy function
might return None to indicate no value. If get_policy fails because of
an RPC timeout however, then it needs to raise an exception rather
than possibly incorrectly returning "no policy".

In comparison, if you're returning something that should always have a
value, then you should never return None. In such a case, None would
indicate that something went wrong, but it lacks any information about
*what* went wrong. An exception, on the other hand, is ideal for
carrying that sort of information. Returning None in such a scenario
is also risky in that the caller may not be expecting None and may try
to treat it as a normal value. In some cases this may even appear to
work until something breaks in a completely different part of the
code.

In the example given, the case where None is returned is clearly an
exceptional case (non-success) even though no exception was raised by
requests.get. You should raise an exception there instead of returning
None to allow the caller better control over the situation.

[toc] | [prev] | [next] | [standalone]

#98273

On Monday, 2 November 2015 21:59:45 UTC+1, Ian  wrote:


> I'm having a hard time understanding what question you're asking. 

:)
I am really having a hard time to explain the problem as English is not my first language.

> You
> have a lot of discussion about where to handle exceptions, 

That's right. I am not sure where to handle exceptions and at the same time to have the code clean as possible.

For example John from the previous post suggested raising exceptions in the get_html procedure. That would mean that I should put every call of get_html function to the try/except block.

In this particular example, I would like just to know whether I managed to get_html or not.
In case I am experiencing some problems inside of the get_html function I could maybe retry several times, but this function should return the html or inform the caller that it is unable to get the html. For caller, the reason why get_html function couldn't get the html is not important.

For now, I am thinking that maybe I should use an approach in which I will have except block in get_html that will be responsible for several exceptions, something like this:

def get_html(url):

    try:
        wpage = requests.get(url)
    except ConnectionError, exception_a, exceptionb...:
        flog('warning', 'ConnectionError exception', log_except=True)
        raise NoHtml

    return

In that way the caller can put get_html function in try/except block but just paying an attention to the NoHtml exception which is much clearer than to have except ConnectionError
except exception_a
except exceptionb...: 
in every call of the get_html.

> but the
> examples that you show are of logging the the exception and re-raising
> it, which is a good practice but distinct from handling the exception.

I mentioned logging in get_html because I don't know what else I should do with an exception in get_html function.

> Then you seem to be asking about the practice of returning None as an
> alternative to propagating the exception, but the example you show
> only returns None in a case where no exception was raised in the first
> place.

I returned None based of the wpage.status_code == requests.codes.ok.
I assume that if this condition is true, everything went well.
Otherwise, I could forget to put some explicit exceptions in the except block so try block will pass even if certain exception has happened.
I read that catching all exception with except Exception as e: is not a good option. 

> None is arguably reasonable to return when it makes sense as a
> non-exceptional value in the context. For example, if you're looking
> up a policy that may or may not exist, then your get_policy function
> might return None to indicate no value. If get_policy fails because of
> an RPC timeout however, then it needs to raise an exception rather
> than possibly incorrectly returning "no policy".

If I understood correctly, in my case I could ask for getting the html from non existent web page or I can have communication (network) problem.
In any of these two cases if I return None I will newer know that web page doesn't exist. Good point.

> In comparison, if you're returning something that should always have a
> value, then you should never return None. In such a case, None would
> indicate that something went wrong, but it lacks any information about
> *what* went wrong. An exception, on the other hand, is ideal for
> carrying that sort of information. Returning None in such a scenario
> is also risky in that the caller may not be expecting None and may try
> to treat it as a normal value. In some cases this may even appear to
> work until something breaks in a completely different part of the
> code.

So my get_html function should never return None in that case.
 
> In the example given, the case where None is returned is clearly an
> exceptional case (non-success) even though no exception was raised by
> requests.get. You should raise an exception there instead of returning
> None to allow the caller better control over the situation.

I am kind of getting the philosophy of exceptions but I still have some dilemmas as you could see above. 

Thank you very much for your comments.

Regards.

[toc] | [prev] | [next] | [standalone]

#98277

Hi,
If I may, I feel you are tying to address a few questions at the same time, although they are related
1. Where to put the try/except block, inside or outside the function
2. How to deal with un-anticipated exceptions
3. How to keep record
My personal feelings are:
1. Kind of prefer try/except block outside the function though. This way it looks clean and easy to follow. In terms of logging the url, since you pass it to the function anyway, there should be ways to keep this option, even with the block written outside the function.
2. From code development perspective, would suggest you follow Test Driven Development (TDD) approach. Nobody can anticipate all of the possible outcomes, but I'm sure you have a pretty good idea about the most likely scenarios, and just make sure that your code is suitable for these scenarios. When future new exceptions arise, you can always refactor your code as needed.
3. Feel it's a personal choice here as being pointed out earlier. No mater which way you go, you just have to follow it up and deal with it, trace it back to the root cause. I kind of prefer logging, so to keep a good record for myself, and you can make it very informative.
Hope this helps.
All the best,
Tian

[toc] | [prev] | [next] | [standalone]

#98552

Hi, 

You are right. I am trying to address a few questions at the same time.
As English is not my first language, I can only say that you have addressed them very well. Thanks. 
1. Where to put the try/except block, inside or outside the function 
2. How to deal with un-anticipated exceptions 
3. How to keep record 

1.	If try/except block is outside the function, I have to always put function call in try/except block, even for logging. From my point of view, I could put at least logging in the function and then re raise the error. 
2.	I have just read what TDD is. I should write my first unit test to get a grip. Anyway, I have a program in which I am looping through several of records. If something happens I would like to put the warning in the log. In that case I need to put for loop in try/except Exception as e: block in order to log an unknown exception 
3.	For me, logging is necessary. Every script (only few of them) I have done, has a underlying log.

Thanks.

Regards.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web