Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #98002 > unrolled thread

Modern recommended exception handling practices?

Back to article view | Back to comp.lang.python

Contents

  Modern recommended exception handling practices? vasudevram <vasudevram@gmail.com> - 2015-10-29 10:43 -0700
    Re: Modern recommended exception handling practices? Steven D'Aprano <steve@pearwood.info> - 2015-11-03 17:47 +1100
      Re: Modern recommended exception handling practices? Chris Angelico <rosuav@gmail.com> - 2015-11-03 18:16 +1100
        Re: Modern recommended exception handling practices? Steven D'Aprano <steve@pearwood.info> - 2015-11-03 18:22 +1100
          Re: Modern recommended exception handling practices? Chris Angelico <rosuav@gmail.com> - 2015-11-03 18:52 +1100
        Re: Modern recommended exception handling practices? Paul Rubin <no.email@nospam.invalid> - 2015-11-03 11:59 -0800

#98002 — Modern recommended exception handling practices?

Hi list,

Are there any modern (i.e. for current versions of Python 2 and 3) recommended exception handling practices?

I've used some simple practices myself that I worked out, and seemed to make sense, but a) they may not cover many common cases (did it for the needs of a few specific projects), and there may of course be better methods.

I've googled for some relevant terms relating to my question, and found some links, including on python.org pages and StackOverflow, but at least the SO one I saw is from '09, so could well be outdated.

I realize that there may not be any recommended standard practices for this, or they may be more than one set, because of differences of opinion. For example, several years back, there was a somewhat controversial thread about, IIRC, checked vs. unchecked Java exceptions, maybe triggered by some post on Elliot Rusty Harold's site (he used to write a lot about Java, both blog posts and books).

Like to hear what people think and have to say about this, anyway, and more importantly, what they do and have found useful in practice.

I also realize my question may be too broad. Please feel free to suggest ways of constraining the problem. Maybe I will myself, in a reply in this thread.

Thanks in advance to all who reply.

- Vasudev Ram
jugad2.blogspot.com

[toc] | [next] | [standalone]

#98134

Hi Vasudev, and welcome.

Sorry for the delay in replying to your questions. My answers inline, below.


On Fri, 30 Oct 2015 04:43 am, vasudevram wrote:

> Are there any modern (i.e. for current versions of Python 2 and 3)
> recommended exception handling practices?

When you say "modern", it sounds like you want to contrast them with "old
fashioned and obsolete" exception-handling practices. I don't know if there
are any obsolete exception-handling practices in Python.

I would follow these rules, as best I am able to:


(1) Remember that, in general, exception catching is very course-grained.
You can catch a particular kind of exception, but you don't know which
specific line of code or expression caused it, the reason why, or where in
the `try` block it was raised. The entire `try` clause is as fine-grained
as you get.

One consequence is that it is easy to write exceptional handling code which
is wrong and buggy:

http://blogs.msdn.com/b/oldnewthing/archive/2004/04/22/118161.aspx

http://blogs.msdn.com/b/oldnewthing/archive/2005/01/14/352949.aspx


To avoid this, the `try` block should cover the least amount of code
possible. For example, suppose you have this:

    try:
        result = calculate(mylist.index(x), func(key))
    except ValueError:
        # thrown by mylist.index if x is not found
        result = "x not found"


That code is probably unsafe, because you can't be sure that either
func(key) or calculate(...) itself might not throw a ValueError. Unless you
can guarantee that there are no circumstances where that might happen, or
if it doesn't matter which of the three expressions throws the ValueError,
you are better off writing this:

    try:
        idx = mylist.index(x)
    except ValueError:
        result = "x not found"
    else:
        result = calculate(idx, func(key))


(2) Catch only the exceptions that you expect and can genuinely recover
from. Don't use `try...except` as a way to cover up and hide the fact that
an error occurred.

One of my favourite quotes about programming is this one:

   "I find it amusing when novice programmers believe their main
    job is preventing programs from crashing. ... More experienced
    programmers realize that correct code is great, code that
    crashes could use improvement, but incorrect code that doesn’t
    crash is a horrible nightmare."
    -- Chris Smith

In context of Python, "crash" means "raise an exception". Exceptions are
your best friend: they show you were your bugs are. How can you fix bugs if
you don't know where they are?

You should only catch the exceptions that you expect to occur, and can
recover from. Everything else should be allowed to raise an exception, so
you can *see* that something is broken. Now you know that something in your
code has a bug, and needs to be fixed.

Consequently, you should never (well, *almost* never) be tempted to write a
bare `except` clause, or even `except Exception`. They are almost always
too broad, and catch too much.

Bare `except` clauses are very possibly *literally the worst* thing that you
can write in Python:

https://realpython.com/blog/python/the-most-diabolical-python-antipattern/


(3) One exception to rule (2) is that sometimes it is useful to surround
your entire main program with a `try...except`, for the purposes of logging
errors and presenting a nice clean error message to your users:

    try:
        main()
    except Exception as err:
        # Catch any and all unexpected exceptions.
        log(err)
        alert("A serious problem has occurred and MyApplication must quit. " 
              "Please contact the help desk on 999-999-999.")
        sys.exit(1)


(4) Remember that Python matches exceptions from top down, so more specific
exceptions (subclasses of a less specific exception) need to come first. So
if we write this:

    try:
        ...
    except Exception:
        print("something unexpected occurred")
    except TypeError:
        print("this is expected")


the TypeError will be caught by the first except clause, and treated as
something unexpected. Swap the order of the except clauses, and you will be
fine:

    try:
        ...
    except TypeError:
        print("this is expected")
    except Exception:
        print("something unexpected occurred")


(5) Remember that often you can avoid exceptions instead of catching
them. "Look Before You Leap" (LBYL) may be a perfectly good alternative:

    if item in mylist:
        idx = mylist.index(item)
        process(idx)
    else:
        result = "not found"


but be sensitive to the amount of work done. The above code searches the
list twice instead of just once.


(6) In general, you can consider setting up a "try" block to be practically
free in Python (which is the opposite to Java, where exception handling is
always expensive). If no exception occurs, there is virtually no difference
in speed between:

    func(x)

and

    try:
        func(x)
    except:
        pass


but if an exception does occur, actually catching it is quite costly. This
idiom is usually called "Easier to Ask for Forgiveness than Permission"
(EAFP).


(7) So which is faster, LBYL or catching the exception? That is extremely
sensitive to not just the specific operations being performed, but how
often the exceptional cases occur. In general, you must measure your code
to know.

But as a very rough rule of thumb, consider looking up a key in a dict:

    if key in mydict:
        result = mydict[key]

versus

    try:
        result = mydict[key]
    except KeyError:
        pass


In my experience, catching the KeyError is about ten times more costly than
testing for the key's presence. So if your keys are missing more than one
time in ten, it's probably better to use LBYL.


(8) Remember that this is called *exception handling*, not "error handling".
In Python, exceptions are not just for errors.

For example, Python extensively uses StopIteration for flow-control. When an
iterator runs out of items, it raises an exception, which the Python
interpreter catches to recognise the end of the iterator. You can do the
same in your own code, invent your own protocols that use exceptions to
signal exceptional cases.

Some very highly respected people like Joel Spolsky and Raymond Chen dislike
exceptions as flow-control, calling them just a thinly disguised GOTO:

http://www.joelonsoftware.com/items/2003/10/13.html

http://blogs.msdn.com/b/oldnewthing/archive/2005/01/06/347666.aspx

and while it is true that exceptions are a kind of GOTO, that doesn't mean
much. For-loops, and if-elif-else are a kind of GOTO too, and we don't
worry about them. (For example, you can't use an exception to jump *into* a
function, only out of one.)


See also:

http://c2.com/cgi/wiki?AvoidExceptionsWheneverPossible

http://c2.com/cgi/wiki?ExceptionsAreOurFriends




[...]
> I realize that there may not be any recommended standard practices for
> this, or they may be more than one set, because of differences of opinion.
> For example, several years back, there was a somewhat controversial thread
> about, IIRC, checked vs. unchecked Java exceptions, maybe triggered by
> some post on Elliot Rusty Harold's site (he used to write a lot about
> Java, both blog posts and books).

Java is, as far as I know, the only language with checked exceptions.
Certainly Python doesn't have them. I know that checked exceptions are very
controversial in the Java community, but I understand that Java's creator
James Gosling eventually came out to say that they were a mistake.

In Python's case, you should document what exceptions you expect your
functions to raise, but you must not assume that such documentation is ever
complete. For example, consider the operator "+". We know that + will raise
a TypeError:

py> 2 + "2"
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for +: 'int' and 'str'

but if one of the operands is a custom type, it could raise *anything*:


py> class X:
...     def __add__(self, other):
...             raise OSError
...
py> X() + 2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in __add__
OSError

Does this mean you should *catch* everything? No. See Rule (2) above. Unless
you are expecting an X instance in your code, the presence of one is
probably a bug. The *unexpected* OSError will be the exception that reveals
this bug, and allows you to fix it.




-- 
Steven

[toc] | [prev] | [next] | [standalone]

#98136

On Tue, Nov 3, 2015 at 5:47 PM, Steven D'Aprano <steve@pearwood.info> wrote:
> On Fri, 30 Oct 2015 04:43 am, vasudevram wrote:
>
>> Are there any modern (i.e. for current versions of Python 2 and 3)
>> recommended exception handling practices?
>
> When you say "modern", it sounds like you want to contrast them with "old
> fashioned and obsolete" exception-handling practices. I don't know if there
> are any obsolete exception-handling practices in Python.

Aside from string exceptions and the "except Type, e:" syntax, I would
agree with you. Actually, I can't think of any "obsolete
exception-handling practices" in any language. Exception handling is
pretty straight-forward: you raise an exception in one place, and you
catch it in another.

> Bare `except` clauses are very possibly *literally the worst* thing that you
> can write in Python:
>
> https://realpython.com/blog/python/the-most-diabolical-python-antipattern/

I would actually like to disallow the bare except, in the same way
that I would disallow the Py2 input() function. In the rare cases
where you really do want to catch *every* exception (eg at a boundary
between a web server and a request handler, where you would log the
exception and return a 500), it should be spelled "except
BaseException [as e]:", same as you should use "eval(raw_input())" in
those extremely rare cases where you actually want to take keyboard
input and evaluate it.

But more generally, the over-broad exception handler is a nasty anti-pattern.

> (5) Remember that often you can avoid exceptions instead of catching
> them. "Look Before You Leap" (LBYL) may be a perfectly good alternative:
>
>     if item in mylist:
>         idx = mylist.index(item)
>         process(idx)
>     else:
>         result = "not found"
>
>
> but be sensitive to the amount of work done. The above code searches the
> list twice instead of just once.

Not to mention having race condition possibilities. There are a few
places where this is useful, though:

start_time = time()
work_done = do_some_work()
time_spent = time()-start_time or 1
print(f"Did {work_done} jobs in {time_spent} secs: {work_done/time_spent} j/s")

The "or 1" is a quick check that means we don't divide by zero. The
performance figure becomes meaningless, but if this is a rare case,
that's probably fine.

> (7) So which is faster, LBYL or catching the exception? That is extremely
> sensitive to not just the specific operations being performed, but how
> often the exceptional cases occur. In general, you must measure your code
> to know.
>
> But as a very rough rule of thumb, consider looking up a key in a dict:
>
>     if key in mydict:
>         result = mydict[key]
>
> versus
>
>     try:
>         result = mydict[key]
>     except KeyError:
>         pass
>
>
> In my experience, catching the KeyError is about ten times more costly than
> testing for the key's presence. So if your keys are missing more than one
> time in ten, it's probably better to use LBYL.

This is actually a tribute to dict performance for key lookup (since
that's one of the important operations in a hashtable). It's NOT the
case for a list.


Exceptions are the "other way" to return something. In a function that
always returns a string, returning None is distinguishable (in the
same way that a C function can return a NULL pointer); in a function
that returns absolutely any object, the only way to signal "no object
to return" is to raise an exception. That's why StopIteration exists,
for instance. Exceptions are a normal part of program flow - they
signal an "exceptional condition" in some small area, but it's normal
to cope with exceptional conditions.

ChrisA

[toc] | [prev] | [next] | [standalone]

#98138

On Tue, 3 Nov 2015 06:16 pm, Chris Angelico wrote:

> Not to mention having race condition possibilities.

Arrggghhh! I knew there was something else I wanted to say.

You're right. Sometimes you *have* to use exception handling code. Take this
for example:


if os.path.exists(pathname):
    f = open(pathname)

That might be good enough for a quick and dirty script, but it's wrong,
because there's a race condition between the time you check whether the
file exists and the time you actually try to open it. (There are other
problems too: just because the file exists doesn't mean you have read
permission to it.) A lot can happen in the few microseconds between
checking for the existence of the file and actually opening it -- the file
could be renamed or deleted.

https://en.wikipedia.org/wiki/Time_of_check_to_time_of_use


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#98139

On Tue, Nov 3, 2015 at 6:22 PM, Steven D'Aprano <steve@pearwood.info> wrote:
> A lot can happen in the few microseconds between
> checking for the existence of the file and actually opening it -- the file
> could be renamed or deleted.

And a lot of microseconds can happen between two opcodes, too. Even
inside a Python script, it's possible for threads or other arbitrary
code execution to get in your way:

if "foo" in counters:
    # context switch here!
    process(counters["foo"])

Garbage collection can happen at any time. Here's an (admittedly
arbitrary) example of how the above could be broken:

>>> class X:
...     def __init__(self, name, dict):
...         self.dict = dict; self.name = name
...         self.dict[self.name] = 0
...     def frob(self):
...         self.dict[self.name] += 1
...     def __del__(self):
...         del self.dict[self.name]
...
>>> counters = {}
>>> x = X("foo", counters)
>>> x.refcycle = x
>>> counters
{'foo': 0}
>>> del x
>>> counters
{'foo': 0}
>>> gc.collect()
10
>>> counters
{}

If the cycle-detecting garbage collector happens to be called
immediately after the 'if', you'll get an exception.

So I suppose what you might do is this:

try:
    # Optimization: Since a lot of these names won't be
    # in the dict, we check first rather than relying on the
    # exception. Since counters get removed in the __del__
    # method, we can't depend 100% on the 'in' check,
    # but an unnecessary try block is cheap.
    if "foo" in counters:
        process(counters["foo"])
except KeyError:
    pass

But any time you need a block comment to justify your code, you'd
better be REALLY sure the performance benefit is worth the complexity.

For reliability, expect exceptions.

ChrisA

[toc] | [prev] | [next] | [standalone]

#98187

Chris Angelico <rosuav@gmail.com> writes:
> Aside from string exceptions and the "except Type, e:" syntax, I would
> agree with you. Actually, I can't think of any "obsolete
> exception-handling practices" in any language. 

I'd say that context managers are a big recent improvement in Python
over dealing with a lot of exceptions explicitly.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web