Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #79373 > unrolled thread

Re: When to use assert

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: When to use assert Anton <anton.schattenfeld@gmail.com> - 2014-10-21 00:48 -0700
    Re: When to use assert Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-10-22 12:44 +1100
      Re: When to use assert Chris Angelico <rosuav@gmail.com> - 2014-10-22 12:53 +1100
        Re: When to use assert Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-10-23 02:49 +1100
          Re: When to use assert Chris Angelico <rosuav@gmail.com> - 2014-10-23 03:01 +1100
          Re: When to use assert Dan Stromberg <drsalists@gmail.com> - 2014-10-22 09:18 -0700
            Re: When to use assert Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-10-24 18:49 +1100
              Re: When to use assert Chris Angelico <rosuav@gmail.com> - 2014-10-24 19:18 +1100
                Re: When to use assert Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-10-26 20:39 +1100
                  Re: When to use assert Chris Angelico <rosuav@gmail.com> - 2014-10-26 20:51 +1100

#79373 — Re: When to use assert

On Saturday, November 16, 2013 11:35:50 PM UTC-8, Steven D'Aprano wrote:
> The question of when to use the assert statement comes up occasionally, 
> 
> usually in response to somebody misusing it, so I thought I'd write a 
> 
> post describing when and why to use assertions, and when not to.
> 
> 
> 
> For those who aren't aware of it, Python's "assert" checks a condition, 
> 
> if it is true it does nothing, and if it is false it raises an 
> 
> AssertionError with an optional error message. For example:
> 
> 
> 
> py> x = 23
> 
> py> assert x > 0, "x is not zero or negative"
> 
> py> assert x%2 == 0, "x is not an even number"
> 
> Traceback (most recent call last):
> 
>   File "<stdin>", line 1, in <module>
> 
> AssertionError: x is not an even number
> 
> 
> 
> 
> 
> Many people use asserts as a quick and easy way to raise an exception if 
> 
> an argument is given the wrong value. But this is wrong, dangerously 
> 
> wrong, for two reasons. The first is that AssertionError is usually the 
> 
> wrong error to give when testing function arguments. You wouldn't write 
> 
> code like this:
> 
> 
> 
> if not isinstance(x, int):
> 
>     raise AssertionError("not an int")
> 
> 
> 
> you'd raise TypeError instead. "assert" raises the wrong sort of 
> 
> exception.
> 
> 
> 
> But, and more dangerously, there's a twist with assert: it can be 
> 
> compiled away and never executed, if you run Python with the -O or -OO 
> 
> optimization flags, and consequently there is no guarantee that assert 
> 
> statements will actually be run. When using assert properly, this is a 
> 
> feature, but when assert is used inappropriately, it leads to code that 
> 
> is completely broken when running with the -O flag.
> 
> 
> 
> When should use assert? In no particular order, assertions should be used 
> 
> for:
> 
> 
> 
> * defensive programming;
> 
> * runtime checks on program logic;
> 
> * checking contracts (e.g. pre-conditions and post-conditions);
> 
> * program invariants; and 
> 
> * checked documentation.
> 
> 
> 
> (It's also acceptable to use assert when testing code, as a sort of quick-
> 
> and-dirty poor man's unit testing, so long as you accept that the tests 
> 
> simply won't do anything if you run with the -O flag. And I sometimes use 
> 
> "assert False" in code to mark code branches that haven't been written 
> 
> yet, and I want them to fail. Although "raise NotImplementedError" is 
> 
> probably better for that, if a little more verbose.)
> 
> 
> 
> Opinions on assertions vary, because they can be a statement of 
> 
> confidence about the correctness of the code. If you're certain that the 
> 
> code is correct, then assertions are pointless, since they will never 
> 
> fail and you can safely remove them. If you're certain the checks can 
> 
> fail (e.g. when testing input data provided by the user), then you dare 
> 
> not use assert since it may be compiled away and then your checks will be 
> 
> skipped.
> 
> 
> 
> It's the situations in between those two that are interesting, times when 
> 
> you're certain the code is correct but not *quite* absolutely certain. 
> 
> Perhaps you've missed some odd corner case (we're all only human). In 
> 
> this case an extra runtime check helps reassure you that any errors will 
> 
> be caught as early as possible rather than in distant parts of the code.
> 
> 
> 
> (This is why assert can be divisive. Since we vary in our confidence 
> 
> about the correctness of code, one person's useful assert is another 
> 
> person's useless runtime test.)
> 
> 
> 
> Another good use for asserts is checking program invariants. An invariant 
> 
> is some condition which you can rely on to be true unless a bug causes it 
> 
> to become false. If there's a bug, better to find out as early as 
> 
> possible, so we make a test for it, but we don't want to slow the code 
> 
> down with such tests. Hence assert, which can be turned on in development 
> 
> and off in production.
> 
> 
> 
> An example of an invariant might be, if your function expects a database 
> 
> connection to be open when it starts, and promises that it will still be 
> 
> open when it returns, that's an invariant of the function:
> 
> 
> 
> def some_function(arg):
> 
>     assert not DB.closed()
> 
>     ... # code goes here
> 
>     assert not DB.closed()
> 
>     return result
> 
> 
> 
> 
> 
> Assertions also make good checked comments. Instead of writing a comment:
> 
> 
> 
> # when we reach here, we know that n > 2
> 
> 
> 
> you can ensure it is checked at runtime by turning it into an assert:
> 
> 
> 
> assert n > 2
> 
> 
> 
> Assertions are also a form of defensive programming. You're not 
> 
> protecting against errors in the code as it is now, but protecting 
> 
> against changes which introduce errors later. Ideally, unit tests will 
> 
> pick those up, but let's face it, even when tests exist at all, they're 
> 
> often incomplete. Build-bots can be down and nobody notices for weeks, or 
> 
> people forget to run tests before committing code. Having an internal 
> 
> check is another line of defence against errors sneaking in, especially 
> 
> those which don't noisily fail but cause the code to malfunction and 
> 
> return incorrect results.
> 
> 
> 
> Suppose you have a series of if...elif blocks, where you know ahead of 
> 
> time what values some variable is expected to have:
> 
> 
> 
> # target is expected to be one of x, y, or z, and nothing else.
> 
> if target == x:
> 
>     run_x_code()
> 
> elif target == y:
> 
>     run_y_code()
> 
> else:
> 
>     run_z_code()
> 
> 
> 
> 
> 
> Assume that this code is completely correct now. But will it stay 
> 
> correct? Requirements change. Code changes. What happens if the 
> 
> requirements change to allow target = w, with associated action 
> 
> run_w_code? If we change the code that sets target, but neglect to change 
> 
> this block of code, it will wrongly call run_z_code() and Bad Things will 
> 
> occur. It would be good to write this block of code defensively, so that 
> 
> it will either be correct, or fail immediately, even in the face of 
> 
> future changes.
> 
> 
> 
> The comment at the start of the block is a good first step, but people 
> 
> are notorious for failing to read and update comments. Chances are it 
> 
> will soon be obsolete. But with an assertion, we can both document the 
> 
> assumptions of this block, and cause a clean, immediate failure if they 
> 
> are violated:
> 
> 
> 
> assert target in (x, y, z)
> 
> if target == x:
> 
>     run_x_code()
> 
> elif target == y:
> 
>     run_y_code()
> 
> else:
> 
>     assert target == z
> 
>     run_z_code()
> 
> 
> 
> 
> 
> Here, the assertions are both defensive programming and checked 
> 
> documentation. I consider this to be a far superior solution than this:
> 
> 
> 
> if target == x:
> 
>     run_x_code()
> 
> elif target == y:
> 
>     run_y_code()
> 
> elif target == z:
> 
>     run_z_code()
> 
> else:
> 
>     # This can never happen. But just in case it does...
> 
>     raise RuntimeError("an unexpected error occurred")
> 
> 
> 
> 
> 
> This tempts some helpful developer to "clean it up" by removing the 
> 
> "unnecessary" test for value==c and removing the "dead code" of the 
> 
> RuntimeError. Besides, "unexpected error" messages are embarrassing when 
> 
> they occur, and they will.
> 
> 
> 
> Design by contract is another good use of assertions. In design by 
> 
> contract, we consider that functions make "contracts" with their callers. 
> 
> E.g. something like this:
> 
> 
> 
> "If you pass me an non-empty string, I guarantee to return the first 
> 
> character of that string converted to uppercase."
> 
> 
> 
> If the contract is broken by either the function or the code calling it, 
> 
> the code is buggy. We say that functions have pre-conditions (the 
> 
> constraints that arguments are expected to have) and post-conditions (the 
> 
> constraints on the return result). So this function might be coded as:
> 
> 
> 
> def first_upper(astring):
> 
>     assert isinstance(astring, str) and len(astring) > 0
> 
>     result = astring[0].upper()
> 
>     assert isinstance(result, str) and len(result) == 1
> 
>     assert result == result.upper()
> 
>     return result
> 
> 
> 
> 
> 
> The aim of Design By Contract is that in a correct program, the pre-
> 
> conditions and post-conditions will always hold. Assertions are typically 
> 
> used, since (so the idea goes) by the time we release the bug-free 
> 
> program and put it into production, the program will be correct and we 
> 
> can safely remove the checks.
> 
> 
> 
> Here's my advice when *not* to use assertions:
> 
> 
> 
> * Never use them for testing user-supplied data, or for anything 
> 
>   where the check must take place under all circumstances.
> 
> 
> 
> * Don't use assert for checking anything that you expect might fail
> 
>   in the ordinary use of your program. Assertions are for extraordinary
> 
>   failure conditions. Your users should never see an AssertionError;
> 
>   if they do, it's a bug to be fixed.
> 
> 
> 
> * In particular, don't use assert just because it's shorter than an
> 
>   explicit test followed by a raise. Assert is not a shortcut for
> 
>   lazy coders.
> 
> 
> 
> * Don't use them for checking input arguments to public library 
> 
>   functions (private ones are okay) since you don't control the 
> 
>   caller and can't guarantee that it will never break the 
> 
>   function's contract.
> 
> 
> 
> * Don't use assert for any error which you expect to recover from.
> 
>   In other words, you've got no reason to catch an AssertionError
> 
>   exception in production code.
> 
> 
> 
> * Don't use so many assertions that they obscure the code.
> 
> 
> 
> 
> 
> 
> 
> -- 
> 
> Steven


I use ORM and often need to write a function that either takes an id of the record or already loaded model object.
So I end up writing a piece of code like below:

def do_something(instance):
    if isinstance(instance_or_id, int):
    	instance = Model.get(instance)

	assert isinstance(instance, Model)

	# Code that assumes that instance is an object of type Model
   
do_somthing is not a part of public library, though it is a public function, which can and intended to be used by other programmers; and assert should never happen if a client uses the function as planned.
I wonder if this use-case is controversial to this part:

> Many people use asserts as a quick and easy way to raise an exception if 
> 
> an argument is given the wrong value. But this is wrong, dangerously 
> 
> wrong, for two reasons. The first is that AssertionError is usually the 
> 
> wrong error to give when testing function arguments. You wouldn't write 
> 
> code like this:
> 
> 
> 
> if not isinstance(x, int):
> 
>     raise AssertionError("not an int")
> 
> 
> 
> you'd raise TypeError instead. "assert" raises the wrong sort of 
> 
> exception.
> 

Thanks,
Anton.

[toc] | [next] | [standalone]

#79407

Anton wrote:

> I use ORM and often need to write a function that either takes an id of
> the record or already loaded model object. So I end up writing a piece of
> code like below:
> 
> def do_something(instance):
>     if isinstance(instance_or_id, int):
>         instance = Model.get(instance)
>         assert isinstance(instance, Model)
>         # Code that assumes that instance is an object of type Model
>    
> do_somthing is not a part of public library, though it is a public
> function, which can and intended to be used by other programmers; and
> assert should never happen if a client uses the function as planned.

I think you mean the assert should never fail.

That seems like a reasonable use for assert, with a proviso below. It's
behaving like a checked comment or a post-condition test: asserting that
Model.get returns a Model instance.

But, the idea of *requiring* Model.get to return a Model instance may be
inadvisable. It goes against duck-typing, and it prevents Model from making
some kinds of implementation changes that might break your post-condition
that get() always returns an instance. For example, it might return a proxy
object instead, and then your assert will fail.


> I 
> wonder if this use-case is controversial to this part:
> 
>> Many people use asserts as a quick and easy way to raise an exception if
>> an argument is given the wrong value. But this is wrong, dangerously
>> wrong, for two reasons. The first is that AssertionError is usually the
>> wrong error to give when testing function arguments. You wouldn't write
>> code like this:
>> 
>> if not isinstance(x, int):
>>     raise AssertionError("not an int")
>> 
>> you'd raise TypeError instead. "assert" raises the wrong sort of
>> exception.

No, because the nature of the exception depends on the intent of the test
and the audience who is expected to see it. In an ideal world,
AssertionError should never be seen by the end user, or the developer
calling your code (assuming that she obeys the documented requirements of
your code). A failed assert should be considered an internal error, which
the user never sees. Since the failure:

"Model.get has stopped returning Model instances"

is likely to be an internal problem ("oops, I broke the get() method, better
fix that") rather than an expected error, using assert is okay.

What would *not* be okay is something like this:


def do_something(instance_or_id):
    if isinstance(instance_or_id, int):
        instance = Model.get(instance_or_id)
    assert isinstance(instance, Model)
    # Code that assumes that instance is an object of type Model


since that fails with (for example) do_something(None): either the assert is
not checked at all, and there will be some mysterious failure deep inside
your code, or the caller will see AssertionError instead of TypeError,
violating user expectations and good design that type errors should raise
TypeError.

A better way to write this might be to have Model.get() responsible for the
error checking, and then just delegate to it:

class Model:
    def get(self, obj):
        if isinstance(obj, Model):
            return obj
        elif isinstance(obj, int):
            model = Model("do something here")
            return model
        raise TypeError('expected an int ID or a Model instance')


def do_something(instance_or_id):
    instance = Model.get(instance_or_id)
    assert isinstance(instance, Model)
    # Code that assumes that instance is an object of type Model


That means that the logic for what is acceptable as a Model is all in one
place, namely the Model.get method, and callers don't need to care about
the pre-condition "argument is a Model or an integer ID", they only need to
care about the post-condition "result of get() is a Model".



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#79408

On Wed, Oct 22, 2014 at 12:44 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> def do_something(instance_or_id):
>     instance = Model.get(instance_or_id)
>     assert isinstance(instance, Model)
>     # Code that assumes that instance is an object of type Model
>
>
> That means that the logic for what is acceptable as a Model is all in one
> place, namely the Model.get method, and callers don't need to care about
> the pre-condition "argument is a Model or an integer ID", they only need to
> care about the post-condition "result of get() is a Model".

And at that point, the assertion is redundant, on par with:

a = len(seq)
assert isinstance(a, int)

because you shouldn't have to assert what's part of a function's guarantee.

ChrisA

[toc] | [prev] | [next] | [standalone]

#79450

Chris Angelico wrote:

> On Wed, Oct 22, 2014 at 12:44 PM, Steven D'Aprano
> <steve+comp.lang.python@pearwood.info> wrote:
>> def do_something(instance_or_id):
>>     instance = Model.get(instance_or_id)
>>     assert isinstance(instance, Model)
>>     # Code that assumes that instance is an object of type Model
>>
>>
>> That means that the logic for what is acceptable as a Model is all in one
>> place, namely the Model.get method, and callers don't need to care about
>> the pre-condition "argument is a Model or an integer ID", they only need
>> to care about the post-condition "result of get() is a Model".
> 
> And at that point, the assertion is redundant, on par with:
> 
> a = len(seq)
> assert isinstance(a, int)
> 
> because you shouldn't have to assert what's part of a function's
> guarantee.

That depends on how well you trust the function, how paranoid you are, and
whether you wish to include a checked comment, among other considerations.
I'd prefer to write the above than:

a = len(seq)
# At this point, a is an int.

because comments inside code that aren't checked are technically known
as "lies" <http://import-that.dreamwidth.org/956.html>. Ha ha only serious.

I wouldn't write such a comment for len() since I would expect anyone
reading the code to know what len() does, but the same doesn't necessarily
apply for every function call I make.

Checking the post-condition of a built-in like len() is too paranoid for my
tastes, as len() enforces the rule that __len__() methods return a
non-negative integer, and there are millions of lines of Python code around
the world calling len(). Somebody surely have noticed by now if len()
violated that post-condition. But for a function from my own code base,
where I might only have dozens of users (oh to have that many!) and fewer
unit tests than perhaps I ought to, I might not be quite so confident that
the assertion was redundant. To err is human, and so there are occasions
when it is appropriate to trust but verify.

As I wrote:

    This is why assert can be divisive. Since we can vary in our 
    confidence about the correctness of code, one person's useful 
    assertion may be another person's useless runtime test.

http://import-that.dreamwidth.org/676.html


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#79452

On Thu, Oct 23, 2014 at 2:49 AM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> Chris Angelico wrote:
>
>> And at that point, the assertion is redundant, on par with:
>>
>> a = len(seq)
>> assert isinstance(a, int)
>>
>> because you shouldn't have to assert what's part of a function's
>> guarantee.
>
> That depends on how well you trust the function, how paranoid you are, and
> whether you wish to include a checked comment, among other considerations.
> I'd prefer to write the above than:
>
> a = len(seq)
> # At this point, a is an int.
>
> because comments inside code that aren't checked are technically known
> as "lies" <http://import-that.dreamwidth.org/956.html>. Ha ha only serious.
> ...
> Checking the post-condition of a built-in like len() is too paranoid for my
> tastes...
>
> As I wrote:
>
>     This is why assert can be divisive. Since we can vary in our
>     confidence about the correctness of code, one person's useful
>     assertion may be another person's useless runtime test.

I agree that the assert is preferable to the comment. But maybe my
level of paranoia is just lower than most people's, as I wouldn't
bother checking the post-conditions of pretty much anything. Do you
put these assertions every time you call the function? Every time
you're depending on its return value? At what point should you stop
writing Python code and start using a language with a lot more
boilerplate and checking (I believe Haskell fits that, though I'm not
overly familiar with the language)?

This is the job of a test suite. You don't pepper your code with
assertions to the effect that "I just pushed something onto my queue,
it should now have this item in it"; you create a test case for it,
and verify your function there. In the rest of the code, you trust
that your test suite passes, and don't waste time with assertions.

Or is that insufficiently paranoid?

ChrisA

[toc] | [prev] | [next] | [standalone]

#79454

On Wed, Oct 22, 2014 at 9:01 AM, Chris Angelico <rosuav@gmail.com> wrote:
> On Thu, Oct 23, 2014 at 2:49 AM, Steven D'Aprano
> <steve+comp.lang.python@pearwood.info> wrote:
>> Chris Angelico wrote:
> I agree that the assert is preferable to the comment. But maybe my
> level of paranoia is just lower than most people's, as I wouldn't
> bother checking the post-conditions of pretty much anything. Do you
> put these assertions every time you call the function? Every time
> you're depending on its return value? At what point should you stop
> writing Python code and start using a language with a lot more
> boilerplate and checking (I believe Haskell fits that, though I'm not
> overly familiar with the language)?

I like to use assertions and "if cond: raise ValueError('foo')" a lot.

I think Eiffel may be the poster-child for a language with
pre-conditions, post-conditions and assertions.

I think you're in good company - a lot of developers don't use assertions much.

I like assertions, because they tend to stop bugs pretty quickly.  If
you have 3 functions, one calling another calling another, assertions
in each can keep you from having to backtrack among them when
debugging, instead going directly to the problem's source.

> This is the job of a test suite.

Test suites are great, and I can't really question your reliance on
them.  I love having lots of automated tests.  But for the reason I
described above, I still like having lots of assertions.

> You don't pepper your code with
> assertions to the effect that "I just pushed something onto my queue,
> it should now have this item in it"; you create a test case for it,
> and verify your function there. In the rest of the code, you trust
> that your test suite passes, and don't waste time with assertions.

I wouldn't test that a value was added to a queue immediately after
adding it.  That's excessive, and may even require an abstraction
violation.

But if, for example, I have a string with 3 known-good values, I'll
if/elif/elif/else, and make the else raise an AssertionError.  The
assertion should never fire, but if the code changes, it could, and if
there's a typo somewhere, it could then too.

> Or is that insufficiently paranoid?

With good tests, you're probably fine.

[toc] | [prev] | [next] | [standalone]

#79560

Dan Stromberg wrote:

> I like to use assertions and "if cond: raise ValueError('foo')" a lot.
> 
> I think Eiffel may be the poster-child for a language with
> pre-conditions, post-conditions and assertions.

Yes. I don't think Eiffel is the only language with explicit support for
testing invariants, but it's probably the best known one.

https://archive.eiffel.com/doc/online/eiffel50/intro/language/invitation-07.html

Others include Ada2012, Mercury, D, Perl6, Cobra, Clojure, and others. Cobra
is especially interesting as the syntax is heavily influenced by Python's.

http://cobra-language.com/trac/cobra/wiki/Contracts


> I think you're in good company - a lot of developers don't use assertions
> much.

"You" in this context is Chris Angelico.


> I like assertions, because they tend to stop bugs pretty quickly.  If
> you have 3 functions, one calling another calling another, assertions
> in each can keep you from having to backtrack among them when
> debugging, instead going directly to the problem's source.

Yes, the purpose of assertions is to help errors be discovered as close to
the cause as possible, rather than somewhere much later on. Consider:

addresses = [get_address(name) for name in database]
# ... much later on ...
for i, address in enumerate(addresses):
    if some_condition():
        addresses[i] = modify(address)
# ... much later on ...
for address in list_of_addresses:
    process_non_empty_address(address)

where you can't easily modify or check the get_address() and modify()
functions. If you have a failure in process_non_empty_address, due to a
violation of the "address must not be empty" invariant, which function is
to blame?

You could wrap them and find out that way:

from somewhere import get_address as _get_addr

def get_address(*args, **kwargs):
    result = _get_addr(*args, **kwargs)
    if not result:
        raise RuntimeError("bug in get_address")
    return result


and under some circumstances that's a good strategy, but often you just want
to determine which function is violating the constraint, fix that one, and
leave the other unmodified.

addresses = [get_address(name) for name in database]
assert all(address for address in addresses)
# ... much later on ...
for i, address in enumerate(addresses):
    if some_condition():
        addresses[i] = modify(address)
        assert addresses[i]


will either identify the culprit, or at least prove that neither
get_address() nor modify() are to blame. Because you're using an assertion,
it's easy to leave the asserts in place forever, and disable them by
passing -O to the Python interpreter in production.

[Aside: it would be nice if Python did it the other way around, and require
a --debugging switch to turn assertions on. Oh well.]


>> This is the job of a test suite.
> 
> Test suites are great, and I can't really question your reliance on
> them.  I love having lots of automated tests.  But for the reason I
> described above, I still like having lots of assertions.

Assertions and test suites are complementary, not in opposition, like belt
and braces. Assertions insure that the code branch will be tested if it is
ever exercised, something test suites can't in general promise. Here's a
toy example:

def some_function(value):
    import random
    random.seed(value)
    if random.random() == 0.25000375:
        assert some_condition
    else:
        pass


Try writing a unit test that guarantees to test the some_condition
branch :-)

[Actually, it's not that hard, if you're willing to monkey-patch the random
module. But you may have reasons for wanting to avoid such drastic
measures.]

I don't know what value will cause some_function() to take the
some_condition branch, but I know that if it ever takes that branch, the
assert will guard it, regardless of whether or not I've written a unit test
to cover that situation.


>> You don't pepper your code with
>> assertions to the effect that "I just pushed something onto my queue,
>> it should now have this item in it"; you create a test case for it,
>> and verify your function there. In the rest of the code, you trust
>> that your test suite passes, and don't waste time with assertions.
> 
> I wouldn't test that a value was added to a queue immediately after
> adding it.  That's excessive, and may even require an abstraction
> violation.
> 
> But if, for example, I have a string with 3 known-good values, I'll
> if/elif/elif/else, and make the else raise an AssertionError.  The
> assertion should never fire, but if the code changes, it could, and if
> there's a typo somewhere, it could then too.

I like this style:

assert x in (a, b, c)
if x == a:
    do_this()
elif x == b:
    do_that()
else:
    assert x == c
    do_something_else()


Why do I prefer that? To defend against future code changes. Defensive
programming can defend not only against bugs in the current code, but also
bugs in *future* code. Consider this common case:

# x is either a or b.
if x == a:
    do_this()
elif x == b:
    do_that()
else:  # x must be c.
    do_something_else()

Comments lie, and people don't read them. The comment at the top of the
suite is already out of date. I'm sure every experienced programmer has
seen situations like that -- or even written code like that. What happens
when the requirements change and x can also take the value d?

With an eye to defensive programming, the first improvement is:

# x is a, b or c.
if x == a:
    do_this()
elif x == b:
    do_that()
elif x == c:
    do_something_else()
else:
    # This cannot possibly happen, but just in case it does...
    raise RuntimeError('an unexpected error occurred')


except that still relies on the next maintainer reading the comment. It's
vulnerable to somebody "fixing" the code by removing the dead code and the
redundant test for x == c. Besides, it's embarrassing to see "an unexpected
error occurred" messages in production, and you know you will.

A better solution is to use assertions as checked comments. If the
requirements change and you forget to update this part of the code, the
assertions will fail:

assert x in (a, b, c)
if x == a:
    do_this()
elif x == b:
    do_that()
else:
    assert x == c
    do_something_else()


>> Or is that insufficiently paranoid?
> 
> With good tests, you're probably fine.

Is it possible to be too paranoid when it comes to tests?

:-)


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#79563

On Fri, Oct 24, 2014 at 6:49 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> addresses = [get_address(name) for name in database]
> assert all(address for address in addresses)
> # ... much later on ...
> for i, address in enumerate(addresses):
>     if some_condition():
>         addresses[i] = modify(address)
>         assert addresses[i]
>
>
> will either identify the culprit, or at least prove that neither
> get_address() nor modify() are to blame. Because you're using an assertion,
> it's easy to leave the asserts in place forever, and disable them by
> passing -O to the Python interpreter in production.

The first assertion is fine, assuming that the emptiness of your
address corresponds to falsiness as defined in Python. (This could be
safe to assume, if the address is an object that knows how to boolify
itself.) The second assertion then proves that modify() isn't
returning nothing, but that might be better done by sticking the
assertion into modify itself.

And that's what I'm talking about: checking a function's postcondition
with an assert implies putting that assertion after every call, and
anything that you have to do every time you call a function belongs
inside that function. Imagine writing this kind of defensive code, and
then having lots of places that call modify()... and missing the
assert in some of them. Can you trust what you're getting back?

> [Aside: it would be nice if Python did it the other way around, and require
> a --debugging switch to turn assertions on. Oh well.]

Maybe. But unless someone actually tests that their assertions are
being run, there's the risk that they're flying blind and assuming
that it's all happening. There'll be all these lovely "checked
comments"... or so people think. Nobody ever runs the app with
--debugging, so nobody ever sees anything.

> Assertions and test suites are complementary, not in opposition, like belt
> and braces. Assertions insure that the code branch will be tested if it is
> ever exercised, something test suites can't in general promise. Here's a
> toy example:
>
> def some_function(value):
>     import random
>     random.seed(value)
>     if random.random() == 0.25000375:
>         assert some_condition
>     else:
>         pass
>
>
> Try writing a unit test that guarantees to test the some_condition
> branch :-)
>
> [Actually, it's not that hard, if you're willing to monkey-patch the random
> module. But you may have reasons for wanting to avoid such drastic
> measures.]

Easy. You craft a test case that passes the right argument, and then
have the test case test itself. You will want to have a script that
figures out what seed value to use:

def find_seed(value):
    import random
    for seed in range(1000000): # Restrict to not-too-many tries
        random.seed(seed)
        if random.random() == value: return seed

I didn't say it'd be efficient to run, but hey, it's easy. I've no
idea how many bits of internal state the default Python RNGs use, but
testing a million seeds took a notable amount of time, so I told it to
fail after that many. (And I didn't find one that gave that result.)

But actually, it would be really simple to monkey-patch. And in any
non-toy situation, there's probably something more significant being
tested here... unless you really are probing a random number generator
or something, in which case you probably know more about its
internals.

> I like this style:
>
> assert x in (a, b, c)
> if x == a:
>     do_this()
> elif x == b:
>     do_that()
> else:
>     assert x == c
>     do_something_else()

If all your branches are simple function calls, I'd be happy with a
KeyError instead of an AssertionError or RuntimeError.

{a:do_this, b:do_that, c:do_something_else}[x]()

I was talking to a student this week who had a long if/elif chain that
translated keywords into values, something like this:

def get_whatever_value(kwd):
    if kwd == 'value_should_be_50':
        return 50
    elif kwd == 'value_wants_to_be_75':
        return 75
    elif kwd == 'one_hundred':
        return 100

There was no 'else' clause, so in the event of an incorrect keyword,
it would return None. Now, I could have advised adding an "else
ValueError" or an assertion, but my preferred technique here is a
simple dict lookup. Simpler AND guarantees that all inputs are
checked.

>>> Or is that insufficiently paranoid?
>>
>> With good tests, you're probably fine.
>
> Is it possible to be too paranoid when it comes to tests?

Yeah, it is. I said earlier about checking that len() returns an
integer. The only way[1] for len(some_object) to return a non-integer
is for someone to have shadowed len, and if you're asserting to see if
someone's maliciously shadowing builtins, you *really* need a hobby.
But hey. Maybe asserting *is* your hobby!

ChrisA

[1] Cue the response pointing out some way that it'll return something
else. I wasn't able to break it, though.

[toc] | [prev] | [next] | [standalone]

#79710

Chris Angelico wrote:

> On Fri, Oct 24, 2014 at 6:49 PM, Steven D'Aprano
> <steve+comp.lang.python@pearwood.info> wrote:
>> addresses = [get_address(name) for name in database]
>> assert all(address for address in addresses)
>> # ... much later on ...
>> for i, address in enumerate(addresses):
>>     if some_condition():
>>         addresses[i] = modify(address)
>>         assert addresses[i]
>>
>>
>> will either identify the culprit, or at least prove that neither
>> get_address() nor modify() are to blame. Because you're using an
>> assertion, it's easy to leave the asserts in place forever, and disable
>> them by passing -O to the Python interpreter in production.
> 
> The first assertion is fine, assuming that the emptiness of your
> address corresponds to falsiness as defined in Python. (This could be
> safe to assume, if the address is an object that knows how to boolify
> itself.) The second assertion then proves that modify() isn't
> returning nothing, but that might be better done by sticking the
> assertion into modify itself.

Sure. If you can modify the function, getting it to check it's own
post-condition is a good strategy. But you might not be able to do that.
Perhaps you don't have the source code, or its from a third-party library
and you don't want to have to keep hacking your copy every time it's
updated, or you're calling an external library written in Java.

(And yes, there are work-arounds for those scenarios too. Aren't choices
wonderful?)


> And that's what I'm talking about: checking a function's postcondition
> with an assert implies putting that assertion after every call, 

There's no rule that you have to use an assertion after *every* call to a
function. If you can check the post-condition inside the function, that's
great. If you can't, then use your judgement. More asserts are not
necessarily better, just as more unit tests are not necessarily better.

You have to make a value judgement based on how much you trust the code,
trust your own skills, trust your colleagues who will be maintaining the
code in your absence, versus the maximum density of asserts per line of
other code.


> and 
> anything that you have to do every time you call a function belongs
> inside that function. Imagine writing this kind of defensive code, and
> then having lots of places that call modify()... and missing the
> assert in some of them. Can you trust what you're getting back?

Well, if the function is perfect, then adding more assertions won't uncover
bugs that aren't there. And if the function is buggy, then adding more
assertions will increase the likelihood that you'll discover those bugs
earlier rather than later. How much do you trust the function?

One test is better than no tests at all, but a thousand tests is not
necessarily better than ten. The same applies to assertions. One might say
that assertions can be considered tests that automatically run every time
you run your code (in debugging mode), rather than needing to remember to
run a special test program.

In the absence of correctness proofs for your code, anything you do (unit
tests, assertions, regression tests, etc.) is just a statistically sampling
of all the potential paths your code might take, hoping to capture bugs. As
any statistician will tell you, eventually you will reach the point of
diminishing returns, where adding more samples isn't worth the cost. That
point will depend on the cost in programmer effort, the performance
implications, and your confidence in the code.

 
>> [Aside: it would be nice if Python did it the other way around, and
>> [require
>> a --debugging switch to turn assertions on. Oh well.]
> 
> Maybe. But unless someone actually tests that their assertions are
> being run, there's the risk that they're flying blind and assuming
> that it's all happening. There'll be all these lovely "checked
> comments"... or so people think. Nobody ever runs the app with
> --debugging, so nobody ever sees anything.

Absolutely. And if you write unit tests but never actually run them, the
same thing applies. You need a process that actually runs the code in
debugging mode, and/or runs the tests.

Or if the unit tests all pass but don't actually do anything useful?

    def test_critical_code_path(self):
        # Make sure application works correctly.
        self.assertTrue(True)

Do I have to link to the DailyWTF showing that some people actually do write
tests like this? Sure, why not :-) 

http://thedailywtf.com/articles/Productive-Testing

Bad programmers can screw up anything.


>> Assertions and test suites are complementary, not in opposition, like
>> belt and braces. Assertions insure that the code branch will be tested if
>> it is ever exercised, something test suites can't in general promise.
>> Here's a toy example:
>>
>> def some_function(value):
>>     import random
>>     random.seed(value)
>>     if random.random() == 0.25000375:
>>         assert some_condition
>>     else:
>>         pass
>>
>>
>> Try writing a unit test that guarantees to test the some_condition
>> branch :-)
>>
>> [Actually, it's not that hard, if you're willing to monkey-patch the
>> [random
>> module. But you may have reasons for wanting to avoid such drastic
>> measures.]
> 
> Easy. You craft a test case that passes the right argument, and then
> have the test case test itself. You will want to have a script that
> figures out what seed value to use:

The Mersenne Twister isn't cryptographically strong, but even so, I'm pretty
sure that it is computationally infeasible unless you get amazingly lucky.
Given 624 sequential random values, it is possible to predict all
subsequent values. But predicting what seed will produce a specific value
is probably ... difficult. But feel free to try it:

> def find_seed(value):
>     import random
>     for seed in range(1000000): # Restrict to not-too-many tries
>         random.seed(seed)
>         if random.random() == value: return seed
> 
> I didn't say it'd be efficient to run, but hey, it's easy. 

Using the same logic, it's "easy" to break sha512 checksums. Just iterate
over all possible files the same size as the target file until you find one
which has the same checksum.

> I've no 
> idea how many bits of internal state the default Python RNGs use, 

A *lot*.

import random
random.getstate()

Enjoy :-)


> But actually, it would be really simple to monkey-patch. 

Like I said :-)

But just because you can monkey-patch, doesn't mean you will. Perhaps you
have a policy not to, or just don't like doing so because you're still
recovering from the pain of working with Ruby developers.

http://devblog.avdi.org/2008/02/23/why-monkeypatching-is-destroying-ruby/


> And in any 
> non-toy situation, there's probably something more significant being
> tested here... unless you really are probing a random number generator
> or something, in which case you probably know more about its
> internals.
> 
>> I like this style:
>>
>> assert x in (a, b, c)
>> if x == a:
>>     do_this()
>> elif x == b:
>>     do_that()
>> else:
>>     assert x == c
>>     do_something_else()
> 
> If all your branches are simple function calls, I'd be happy with a
> KeyError instead of an AssertionError or RuntimeError.
> 
> {a:do_this, b:do_that, c:do_something_else}[x]()

Nah, can't do that, because the "that's an old hack and an obsolete Python
idiom, you have to use nested ternary ifs" crowd will get angry at you.

*wink*

Translating a chain of if...elif to a dict lookup is a good, hopefully not
obsolete, Python idiom. But it doesn't work so well when the blocks are
blocks of code, and it fails completely if any of the blocks contains a
return, break or continue.



[...]
>>>> Or is that insufficiently paranoid?
>>>
>>> With good tests, you're probably fine.
>>
>> Is it possible to be too paranoid when it comes to tests?
> 
> Yeah, it is. 

I was actually trying to be funny. Sorry for the fail.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#79711

On Sun, Oct 26, 2014 at 8:39 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> In the absence of correctness proofs for your code, anything you do (unit
> tests, assertions, regression tests, etc.) is just a statistically sampling
> of all the potential paths your code might take, hoping to capture bugs. As
> any statistician will tell you, eventually you will reach the point of
> diminishing returns, where adding more samples isn't worth the cost. That
> point will depend on the cost in programmer effort, the performance
> implications, and your confidence in the code.

That's about the size of it, I guess. How much testing versus how much
time actually spent coding?

I'm pretty sure a lot of my code has the wrong ratio... I'm just not
sure, for a lot of the code, which direction it's wrong :)

ChrisA

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web