Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #24621 > unrolled thread

Re: retry many times decorator

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: retry many times decorator andrea crotti <andrea.crotti.0@gmail.com> - 2012-06-28 17:26 +0100
    Re: retry many times decorator Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-06-28 17:43 +0000
      Re: retry many times decorator Andrea Crotti <andrea.crotti.0@gmail.com> - 2012-06-28 22:11 +0100
      Re: retry many times decorator andrea crotti <andrea.crotti.0@gmail.com> - 2012-06-29 09:53 +0100
        Re: retry many times decorator Justin Barber <barber.justin@gmail.com> - 2012-06-29 05:45 -0700
        Re: retry many times decorator Justin Barber <barber.justin@gmail.com> - 2012-06-29 05:45 -0700

#24621 — Re: retry many times decorator

> Returning a boolean isn't very Pythonic. It would be better, IMHO, if
> it could swallow a specified exception (or specified exceptions?)
> raised when an attempt failed, up to the maximum permitted number of
> attempts. If the final attempt fails, propagate the exception.
> --
> http://mail.python.org/mailman/listinfo/python-list


Yes right I also didn't like it..  Now it become something as below,
so I capture every possible exception (since it must be generic) and
log what exception was raised.  I'm not re-raising because if it fails
and it's fatal I should just exit, and if it's not fatal it should
just continue..

class retry_n_times:
    def __init__(self, ntimes=3, timeout=3, fatal=True):
        self.ntimes = ntimes
        self.timeout = timeout
        self.fatal = fatal

    def __call__(self, func):
        def _retry_n_times(*args, **kwargs):
            attempts = 0
            while True:
                logger.debug("Attempt number %s of %s" % (attempts,
func.__name__))
                try:
                    ret = func(*args, **kwargs)
                except Exception as e:
                    logger.error("Got exception %s with error %s" %
(type(e), str(e)))
                    sleep(self.timeout)
                else:
                    return ret

                attempts += 1
                if attempts == self.ntimes:
                    logger.error("Giving up the attempts while running
%s" % func.__name__)
                    if self.fatal:
                        exit(100)

        return _retry_n_times

[toc] | [next] | [standalone]

#24628

On Thu, 28 Jun 2012 17:26:36 +0100, andrea crotti wrote:

>> Returning a boolean isn't very Pythonic. It would be better, IMHO, if
>> it could swallow a specified exception (or specified exceptions?)
>> raised when an attempt failed, up to the maximum permitted number of
>> attempts. If the final attempt fails, propagate the exception. --
>> http://mail.python.org/mailman/listinfo/python-list
> 
> 
> Yes right I also didn't like it..  Now it become something as below, so
> I capture every possible exception (since it must be generic) 

I disagree. If you make a coding error in your function, why do you think 
it is useful to retry that buggy code over and over again? It's never 
going to get less buggy unless you see the exception and fix the bug.

For any operation that you want to retry, identify the *temporary* 
errors, catch them, and retry the request. *Permanent* errors should 
immediately fail, without retrying. *Unexpected* errors should not be 
caught, since they probably represent a bug in your code.

E.g. when trying to read a HTTP resource, there is no point retrying a 
400 Bad Request repeatedly, and it is rather anti-social to keep 
hammering a website when it's already told you that your syntax is wrong. 
Likewise, if you get 401, there is no point in retrying with the exact 
same request -- you need a *different* request, this time using 
authentication. A 403 is a permanent error and there's no point in 
retrying. A 404 might be temporary (although it probably isn't). A 503 or 
504 is likely to be temporary. And so forth.

http://en.wikipedia.org/wiki/List_of_HTTP_status_codes

And if you get an unexpected AttributeError: 'NoneType' object has no 
attribute 'read' (as I just did today, in one of my scripts), that's a 
bug that should be fixed.

Almost any other resource you might try is likely to have similar 
responses: some errors will indicate temporary failures, others will be 
permanent, and some will indicate a bug in your code. I think it is rude 
and pointless to repeatedly retry the same failed request if it cannot 
possibly succeed.

I also find that exponential backoff for the delay is best. E.g. wait 1 
second after the first failed attempt, 2 seconds after the second, 4 
seconds after the third, 8 seconds after the fourth, etc.


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#24636

On 06/28/2012 06:43 PM, Steven D'Aprano wrote:
> On Thu, 28 Jun 2012 17:26:36 +0100, andrea crotti wrote:
>
>
> I disagree. If you make a coding error in your function, why do you think
> it is useful to retry that buggy code over and over again? It's never
> going to get less buggy unless you see the exception and fix the bug.
>
> For any operation that you want to retry, identify the *temporary*
> errors, catch them, and retry the request. *Permanent* errors should
> immediately fail, without retrying. *Unexpected* errors should not be
> caught, since they probably represent a bug in your code.

Ah well maybe I wasn't clear, but I'm not going to retry random things, 
I will only decorate
the functions that I know for sure that could go wrong for temporary 
network problems.

For example they told me that sometimes mysql just doesn't respond in 
time for some reasons,
but there's nothing permanently wrong, so retrying is the best option..

It would be good of course, however, to catch the exceptions that are 
known to be permanent problems
in the function at least, and leave the retry as last resource..

Thanks for the idea of the exponential backoff, which is also a better 
name than timeout for the variable..

[toc] | [prev] | [next] | [standalone]

#24650

On the other hand now that I think again even supposing there is a
permanent error like MySql completely down, retrying continuosly
won't do any harm anyway because the machine will not be able to do
anything else anyway, when someone will fix MySql it would
restart again without human intervention.

So I think I could even just let it retry and use maybe a SMTPHanlder
for the logging errors, to make the notification of problems very
quick..

[toc] | [prev] | [next] | [standalone]

#24652

On Friday, June 29, 2012 4:53:43 AM UTC-4, andrea crotti wrote:
> On the other hand now that I think again even supposing there is a
> permanent error like MySql completely down, retrying continuosly
> won't do any harm anyway because the machine will not be able to do
> anything else anyway, when someone will fix MySql it would
> restart again without human intervention.
> 
> So I think I could even just let it retry and use maybe a SMTPHanlder
> for the logging errors, to make the notification of problems very
> quick..

Rather then write a decorator, sounds like you should write your own class or method connecting to the database that has the ability to retry. Are you finding you are decorating only db calls or are you decorating all sorts of stuff?

[toc] | [prev] | [next] | [standalone]

#24653

On Friday, June 29, 2012 4:53:43 AM UTC-4, andrea crotti wrote:
> On the other hand now that I think again even supposing there is a
> permanent error like MySql completely down, retrying continuosly
> won't do any harm anyway because the machine will not be able to do
> anything else anyway, when someone will fix MySql it would
> restart again without human intervention.
> 
> So I think I could even just let it retry and use maybe a SMTPHanlder
> for the logging errors, to make the notification of problems very
> quick..

Rather then write a decorator, sounds like you should write your own class or method connecting to the database that has the ability to retry. Are you finding you are decorating only db calls or are you decorating all sorts of stuff?

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web