Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!feeder.erje.net!eu.feeder.erje.net!newsfeed.xs4all.nl!newsfeed3.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.000 X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'api.': 0.05; 'argument': 0.05; 'cache': 0.07; 'class,': 0.07; 'arguments': 0.09; 'differently.': 0.09; 'except:': 0.09; 'exception,': 0.09; 'formatting': 0.09; 'item,': 0.09; 'methods,': 0.09; 'oh,': 0.09; 'override': 0.09; 'parsed': 0.09; 'try:': 0.09; 'wrapped': 0.09; 'cc:addr:python-list': 0.11; 'def': 0.12; 'mostly': 0.14; 'thread': 0.14; '**kwargs)': 0.16; 'args,': 0.16; 'backwards': 0.16; 'behave': 0.16; 'cases)': 0.16; 'concurrent': 0.16; 'handling,': 0.16; 'have)': 0.16; 'kwargs': 0.16; 'log.': 0.16; 'merely': 0.16; 'overridden': 0.16; 'red,': 0.16; 'statements,': 0.16; 'subclass': 0.16; 'subclasses': 0.16; 'subject:Thread': 0.16; 'true:': 0.16; 'do,': 0.16; 'wrote:': 0.18; 'code.': 0.18; 'looked': 0.18; 'module': 0.19; 'finished': 0.19; 'normally': 0.19; "python's": 0.19; 'thu,': 0.19; 'written': 0.21; '(in': 0.22; 'aug': 0.22; 'email addr:gmail.com>': 0.22; 'cc:addr:python.org': 0.22; 'print': 0.22; 'error': 0.23; 'exists': 0.24; 'module,': 0.24; 'cc:2**0': 0.24; 'handling': 0.26; 'mention': 0.26; 'pass': 0.26; 'header:In-Reply-To:1': 0.27; 'function': 0.29; 'wondering': 0.29; 'chris': 0.29; "we'd": 0.29; 'thus': 0.29; 'returned': 0.30; 'sets': 0.30; 'especially': 0.30; 'message-id:@mail.gmail.com': 0.30; 'went': 0.31; 'code': 0.31; 'easier': 0.31; '-0700,': 0.31; '3.2': 0.31; 'exceptions': 0.31; 'work:': 0.31; 'class': 0.32; 'run': 0.32; 'quite': 0.32; 'standards': 0.33; 'actual': 0.34; 'table': 0.34; 'skip:_ 10': 0.34; 'received:74.125.82': 0.34; 'could': 0.34; 'knowledge': 0.35; 'common': 0.35; 'connection': 0.35; 'except': 0.35; 'something': 0.35; 'point.': 0.35; 'but': 0.35; 'received:google.com': 0.35; 'really': 0.36; 'joined': 0.36; 'doing': 0.36; "didn't": 0.36; 'similar': 0.36; 'two': 0.37; 'others.': 0.38; 'fact': 0.38; 'pm,': 0.38; 'rather': 0.38; 'received:74.125': 0.39; 'enough': 0.39; 'called': 0.40; 'even': 0.60; 'remove': 0.60; 'read': 0.60; 'anytime': 0.60; 'failures': 0.60; 'most': 0.60; 'color': 0.61; 'entire': 0.61; 'first': 0.61; 'making': 0.63; 'our': 0.64; 'provide': 0.64; 'different': 0.65; 'to:addr:gmail.com': 0.65; 'due': 0.66; 'behavior': 0.77; '"it': 0.84; '"it': 0.84; '_bootstrap': 0.84; 'actually,': 0.84; 'right).': 0.84; 'streams': 0.84; 'on?': 0.91; 'serious': 0.97 X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type; bh=vGepYjYu5HJD9EXzJLojyFtx/lA6pdugFCi/mVeD0Uc=; b=dYaK3BKbCWy119ZvD5jddYSc3synY/4RRf/DRP8EAWegOfvGt1iLUSc7pD6ETBmOnu vVNXqetXW+HVRGle74FHku62tzbaSVKNS+5LqjpT3wrWXFBjH3pW5YHZH/YIY9TkhVDV KGENkUJ2+6jF4HC5ceYl+w5A75uOrCjSp6HDTymLez67r9Sesbvxp/YJGXFqydpzI+EB iMQhWKlun6o7za8XLRyUYSVN0neguhoK8yjcYKhoezQHczPwRFOOC6TTIx9BJsC0BN1j uAJ+6tYsXHJDag2IXMXEJsCasIWW8jw4BvOS+KJO8wtM3GhNgtW7LL9EW1tywuU3j6q2 hLFA== X-Gm-Message-State: ALoCoQnUJTrvRVMl+vBGDKB1w/fHzedSSzFhJGAm6jwTFrA9Jm7j+sLU8bJ/GyeT4Q1LdH25mtQD X-Received: by 10.180.188.205 with SMTP id gc13mr41133857wic.66.1409262060404; Thu, 28 Aug 2014 14:41:00 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: <20140828203552.GA23778@arxnet.hu> References: <20140827185514.GA18660@arxnet.hu> <20140828085211.GC2887@arxnet.hu> <20140828183900.GA22771@arxnet.hu> <20140828203552.GA23778@arxnet.hu> From: Chris Kaynor Date: Thu, 28 Aug 2014 14:40:40 -0700 Subject: Re: Thread terminate To: =?UTF-8?Q?Ervin_Heged=C3=BCs?= Content-Type: multipart/alternative; boundary=001a11c385ac819d170501b76474 Cc: "python-list@python.org" X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 196 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1409262067 news.xs4all.nl 2898 [2001:888:2000:d::a6]:58825 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:77239 --001a11c385ac819d170501b76474 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On Thu, Aug 28, 2014 at 1:35 PM, Ervin Heged=C3=BCs wro= te: > On Thu, Aug 28, 2014 at 12:34:18PM -0700, Chris Kaynor wrote: depending on what you are doing with the first two arguments to self._exit= , >> the following might also work: >> >> def run(self): >> try: >> connect_to_db() >> do_another_thing() >> except: >> self._exit(*sys.exc_info()) >> return True > > > The first argument is a status, this is passed to the item, which passed > to thread - so the thread sets that status, and the main loop knows, > which item needs to pass to a thread. Eg. if the DB connection > has failed, it needs to run again, at later. But if that item is > parsed and finished (eg. table exists in db), then the item > is deletable from the queue. > Mostly, I was wondering if you really need to differentiate types of failures at that point. Would it be enough to merely know "it worked" vs "it failed"? Even if you do need to behave differently, can you make the decision based on the type of exception, rather than which stage you were on? If so for either, you can remove the multiple try...except around different statements, and put one big one around the entire run function. This would likely simplify your handling code. > One solution I did not mention before, but is plausible, is to >> monkey-patch/override the threading module to change the behavior >> of _bootstrap_inner or _bootstrap to behave differently. As these are on >> the Thread class, you could override them in your subclass or a base >> subclass (depending on the number of Thread subclasses you have) that >> behaves differently. Perhaps by making it call sys.excepthook. At one >> point, we discussed doing something similar at work as we normally print >> exceptions to custom streams that provide color formatting to make the >> exceptions easier to find and read (exception message printed in red, >> traceback in a dark red), as well as log. > > > oh, that would be a big gun - and I don't feel that knowledge to > made that with security and stability. > > I think this solution will enough at first time :), I didn't used > anytime the threads (in serious project). > Yah, we never did this either, mostly due to the fact that then we'd have to update our code if Python's code was updated. This can be quite problematic, especially as the functions that would need to be overridden are private, and thus not put to the same backwards compatibility standards as most of Python's API. What we did do, however, was wrap threading (in some cases) where we used a thread pool where the actual run function was wrapped in our own error handling, and called out to the desired function. Basically, most of our threads went though common code that looked something like (simplified here): def run(self): while True: func, args, kwargs =3D self.queue.pop() # Actually, returned an instance of a class which provided some join() and cancel() methods, among others. try: func(*args, **kwargs) except BaseExeception: # complex error handling here. Depending on conditions, this might print the error, or might merely cache it in case the work item was joined Basically, the whole code block was a custom implementation of Python's concurrent module, which we could not use as we had to support 2.6. It was also written before 3.2 came out (if I remember the timing right). Chris --001a11c385ac819d170501b76474 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
On T= hu, Aug 28, 2014 at 1:35 PM, Ervin Heged=C3=BCs <airween@gmail.com>= wrote:
On Thu, Aug 28, 2014 at 12:= 34:18PM -0700, Chris Kaynor wrote:
depending on what you are doing with the first two arguments to self._exit,=
the following might also work:=C2=A0
=C2=A0
def run(self):
=C2= =A0 =C2=A0 try:
=C2=A0 =C2=A0 =C2=A0 =C2=A0 connect_to_db()
=C2=A0 = =C2=A0 =C2=A0 =C2=A0 do_another_thing()
=C2=A0 =C2=A0 except:
=C2=A0 =C2=A0 =C2=A0 =C2=A0 self._exit(*sys.exc_info())
=C2=A0 =C2=A0 = =C2=A0 =C2=A0 return True

The first argument is a status, this is passed to the item, which pas= sed
to thread - so the thread sets that status, and the main loop knows,
which item needs to pass to a thread. Eg. if the DB connection
has failed, it needs to run again, at later. But if that item is
parsed and finished (eg. table exists in db), then the item
is deletable from the queue.

Mostly, I = was wondering if you really need to differentiate types of failures at that= point. Would it be enough to merely know "it worked" vs "it= failed"? Even if you do need to behave differently, can you make the = decision based on the type of exception, rather than which stage you were o= n? If so for either, you can remove the multiple try...except around differ= ent statements, and put one big one around the entire run function.

This would likely simplify your handling code.
=C2=A0
One solution I did not mention before, but= is plausible, is to
monkey-patch/override the threading module to change the behavior
of _bo= otstrap_inner or _bootstrap to behave differently. As these are on
the T= hread class, you could override them in your subclass or a base
subclass= (depending on the number of Thread subclasses you have) that
behaves differently. Perhaps by making it call sys.excepthook. At one
po= int, we discussed doing something similar at work as we normally print
e= xceptions to custom streams that provide color formatting to make the
exceptions easier to find and read (exception message printed in red,
tr= aceback in a dark red), as well as log.

oh, that would be a big gun - and I don't feel that knowledge to<= br> made that with security and stability.

I think this solution will enough at first time :), I didn't used
anytime the threads (in serious project).

Yah, we never did this either, mostly due to the fact that then we'd= have to update our code if Python's code was updated. This can be quit= e problematic, especially as the functions that would need to be overridden= are private, and thus not put to the same backwards compatibility standard= s as most of Python's API.

What we did do, however, was wrap threading (in some ca= ses) where we used a thread pool where the actual run function was wrapped = in our own error handling, and called out to the desired function. Basicall= y, most of our threads went though common code that looked something like (= simplified here):

def run(self):
=C2=A0 =C2=A0 while True:
=C2=A0 =C2=A0 =C2=A0 =C2=A0 func, args, kwargs =3D self.queue.pop() #= Actually, returned an instance of a class which provided some join() and c= ancel() methods, among others.
=C2=A0 =C2=A0 =C2=A0 =C2=A0 try:
=C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 func(*args, **kwargs)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 = except BaseExeception:
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0# complex error handling here. Depending on conditions, this might pr= int the error, or might merely cache it in case the work item was joined

Basically, the whole code block was a custom implementa= tion of Python's concurrent module, which we could not use as we had to= support 2.6. It was also written before 3.2 came out (if I remember the ti= ming right).

Chris
--001a11c385ac819d170501b76474--