Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.000 X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'url:pypi': 0.03; '*args,': 0.07; '[0]': 0.07; 'andres': 0.09; 'args,': 0.09; 'logic': 0.09; 'subclass': 0.09; 'tuple': 0.09; 'thread': 0.10; 'example:': 0.11; 'exception': 0.13; 'def': 0.14; 'obviously': 0.15; '>on': 0.16; '>thanks,': 0.16; 'callable,': 0.16; 'consume': 0.16; 'dirs,': 0.16; 'enough.': 0.16; 'exceptions.': 0.16; 'from:addr:cs': 0.16; 'from:addr:zip.com.au': 0.16; 'from:name:cameron simpson': 0.16; 'handling.': 0.16; 'message-id:@cskk.homeip.net': 0.16; 'none).': 0.16; 'received:211.29': 0.16; 'received:211.29.132': 0.16; 'received:cskk.homeip.net': 0.16; 'received:homeip.net': 0.16; 'received:optusnet.com.au': 0.16; 'received:syd.optusnet.com.au': 0.16; 'silly': 0.16; 'simpson': 0.16; 'task:': 0.16; 'then?': 0.16; 'wrote:': 0.16; 'changes': 0.20; '(or': 0.21; 'arguments': 0.22; 'exceptions': 0.22; 'raised': 0.22; 'try:': 0.22; 'pass': 0.22; 'finished': 0.23; 'this:': 0.23; 'cheers,': 0.24; 'seems': 0.24; 'header:In-Reply-To:1': 0.24; 'raise': 0.24; 'header:User- Agent:1': 0.26; 'error': 0.27; 'logging': 0.27; 'actual': 0.29; "i'm": 0.29; '**kwargs):': 0.29; 'queue': 0.29; 'skip:q 20': 0.29; 'task': 0.31; 'code': 0.31; "can't": 0.32; 'core': 0.32; 'url:python': 0.33; "he's": 0.33; 'raising': 0.33; 'received:com.au': 0.33; 'wrap': 0.33; 'to:addr:python-list': 0.35; 'tasks': 0.35; 'something': 0.35; 'but': 0.36; 'url:org': 0.36; 'except': 0.36; 'there': 0.36; 'should': 0.37; 'subject:: ': 0.37; 'charset:us-ascii': 0.37; 'skip:p 20': 0.38; 'pm,': 0.39; 'things': 0.39; 'to:addr:python.org': 0.39; 'subject:-': 0.39; 'sure': 0.40; 'where': 0.40; 'subject:with': 0.40; 'some': 0.40; 'content-disposition:inline': 0.60; 'your': 0.60; 'back': 0.61; 'simple': 0.61; 'more': 0.62; 'above,': 0.63; 'cameron': 0.66; '>def': 0.84; 'guts': 0.84; 'succeeds': 0.84 Date: Sat, 20 Jun 2015 13:14:55 +1000 From: Cameron Simpson To: python-list@python.org Subject: Re: Catching exceptions with multi-processing MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) References: X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.1 cv=eZjABOwH c=1 sm=1 tr=0 a=rgDbx50tNA2z7xLXQOoruw==:117 a=rgDbx50tNA2z7xLXQOoruw==:17 a=ZtCCktOnAAAA:8 a=PO7r1zJSAAAA:8 a=J0QyKEt1u0cA:10 a=kj9zAlcOel0A:10 a=vrnE16BAAAAA:8 a=XAFQembCKUMA:10 a=pGLkceISAAAA:8 a=8AHkEIZyAAAA:8 a=RYoCHCFTe0K8E1ZtUBMA:9 a=CjuIK1q_8ugA:10 X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.20+ Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 55 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1434771749 news.xs4all.nl 2840 [2001:888:2000:d::a6]:33562 X-Complaints-To: abuse@xs4all.nl Path: csiph.com!usenet.pasdenom.info!news.stben.net!border1.nntp.ams1.giganews.com!nntp.giganews.com!newsfeed.xs4all.nl!newsfeed8.news.xs4all.nl!post.news.xs4all.nl!not-for-mail Xref: csiph.com comp.lang.python:92900 On 19Jun2015 18:16, Fabien wrote: >On 06/19/2015 04:25 PM, Andres Riancho wrote: >> My recommendation is that you should pass some extra arguments to the task: >> * A unique task id >> * A result multiprocessing.Queue >> >> When an exception is raised you put (unique_id, exception) to the >>queue. When it succeeds you put (unique_id, None). In the main process >>you consume the queue and do your error handling. >> >> Note that some exceptions can't be serialized, there is where >>tblib [0] comes handy. >> >>[0]https://pypi.python.org/pypi/tblib >> >>Regards, > >Thanks, I wasn't aware of the multiprocessing.Queue workflow. It seems >like its going to require some changes in the actual code of the tasks >though. Did I get it right that I should stop raising exceptions then? > >Something like: > >def task_1(path, q): > # Do stuffs > if dont_work: > q.put(RuntimeError("didnt work")) > return > # finished > q.put(None) > return I would keep your core logic Pythonic, raise exceptions. But I would wrap each task in something to catch any Exception subclass and report back to the queue. Untested example: def subwrapper(q, callable, *args, **kwargs): try: q.put( ('COMPLETED', callable(*args, **kwargs)) ) except Exception as e: q.put( ('FAILED', e, callable, args, kwargs) ) then dispatch tasks like this: pool.map(subwrapper, q, task1, dirs, chunksize=1) and have a thread (or main program) collect things from the queue for logging and other handling. Obviously you might return something more sophisticated that my simple tuple above, but I'm sure you get the idea. Cheers, Cameron Simpson He's silly and he's ignorant, but he's got guts, and guts is enough. - Sgt. Hartmann