Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!feeder.erje.net!eu.feeder.erje.net!newsfeed.xs4all.nl!newsfeed1a.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.024 X-Spam-Evidence: '*H*': 0.95; '*S*': 0.00; 'subject:same': 0.07; '"if': 0.09; 'callback': 0.09; 'caller': 0.09; 'def': 0.12; 'thread': 0.14; 'added.': 0.16; 'subject: \n ': 0.16; 'subject:run': 0.16; 'wanted.': 0.16; 'wrote:': 0.18; 'skip:f 30': 0.19; 'thu,': 0.19; 'example': 0.22; 'otherwise,': 0.22; 'handling': 0.26; 'header:In-Reply-To:1': 0.27; 'am,': 0.29; 'message-id:@mail.gmail.com': 0.30; 'code': 0.31; '25,': 0.31; 'bad.': 0.31; 'sep': 0.31; 'subject:the': 0.34; 'could': 0.34; "can't": 0.35; 'received:google.com': 0.35; 'there': 0.35; 'really': 0.36; 'returning': 0.36; 'subject:?': 0.36; 'should': 0.36; 'easily': 0.37; 'to:addr:python-list': 0.38; 'sure': 0.39; 'to:addr:python.org': 0.39; 'called': 0.40; 'future': 0.60; 'ian': 0.60; 'completed': 0.61; 'complete': 0.62; 'guarantee': 0.63; 'benefit': 0.68; 'subject:skip:T 10': 0.68; 'future,': 0.83 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=YATWwjGI2Hs4fAn6lDCHhsPMMbvBlKJDD7BiEFgreXs=; b=XenS9OJ/q7MoijojK0xjBlsXKwa1t2qVLuRRf0au6Kxef+CsxInjLK7P08SrMnCDKP gIqGpVJsUmzcUckR8le8K5i681e+jgvQVk8mRvQCel1/Aay2m53ZOGq2PPQnbQ6EZ/qp 5In08GWf8P9iEQXwn4RVRV21FnAiQKwGyfg0xJd6HUkykImxyvH+hr2Q/MtLmodzWDkO oNRiwE67VqqIlID8RV/F2UWHZ6LQbhDylsCS3fAassJyp7pPPTulk0nM+QLWvKDBtQj+ rlUy4rsFkSc6E/otgN91JffhnDW0Tta85gJd71iEws0Y2SrSzfet2/RyHzI/nonXNydg Vp+A== X-Received: by 10.70.128.199 with SMTP id nq7mr2703879pdb.156.1411654605418; Thu, 25 Sep 2014 07:16:45 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: <871tr06nhs.fsf@elektro.pacujo.net> References: <871tr06nhs.fsf@elektro.pacujo.net> From: Ian Kelly Date: Thu, 25 Sep 2014 08:16:05 -0600 Subject: Re: ThreadPoolExecutor - callback guaranteed to run in same thread as the submitted function? To: Python Content-Type: text/plain; charset=UTF-8 X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 32 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1411654608 news.xs4all.nl 2836 [2001:888:2000:d::a6]:57292 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:78341 On Thu, Sep 25, 2014 at 3:46 AM, Marko Rauhamaa wrote: > Ian Kelly : > >> The documentation for Future.add_done_callback says: "If the future >> has already completed or been cancelled, fn will be called >> immediately." > > That sounds really bad. > > There should be a guarantee that the callback is not called from the > same thread before returning from Future.add_done_callback. Otherwise, > the caller may have a really difficult time handling preemption and > avoiding deadlocks at the same time. > > Example (pseudocode): > > def callback(self): > with self.lock: > ... > > def xyz(self, f): > with self.lock: > ... > f.add_done_callback(self.callback) > > The code will deadlock if the callback is invoked immediately. Easily solved using a re-entrant lock. Alternatively, one could use synchronization to make sure the future can't complete until all callbacks have been added. That would have the benefit of keeping the callback processing on the same thread as the future, as the OP wanted.