Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]


Groups > comp.lang.python > #78343

Re: ThreadPoolExecutor - callback guaranteed to run in same thread as the submitted function?

Path csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!feeder.erje.net!eu.feeder.erje.net!newsfeed.fsmpi.rwth-aachen.de!eternal-september.org!feeder.eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail
From Marko Rauhamaa <marko@pacujo.net>
Newsgroups comp.lang.python
Subject Re: ThreadPoolExecutor - callback guaranteed to run in same thread as the submitted function?
Date Thu, 25 Sep 2014 17:47:54 +0300
Organization A noiseless patient Spider
Lines 48
Message-ID <87sijf69k5.fsf@elektro.pacujo.net> (permalink)
References <e2f400e6-359e-4070-a822-d734e85e6a75@googlegroups.com> <mailman.14345.1411627213.18130.python-list@python.org> <871tr06nhs.fsf@elektro.pacujo.net> <mailman.14358.1411654608.18130.python-list@python.org>
Mime-Version 1.0
Content-Type text/plain; charset=us-ascii
Injection-Info mx05.eternal-september.org; posting-host="ff5cf27ef3d5b31f034d3b72bdc27a41"; logging-data="511"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX1+URBf3mg1WZVFGlPzwj5xS"
User-Agent Gnus/5.13 (Gnus v5.13) Emacs/23.3 (gnu/linux)
Cancel-Lock sha1:wdpQjt4uKIV23yE5Es22Q1MzcZo= sha1:VhGS6bet6eJGyKX9crShbG0/GJA=
Xref csiph.com comp.lang.python:78343

Show key headers only | View raw


Ian Kelly <ian.g.kelly@gmail.com>:

> On Thu, Sep 25, 2014 at 3:46 AM, Marko Rauhamaa <marko@pacujo.net> wrote:
>> Example (pseudocode):
>>
>>    def callback(self):
>>       with self.lock:
>>           ...
>>
>>    def xyz(self, f):
>>       with self.lock:
>>           ...
>>           f.add_done_callback(self.callback)
>>
>> The code will deadlock if the callback is invoked immediately.
>
> Easily solved using a re-entrant lock.

Not easily. Your state transition might not be complete at the time of
the preemption. Many such issues can be resolved by moving the
preemptive function calls to the end of the critical section, but that
is not always possible, easy or taken into account.

If preemption needs to be prepared for, the real solution is:


   def xyz(self, f):
       with self.lock:
           ...
           def add_callback():
               f.add_done_callback(self.callback)
           self.executor.schedule(add_callback)

which is ugly, of course.

> Alternatively, one could use synchronization to make sure the future
> can't complete until all callbacks have been added. That would have
> the benefit of keeping the callback processing on the same thread as
> the future, as the OP wanted.

You can still keep the callback in the same thread if your executor is
tied to the the same thread as the calling function. That's the classic
event loop. The executor API should be independent of the implementation
(single vs multithreaded), but it should be able to guarantee that it
never makes callbacks preemptively.


Marko

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread


Thread

ThreadPoolExecutor - callback guaranteed to run in same thread as the submitted function? Praveen Gollakota <pgollakota@gmail.com> - 2014-09-24 07:44 -0700
  Re: ThreadPoolExecutor - callback guaranteed to run in same thread as the submitted function? dieter <dieter@handshake.de> - 2014-09-25 08:14 +0200
  Re: ThreadPoolExecutor - callback guaranteed to run in same thread as the submitted function? Ian Kelly <ian.g.kelly@gmail.com> - 2014-09-25 00:39 -0600
    Re: ThreadPoolExecutor - callback guaranteed to run in same thread as the submitted function? Marko Rauhamaa <marko@pacujo.net> - 2014-09-25 12:46 +0300
      Re: ThreadPoolExecutor - callback guaranteed to run in same thread as the submitted function? Ian Kelly <ian.g.kelly@gmail.com> - 2014-09-25 08:16 -0600
        Re: ThreadPoolExecutor - callback guaranteed to run in same thread as the submitted function? Marko Rauhamaa <marko@pacujo.net> - 2014-09-25 17:47 +0300
          Re: ThreadPoolExecutor - callback guaranteed to run in same thread as the submitted function? Ian Kelly <ian.g.kelly@gmail.com> - 2014-09-25 09:40 -0600
          Re: ThreadPoolExecutor - callback guaranteed to run in same thread as the submitted function? Chris Angelico <rosuav@gmail.com> - 2014-09-26 02:04 +1000

csiph-web