Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]


Groups > comp.lang.python > #78345

Re: ThreadPoolExecutor - callback guaranteed to run in same thread as the submitted function?

References <e2f400e6-359e-4070-a822-d734e85e6a75@googlegroups.com> <mailman.14345.1411627213.18130.python-list@python.org> <871tr06nhs.fsf@elektro.pacujo.net> <mailman.14358.1411654608.18130.python-list@python.org> <87sijf69k5.fsf@elektro.pacujo.net>
From Ian Kelly <ian.g.kelly@gmail.com>
Date 2014-09-25 09:40 -0600
Subject Re: ThreadPoolExecutor - callback guaranteed to run in same thread as the submitted function?
Newsgroups comp.lang.python
Message-ID <mailman.14361.1411659684.18130.python-list@python.org> (permalink)

Show all headers | View raw


On Thu, Sep 25, 2014 at 8:47 AM, Marko Rauhamaa <marko@pacujo.net> wrote:
> Ian Kelly <ian.g.kelly@gmail.com>:
>
>> On Thu, Sep 25, 2014 at 3:46 AM, Marko Rauhamaa <marko@pacujo.net> wrote:
>>> Example (pseudocode):
>>>
>>>    def callback(self):
>>>       with self.lock:
>>>           ...
>>>
>>>    def xyz(self, f):
>>>       with self.lock:
>>>           ...
>>>           f.add_done_callback(self.callback)
>>>
>>> The code will deadlock if the callback is invoked immediately.
>>
>> Easily solved using a re-entrant lock.
>
> Not easily. Your state transition might not be complete at the time of
> the preemption. Many such issues can be resolved by moving the
> preemptive function calls to the end of the critical section, but that
> is not always possible, easy or taken into account.

Fair enough. In the simple example above, I see no harm in simply
moving the add_done_callback call outside (after) the "with self.lock"
block altogether. I can imagine more complex examples where the call
is further down the stack and harder to isolate from the lock,
although I would argue that's probably a code smell.

> If preemption needs to be prepared for, the real solution is:
>
>
>    def xyz(self, f):
>        with self.lock:
>            ...
>            def add_callback():
>                f.add_done_callback(self.callback)
>            self.executor.schedule(add_callback)
>
> which is ugly, of course.

YMMV, but I think I would prefer to make the executor.schedule call be
the callback:

    def callback(self, f):
        with self.lock:
            ...

    def xyz(self, f):
        with self.lock:
            ...
            f.add_done_callback(
                functools.partial(self.executor.schedule, self.callback))

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread


Thread

ThreadPoolExecutor - callback guaranteed to run in same thread as the submitted function? Praveen Gollakota <pgollakota@gmail.com> - 2014-09-24 07:44 -0700
  Re: ThreadPoolExecutor - callback guaranteed to run in same thread as the submitted function? dieter <dieter@handshake.de> - 2014-09-25 08:14 +0200
  Re: ThreadPoolExecutor - callback guaranteed to run in same thread as the submitted function? Ian Kelly <ian.g.kelly@gmail.com> - 2014-09-25 00:39 -0600
    Re: ThreadPoolExecutor - callback guaranteed to run in same thread as the submitted function? Marko Rauhamaa <marko@pacujo.net> - 2014-09-25 12:46 +0300
      Re: ThreadPoolExecutor - callback guaranteed to run in same thread as the submitted function? Ian Kelly <ian.g.kelly@gmail.com> - 2014-09-25 08:16 -0600
        Re: ThreadPoolExecutor - callback guaranteed to run in same thread as the submitted function? Marko Rauhamaa <marko@pacujo.net> - 2014-09-25 17:47 +0300
          Re: ThreadPoolExecutor - callback guaranteed to run in same thread as the submitted function? Ian Kelly <ian.g.kelly@gmail.com> - 2014-09-25 09:40 -0600
          Re: ThreadPoolExecutor - callback guaranteed to run in same thread as the submitted function? Chris Angelico <rosuav@gmail.com> - 2014-09-26 02:04 +1000

csiph-web