Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]


Groups > comp.lang.python > #78291 > unrolled thread

ThreadPoolExecutor - callback guaranteed to run in same thread as the submitted function?

Started byPraveen Gollakota <pgollakota@gmail.com>
First post2014-09-24 07:44 -0700
Last post2014-09-26 02:04 +1000
Articles 8 — 5 participants

Back to article view | Back to comp.lang.python


Contents

  ThreadPoolExecutor - callback guaranteed to run in same thread as the submitted function? Praveen Gollakota <pgollakota@gmail.com> - 2014-09-24 07:44 -0700
    Re: ThreadPoolExecutor - callback guaranteed to run in same thread as the submitted function? dieter <dieter@handshake.de> - 2014-09-25 08:14 +0200
    Re: ThreadPoolExecutor - callback guaranteed to run in same thread as the submitted function? Ian Kelly <ian.g.kelly@gmail.com> - 2014-09-25 00:39 -0600
      Re: ThreadPoolExecutor - callback guaranteed to run in same thread as the submitted function? Marko Rauhamaa <marko@pacujo.net> - 2014-09-25 12:46 +0300
        Re: ThreadPoolExecutor - callback guaranteed to run in same thread as the submitted function? Ian Kelly <ian.g.kelly@gmail.com> - 2014-09-25 08:16 -0600
          Re: ThreadPoolExecutor - callback guaranteed to run in same thread as the submitted function? Marko Rauhamaa <marko@pacujo.net> - 2014-09-25 17:47 +0300
            Re: ThreadPoolExecutor - callback guaranteed to run in same thread as the submitted function? Ian Kelly <ian.g.kelly@gmail.com> - 2014-09-25 09:40 -0600
            Re: ThreadPoolExecutor - callback guaranteed to run in same thread as the submitted function? Chris Angelico <rosuav@gmail.com> - 2014-09-26 02:04 +1000

#78291 — ThreadPoolExecutor - callback guaranteed to run in same thread as the submitted function?

FromPraveen Gollakota <pgollakota@gmail.com>
Date2014-09-24 07:44 -0700
SubjectThreadPoolExecutor - callback guaranteed to run in same thread as the submitted function?
Message-ID<e2f400e6-359e-4070-a822-d734e85e6a75@googlegroups.com>
Hello,

In the ThreadPoolExecutor (TPE), is the callback always guaranteed to run in the same thread as the submitted function?

 For example, I tested this with the following code. I ran it many times and it seemed like `func` and `callback` always ran in the same thread.

    import concurrent.futures
    import random
    import threading
    import time

    executor = concurrent.futures.ThreadPoolExecutor(max_workers=3)

    def func(x):
        time.sleep(random.random())
        return threading.current_thread().name

    def callback(future):
        time.sleep(random.random())
        x = future.result()
        cur_thread = threading.current_thread().name
        if (cur_thread != x):
            print(cur_thread, x)

    print('main thread: %s' % threading.current_thread())
    for i in range(10000):
        future = executor.submit(func, i)
        future.add_done_callback(callback)

However, it seemed to fail when I removed the `time.sleep(random.random())` statements, i.e. at least a few `func` functions and `callbacks` **did not** run in the same thread. 

For a project that I am working on, the callback must always run on the same thread as the submitted function, so I wanted to be sure that this is guaranteed by TPE. (And also the results of the test without the random sleep seemed puzzling).

I looked at the source code for executors (see https://hg.python.org/cpython/file/6d44906344f4/Lib/concurrent/futures/_base.py#l297) and it does not seem like we switch the thread to the main thread before we run the callback. But just wanted to be sure. 

Any insight would be appreciated!

Thanks,
Praveen

[toc] | [next] | [standalone]


#78325

Fromdieter <dieter@handshake.de>
Date2014-09-25 08:14 +0200
Message-ID<mailman.14343.1411625683.18130.python-list@python.org>
In reply to#78291
Praveen Gollakota <pgollakota@gmail.com> writes:
> In the ThreadPoolExecutor (TPE), is the callback always guaranteed to run in the same thread as the submitted function?

To answer question of this type (yourself), you may look
at the relevant source code.

I am quite sure that the "submitted function" will be executed
in one of the threads from the thread pool.

I have not looked at the "ThreadPoolExecutor" code. The easiest thing
would be to run the callback in the same thread as the submitted function.
However, it could facilitate use of the "ThreadPoolExecutor"
(less locking) when the synchronisation were done inside
the "ThreadPoolExecutor" and the callbacks were executed in a single
thread (likely the one which has set up the "ThreadPoolExecutor").
A look at the source should tell the details.

[toc] | [prev] | [next] | [standalone]


#78328

FromIan Kelly <ian.g.kelly@gmail.com>
Date2014-09-25 00:39 -0600
Message-ID<mailman.14345.1411627213.18130.python-list@python.org>
In reply to#78291
On Wed, Sep 24, 2014 at 8:44 AM, Praveen Gollakota <pgollakota@gmail.com> wrote:
> However, it seemed to fail when I removed the `time.sleep(random.random())` statements, i.e. at least a few `func` functions and `callbacks` **did not** run in the same thread.
>
> For a project that I am working on, the callback must always run on the same thread as the submitted function, so I wanted to be sure that this is guaranteed by TPE. (And also the results of the test without the random sleep seemed puzzling).

The documentation for Future.add_done_callback says: "If the future
has already completed or been cancelled, fn will be called
immediately." Checking the code, the callback is simply called from
the submitting thread in this case. This makes sense as the thread
that completed the future has already moved onto something else and
may not be readily available or even identifiable.

[toc] | [prev] | [next] | [standalone]


#78332

FromMarko Rauhamaa <marko@pacujo.net>
Date2014-09-25 12:46 +0300
Message-ID<871tr06nhs.fsf@elektro.pacujo.net>
In reply to#78328
Ian Kelly <ian.g.kelly@gmail.com>:

> The documentation for Future.add_done_callback says: "If the future
> has already completed or been cancelled, fn will be called
> immediately."

That sounds really bad.

There should be a guarantee that the callback is not called from the
same thread before returning from Future.add_done_callback. Otherwise,
the caller may have a really difficult time handling preemption and
avoiding deadlocks at the same time.

Example (pseudocode):

   def callback(self):
      with self.lock:
          ...

   def xyz(self, f):
      with self.lock:
          ...
          f.add_done_callback(self.callback)

The code will deadlock if the callback is invoked immediately.


Marko

[toc] | [prev] | [next] | [standalone]


#78341

FromIan Kelly <ian.g.kelly@gmail.com>
Date2014-09-25 08:16 -0600
Message-ID<mailman.14358.1411654608.18130.python-list@python.org>
In reply to#78332
On Thu, Sep 25, 2014 at 3:46 AM, Marko Rauhamaa <marko@pacujo.net> wrote:
> Ian Kelly <ian.g.kelly@gmail.com>:
>
>> The documentation for Future.add_done_callback says: "If the future
>> has already completed or been cancelled, fn will be called
>> immediately."
>
> That sounds really bad.
>
> There should be a guarantee that the callback is not called from the
> same thread before returning from Future.add_done_callback. Otherwise,
> the caller may have a really difficult time handling preemption and
> avoiding deadlocks at the same time.
>
> Example (pseudocode):
>
>    def callback(self):
>       with self.lock:
>           ...
>
>    def xyz(self, f):
>       with self.lock:
>           ...
>           f.add_done_callback(self.callback)
>
> The code will deadlock if the callback is invoked immediately.

Easily solved using a re-entrant lock. Alternatively, one could use
synchronization to make sure the future can't complete until all
callbacks have been added. That would have the benefit of keeping the
callback processing on the same thread as the future, as the OP
wanted.

[toc] | [prev] | [next] | [standalone]


#78343

FromMarko Rauhamaa <marko@pacujo.net>
Date2014-09-25 17:47 +0300
Message-ID<87sijf69k5.fsf@elektro.pacujo.net>
In reply to#78341
Ian Kelly <ian.g.kelly@gmail.com>:

> On Thu, Sep 25, 2014 at 3:46 AM, Marko Rauhamaa <marko@pacujo.net> wrote:
>> Example (pseudocode):
>>
>>    def callback(self):
>>       with self.lock:
>>           ...
>>
>>    def xyz(self, f):
>>       with self.lock:
>>           ...
>>           f.add_done_callback(self.callback)
>>
>> The code will deadlock if the callback is invoked immediately.
>
> Easily solved using a re-entrant lock.

Not easily. Your state transition might not be complete at the time of
the preemption. Many such issues can be resolved by moving the
preemptive function calls to the end of the critical section, but that
is not always possible, easy or taken into account.

If preemption needs to be prepared for, the real solution is:


   def xyz(self, f):
       with self.lock:
           ...
           def add_callback():
               f.add_done_callback(self.callback)
           self.executor.schedule(add_callback)

which is ugly, of course.

> Alternatively, one could use synchronization to make sure the future
> can't complete until all callbacks have been added. That would have
> the benefit of keeping the callback processing on the same thread as
> the future, as the OP wanted.

You can still keep the callback in the same thread if your executor is
tied to the the same thread as the calling function. That's the classic
event loop. The executor API should be independent of the implementation
(single vs multithreaded), but it should be able to guarantee that it
never makes callbacks preemptively.


Marko

[toc] | [prev] | [next] | [standalone]


#78345

FromIan Kelly <ian.g.kelly@gmail.com>
Date2014-09-25 09:40 -0600
Message-ID<mailman.14361.1411659684.18130.python-list@python.org>
In reply to#78343
On Thu, Sep 25, 2014 at 8:47 AM, Marko Rauhamaa <marko@pacujo.net> wrote:
> Ian Kelly <ian.g.kelly@gmail.com>:
>
>> On Thu, Sep 25, 2014 at 3:46 AM, Marko Rauhamaa <marko@pacujo.net> wrote:
>>> Example (pseudocode):
>>>
>>>    def callback(self):
>>>       with self.lock:
>>>           ...
>>>
>>>    def xyz(self, f):
>>>       with self.lock:
>>>           ...
>>>           f.add_done_callback(self.callback)
>>>
>>> The code will deadlock if the callback is invoked immediately.
>>
>> Easily solved using a re-entrant lock.
>
> Not easily. Your state transition might not be complete at the time of
> the preemption. Many such issues can be resolved by moving the
> preemptive function calls to the end of the critical section, but that
> is not always possible, easy or taken into account.

Fair enough. In the simple example above, I see no harm in simply
moving the add_done_callback call outside (after) the "with self.lock"
block altogether. I can imagine more complex examples where the call
is further down the stack and harder to isolate from the lock,
although I would argue that's probably a code smell.

> If preemption needs to be prepared for, the real solution is:
>
>
>    def xyz(self, f):
>        with self.lock:
>            ...
>            def add_callback():
>                f.add_done_callback(self.callback)
>            self.executor.schedule(add_callback)
>
> which is ugly, of course.

YMMV, but I think I would prefer to make the executor.schedule call be
the callback:

    def callback(self, f):
        with self.lock:
            ...

    def xyz(self, f):
        with self.lock:
            ...
            f.add_done_callback(
                functools.partial(self.executor.schedule, self.callback))

[toc] | [prev] | [next] | [standalone]


#78346

FromChris Angelico <rosuav@gmail.com>
Date2014-09-26 02:04 +1000
Message-ID<mailman.14362.1411661084.18130.python-list@python.org>
In reply to#78343
On Fri, Sep 26, 2014 at 1:40 AM, Ian Kelly <ian.g.kelly@gmail.com> wrote:
> Fair enough. In the simple example above, I see no harm in simply
> moving the add_done_callback call outside (after) the "with self.lock"
> block altogether. I can imagine more complex examples where the call
> is further down the stack and harder to isolate from the lock,
> although I would argue that's probably a code smell.

I'd definitely call it code smell. What happens with this?

with lock:
    def inner():
        do_more_work()
    f.add_done_callback(inner)

The lock's possibly released before the callback happens, or possibly
not, but either way, the code *looks* as if it's all happening inside
the "with lock" block. Recipe for confusion IMO.

ChrisA

[toc] | [prev] | [standalone]


Back to top | Article view | comp.lang.python


csiph-web