Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.lang.python > #78291 > unrolled thread
| Started by | Praveen Gollakota <pgollakota@gmail.com> |
|---|---|
| First post | 2014-09-24 07:44 -0700 |
| Last post | 2014-09-26 02:04 +1000 |
| Articles | 8 — 5 participants |
Back to article view | Back to comp.lang.python
ThreadPoolExecutor - callback guaranteed to run in same thread as the submitted function? Praveen Gollakota <pgollakota@gmail.com> - 2014-09-24 07:44 -0700
Re: ThreadPoolExecutor - callback guaranteed to run in same thread as the submitted function? dieter <dieter@handshake.de> - 2014-09-25 08:14 +0200
Re: ThreadPoolExecutor - callback guaranteed to run in same thread as the submitted function? Ian Kelly <ian.g.kelly@gmail.com> - 2014-09-25 00:39 -0600
Re: ThreadPoolExecutor - callback guaranteed to run in same thread as the submitted function? Marko Rauhamaa <marko@pacujo.net> - 2014-09-25 12:46 +0300
Re: ThreadPoolExecutor - callback guaranteed to run in same thread as the submitted function? Ian Kelly <ian.g.kelly@gmail.com> - 2014-09-25 08:16 -0600
Re: ThreadPoolExecutor - callback guaranteed to run in same thread as the submitted function? Marko Rauhamaa <marko@pacujo.net> - 2014-09-25 17:47 +0300
Re: ThreadPoolExecutor - callback guaranteed to run in same thread as the submitted function? Ian Kelly <ian.g.kelly@gmail.com> - 2014-09-25 09:40 -0600
Re: ThreadPoolExecutor - callback guaranteed to run in same thread as the submitted function? Chris Angelico <rosuav@gmail.com> - 2014-09-26 02:04 +1000
| From | Praveen Gollakota <pgollakota@gmail.com> |
|---|---|
| Date | 2014-09-24 07:44 -0700 |
| Subject | ThreadPoolExecutor - callback guaranteed to run in same thread as the submitted function? |
| Message-ID | <e2f400e6-359e-4070-a822-d734e85e6a75@googlegroups.com> |
Hello,
In the ThreadPoolExecutor (TPE), is the callback always guaranteed to run in the same thread as the submitted function?
For example, I tested this with the following code. I ran it many times and it seemed like `func` and `callback` always ran in the same thread.
import concurrent.futures
import random
import threading
import time
executor = concurrent.futures.ThreadPoolExecutor(max_workers=3)
def func(x):
time.sleep(random.random())
return threading.current_thread().name
def callback(future):
time.sleep(random.random())
x = future.result()
cur_thread = threading.current_thread().name
if (cur_thread != x):
print(cur_thread, x)
print('main thread: %s' % threading.current_thread())
for i in range(10000):
future = executor.submit(func, i)
future.add_done_callback(callback)
However, it seemed to fail when I removed the `time.sleep(random.random())` statements, i.e. at least a few `func` functions and `callbacks` **did not** run in the same thread.
For a project that I am working on, the callback must always run on the same thread as the submitted function, so I wanted to be sure that this is guaranteed by TPE. (And also the results of the test without the random sleep seemed puzzling).
I looked at the source code for executors (see https://hg.python.org/cpython/file/6d44906344f4/Lib/concurrent/futures/_base.py#l297) and it does not seem like we switch the thread to the main thread before we run the callback. But just wanted to be sure.
Any insight would be appreciated!
Thanks,
Praveen
[toc] | [next] | [standalone]
| From | dieter <dieter@handshake.de> |
|---|---|
| Date | 2014-09-25 08:14 +0200 |
| Message-ID | <mailman.14343.1411625683.18130.python-list@python.org> |
| In reply to | #78291 |
Praveen Gollakota <pgollakota@gmail.com> writes: > In the ThreadPoolExecutor (TPE), is the callback always guaranteed to run in the same thread as the submitted function? To answer question of this type (yourself), you may look at the relevant source code. I am quite sure that the "submitted function" will be executed in one of the threads from the thread pool. I have not looked at the "ThreadPoolExecutor" code. The easiest thing would be to run the callback in the same thread as the submitted function. However, it could facilitate use of the "ThreadPoolExecutor" (less locking) when the synchronisation were done inside the "ThreadPoolExecutor" and the callbacks were executed in a single thread (likely the one which has set up the "ThreadPoolExecutor"). A look at the source should tell the details.
[toc] | [prev] | [next] | [standalone]
| From | Ian Kelly <ian.g.kelly@gmail.com> |
|---|---|
| Date | 2014-09-25 00:39 -0600 |
| Message-ID | <mailman.14345.1411627213.18130.python-list@python.org> |
| In reply to | #78291 |
On Wed, Sep 24, 2014 at 8:44 AM, Praveen Gollakota <pgollakota@gmail.com> wrote: > However, it seemed to fail when I removed the `time.sleep(random.random())` statements, i.e. at least a few `func` functions and `callbacks` **did not** run in the same thread. > > For a project that I am working on, the callback must always run on the same thread as the submitted function, so I wanted to be sure that this is guaranteed by TPE. (And also the results of the test without the random sleep seemed puzzling). The documentation for Future.add_done_callback says: "If the future has already completed or been cancelled, fn will be called immediately." Checking the code, the callback is simply called from the submitting thread in this case. This makes sense as the thread that completed the future has already moved onto something else and may not be readily available or even identifiable.
[toc] | [prev] | [next] | [standalone]
| From | Marko Rauhamaa <marko@pacujo.net> |
|---|---|
| Date | 2014-09-25 12:46 +0300 |
| Message-ID | <871tr06nhs.fsf@elektro.pacujo.net> |
| In reply to | #78328 |
Ian Kelly <ian.g.kelly@gmail.com>:
> The documentation for Future.add_done_callback says: "If the future
> has already completed or been cancelled, fn will be called
> immediately."
That sounds really bad.
There should be a guarantee that the callback is not called from the
same thread before returning from Future.add_done_callback. Otherwise,
the caller may have a really difficult time handling preemption and
avoiding deadlocks at the same time.
Example (pseudocode):
def callback(self):
with self.lock:
...
def xyz(self, f):
with self.lock:
...
f.add_done_callback(self.callback)
The code will deadlock if the callback is invoked immediately.
Marko
[toc] | [prev] | [next] | [standalone]
| From | Ian Kelly <ian.g.kelly@gmail.com> |
|---|---|
| Date | 2014-09-25 08:16 -0600 |
| Message-ID | <mailman.14358.1411654608.18130.python-list@python.org> |
| In reply to | #78332 |
On Thu, Sep 25, 2014 at 3:46 AM, Marko Rauhamaa <marko@pacujo.net> wrote: > Ian Kelly <ian.g.kelly@gmail.com>: > >> The documentation for Future.add_done_callback says: "If the future >> has already completed or been cancelled, fn will be called >> immediately." > > That sounds really bad. > > There should be a guarantee that the callback is not called from the > same thread before returning from Future.add_done_callback. Otherwise, > the caller may have a really difficult time handling preemption and > avoiding deadlocks at the same time. > > Example (pseudocode): > > def callback(self): > with self.lock: > ... > > def xyz(self, f): > with self.lock: > ... > f.add_done_callback(self.callback) > > The code will deadlock if the callback is invoked immediately. Easily solved using a re-entrant lock. Alternatively, one could use synchronization to make sure the future can't complete until all callbacks have been added. That would have the benefit of keeping the callback processing on the same thread as the future, as the OP wanted.
[toc] | [prev] | [next] | [standalone]
| From | Marko Rauhamaa <marko@pacujo.net> |
|---|---|
| Date | 2014-09-25 17:47 +0300 |
| Message-ID | <87sijf69k5.fsf@elektro.pacujo.net> |
| In reply to | #78341 |
Ian Kelly <ian.g.kelly@gmail.com>:
> On Thu, Sep 25, 2014 at 3:46 AM, Marko Rauhamaa <marko@pacujo.net> wrote:
>> Example (pseudocode):
>>
>> def callback(self):
>> with self.lock:
>> ...
>>
>> def xyz(self, f):
>> with self.lock:
>> ...
>> f.add_done_callback(self.callback)
>>
>> The code will deadlock if the callback is invoked immediately.
>
> Easily solved using a re-entrant lock.
Not easily. Your state transition might not be complete at the time of
the preemption. Many such issues can be resolved by moving the
preemptive function calls to the end of the critical section, but that
is not always possible, easy or taken into account.
If preemption needs to be prepared for, the real solution is:
def xyz(self, f):
with self.lock:
...
def add_callback():
f.add_done_callback(self.callback)
self.executor.schedule(add_callback)
which is ugly, of course.
> Alternatively, one could use synchronization to make sure the future
> can't complete until all callbacks have been added. That would have
> the benefit of keeping the callback processing on the same thread as
> the future, as the OP wanted.
You can still keep the callback in the same thread if your executor is
tied to the the same thread as the calling function. That's the classic
event loop. The executor API should be independent of the implementation
(single vs multithreaded), but it should be able to guarantee that it
never makes callbacks preemptively.
Marko
[toc] | [prev] | [next] | [standalone]
| From | Ian Kelly <ian.g.kelly@gmail.com> |
|---|---|
| Date | 2014-09-25 09:40 -0600 |
| Message-ID | <mailman.14361.1411659684.18130.python-list@python.org> |
| In reply to | #78343 |
On Thu, Sep 25, 2014 at 8:47 AM, Marko Rauhamaa <marko@pacujo.net> wrote:
> Ian Kelly <ian.g.kelly@gmail.com>:
>
>> On Thu, Sep 25, 2014 at 3:46 AM, Marko Rauhamaa <marko@pacujo.net> wrote:
>>> Example (pseudocode):
>>>
>>> def callback(self):
>>> with self.lock:
>>> ...
>>>
>>> def xyz(self, f):
>>> with self.lock:
>>> ...
>>> f.add_done_callback(self.callback)
>>>
>>> The code will deadlock if the callback is invoked immediately.
>>
>> Easily solved using a re-entrant lock.
>
> Not easily. Your state transition might not be complete at the time of
> the preemption. Many such issues can be resolved by moving the
> preemptive function calls to the end of the critical section, but that
> is not always possible, easy or taken into account.
Fair enough. In the simple example above, I see no harm in simply
moving the add_done_callback call outside (after) the "with self.lock"
block altogether. I can imagine more complex examples where the call
is further down the stack and harder to isolate from the lock,
although I would argue that's probably a code smell.
> If preemption needs to be prepared for, the real solution is:
>
>
> def xyz(self, f):
> with self.lock:
> ...
> def add_callback():
> f.add_done_callback(self.callback)
> self.executor.schedule(add_callback)
>
> which is ugly, of course.
YMMV, but I think I would prefer to make the executor.schedule call be
the callback:
def callback(self, f):
with self.lock:
...
def xyz(self, f):
with self.lock:
...
f.add_done_callback(
functools.partial(self.executor.schedule, self.callback))
[toc] | [prev] | [next] | [standalone]
| From | Chris Angelico <rosuav@gmail.com> |
|---|---|
| Date | 2014-09-26 02:04 +1000 |
| Message-ID | <mailman.14362.1411661084.18130.python-list@python.org> |
| In reply to | #78343 |
On Fri, Sep 26, 2014 at 1:40 AM, Ian Kelly <ian.g.kelly@gmail.com> wrote:
> Fair enough. In the simple example above, I see no harm in simply
> moving the add_done_callback call outside (after) the "with self.lock"
> block altogether. I can imagine more complex examples where the call
> is further down the stack and harder to isolate from the lock,
> although I would argue that's probably a code smell.
I'd definitely call it code smell. What happens with this?
with lock:
def inner():
do_more_work()
f.add_done_callback(inner)
The lock's possibly released before the callback happens, or possibly
not, but either way, the code *looks* as if it's all happening inside
the "with lock" block. Recipe for confusion IMO.
ChrisA
[toc] | [prev] | [standalone]
Back to top | Article view | comp.lang.python
csiph-web