Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #103056 > unrolled thread

Re: Passing data across callbacks in ThreadPoolExecutor

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Passing data across callbacks in ThreadPoolExecutor Ian Kelly <ian.g.kelly@gmail.com> - 2016-02-17 09:24 -0700

#103056 — Re: Passing data across callbacks in ThreadPoolExecutor

On Tue, Feb 16, 2016 at 2:48 PM, Joseph L. Casale
<jcasale@activenetwerx.com> wrote:
> What is the pattern for chaining execution of tasks with ThreadPoolExecutor?
> Callbacks is not an adequate facility as each task I have will generate new output.

Can you specify in more detail what your use case is?

If you don't mind having threads sitting around waiting, you can just
submit each chained task at the start and have each task wait on the
futures of its dependencies. The advantage of this is that it's easy
to conceptualize the dependency graph of the tasks. The disadvantage
is that it eats up extra threads. You'll probably want to increase the
size of the thread pool to handle the waiting tasks (or use a separate
ThreadPoolExecutor for each chained task).

E.g.:

def task2(input, f1):
    f1_result = f1.result()  # Wait for f1 to be done.
    result = frobnicate(input, f1_result)
    return result

f1 = executor.submit(task1, input)
f2 = executor.submit(task2, input, f1)

f1.add_done_callback(handle_task1_done)
f2.add_done_callback(handle_task2_done)



Otherwise, is there some reason you can't use multiple callbacks, one
to handle the task's output and one to submit the chained task?

E.g.:

def chain_task2(input, f2):
    f2 = executor.submit(task2, input, f2.result())
    f2.add_done_callback(handle_task2_done)

f1 = executor.submit(task1, input)
f1.add_done_callback(handle_task1_done)
f1.add_done_callback(functools.partial(chain_task2, input))

[toc] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web