Path: csiph.com!fu-berlin.de!uni-berlin.de!not-for-mail
From: "Joseph L. Casale" <jcasale@activenetwerx.com>
Newsgroups: comp.lang.python
Subject: Re: Passing data across callbacks in ThreadPoolExecutor
Date: Thu, 18 Feb 2016 19:06:29 +0000
Lines: 41
Message-ID: <mailman.18.1455822696.2289.python-list@python.org>
References: <fe3d0dcf47cb4f378b7a2d2e38d00ed4@activenetwerx.com>, <CALwzid=vU_MYEyqimR9tHT7Q86PdUKVu3dVvkn21wDKPSpMiyA@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Thread-Topic: Passing data across callbacks in ThreadPoolExecutor
Thread-Index: AQHRaQOLnYiuCC9hLkqY+oiLW/4uEJ8w4q2AgAFIQtA=
In-Reply-To: <CALwzid=vU_MYEyqimR9tHT7Q86PdUKVu3dVvkn21wDKPSpMiyA@mail.gmail.com>
Accept-Language: en-CA, en-US
Content-Language: en-CA
Precedence: list
Xref: csiph.com comp.lang.python:103144

On Thur, Feb 17, 2016 at 9:24 AM, Ian Kelly <ian.g.kelly@gmail.com> wrote:
>> What is the pattern for chaining execution of tasks with ThreadPoolExecu=
tor?
>> Callbacks is not an adequate facility as each task I have will generate =
new output.
>
> Can you specify in more detail what your use case is?
>
> If you don't mind having threads sitting around waiting, you can just
> submit each chained task at the start and have each task wait on the
> futures of its dependencies. The advantage of this is that it's easy
> to conceptualize the dependency graph of the tasks. The disadvantage
> is that it eats up extra threads. You'll probably want to increase the
> size of the thread pool to handle the waiting tasks (or use a separate
> ThreadPoolExecutor for each chained task).

The thing with callbacks is each one gets the original tasks result. So the
callback can't pass a new task result up and/or cancel the task "set".

> Otherwise, is there some reason you can't use multiple callbacks, one
> to handle the task's output and one to submit the chained task?
>=20
> E.g.:
>=20
> def chain_task2(input, f2):
>     f2 =3D executor.submit(task2, input, f2.result())
>     f2.add_done_callback(handle_task2_done)
>=20
> f1 =3D executor.submit(task1, input)
> f1.add_done_callback(handle_task1_done)
> f1.add_done_callback(functools.partial(chain_task2, input))

Closing over the executer instance seems potentially race'y. What happens w=
hen
the last task finishes and the executer checks (locking the queue) and the =
task also
wants to add more work. I'll check the source, but I had hoped for a more s=
uccinct way.

Thanks,
jlc