Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #71130

Re: Real-world use of concurrent.futures

Show all headers | View raw

On 08/05/2014 21:44, Ian Kelly wrote:
> I don't think it needs to be "messy". Something like this should do
> the trick, I think:
>
> from concurrent.futures import *
> from itertools import islice
>
> def batched_pool_runner(f, iterable, pool, batch_size):
>   it = iter(iterable)
>   # Submit the first batch of tasks.
>   futures = set(pool.submit(f, x) for x in islice(it, batch_size))
>   while futures:
>     done, futures = wait(futures, return_when=FIRST_COMPLETED)
>     # Replenish submitted tasks up to the number that completed.
>     futures.update(pool.submit(f, x) for x in islice(it, len(done)))
>     yield from done
>

Thank you, that's very neat. It's just the sort of thing I was looking
for. Nice use of itertools.islice and "yield from".

I'll try this out in the next few days and report back.

- Andrew

Back to comp.lang.python | Previous | Next | Find similar | Unroll thread

Thread

Re: Real-world use of concurrent.futures Andrew McLean <andrew@andros.org.uk> - 2014-05-08 22:18 +0100

csiph-web