Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #71130 > unrolled thread

Re: Real-world use of concurrent.futures

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Real-world use of concurrent.futures Andrew McLean <andrew@andros.org.uk> - 2014-05-08 22:18 +0100

#71130 — Re: Real-world use of concurrent.futures

On 08/05/2014 21:44, Ian Kelly wrote:
> I don't think it needs to be "messy". Something like this should do
> the trick, I think:
>
> from concurrent.futures import *
> from itertools import islice
>
> def batched_pool_runner(f, iterable, pool, batch_size):
>   it = iter(iterable)
>   # Submit the first batch of tasks.
>   futures = set(pool.submit(f, x) for x in islice(it, batch_size))
>   while futures:
>     done, futures = wait(futures, return_when=FIRST_COMPLETED)
>     # Replenish submitted tasks up to the number that completed.
>     futures.update(pool.submit(f, x) for x in islice(it, len(done)))
>     yield from done
>

Thank you, that's very neat. It's just the sort of thing I was looking
for. Nice use of itertools.islice and "yield from".

I'll try this out in the next few days and report back.

- Andrew

[toc] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web