Path: csiph.com!usenet.pasdenom.info!aioe.org!news.stack.nl!newsfeed.xs4all.nl!newsfeed1.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Sender: Andrew McLean <andrew.l.mclean@gmail.com>
Date: Thu, 08 May 2014 22:18:51 +0100
From: Andrew McLean <andrew@andros.org.uk>
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.5.0
MIME-Version: 1.0
To: python-list@python.org
Subject: Re: Real-world use of concurrent.futures
References: <536BD338.4070004@andros.org.uk> <CALwzid=WJA2o_k7qBkCpWFNmuQW-e0GGkgtGEFe_QPGN5-gX2g@mail.gmail.com>
In-Reply-To: <CALwzid=WJA2o_k7qBkCpWFNmuQW-e0GGkgtGEFe_QPGN5-gX2g@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.9798.1399583933.18130.python-list@python.org>
Lines: 25
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:71130

On 08/05/2014 21:44, Ian Kelly wrote:
> I don't think it needs to be "messy". Something like this should do
> the trick, I think:
>
> from concurrent.futures import *
> from itertools import islice
>
> def batched_pool_runner(f, iterable, pool, batch_size):
>   it = iter(iterable)
>   # Submit the first batch of tasks.
>   futures = set(pool.submit(f, x) for x in islice(it, batch_size))
>   while futures:
>     done, futures = wait(futures, return_when=FIRST_COMPLETED)
>     # Replenish submitted tasks up to the number that completed.
>     futures.update(pool.submit(f, x) for x in islice(it, len(done)))
>     yield from done
>

Thank you, that's very neat. It's just the sort of thing I was looking
for. Nice use of itertools.islice and "yield from".

I'll try this out in the next few days and report back.

- Andrew