Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!feeder.erje.net!eu.feeder.erje.net!feeds.phibee-telecom.net!newsfeed.xs4all.nl!newsfeed2.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Sender: Andrew McLean <andrew.l.mclean@gmail.com>
Date: Tue, 13 May 2014 21:09:28 +0100
From: Andrew McLean <andrew@andros.org.uk>
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.5.0
MIME-Version: 1.0
To: python-list@python.org
Subject: Re: Real-world use of concurrent.futures
References: <536BD338.4070004@andros.org.uk> <CALwzid=WJA2o_k7qBkCpWFNmuQW-e0GGkgtGEFe_QPGN5-gX2g@mail.gmail.com>
In-Reply-To: <CALwzid=WJA2o_k7qBkCpWFNmuQW-e0GGkgtGEFe_QPGN5-gX2g@mail.gmail.com>
Content-Type: multipart/alternative; boundary="------------000406050508010900070909"
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.9979.1400011769.18130.python-list@python.org>
Lines: 165
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:71510

This is a multi-part message in MIME format.
--------------000406050508010900070909
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

On 08/05/2014 21:44, Ian Kelly wrote:
> On May 8, 2014 12:57 PM, "Andrew McLean" <lists@andros.org.uk
> <mailto:lists@andros.org.uk>> wrote:
> > So far so good. However, I thought this would be an opportunity to
> > explore concurrent.futures and to see whether it offered any benefits
> > over the more explicit approach discussed above. The problem I am having
> > is that all the discussions I can find of the use of concurrent.futures
> > show use with toy problems involving just a few tasks. The url
> > downloader in the documentation is typical, it proceeds as follows:
> >
> > 1. Get an instance of concurrent.futuresThreadPoolExecutor
> > 2. Submit a few tasks to the executer
> > 3. Iterate over the results using concurrent.futures.as_completed
> >
> > That's fine, but I suspect that isn't a helpful pattern if I have a very
> > large number of tasks. In my case I could run out of memory if I tried
> > submitting all of the tasks to the executor before processing any of the
> > results.
>
> I thought that ThreadPoolExecutor.map would handle this transparently
> if you passed it a lazy iterable such as a generator.  From my testing
> though, that seems not to be the case; with a generator of 100000
> items and a pool of 2 workers, the entire generator was consumed
> before any results were returned.
>
> > I'm guessing what I want to do is, submit tasks in batches of perhaps a
> > few hundred, iterate over the results until most are complete, then
> > submit some more tasks and so on. I'm struggling to see how to do this
> > elegantly without a lot of messy code just there to do "bookkeeping".
> > This can't be an uncommon scenario. Am I missing something, or is this
> > just not a job suitable for futures?
>
> I don't think it needs to be "messy". Something like this should do
> the trick, I think:
>
> from concurrent.futures import *
> from itertools import islice
>
> def batched_pool_runner(f, iterable, pool, batch_size):
>   it = iter(iterable)
>   # Submit the first batch of tasks.
>   futures = set(pool.submit(f, x) for x in islice(it, batch_size))
>   while futures:
>     done, futures = wait(futures, return_when=FIRST_COMPLETED)
>     # Replenish submitted tasks up to the number that completed.
>     futures.update(pool.submit(f, x) for x in islice(it, len(done)))
>     yield from done

That worked very nicely, thank you.  I think that would make a good
recipe, whether for the documentation or elsewhere. I suspect I'm not
the only person that would benefit from something to bridge the gap
between a toy example and something practical.

Andrew




--------------000406050508010900070909
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

<html>
  <head>
    <meta content="text/html; charset=ISO-8859-1"
      http-equiv="Content-Type">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <div class="moz-cite-prefix">On 08/05/2014 21:44, Ian Kelly wrote:<br>
    </div>
    <blockquote
cite="mid:CALwzid=WJA2o_k7qBkCpWFNmuQW-e0GGkgtGEFe_QPGN5-gX2g@mail.gmail.com"
      type="cite">
      <div dir="ltr">On May 8, 2014 12:57 PM, "Andrew McLean" &lt;<a
          moz-do-not-send="true" href="mailto:lists@andros.org.uk">lists@andros.org.uk</a>&gt;

        wrote:<br>
        &gt; So far so good. However, I thought this would be an
        opportunity to<br>
        &gt; explore concurrent.futures and to see whether it offered
        any benefits<br>
        &gt; over the more explicit approach discussed above. The
        problem I am having<br>
        &gt; is that all the discussions I can find of the use of
        concurrent.futures<br>
        &gt; show use with toy problems involving just a few tasks. The
        url<br>
        &gt; downloader in the documentation is typical, it proceeds as
        follows:<br>
        &gt;<br>
        &gt; 1. Get an instance of concurrent.futuresThreadPoolExecutor<br>
        &gt; 2. Submit a few tasks to the executer<br>
        &gt; 3. Iterate over the results using
        concurrent.futures.as_completed<br>
        &gt;<br>
        &gt; That's fine, but I suspect that isn't a helpful pattern if
        I have a very<br>
        &gt; large number of tasks. In my case I could run out of memory
        if I tried<br>
        &gt; submitting all of the tasks to the executor before
        processing any of the<br>
        &gt; results.<br>
        <br>
        I thought that ThreadPoolExecutor.map would handle this
        transparently if you passed it a lazy iterable such as a
        generator. &nbsp;From my testing though, that seems not to be the
        case; with a generator of 100000 items and a pool of 2 workers,
        the entire generator was consumed before any results were
        returned.<br>
        <br>
        &gt; I'm guessing what I want to do is, submit tasks in batches
        of perhaps a<br>
        &gt; few hundred, iterate over the results until most are
        complete, then<br>
        &gt; submit some more tasks and so on. I'm struggling to see how
        to do this<br>
        &gt; elegantly without a lot of messy code just there to do
        "bookkeeping".<br>
        &gt; This can't be an uncommon scenario. Am I missing something,
        or is this<br>
        &gt; just not a job suitable for futures?<br>
        <br>
        I don't think it needs to be "messy". Something like this should
        do the trick, I think:<br>
        <br>
        from concurrent.futures import *<br>
        <div>from itertools import islice<br>
          <div><br>
          </div>
          <div>def batched_pool_runner(f, iterable, pool, batch_size):</div>
          <div>&nbsp; it = iter(iterable)</div>
          <div>&nbsp; # Submit the first batch of tasks.<br>
            &nbsp; futures = set(pool.submit(f, x) for x in islice(it,
            batch_size))<br>
            &nbsp; while futures:<br>
            &nbsp; &nbsp; done, futures = wait(futures,
            return_when=FIRST_COMPLETED)</div>
          <div>&nbsp; &nbsp; # Replenish submitted tasks up to the number that
            completed.<br>
            &nbsp; &nbsp; futures.update(pool.submit(f, x) for x in islice(it,
            len(done)))</div>
          <div>&nbsp; &nbsp; yield from done<br>
          </div>
        </div>
      </div>
    </blockquote>
    <br>
    That worked very nicely, thank you.&nbsp; I think that would make a good
    recipe, whether for the documentation or elsewhere. I suspect I'm
    not the only person that would benefit from something to bridge the
    gap between a toy example and something practical.<br>
    <br>
    Andrew<br>
    <br>
    <br>
    <br>
  </body>
</html>

--------------000406050508010900070909--