Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!feeder.erje.net!eu.feeder.erje.net!eternal-september.org!feeder.eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail From: Marko Rauhamaa Newsgroups: comp.lang.python Subject: Re: Real-world use of concurrent.futures Date: Thu, 08 May 2014 22:56:02 +0300 Organization: A noiseless patient Spider Lines: 16 Message-ID: <87ppjot57x.fsf@elektro.pacujo.net> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Injection-Info: mx05.eternal-september.org; posting-host="ff5cf27ef3d5b31f034d3b72bdc27a41"; logging-data="6153"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX18P1G9PHISA9HueshSPNkuZ" User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.3 (gnu/linux) Cancel-Lock: sha1:XE3ErluA/fk0KXQtEzcx6MxC7Jc= sha1:Hh9Wszt5VJnbdukD86ajAxUQ3BY= Xref: csiph.com comp.lang.python:71125 Andrew McLean : > That's fine, but I suspect that isn't a helpful pattern if I have a > very large number of tasks. In my case I could run out of memory if I > tried submitting all of the tasks to the executor before processing > any of the results. This is related to flow control. You'll need an object for each flow (transaction). When new work comes in from the network, you'll have to see if you are hitting the maximum number of pending transactions, and not start another one before previous transactions have been processed. Whenever a transaction is completed, you pull in more work. Marko