Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder1.news.weretis.net!feeder4.news.weretis.net!rt.uk.eu.org!newsfeed.xs4all.nl!newsfeed2a.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.014 X-Spam-Evidence: '*H*': 0.97; '*S*': 0.00; 'output': 0.05; 'explicit': 0.07; 'iterate': 0.09; 'lookup': 0.09; 'record.': 0.09; 'subject:skip:c 10': 0.09; 'thread': 0.14; 'csv': 0.16; 'executor': 0.16; 'from:addr:lists': 0.16; 'guessing': 0.16; 'length)': 0.16; 'messy': 0.16; 'proceeds': 0.16; 'received:74.55.86': 0.16; 'received:74.55.86.74': 0.16; 'received:smtp.webfaction.com': 0.16; 'received:webfaction.com': 0.16; 'uncommon': 0.16; 'discussions': 0.16; 'written': 0.21; 'input': 0.22; 'memory': 0.22; 'putting': 0.22; 'header:User- Agent:1': 0.23; 'removed.': 0.24; 'helpful': 0.24; 'file.': 0.24; 'performing': 0.26; 'second': 0.26; 'tried': 0.27; 'andrew': 0.30; 'involving': 0.30; "i'm": 0.30; 'code': 0.31; 'submitting': 0.31; 'complete,': 0.31; 'fine,': 0.31; 'file': 0.32; 'probably': 0.32; 'skip:c 30': 0.32; 'run': 0.32; 'addresses': 0.33; 'checking': 0.33; 'trouble': 0.34; 'could': 0.34; 'problem': 0.35; "can't": 0.35; 'offered': 0.35; 'good.': 0.35; 'but': 0.35; 'there': 0.35; 'really': 0.36; 'similar': 0.36; 'example,': 0.37; 'problems': 0.38; 'follows:': 0.38; 'tasks': 0.38; 'to:addr:python-list': 0.38; 'to:addr:python.org': 0.39; 'how': 0.40; 'skip:u 10': 0.60; 'read': 0.60; 'results.': 0.60; 'most': 0.60; 'show': 0.63; 'more': 0.64; 'details': 0.65; 'benefit': 0.68; 'invalid': 0.68; 'tasks.': 0.68; 'results': 0.69; 'containing': 0.69; 'records': 0.73; 'fields,': 0.84; 'futures?': 0.84; 'toy': 0.84; 'amongst': 0.91 Date: Thu, 08 May 2014 19:55:52 +0100 From: Andrew McLean User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.5.0 MIME-Version: 1.0 To: python-list@python.org Subject: Real-world use of concurrent.futures Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 48 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1399575348 news.xs4all.nl 2930 [2001:888:2000:d::a6]:57674 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:71118 I have a problem that would benefit from a multithreaded implementation and having trouble understanding how to approach it using concurrent.futures. The details don't really matter, but it will probably help to be explicit. I have a large CSV file that contains a lot of fields, amongst them one containing email addresses. I want to write a program that validates the email addresses by checking that the domain names have a valid MX record. The output will be a copy of the file with any invalid email addresses removed. Because of latency in the DNS lookup this could benefit from multithreading. I have written similar code in the past using explicit threads communicating via queues. For this example, I could have a thread that read the file using csv.DictReader, putting dicts containing records from the input file into a (finite length) queue. Then I would have a number of worker threads reading the queue, performing the validation and putting validated results in a second queue. A final thread would read from the second queue writing the results to the output file. So far so good. However, I thought this would be an opportunity to explore concurrent.futures and to see whether it offered any benefits over the more explicit approach discussed above. The problem I am having is that all the discussions I can find of the use of concurrent.futures show use with toy problems involving just a few tasks. The url downloader in the documentation is typical, it proceeds as follows: 1. Get an instance of concurrent.futuresThreadPoolExecutor 2. Submit a few tasks to the executer 3. Iterate over the results using concurrent.futures.as_completed That's fine, but I suspect that isn't a helpful pattern if I have a very large number of tasks. In my case I could run out of memory if I tried submitting all of the tasks to the executor before processing any of the results. I'm guessing what I want to do is, submit tasks in batches of perhaps a few hundred, iterate over the results until most are complete, then submit some more tasks and so on. I'm struggling to see how to do this elegantly without a lot of messy code just there to do "bookkeeping". This can't be an uncommon scenario. Am I missing something, or is this just not a job suitable for futures? Regards, Andrew