Path: csiph.com!usenet.pasdenom.info!dedibox.gegeweb.org!gegeweb.eu!nntpfeed.proxad.net!proxad.net!feeder1-2.proxad.net!usenet-fr.net!nerim.net!novso.com!newsfeed.xs4all.nl!newsfeed3.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Return-Path: <python-python-list@m.gmane.org>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.013
X-Spam-Evidence: '*H*': 0.97; '*S*': 0.00; 'output': 0.05; 'explicit': 0.07; 'lookup': 0.09; 'received:80.91': 0.09; 'received:80.91.229': 0.09; 'received:gmane.org': 0.09; 'received:list': 0.09; 'record.': 0.09; 'subject:skip:c 10': 0.09; 'things,': 0.09; 'jan': 0.12; 'thread': 0.14; 'csv': 0.16; 'length)': 0.16; 'received:80.91.229.3': 0.16; 'received:plane.gmane.org': 0.16; 'reedy': 0.16; 'discussions': 0.16; 'wrote:': 0.18; 'module': 0.19; 'written': 0.21; 'input': 0.22; 'putting': 0.22; 'header:User-Agent:1': 0.23; 'removed.': 0.24; 'file.': 0.24; 'performing': 0.26; 'second': 0.26; 'header:X -Complaints-To:1': 0.27; 'header:In-Reply-To:1': 0.27; 'andrew': 0.30; 'involving': 0.30; 'code': 0.31; 'file': 0.32; 'probably': 0.32; 'addresses': 0.33; 'checking': 0.33; 'trouble': 0.34; 'could': 0.34; 'problem': 0.35; 'offered': 0.35; 'good.': 0.35; 'but': 0.35; 'really': 0.36; 'similar': 0.36; 'example,': 0.37; 'problems': 0.38; 'to:addr:python-list': 0.38; 'pm,': 0.38; 'to:addr:python.org': 0.39; 'received:org': 0.40; 'how': 0.40; 'skip:u 10': 0.60; 'read': 0.60; 'new': 0.61; 'received:173': 0.61; 'show': 0.63; 'more': 0.64; 'details': 0.65; 'benefit': 0.68; 'invalid': 0.68; 'tasks.': 0.68; 'results': 0.69; 'containing': 0.69; 'records': 0.73; '3.4': 0.84; 'fields,': 0.84; 'received:fios.verizon.net': 0.84; 'toy': 0.84; 'amongst': 0.91
X-Injected-Via-Gmane: http://gmane.org/
To: python-list@python.org
From: Terry Reedy <tjreedy@udel.edu>
Subject: Re: Real-world use of concurrent.futures
Date: Thu, 08 May 2014 16:27:03 -0400
References: <536BD338.4070004@andros.org.uk>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Gmane-NNTP-Posting-Host: pool-173-75-254-207.phlapa.fios.verizon.net
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.5.0
In-Reply-To: <536BD338.4070004@andros.org.uk>
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups: comp.lang.python
Message-ID: <mailman.9796.1399580844.18130.python-list@python.org>
Lines: 34
NNTP-Posting-Host: 2001:888:2000:d::a6
X-Trace: 1399580844 news.xs4all.nl 2867 [2001:888:2000:d::a6]:35502
X-Complaints-To: abuse@xs4all.nl
Xref: csiph.com comp.lang.python:71128

On 5/8/2014 2:55 PM, Andrew McLean wrote:
> I have a problem that would benefit from a multithreaded implementation
> and having trouble understanding how to approach it using
> concurrent.futures.
>
> The details don't really matter, but it will probably help to be
> explicit. I have a large CSV file that contains a lot of fields, amongst
> them one containing email addresses. I want to write a program that
> validates the email addresses by checking that the domain names have a
> valid MX record. The output will be a copy of the file with any invalid
> email addresses removed. Because of latency in the DNS lookup this could
> benefit from multithreading.
>
> I have written similar code in the past using explicit threads
> communicating via queues. For this example, I could have a thread that
> read the file using csv.DictReader, putting dicts containing records
> from the input file into a (finite length) queue. Then I would have a
> number of worker threads reading the queue, performing the validation
> and putting validated results in a second queue. A final thread would
> read from the second queue writing the results to the output file.
>
> So far so good. However, I thought this would be an opportunity to
> explore concurrent.futures and to see whether it offered any benefits
> over the more explicit approach discussed above. The problem I am having
> is that all the discussions I can find of the use of concurrent.futures
> show use with toy problems involving just a few tasks.

You might look as the new asyncio module in 3.4 (backport available on 
pypi, I believe). Among other things, it uses a variation on 
concurrent.futures. It includes timeouts.

-- 
Terry Jan Reedy