Path: csiph.com!usenet.pasdenom.info!news.albasani.net!rt.uk.eu.org!newsfeed.xs4all.nl!newsfeed2a.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Sender: Andrew McLean <andrew.l.mclean@gmail.com>
Date: Thu, 08 May 2014 20:25:43 +0100
From: Andrew McLean <andrew@andros.org.uk>
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.5.0
MIME-Version: 1.0
To: python-list@python.org
Subject: Re: Real-world use of concurrent.futures
References: <536BD338.4070004@andros.org.uk> <CAPTjJmpc_HeuyiqW=ZSo6cRp3wRBdLq6r2gDcNxh+-4sjNSx1A@mail.gmail.com>
In-Reply-To: <CAPTjJmpc_HeuyiqW=ZSo6cRp3wRBdLq6r2gDcNxh+-4sjNSx1A@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.9795.1399577599.18130.python-list@python.org>
Lines: 49
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:71123

On 08/05/2014 20:06, Chris Angelico wrote:
> On Fri, May 9, 2014 at 4:55 AM, Andrew McLean <lists@andros.org.uk> wrote:
>> Because of latency in the DNS lookup this could
>> benefit from multithreading.
> Before you go too far down roads that are starting to look
> problematic: A DNS lookup is a UDP packet out and a UDP packet in
> (ignoring the possibility of TCP queries, which you probably won't be
> doing here). Maybe it would be easier to implement it as asynchronous
> networking? I don't know that Python makes it easy for you to
> construct DNS requests and parse DNS responses; that's something more
> in Pike's line of work. But it may be more possible to outright do the
> DNS query asynchronously. TBH I haven't looked into it; but it's
> another option to consider.
>
> Separately from your programming model, though, how are you handling
> timeouts? Any form of DNS error (NXDOMAIN being the most likely), and
> the sort-of-error-but-not-error state of getting a response with no
> answer, indicates that the address is invalid; but what if you just
> don't hear back from the server? Will that mark addresses off as dead?
>
> ChrisA

I've done this on a very small scale in the past. I used

http://www.dnspython.org/

to do the heavy lifting. The relevant bits of code looks like:

> # Set up the default dns resolver and add a cache
> dns.resolver.default_resolver = dns.resolver.Resolver()
> dns.resolver.default_resolver.cache = dns.resolver.Cache()
and
>     try:
>         result = dns.resolver.query(domain, 'MX')
>         return True
>     except dns.resolver.NXDOMAIN:
>         return False
>     except dns.resolver.NoAnswer:
>         return False
>     except dns.resolver.Timeout:
>         print "*** timeout looking for the MX record for the domain:
> %s" % domain
>         return False

You are right, I'll need to do something more sophisticated when I
encounter a timeout, but I think that's a matter of detail.

Andrew