Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #53384 > unrolled thread

gethostbyname_ex(hostname) extremely slow (crossposted from stackoverflow)

Back to article view | Back to comp.lang.python

Contents

  gethostbyname_ex(hostname) extremely slow (crossposted from stackoverflow) anntzer.lee@gmail.com - 2013-08-31 17:03 -0700
    Re: gethostbyname_ex(hostname) extremely slow (crossposted from stackoverflow) Roy Smith <roy@panix.com> - 2013-08-31 21:01 -0400
      Re: gethostbyname_ex(hostname) extremely slow (crossposted from stackoverflow) anntzer.lee@gmail.com - 2013-08-31 21:51 -0700
        Re: gethostbyname_ex(hostname) extremely slow (crossposted from stackoverflow) Michael Torrie <torriem@gmail.com> - 2013-08-31 23:06 -0600
          Re: gethostbyname_ex(hostname) extremely slow (crossposted from stackoverflow) anntzer.lee@gmail.com - 2013-09-01 10:13 -0700
        Re: gethostbyname_ex(hostname) extremely slow (crossposted from stackoverflow) Roy Smith <roy@panix.com> - 2013-09-01 06:57 -0400
    Re: gethostbyname_ex(hostname) extremely slow (crossposted from stackoverflow) Chris Angelico <rosuav@gmail.com> - 2013-09-01 21:37 +1000
      Re: gethostbyname_ex(hostname) extremely slow (crossposted from stackoverflow) anntzer.lee@gmail.com - 2013-09-01 13:37 -0700
        Re: gethostbyname_ex(hostname) extremely slow (crossposted from stackoverflow) Chris Angelico <rosuav@gmail.com> - 2013-09-02 07:03 +1000
          Re: gethostbyname_ex(hostname) extremely slow (crossposted from stackoverflow) anntzer.lee@gmail.com - 2013-09-01 22:28 -0700
            Re: gethostbyname_ex(hostname) extremely slow (crossposted from stackoverflow) Chris Angelico <rosuav@gmail.com> - 2013-09-02 20:42 +1000
            Re: gethostbyname_ex(hostname) extremely slow (crossposted from stackoverflow) Roy Smith <roy@panix.com> - 2013-09-02 08:45 -0400
              Re: gethostbyname_ex(hostname) extremely slow (crossposted from stackoverflow) anntzer.lee@gmail.com - 2013-09-02 10:52 -0700
                Re: gethostbyname_ex(hostname) extremely slow (crossposted from stackoverflow) Chris Angelico <rosuav@gmail.com> - 2013-09-03 07:12 +1000

#53384 — gethostbyname_ex(hostname) extremely slow (crossposted from stackoverflow)

Hi,

At startup, IPython (qtconsole) calls "socket.gethostbyname_ex(socket.gethostname())[2]" to find a list of IP addresses that point to the machine. On a Linux server that I manage this call is extremely slow (>20s)... which I have trouble understanding as "ip addr show" seems to give the same information nearly instantaneously. Is there anything I can do to make this faster? Can this be a network configuration issue (I am behind a router)?

This issue is independent of IPython:

$ time python -c 'import socket; print(socket.gethostbyname_ex(socket.gethostname())[2])' 
['192.168.0.102']
python -c   0.07s user 0.02s system 0% cpu 28.190 total

Thanks.

Antony

[toc] | [next] | [standalone]

#53392

In article <b9f77b6f-3a65-407a-aff5-5677be2ba228@googlegroups.com>,
 anntzer.lee@gmail.com wrote:

> Hi,
> 
> At startup, IPython (qtconsole) calls 
> "socket.gethostbyname_ex(socket.gethostname())[2]" to find a list of IP 
> addresses that point to the machine. On a Linux server that I manage this 
> call is extremely slow (>20s)... which I have trouble understanding as "ip 
> addr show" seems to give the same information nearly instantaneously. Is 
> there anything I can do to make this faster? Can this be a network 
> configuration issue (I am behind a router)?

It's almost certainly some sort of configuration issue which is causing 
some DNS query to timeout.  The first step, is to figure out which part 
is taking so long.  Open up a python shell and run:

>>> name = socket.gethostname()

see how long that takes and what it returns.  Then, assuming it returns 
a string containing your hostname (massive handwave about what that 
actually means), try

>>> socket.gethostbyname_ex(name)

and see how long that takes and what it returns.  At least at that point 
you'll have cut the problem in half.

If I had to guess, you've got a missing PTR record, because that's what 
most people screw up.

[toc] | [prev] | [next] | [standalone]

#53397

It is the call to gethostbyname_ex that is very slow.  The call to gethostname is quick (and returns the same string as /usr/bin/hostname).

On Saturday, August 31, 2013 6:01:00 PM UTC-7, Roy Smith wrote:
> In article <b9f77b6f-3a65-407a-aff5-5677be2ba228@googlegroups.com>,
> 
>  anntzer.lee@gmail.com wrote:
> 
> 
> 
> > Hi,
> 
> > 
> 
> > At startup, IPython (qtconsole) calls 
> 
> > "socket.gethostbyname_ex(socket.gethostname())[2]" to find a list of IP 
> 
> > addresses that point to the machine. On a Linux server that I manage this 
> 
> > call is extremely slow (>20s)... which I have trouble understanding as "ip 
> 
> > addr show" seems to give the same information nearly instantaneously. Is 
> 
> > there anything I can do to make this faster? Can this be a network 
> 
> > configuration issue (I am behind a router)?
> 
> 
> 
> It's almost certainly some sort of configuration issue which is causing 
> 
> some DNS query to timeout.  The first step, is to figure out which part 
> 
> is taking so long.  Open up a python shell and run:
> 
> 
> 
> >>> name = socket.gethostname()
> 
> 
> 
> see how long that takes and what it returns.  Then, assuming it returns 
> 
> a string containing your hostname (massive handwave about what that 
> 
> actually means), try
> 
> 
> 
> >>> socket.gethostbyname_ex(name)
> 
> 
> 
> and see how long that takes and what it returns.  At least at that point 
> 
> you'll have cut the problem in half.
> 
> 
> 
> If I had to guess, you've got a missing PTR record, because that's what 
> 
> most people screw up.

[toc] | [prev] | [next] | [standalone]

#53398

On 08/31/2013 10:51 PM, anntzer.lee@gmail.com wrote:
> It is the call to gethostbyname_ex that is very slow.  The call to
> gethostname is quick (and returns the same string as
> /usr/bin/hostname).

What gethostbyname_ex and /usr/bin/hostname do are very different
things.  gethostbyname_ex does a DNS lookup against a server.
/usr/bin/hostname just checks a local computer setting.  I don't see why
you are comparing the two.  /usr/bin/hostname is not going to help you
find a list of IP addresses that point to a machine.

[toc] | [prev] | [next] | [standalone]

#53420

On Saturday, August 31, 2013 10:06:43 PM UTC-7, Michael Torrie wrote:
> On 08/31/2013 10:51 PM, anntzer.lee@gmail.com wrote:
> 
> > It is the call to gethostbyname_ex that is very slow.  The call to
> > gethostname is quick (and returns the same string as
> > /usr/bin/hostname).
> 
> What gethostbyname_ex and /usr/bin/hostname do are very different
> things.  gethostbyname_ex does a DNS lookup against a server.
> /usr/bin/hostname just checks a local computer setting.  I don't see why
> you are comparing the two.  /usr/bin/hostname is not going to help you
> find a list of IP addresses that point to a machine.

I was just replying to the previous comment "name = socket.gethostname() see how long that takes and what it returns.  Then, assuming it returns a string containing your hostname (massive handwave about what that actually means)", saying that gethostname resolves my own hostname instantaneously.

[toc] | [prev] | [next] | [standalone]

#53410

In article <48c8f8ca-d8c1-4a60-ba7f-8e8b00993f15@googlegroups.com>,
 anntzer.lee@gmail.com wrote:

> It is the call to gethostbyname_ex that is very slow.  The call to 
> gethostname is quick (and returns the same string as /usr/bin/hostname).

First, please stop posting with Google Groups.  It makes a total mess of 
the quoted text.

Next, what happens when you use the command-line tools (nslookup or dig) 
to translate your hostname to an IP address, and what happens if you 
then try to reverse translate that IP address back to a name?

> This issue is independent of IPython:
> 
> $ time python -c 'import socket; 
> print(socket.gethostbyname_ex(socket.gethostname())[2])' 
> ['192.168.0.102']
> python -c   0.07s user 0.02s system 0% cpu 28.190 total

The real point is not that it's independent of IPython, it has nothing 
to do with Python at all.  What you've got here is a network 
configuration issue.  You would do better to recreate this problem with 
the native OS command line tools and then ask about it on a forum 
dedicated to your operating system.

[toc] | [prev] | [next] | [standalone]

#53413

On Sun, Sep 1, 2013 at 10:03 AM,  <anntzer.lee@gmail.com> wrote:
> At startup, IPython (qtconsole) calls "socket.gethostbyname_ex(socket.gethostname())[2]" to find a list of IP addresses that point to the machine. On a Linux server that I manage this call is extremely slow (>20s)... which I have trouble understanding as "ip addr show" seems to give the same information nearly instantaneously. Is there anything I can do to make this faster? Can this be a network configuration issue (I am behind a router)?

Yes, it most definitely CAN be a network config issue. The C function
you want to be calling is getifaddrs(), and I don't think there's a
way to call that from core Python. But a Google search for 'python
getifaddrs' shows up a few third-party modules that might be of use to
you; that'd be a lot quicker and more reliable than trying to look up
your own hostname and depending on the results.

ChrisA

[toc] | [prev] | [next] | [standalone]

#53430

On Sunday, September 1, 2013 4:37:34 AM UTC-7, Chris Angelico wrote:

> Yes, it most definitely CAN be a network config issue. The C function
> you want to be calling is getifaddrs(), and I don't think there's a
> way to call that from core Python. But a Google search for 'python
> getifaddrs' shows up a few third-party modules that might be of use to
> you; that'd be a lot quicker and more reliable than trying to look up
> your own hostname and depending on the results.
> 
> ChrisA

I tried using netifaces (https://pypi.python.org/pypi/netifaces) which seems to rely on getifaddrs (according to the doc, I didn't check the source).  Again, it returns nearly instantaneously the correct IP address.

[toc] | [prev] | [next] | [standalone]

#53433

On Mon, Sep 2, 2013 at 6:37 AM,  <anntzer.lee@gmail.com> wrote:
> On Sunday, September 1, 2013 4:37:34 AM UTC-7, Chris Angelico wrote:
>
>> Yes, it most definitely CAN be a network config issue. The C function
>> you want to be calling is getifaddrs(), and I don't think there's a
>> way to call that from core Python. But a Google search for 'python
>> getifaddrs' shows up a few third-party modules that might be of use to
>> you; that'd be a lot quicker and more reliable than trying to look up
>> your own hostname and depending on the results.
>>
>> ChrisA
>
> I tried using netifaces (https://pypi.python.org/pypi/netifaces) which seems to rely on getifaddrs (according to the doc, I didn't check the source).  Again, it returns nearly instantaneously the correct IP address.

Perfect!

ChrisA

[toc] | [prev] | [next] | [standalone]

#53457

On Sunday, September 1, 2013 2:03:56 PM UTC-7, Chris Angelico wrote:

> > I tried using netifaces (https://pypi.python.org/pypi/netifaces) which seems to rely on getifaddrs (according to the doc, I didn't check the source).  Again, it returns nearly instantaneously the correct IP address.
> 
> Perfect!
> 
> ChrisA

Not really for my use case -- it isn't that *I* want to know my public IP address, but rather that IPython wants to know it.  Of course I could patch IPython's source to use netifaces but that sounds like an overkill.

As it happens I found a better way: just add the proper entry to /etc/hosts.

Still, thanks for the suggestions.

Antony

[toc] | [prev] | [next] | [standalone]

#53470

On Mon, Sep 2, 2013 at 3:28 PM,  <anntzer.lee@gmail.com> wrote:
> On Sunday, September 1, 2013 2:03:56 PM UTC-7, Chris Angelico wrote:
>
>> > I tried using netifaces (https://pypi.python.org/pypi/netifaces) which seems to rely on getifaddrs (according to the doc, I didn't check the source).  Again, it returns nearly instantaneously the correct IP address.
>>
>> Perfect!
>>
>> ChrisA
>
> Not really for my use case -- it isn't that *I* want to know my public IP address, but rather that IPython wants to know it.  Of course I could patch IPython's source to use netifaces but that sounds like an overkill.
>
> As it happens I found a better way: just add the proper entry to /etc/hosts.

Ah, I see what you mean.

If it were possible with core Python, I would recommend raising a bug
with IPython, as the current method is susceptible to external issues.
But to just get your problem solved, yes, a host file entry sounds
like it's the best way to go.

ChrisA

[toc] | [prev] | [next] | [standalone]

#53481

In article <00843d58-db21-4cf0-9430-85362a1dd66f@googlegroups.com>,
 anntzer.lee@gmail.com wrote:

> As it happens I found a better way: just add the proper entry to /etc/hosts.

You have not found a better way.  You still have a network (or more 
specifically, DNS) configuration that's broken.

What you have found is a pragmatic way to solve your immediate problem 
and get some work done.  That is certainly useful (and I've done it 
plenty of times), but you need to understand that what you've done is 
hidden the problem, not solved it.

[toc] | [prev] | [next] | [standalone]

#53517

On Monday, September 2, 2013 5:45:26 AM UTC-7, Roy Smith wrote:
> In article <00843d58-db21-4cf0-9430-85362a1dd66f@googlegroups.com>,
>  anntzer.lee@gmail.com wrote:
> 
> > As it happens I found a better way: just add the proper entry to /etc/hosts.
> 
> You have not found a better way.  You still have a network (or more 
> specifically, DNS) configuration that's broken.
> 
> What you have found is a pragmatic way to solve your immediate problem 
> and get some work done.  That is certainly useful (and I've done it 
> plenty of times), but you need to understand that what you've done is 
> hidden the problem, not solved it.

To be honest, knowing nothing about DNS configuration, I don't even know if adding the entry to /etc/hosts is the "proper" fix or if the issue should be fixed somewhere else (or perhaps "didn't know", as you seem to imply that that is not the correct way).

[toc] | [prev] | [next] | [standalone]

#53528

On Tue, Sep 3, 2013 at 3:52 AM,  <anntzer.lee@gmail.com> wrote:
> To be honest, knowing nothing about DNS configuration, I don't even know if adding the entry to /etc/hosts is the "proper" fix or if the issue should be fixed somewhere else (or perhaps "didn't know", as you seem to imply that that is not the correct way).

Since you can't change the code, it's almost certainly the best
solution available to you. It's more-or-less equivalent to speeding up
your DNS lookups massively. The main downside is that if something
changes, you need to change it in DNS and in your hosts file; as Roy
says, it's a pragmatic solution rather than a perfect one.

ChrisA

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web