Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #178949 > unrolled thread

TTL expired in transit to qemu virtual machine.

Back to article view | Back to linux.debian.user

Contents

  TTL expired in transit to qemu virtual machine. Mimiko <vbvbrj@gmail.com> - 2017-03-17 12:00 +0100
    Re: TTL expired in transit to qemu virtual machine. Dominik George <nik@naturalnet.de> - 2017-03-17 12:30 +0100
      Re: TTL expired in transit to qemu virtual machine. Mimiko <vbvbrj@gmail.com> - 2017-03-20 07:30 +0100
    Re: TTL expired in transit to qemu virtual machine. Igor Cicimov <icicimov@gmail.com> - 2017-03-18 06:30 +0100
      Re: TTL expired in transit to qemu virtual machine. Mimiko <vbvbrj@gmail.com> - 2017-03-20 07:50 +0100
        Re: TTL expired in transit to qemu virtual machine. davidson@freevolt.org - 2017-03-21 01:30 +0100
          Re: TTL expired in transit to qemu virtual machine. Mimiko <vbvbrj@gmail.com> - 2017-03-23 07:40 +0100
            Re: TTL expired in transit to qemu virtual machine. Greg Wooledge <wooledg@eeg.ccf.org> - 2017-03-23 13:20 +0100
              Re: TTL expired in transit to qemu virtual machine. Mimiko <vbvbrj@gmail.com> - 2017-03-24 12:00 +0100

#178949 — TTL expired in transit to qemu virtual machine.

Hello.

I've setup qemu/kvm and installed several virtual machines. Access and ping to some virtuals are ok, but one have a stable problem not receiving 
correctly packets. First, this is the environment:

 >uname -a
Linux 3.2.0-4-amd64 #1 SMP Debian 3.2.84-1 x86_64 GNU/Linux

 >libvirtd --version
libvirtd (libvirt) 0.9.12.3

 >cat /etc/network/interfaces
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet manual

auto eth1
iface eth1 inet manual

auto bond0
iface bond0 inet manual
         bond-slaves eth0 eth1
         bond-mode balance-alb
         bond-miimon 100
         bond-downdelay 200
         bond-updelay 200

auto br0
iface br0 inet static
         address 10.10.10.10
         netmask 255.255.0.0
         vlan-raw-device bond0
         bridge_ports bond0
         bridge_stp off
         bridge_fd 0
         bridge_ageing 0
         bridge_maxwait 0

Virtual machines connects to LAN using bridge:
 >virt-install .... --network=bridge:br0,model=virtio ....

All virtuals has networking configuret like that. Also in iptables is an entry to allow access to virtuals:

 >iptables -L FORWARD -nv
Chain FORWARD (policy DROP 0 packets, 0 bytes)
  pkts bytes target     prot opt in     out     source               destination
  X    X     ACCEPT     all  --  br0    br0     0.0.0.0/0            0.0.0.0/0

Most virtuals does not have networking problems, but some times they can't be reached. For now only one virtual machines have this problem:
 From windows machine ping virtual machine:

 >ping 10.10.10.3

Reply from 10.10.10.10: TTL expired in transit.
Reply from 10.10.10.10: TTL expired in transit.
Reply from 10.10.10.10: TTL expired in transit.
Reply from 10.10.10.10: TTL expired in transit.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Reply from 10.10.10.10: TTL expired in transit.
Reply from 10.10.10.10: TTL expired in transit.
Reply from 10.10.10.10: TTL expired in transit.
Reply from 10.10.10.10: TTL expired in transit.

 >tracert -d 10.10.10.3

Tracing route to 10.10.10.3 over a maximum of 30 hops

   1    <1 ms    <1 ms    <1 ms  10.10.10.10
   2    <1 ms    <1 ms    <1 ms  10.10.10.10
   3    <1 ms    <1 ms     *     10.10.10.10
   4     1 ms    <1 ms    <1 ms  10.10.10.10
   5    <1 ms    <1 ms     *     10.10.10.10

So packet goes round on interfaces of server hosting virtuals.

Virtuals are linux different flavour and one windows. This problem may occur on any of this virtuals.

I've observed that for this particular virtual, which have problem, the arp table of host assigned self mac to the virtual's IP, not the mac 
configured for virtual machine.

What could be the problem?

-- 
Mimiko desu.

[toc] | [next] | [standalone]

#178950

Hi,

> >iptables -L FORWARD -nv
>Chain FORWARD (policy DROP 0 packets, 0 bytes)
>pkts bytes target     prot opt in     out     source              
>destination
>X    X     ACCEPT     all  --  br0    br0     0.0.0.0/0           
>0.0.0.0/0

What is that supposed to do?

Forwarding on the IP layer, by definition, happens between different interfaces.

Although this rule is most likely a no-op, you should remove it to reduce confusion.

-nik

[toc] | [prev] | [next] | [standalone]

#179029

On 17.03.2017 13:21, Dominik George wrote:
>>> iptables -L FORWARD -nv
>> Chain FORWARD (policy DROP 0 packets, 0 bytes)
>> pkts bytes target     prot opt in     out     source
>> destination
>> X    X     ACCEPT     all  --  br0    br0     0.0.0.0/0
>> 0.0.0.0/0
> What is that supposed to do?
>
> Forwarding on the IP layer, by definition, happens between different interfaces.
>
> Although this rule is most likely a no-op, you should remove it to reduce confusion.

This rule allows traffic to virtuals. Without this rule virtuals does not have access to LAN. Yes, I know that bridge setup does not need iptables, 
but reality is this.


-- 
Mimiko desu.

[toc] | [prev] | [next] | [standalone]

#178986

[Multipart message — attachments visible in raw view] — view raw

Hi Mimiko,

On Fri, Mar 17, 2017 at 9:58 PM, Mimiko <vbvbrj@gmail.com> wrote:

> Hello.
>
> I've setup qemu/kvm and installed several virtual machines. Access and
> ping to some virtuals are ok, but one have a stable problem not receiving
> correctly packets. First, this is the environment:
>
> >uname -a
> Linux 3.2.0-4-amd64 #1 SMP Debian 3.2.84-1 x86_64 GNU/Linux
>

That's an really old kernel, I don't start anything virtual these days
without at least 3.13.x kernel.


>
> >libvirtd --version
> libvirtd (libvirt) 0.9.12.3
>
> >cat /etc/network/interfaces
> auto lo
> iface lo inet loopback
>
> auto eth0
> iface eth0 inet manual
>
> auto eth1
> iface eth1 inet manual
>
> auto bond0
> iface bond0 inet manual
>         bond-slaves eth0 eth1
>         bond-mode balance-alb
>         bond-miimon 100
>         bond-downdelay 200
>         bond-updelay 200
>
> auto br0
> iface br0 inet static
>         address 10.10.10.10
>         netmask 255.255.0.0
>         vlan-raw-device bond0
>         bridge_ports bond0
>         bridge_stp off
>         bridge_fd 0
>         bridge_ageing 0
>         bridge_maxwait 0
>

Hmmm, this doesn't make much sense to me, more specifically this part:

        vlan-raw-device bond0
        bridge_ports bond0

Whats the purpose exactly of the vlan? Usually, and that is how I do it,
you would split the VLAN's coming from the switch trunk port over the bond
and attach them to separate bridges lets say:

# VLAN51
auto br-vlan51
iface br-vlan51 inet manual
    vlan-raw-device bond0
    bridge_ports *bond0.51*
    bridge_stp off
    bridge_maxwait 0
    bridge_fd 0

# VLAN52
auto br-vlan52
iface br-vlan52 inet manual
    vlan-raw-device bond0
    bridge_ports *bond0.52*
    bridge_stp off
    bridge_maxwait 0
    bridge_fd 0

If the intention was to pass through the tagged traffic to the VM's then
vlan-raw-device part is not needed at all.


> Virtual machines connects to LAN using bridge:
> >virt-install .... --network=bridge:br0,model=virtio ....
>
> All virtuals has networking configuret like that. Also in iptables is an
> entry to allow access to virtuals:
>
> >iptables -L FORWARD -nv
> Chain FORWARD (policy DROP 0 packets, 0 bytes)
>  pkts bytes target     prot opt in     out     source
>  destination
>  X    X     ACCEPT     all  --  br0    br0     0.0.0.0/0
> 0.0.0.0/0
>
> Most virtuals does not have networking problems, but some times they can't
> be reached. For now only one virtual machines have this problem:
> From windows machine ping virtual machine:
>
> >ping 10.10.10.3
>
> Reply from 10.10.10.10: TTL expired in transit.
> Reply from 10.10.10.10: TTL expired in transit.
> Reply from 10.10.10.10: TTL expired in transit.
> Reply from 10.10.10.10: TTL expired in transit.
> Request timed out.
> Request timed out.
> Request timed out.
> Request timed out.
> Request timed out.
> Request timed out.
> Reply from 10.10.10.10: TTL expired in transit.
> Reply from 10.10.10.10: TTL expired in transit.
> Reply from 10.10.10.10: TTL expired in transit.
> Reply from 10.10.10.10: TTL expired in transit.
>
> >tracert -d 10.10.10.3
>
> Tracing route to 10.10.10.3 over a maximum of 30 hops
>
>   1    <1 ms    <1 ms    <1 ms  10.10.10.10
>   2    <1 ms    <1 ms    <1 ms  10.10.10.10
>   3    <1 ms    <1 ms     *     10.10.10.10
>   4     1 ms    <1 ms    <1 ms  10.10.10.10
>   5    <1 ms    <1 ms     *     10.10.10.10
>
> So packet goes round on interfaces of server hosting virtuals.
>

Yep typical routing loop.


>
> Virtuals are linux different flavour and one windows. This problem may
> occur on any of this virtuals.
>
> I've observed that for this particular virtual, which have problem, the
> arp table of host assigned self mac to the virtual's IP, not the mac
> configured for virtual machine.
>

That's strange indeed, except if br0 is used by something else like libvrit
network that sets up the interface for proxy-arp. What's the output of:

# brctl showmacs br0
# ip route show
# arp -n

on the host, and:

# ip link show
# ip route show
# arp -n

on the problematic vm and on one of the good vm's?

To find the loop I would start by doing ping between good and bad vm (both
directions in turns) and check the traffic on the host interface:

# tcpdump -ennqt -i br0 \( arp or icmp \)

and corresponding network devices on both vm's too.


>
> What could be the problem?
>
>
Any sysctl settings you might have changed on the host?


> --
> Mimiko desu.
>
>

[toc] | [prev] | [next] | [standalone]

#179030

On 18.03.2017 07:22, Igor Cicimov wrote:
>     >uname -a
>     Linux 3.2.0-4-amd64 #1 SMP Debian 3.2.84-1 x86_64 GNU/Linux
>
> That's an really old kernel, I don't start anything virtual these days without at least 3.13.x kernel.

I regularly do apt-get upgrade, but not to next Debian version. So, how this kernel be old for Debian 7?

> Hmmm, this doesn't make much sense to me, more specifically this part:
>
>         vlan-raw-device bond0
>         bridge_ports bond0
>
> Whats the purpose exactly of the vlan? Usually, and that is how I do it, you would split the VLAN's coming from the switch trunk port over the bond
> and attach them to separate bridges lets say:

Don't mind that line. I putted it as a reminder to continue extend networking in using vlans. For now there is no vlan used.


> That's strange indeed, except if br0 is used by something else like libvrit network that sets up the interface for proxy-arp. What's the output of:
>
> # brctl showmacs br0

port no mac addr                is local?       ageing timer
   1     bridge mac              yes                0.00
   8     fe:54:00:13:76:5b       yes                0.00
   5     fe:54:00:44:ac:39       yes                0.00
   6     fe:54:00:a8:35:fb       yes                0.00
   2     fe:54:00:c2:64:2d       yes                0.00
   4     fe:54:00:d5:17:fb       yes                0.00
   3     fe:54:00:de:83:2d       yes                0.00


> # ip route show

10.0.0.0/8 via 10.10.10.254 dev br0
10.10.0.0/16 dev br0  proto kernel  scope link  src 10.10.10.10


> # arp -n

gives a long list for local and virtual ip/mac

>
> on the host, and:
>
> # ip link show

I can't connect using ssh, only vnc.
This shows lo and eth0 with corresponding mac and is UP

> # ip route show
10.10.0.0/16 dev eth0 proto kernel scope link src 10.10.10.3
169.254.0.0/16 dev eth0 scope link metric 1002
default via 10.10.10.254 dev eth0

> # arp -n

gives a long list for local and virtual ip/mac


>
> on the problematic vm and on one of the good vm's?

Thouse 3 commands shows the same (except ip and mac) on both vm's.


>
> To find the loop I would start by doing ping between good and bad vm (both directions in turns) and check the traffic on the host interface:
>
> # tcpdump -ennqt -i br0 \( arp or icmp \)
>
> and corresponding network devices on both vm's too.

This gives me lot of output regarding arp and icmp on LAN.

> Any sysctl settings you might have changed on the host?

net.ipv4.ip_forward=1
net.ipv4.conf.all.arp_ignore=1
net.ipv4.conf.all.arp_announce=2


-- 
Mimiko desu.

[toc] | [prev] | [next] | [standalone]

#179068

On Mon, 20 Mar 2017, Mimiko wrote:
> On 18.03.2017 07:22, Igor Cicimov wrote:
>>     >uname -a
>>     Linux 3.2.0-4-amd64 #1 SMP Debian 3.2.84-1 x86_64 GNU/Linux
>> 
>> That's an really old kernel, I don't start anything virtual these days 
>> without at least 3.13.x kernel.
>
> I regularly do apt-get upgrade, but not to next Debian version. So, how this 
> kernel be old for Debian 7?

TLDR:

  1. I have no clues to offer regarding OP's virtualisation/networking
  troubles, but I wish them the best of fortune.

  2. OP's kernel version does not look quite current to me, even for
  wheezy (debian 7) and it might be a matter of concern (though I claim
  no particular expertise).

Longer version:

It is not clear to me whether your question

> I regularly do apt-get upgrade, but not to next Debian version. So, how this 
> kernel be old for Debian 7?

is a request for information, or merely rhetorical (ie, an assertion
that your kernel is in fact current relative to debian 7).

In case you were asking, I'm still using debian 7 too,

   $ cat /etc/debian_version
   7.11

and, for what it's worth, in case it interests you

   $ uname -a
   Linux leviathan 3.2.0-4-686-pae #1 SMP Debian 3.2.86-1 i686 GNU/Linux

I notice the version you posted here (3.2.84-1) is slightly older than
mine. If like myself you plan on running the 3.2.0-4-* release for a
while, It might[1] be worth upgrading at least to version 3.2.86-1:

   https://security-tracker.debian.org/tracker/CVE-2016-5195

There is also a newer kernel release available, for those who choose
to install it explicitly[3], from the wheezy-backports repository[2],
(see the 3.16.0-0.bpo releases near the end):

   $ apt-cache search '^linux-image-*'
   linux-image-3.2.0-4-486 - Linux 3.2 for older PCs
   linux-image-3.2.0-4-686-pae - Linux 3.2 for modern PCs
   linux-image-3.2.0-4-686-pae-dbg - Debugging symbols for Linux 3.2.0-4-686-pae
   linux-image-3.2.0-4-amd64 - Linux 3.2 for 64-bit PCs
   linux-image-3.2.0-4-rt-686-pae - Linux 3.2 for modern PCs, PREEMPT_RT
   linux-image-3.2.0-4-rt-686-pae-dbg - Debugging symbols for Linux 3.2.0-4-rt-686-pae
   linux-image-2.6-486 - Linux for older PCs (dummy package)
   linux-image-2.6-686 - Linux for modern PCs (dummy package)
   linux-image-2.6-686-bigmem - Linux for PCs with 4GB+ RAM (dummy package)
   linux-image-2.6-686-pae - Linux for modern PCs (dummy package)
   linux-image-2.6-amd64 - Linux for 64-bit PCs (dummy package)
   linux-image-486 - Linux for older PCs (meta-package)
   linux-image-686 - Linux for modern PCs (dummy package)
   linux-image-686-bigmem - Linux for PCs with 4GB+ RAM (dummy package)
   linux-image-686-pae - Linux for modern PCs (meta-package)
   linux-image-amd64 - Linux for 64-bit PCs (meta-package)
   linux-image-rt-686-pae - Linux for modern PCs (meta-package), PREEMPT_RT
   linux-image-3.16.0-0.bpo.4-586 - Linux 3.16 for older PCs
   linux-image-3.16.0-0.bpo.4-686-pae - Linux 3.16 for modern PCs
   linux-image-3.16.0-0.bpo.4-686-pae-dbg - Debugging symbols for Linux 3.16.0-0.bpo.4-686-pae
   linux-image-3.16.0-0.bpo.4-amd64 - Linux 3.16 for 64-bit PCs
   linux-image-586 - Linux for older PCs (meta-package)
   linux-image-686-pae-dbg - Debugging symbols for Linux 686-pae configuration (meta-package)

But regarding backports, do see https://backports.debian.org/FAQ/

| Q: Is there security support for packages from backports.debian.org?
| 
| A: Unfortunately not. This is done on a best effort basis by the
| people who track the package, usually the ones who originally did
| upload the package into backports. When security related bugs are
| fixed in Debian unstable the backporter is permitted to upload the
| package from directly there instead of having to wait until the fix
| hits testing. You can see the open issues for wheezy-backports in
| the security tracker (though there may be false positives too, the
| version compare isn't perfect yet)

[1] The security-tracker page for CVE-2016-5195 lists version 78-1 as
     'vulnerable', and 86-1 as 'fixed'. It is unclear to me what this
     is supposed to say about OP's kernel, which is apparently at 84-1.

[2] https://backports.debian.org/Instructions/#index2h2
[3] https://backports.debian.org/Instructions/#index3h2

[toc] | [prev] | [next] | [standalone]

#179177

On 21.03.2017 02:25, davidson@freevolt.org wrote:
> It is not clear to me whether your question
>
>> I regularly do apt-get upgrade, but not to next Debian version. So, how this kernel be old for Debian 7?
>
> is a request for information, or merely rhetorical (ie, an assertion
> that your kernel is in fact current relative to debian 7).
>
> In case you were asking, I'm still using debian 7 too,
>
>   $ cat /etc/debian_version
>   7.11
>
> and, for what it's worth, in case it interests you
>
>   $ uname -a
>   Linux leviathan 3.2.0-4-686-pae #1 SMP Debian 3.2.86-1 i686 GNU/Linux
>
> I notice the version you posted here (3.2.84-1) is slightly older than
> mine. If like myself you plan on running the 3.2.0-4-* release for a
> while, It might[1] be worth upgrading at least to version 3.2.86-1:

Well, doing regulat apt-get update && apt-get upgrade, I fought that kernel is also upgraded. I've seen this several times. How comes it wasn't 
updated to 3.2.86-1.

As for apt-cache search '^linux-image-*' :
linux-image-3.2.0-4-amd64 - Linux 3.2 for 64-bit PCs
linux-image-3.2.0-4-amd64-dbg - Debugging symbols for Linux 3.2.0-4-amd64
linux-image-3.2.0-4-rt-amd64 - Linux 3.2 for 64-bit PCs, PREEMPT_RT
linux-image-3.2.0-4-rt-amd64-dbg - Debugging symbols for Linux 3.2.0-4-rt-amd64
linux-image-2.6-amd64 - Linux for 64-bit PCs (dummy package)
linux-image-amd64 - Linux for 64-bit PCs (meta-package)
linux-image-rt-amd64 - Linux for 64-bit PCs (meta-package), PREEMPT_RT

Then doing apt-get install linux-image-amd64 says it already latest version.

Anyway, this slightly different (minor) version difference could be the problem?

-- 
Mimiko desu.

[toc] | [prev] | [next] | [standalone]

#179191

On Thu, Mar 23, 2017 at 08:36:38AM +0200, Mimiko wrote:
> Well, doing regulat apt-get update && apt-get upgrade, I fought that kernel 
> is also upgraded. I've seen this several times. How comes it wasn't updated 
> to 3.2.86-1.

Run "uname -a" (and also "uptime") and post the results.  After a kernel
package is upgraded, you do have to reboot to run the new kernel, and
uptime will tell us how long ago you did that.

> As for apt-cache search '^linux-image-*' :
> linux-image-3.2.0-4-amd64 - Linux 3.2 for 64-bit PCs
[...]

> Then doing apt-get install linux-image-amd64 says it already latest version.

It's important to note that the kernel package name is *not* the same
as the exact kernel version.  That's why you want to run "uname -a".
It will show both of them.

The kernel package name 3.2.0-4-amd64 refers to the ABI (module
compatibility) of the kernel.  A module built for any kernel with this
package name will work with any other version, so you don't have to
rebuild all of your third party kernel modules every time you upgrade
the kernel.

[toc] | [prev] | [next] | [standalone]

#179242

On 23.03.2017 14:14, Greg Wooledge wrote:
> Run "uname -a" (and also "uptime") and post the results.

You a right. I didn't do a restart after last kernel update.

After a restart, TTL problem didn't occur anymore.

Also I've noted that the particular vm had INPUT default policy to DROP and no icmp accepted. Strange that some time I did get an answer on ping.
Accepting icmp requests gives me stable replies.

Thank you all for answering.

-- 
Mimiko desu.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web