Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #204134 > unrolled thread

IPv6 router is not forwarding packets

Started byRainer Dorsch <ml@bokomoko.de>
First post2019-01-06 19:20 +0100
Last post2019-01-06 20:30 +0100
Articles 18 — 4 participants

Back to article view | Back to linux.debian.user

Contents

  IPv6 router is not forwarding packets Rainer Dorsch <ml@bokomoko.de> - 2019-01-06 19:20 +0100
    Re: IPv6 router is not forwarding packets Ulf Volmer <u.volmer@u-v.de> - 2019-01-06 19:40 +0100
      Re: IPv6 router is not forwarding packets Pascal Hambourg <pascal@plouf.fr.eu.org> - 2019-01-06 19:50 +0100
        Re: IPv6 router is not forwarding packets Rainer Dorsch <ml@bokomoko.de> - 2019-01-06 20:30 +0100
          Re: IPv6 router is not forwarding packets Pascal Hambourg <pascal@plouf.fr.eu.org> - 2019-01-06 20:40 +0100
            Re: IPv6 router is not forwarding packets Ulf Volmer <u.volmer@u-v.de> - 2019-01-06 21:00 +0100
          Re: IPv6 router is not forwarding packets Georgi Naplatanov <gosho@oles.biz> - 2019-01-06 20:40 +0100
      Re: IPv6 router is not forwarding packets Rainer Dorsch <ml@bokomoko.de> - 2019-01-06 20:10 +0100
        Re: IPv6 router is not forwarding packets Pascal Hambourg <pascal@plouf.fr.eu.org> - 2019-01-06 20:30 +0100
          Re: IPv6 router is not forwarding packets Rainer Dorsch <ml@bokomoko.de> - 2019-01-06 20:50 +0100
            Re: IPv6 router is not forwarding packets Ulf Volmer <u.volmer@u-v.de> - 2019-01-06 21:00 +0100
              Re: IPv6 router is not forwarding packets Rainer Dorsch <ml@bokomoko.de> - 2019-01-06 21:30 +0100
                Re: IPv6 router is not forwarding packets Rainer Dorsch <ml@bokomoko.de> - 2019-01-06 21:40 +0100
                  Re: IPv6 router is not forwarding packets Ulf Volmer <u.volmer@u-v.de> - 2019-01-06 21:50 +0100
                    Re: IPv6 router is not forwarding packets Rainer Dorsch <ml@bokomoko.de> - 2019-01-07 23:10 +0100
                      Re: IPv6 router is not forwarding packets Ulf Volmer <u.volmer@u-v.de> - 2019-01-08 03:50 +0100
    Re: IPv6 router is not forwarding packets Pascal Hambourg <pascal@plouf.fr.eu.org> - 2019-01-06 19:50 +0100
      Re: IPv6 router is not forwarding packets Rainer Dorsch <ml@bokomoko.de> - 2019-01-06 20:30 +0100

#204134 — IPv6 router is not forwarding packets

FromRainer Dorsch <ml@bokomoko.de>
Date2019-01-06 19:20 +0100
SubjectIPv6 router is not forwarding packets
Message-ID<xdlbH-5JS-5@gated-at.bofh.it>
Hello,

I tried to configure a stretch based IPv6 router.

My setup is:

ISP router -> Stretch router (home) -> Endpoint (mohot)

I can ssh from the mohot to home (ssh 2a02:8070:898f:e4f8:d263:b4ff:fe00:325c) 
and I can ssh from the home to the internet (using IPv6), but I cannot ssh 
from the mohot (endpoint) to the internet:

rd@mohot:~$ ssh -vvv 2a03:4000:6:52b6::
OpenSSH_7.7p1 Debian-3, OpenSSL 1.0.2o  27 Mar 2018
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolve_canonicalize: hostname 2a03:4000:6:52b6:: is address
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to 2a03:4000:6:52b6:: [2a03:4000:6:52b6::] port 22.

IPv6 forwarding is enabled on the home (stretch router):

# cat /proc/sys/net/ipv6/conf/all/forwarding 
1
#

For me it seems that home is dropping the request, if that is the case, is 
there a good way to find out why that happens?

Any hint is welcome... 

Here is the interface config and routing configuration:

ISP Router:
-----------

My upstream connection gives me a dynamic IPv6 prefix: 2a02:8070:898f:e400::/56
and has an IPv6 address 2a02:8070:8900::30a0:caa7:42e0:93d2

Stretch Router:
---------------

The stretch router has an upstream interface

root@home:/etc# ip addr show dev eth0.1
4: eth0.1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state 
UP group default qlen 1000
    link/ether d0:63:b4:00:32:5c brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.30/24 brd 192.168.0.255 scope global eth0.1
       valid_lft forever preferred_lft forever
    inet6 2a02:8070:898f:e400:d263:b4ff:fe00:325c/64 scope global mngtmpaddr 
dynamic 
       valid_lft 6968sec preferred_lft 3368sec
    inet6 fe80::d263:b4ff:fe00:325c/64 scope link 
       valid_lft forever preferred_lft forever

and a downstream interface

root@home:/etc# ip addr show dev eth0.7 
10: eth0.7@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue 
state UP group default qlen 1000 
   link/ether d0:63:b4:00:32:5c brd ff:ff:ff:ff:ff:ff 
   inet 192.168.7.1/24 brd 192.168.7.255 scope global eth0.7 
      valid_lft forever preferred_lft forever 
   inet6 2a02:8070:898f:e4f8:d263:b4ff:fe00:325c/62 scope global  
      valid_lft forever preferred_lft forever 
   inet6 fe80::d263:b4ff:fe00:325c/64 scope link  
      valid_lft forever preferred_lft forever 
root@home:/etc# 

for prefix delegation, I followed the Debian wiki

https://wiki.debian.org/IPv6PrefixDelegation

And routing information:
root@home:/etc# ip -6 r
2a02:8070:898f:e400::/64 dev eth0.1 proto kernel metric 256  expires 6974sec 
pref medium
2a02:8070:898f:e4f8::/62 dev eth0.7 proto kernel metric 256  pref medium
fe80::/64 dev eth0 proto kernel metric 256  pref medium
fe80::/64 dev eth0.1 proto kernel metric 256  pref medium
fe80::/64 dev eth0.2 proto kernel metric 256  pref medium
fe80::/64 dev eth0.3 proto kernel metric 256  pref medium
fe80::/64 dev eth0.4 proto kernel metric 256  pref medium
fe80::/64 dev eth0.5 proto kernel metric 256  pref medium
fe80::/64 dev eth0.6 proto kernel metric 256  pref medium
fe80::/64 dev eth0.7 proto kernel metric 256  pref medium
default via fe80::e228:6dff:fe43:5776 dev eth0.1 proto ra metric 1024  expires 
1574sec hoplimit 255 pref medium
root@home:/etc# 

IPv6 forwarding is enabled:

root@home:/etc# cat /proc/sys/net/ipv6/conf/all/forwarding 
1
root@home:/etc# cat /proc/sys/net/ipv6/conf/eth0.7/forwarding 
1
root@home:/etc#


Endpoint:
---------
rd@mohot:~$ ip addr show dev eth0 
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP 
group default qlen 1000 
   link/ether d0:63:b4:00:4d:d1 brd ff:ff:ff:ff:ff:ff 
   inet 192.168.7.31/24 brd 192.168.7.255 scope global eth0 
      valid_lft forever preferred_lft forever 
   inet6 2a02:8070:898f:e400:d263:b4ff:fe00:4dd1/64 scope global dynamic 
mngtmpaddr  
      valid_lft 7030sec preferred_lft 3430sec 
   inet6 2a02:8070:898f:e4f8:d263:b4ff:fe00:4dd1/64 scope global dynamic 
mngtmpaddr  
      valid_lft 14215sec preferred_lft 14215sec 
   inet6 fe80::d263:b4ff:fe00:4dd1/64 scope link  
      valid_lft forever preferred_lft forever 
rd@mohot:~$ 

Many thanks 
Rainer

-- 
Rainer Dorsch
http://bokomoko.de/

[toc] | [next] | [standalone]

#204136

FromUlf Volmer <u.volmer@u-v.de>
Date2019-01-06 19:40 +0100
Message-ID<xdlv3-5Qz-5@gated-at.bofh.it>
In reply to#204134
On 06.01.19 18:36, Rainer Dorsch wrote:

> Endpoint:
> ---------
> rd@mohot:~$ ip addr show dev eth0 
> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP 
> group default qlen 1000 
>    link/ether d0:63:b4:00:4d:d1 brd ff:ff:ff:ff:ff:ff 
>    inet 192.168.7.31/24 brd 192.168.7.255 scope global eth0 
>       valid_lft forever preferred_lft forever 
>    inet6 2a02:8070:898f:e400:d263:b4ff:fe00:4dd1/64 scope global dynamic 
> mngtmpaddr  
>       valid_lft 7030sec preferred_lft 3430sec 
>    inet6 2a02:8070:898f:e4f8:d263:b4ff:fe00:4dd1/64 scope global dynamic 
> mngtmpaddr  
>       valid_lft 14215sec preferred_lft 14215sec 
>    inet6 fe80::d263:b4ff:fe00:4dd1/64 scope link  
>       valid_lft forever preferred_lft forever 

I'm confused that you have uplink and downlink addresses configured on
the client. Anyway, can you also post 'ip -6 r' from the client?

best regards
Ulf

[toc] | [prev] | [next] | [standalone]

#204137

FromPascal Hambourg <pascal@plouf.fr.eu.org>
Date2019-01-06 19:50 +0100
Message-ID<xdlEK-5U5-9@gated-at.bofh.it>
In reply to#204136
Le 06/01/2019 à 19:32, Ulf Volmer a écrit :
> On 06.01.19 18:36, Rainer Dorsch wrote:
>>     inet6 2a02:8070:898f:e400:d263:b4ff:fe00:4dd1/64 scope global dynamic mngtmpaddr
>>        valid_lft 7030sec preferred_lft 3430sec
>>     inet6 2a02:8070:898f:e4f8:d263:b4ff:fe00:4dd1/64 scope global dynamic mngtmpaddr
>>        valid_lft 14215sec preferred_lft 14215sec
> 
> I'm confused that you have uplink and downlink addresses configured on
> the client.

Good catch, didn't spot this. This is of course wrong.

BTW, I am a bit surprised by the upstream setup. It wastes a full /64 
just for one address.

[toc] | [prev] | [next] | [standalone]

#204142

FromRainer Dorsch <ml@bokomoko.de>
Date2019-01-06 20:30 +0100
Message-ID<xdmhr-6nB-5@gated-at.bofh.it>
In reply to#204137
Am Sonntag, 6. Januar 2019, 19:49:13 CET schrieb Pascal Hambourg:
> Le 06/01/2019 à 19:32, Ulf Volmer a écrit :
> > On 06.01.19 18:36, Rainer Dorsch wrote:
> >>     inet6 2a02:8070:898f:e400:d263:b4ff:fe00:4dd1/64 scope global dynamic
> >>     mngtmpaddr>>     
> >>        valid_lft 7030sec preferred_lft 3430sec
> >>     
> >>     inet6 2a02:8070:898f:e4f8:d263:b4ff:fe00:4dd1/64 scope global dynamic
> >>     mngtmpaddr>>     
> >>        valid_lft 14215sec preferred_lft 14215sec
> > 
> > I'm confused that you have uplink and downlink addresses configured on
> > the client.
> 
> Good catch, didn't spot this. This is of course wrong.
> 
> BTW, I am a bit surprised by the upstream setup. It wastes a full /64
> just for one address.

Do yo mean the 2a02:8070:898f:e400:d263:b4ff:fe00:325c/64 in

root@home:/etc# ip addr show eth0.1
4: eth0.1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state 
UP group default qlen 1000
    link/ether d0:63:b4:00:32:5c brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.30/24 brd 192.168.0.255 scope global eth0.1
       valid_lft forever preferred_lft forever
    inet6 2a02:8070:898f:e400:d263:b4ff:fe00:325c/64 scope global mngtmpaddr 
dynamic 
       valid_lft 6676sec preferred_lft 3076sec
    inet6 fe80::d263:b4ff:fe00:325c/64 scope link 
       valid_lft forever preferred_lft forever
root@home:/etc# 

?

Isn't the /64 the prefix length and longer prefixes than 64 are not supported in 
ipv6?

Thanks
Rainer

-- 
Rainer Dorsch
http://bokomoko.de/

[toc] | [prev] | [next] | [standalone]

#204145

FromPascal Hambourg <pascal@plouf.fr.eu.org>
Date2019-01-06 20:40 +0100
Message-ID<xdmr7-6qQ-7@gated-at.bofh.it>
In reply to#204142
Le 06/01/2019 à 20:23, Rainer Dorsch a écrit :
> Am Sonntag, 6. Januar 2019, 19:49:13 CET schrieb Pascal Hambourg:
>>
>> BTW, I am a bit surprised by the upstream setup. It wastes a full /64
>> just for one address.
> 
> Do yo mean the 2a02:8070:898f:e400:d263:b4ff:fe00:325c/64 in

Yes.

> Isn't the /64 the prefix length and longer prefixes than 64 are not supported in
> ipv6?

IPv6 supports any prefix size. Only SLAAC (autoconfiguration using RA's) 
requires /64.

[toc] | [prev] | [next] | [standalone]

#204149

FromUlf Volmer <u.volmer@u-v.de>
Date2019-01-06 21:00 +0100
Message-ID<xdmKu-6xF-3@gated-at.bofh.it>
In reply to#204145
On 06.01.19 20:33, Pascal Hambourg wrote:
> Le 06/01/2019 à 20:23, Rainer Dorsch a écrit :
>> Isn't the /64 the prefix length and longer prefixes than 64 are not
>> supported in
>> ipv6?
> 
> IPv6 supports any prefix size. Only SLAAC (autoconfiguration using RA's)
> requires /64.

Usually he will get a /56 from his provider. this are 256 /64 subnets.
I do not see the disadvantage to use one of them for the uplink network.

best regards
Ulf

[toc] | [prev] | [next] | [standalone]

#204146

FromGeorgi Naplatanov <gosho@oles.biz>
Date2019-01-06 20:40 +0100
Message-ID<xdmr8-6qQ-21@gated-at.bofh.it>
In reply to#204142
On 1/6/19 9:23 PM, Rainer Dorsch wrot> Isn't the /64 the prefix length
and longer prefixes than 64 are not supported in
> ipv6?
> 

For IPv6 there are two methods for IP address distribution - Router
Advertisement and DHCPv6.

Router Advertisement doesn't support prefixes longer than /64.

On the other hand Android doesn't support DHCPv6.

Kind regards
Georgi

[toc] | [prev] | [next] | [standalone]

#204141

FromRainer Dorsch <ml@bokomoko.de>
Date2019-01-06 20:10 +0100
Message-ID<xdlY6-6g9-5@gated-at.bofh.it>
In reply to#204136
Hi Ulf,

Am Sonntag, 6. Januar 2019, 19:32:46 CET schrieb Ulf Volmer:
> On 06.01.19 18:36, Rainer Dorsch wrote:
> > Endpoint:
> > ---------
> > rd@mohot:~$ ip addr show dev eth0
> > 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state
> > UP group default qlen 1000
> > 
> >    link/ether d0:63:b4:00:4d:d1 brd ff:ff:ff:ff:ff:ff
> >    inet 192.168.7.31/24 brd 192.168.7.255 scope global eth0
> >    
> >       valid_lft forever preferred_lft forever
> >    
> >    inet6 2a02:8070:898f:e400:d263:b4ff:fe00:4dd1/64 scope global dynamic
> > 
> > mngtmpaddr
> > 
> >       valid_lft 7030sec preferred_lft 3430sec
> >    
> >    inet6 2a02:8070:898f:e4f8:d263:b4ff:fe00:4dd1/64 scope global dynamic
> > 
> > mngtmpaddr
> > 
> >       valid_lft 14215sec preferred_lft 14215sec
> >    
> >    inet6 fe80::d263:b4ff:fe00:4dd1/64 scope link
> >    
> >       valid_lft forever preferred_lft forever
> 
> I'm confused that you have uplink and downlink addresses configured on
> the client. Anyway, can you also post 'ip -6 r' from the client?

I have not explicitly configured the addresses, but I have a dnsmasq running on 
the server which might be responsible for that, though I do not see how, I 
have in dnsmasq.conf

# Do router advertisements for all subnets where we're doing DHCPv6                                                                                                                                                                                                                                  
# Unless overriden by ra-stateless, ra-names, et al, the router                                                                                                                                                                                                                                      
# advertisements will have the M and O bits set, so that the clients                                                                                                                                                                                                                                 
# get addresses and configuration from DHCPv6, and the A bit reset, so the                                                                                                                                                                                                                           
# clients don't use SLAAC addresses.                                                                                                                                                                                                                                                                 
enable-ra

# Advertise delegated prefix based on the IPv6 address of eth0.                                                                                                                                                                                                                                      
dhcp-range = ::1,constructor:eth0.7,   ra-stateless, ra-names, 4h


The output you requested on the mohot (endpoint):

rd@mohot:~$ ip -6 r
2a02:8070:898f:e400::/64 dev eth0 proto kernel metric 256 pref medium
2a02:8070:898f:e4f8::/64 dev eth0 proto kernel metric 256 pref medium
2a02:8070:898f:e4f8::/62 dev eth0 proto kernel metric 256 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
default via fe80::d263:b4ff:fe00:325c dev eth0 proto ra metric 1024 hoplimit 64 
pref medium
default via fe80::e228:6dff:fe43:5776 dev eth0 proto ra metric 1024 hoplimit 
255 pref medium
rd@mohot:~$

I do not understand why there is fe80::e228:6dff:fe43:5776. This is the link 
local address of the ISP router, which does not make sense (?).

but even if I delete entry, I do not get it working: 

root@mohot:~# ip r del default via fe80::e228:6dff:fe43:5776 
root@mohot:~# ip -6 r
2a02:8070:898f:e400::/64 dev eth0 proto kernel metric 256 pref medium
2a02:8070:898f:e4f8::/64 dev eth0 proto kernel metric 256 pref medium
2a02:8070:898f:e4f8::/62 dev eth0 proto kernel metric 256 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
default via fe80::d263:b4ff:fe00:325c dev eth0 proto ra metric 1024 hoplimit 64 
pref medium
root@mohot:~#

does not change anything (visible) and causes timeouts.

What surprises me is that traceroute6 does not even find home (stretch router)

rd@mohot:~$ traceroute6 2a03:4000:6:52b6::  
traceroute to 2a03:4000:6:52b6:: (2a03:4000:6:52b6::), 30 hops max, 80 byte 
packets
 1  * * *
 2  * * *
 3  * * *

but I am not sure how reliable traceroute is.

A direct connection works well:

rd@mohot:~$ ssh fe80::d263:b4ff:fe00:325c%eth0
Linux home 4.18.0-0.bpo.3-armmp #1 SMP Debian 4.18.20-2~bpo9+1 (2018-12-08) 
armv7l

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sun Jan  6 19:46:41 2019 from fe80::d263:b4ff:fe00:4dd1%eth0.7
rd@home:~$ 



Thanks
Rainer



> 
> best regards
> Ulf


-- 
Rainer Dorsch
http://bokomoko.de/

[toc] | [prev] | [next] | [standalone]

#204143

FromPascal Hambourg <pascal@plouf.fr.eu.org>
Date2019-01-06 20:30 +0100
Message-ID<xdmhr-6nB-7@gated-at.bofh.it>
In reply to#204141
Le 06/01/2019 à 19:59, Rainer Dorsch a écrit :
> Am Sonntag, 6. Januar 2019, 19:32:46 CET schrieb Ulf Volmer:
>>
>> I'm confused that you have uplink and downlink addresses configured on
>> the client. Anyway, can you also post 'ip -6 r' from the client?
> 
> I have not explicitly configured the addresses, but I have a dnsmasq running on
> the server which might be responsible for that, though I do not see how, I
> have in dnsmasq.conf
(...)
> rd@mohot:~$ ip -6 r
> 2a02:8070:898f:e400::/64 dev eth0 proto kernel metric 256 pref medium
> 2a02:8070:898f:e4f8::/64 dev eth0 proto kernel metric 256 pref medium
> 2a02:8070:898f:e4f8::/62 dev eth0 proto kernel metric 256 pref medium
> fe80::/64 dev eth0 proto kernel metric 256 pref medium
> default via fe80::d263:b4ff:fe00:325c dev eth0 proto ra metric 1024 hoplimit 64
> pref medium
> default via fe80::e228:6dff:fe43:5776 dev eth0 proto ra metric 1024 hoplimit
> 255 pref medium
> rd@mohot:~$
> 
> I do not understand why there is fe80::e228:6dff:fe43:5776. This is the link
> local address of the ISP router, which does not make sense (?).

Either dnsmasq is doing something nasty, or something else is leaking 
RA's from upstream to downstream. Could it be your VLAN setup ?

> but even if I delete entry, I do not get it working:

You need at least to delete the bogus upstream address too.

You should start with a fully static setup on the host, fix routing, and 
when done you can try to setup SLAAC/DHCPv6.

[toc] | [prev] | [next] | [standalone]

#204147

FromRainer Dorsch <ml@bokomoko.de>
Date2019-01-06 20:50 +0100
Message-ID<xdmAN-6ug-3@gated-at.bofh.it>
In reply to#204143
Am Sonntag, 6. Januar 2019, 20:29:27 CET schrieb Pascal Hambourg:
> You should start with a fully static setup on the host, fix routing, and
> when done you can try to setup SLAAC/DHCPv6.

I agree.

I thought that

iface eth0 inet6 manual

does not do automatic stuff. But after a reboot I still get

root@mohot:~# ip -6 a show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
    inet6 2a02:8070:898f:e400:d263:b4ff:fe00:4dd1/64 scope global dynamic 
mngtmpaddr 
       valid_lft 7126sec preferred_lft 3526sec
    inet6 2a02:8070:898f:e4f8:d263:b4ff:fe00:4dd1/64 scope global dynamic 
mngtmpaddr 
       valid_lft 14019sec preferred_lft 14019sec
    inet6 fe80::d263:b4ff:fe00:4dd1/64 scope link 
       valid_lft forever preferred_lft forever
root@mohot:~# ip -6 r
2a02:8070:898f:e400::/64 dev eth0 proto kernel metric 256 pref medium
2a02:8070:898f:e4f8::/64 dev eth0 proto kernel metric 256 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
default via fe80::d263:b4ff:fe00:325c dev eth0 proto ra metric 1024 hoplimit 64 
pref medium
default via fe80::e228:6dff:fe43:5776 dev eth0 proto ra metric 1024 hoplimit 
255 pref medium
root@mohot:~# 

Is that expected? If yes, is that then dhcpv6 or slaac?

Thanks
Rainer
-- 
Rainer Dorsch
http://bokomoko.de/

[toc] | [prev] | [next] | [standalone]

#204150

FromUlf Volmer <u.volmer@u-v.de>
Date2019-01-06 21:00 +0100
Message-ID<xdmKu-6xF-5@gated-at.bofh.it>
In reply to#204147
On 06.01.19 20:47, Rainer Dorsch wrote:

> does not do automatic stuff. But after a reboot I still get
> 
> root@mohot:~# ip -6 a show eth0
> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
>     inet6 2a02:8070:898f:e400:d263:b4ff:fe00:4dd1/64 scope global dynamic 
> mngtmpaddr 
>        valid_lft 7126sec preferred_lft 3526sec
>     inet6 2a02:8070:898f:e4f8:d263:b4ff:fe00:4dd1/64 scope global dynamic 
> mngtmpaddr 
>        valid_lft 14019sec preferred_lft 14019sec
>     inet6 fe80::d263:b4ff:fe00:4dd1/64 scope link 
>        valid_lft forever preferred_lft forever
> root@mohot:~# ip -6 r
> 2a02:8070:898f:e400::/64 dev eth0 proto kernel metric 256 pref medium
> 2a02:8070:898f:e4f8::/64 dev eth0 proto kernel metric 256 pref medium
> fe80::/64 dev eth0 proto kernel metric 256 pref medium
> default via fe80::d263:b4ff:fe00:325c dev eth0 proto ra metric 1024 hoplimit 64 
> pref medium
> default via fe80::e228:6dff:fe43:5776 dev eth0 proto ra metric 1024 hoplimit 
> 255 pref medium
> root@mohot:~# 
> 
> Is that expected? If yes, is that then dhcpv6 or slaac?

I think this is slaac. But you still get RAs from VLAN 1 and your ISP
router. Are you should that your VLAN separating at layer 2 works as
expected? tcpdump/wireshark is your friend.

best regards
Ulf

[toc] | [prev] | [next] | [standalone]

#204151

FromRainer Dorsch <ml@bokomoko.de>
Date2019-01-06 21:30 +0100
Message-ID<xdndv-6X9-7@gated-at.bofh.it>
In reply to#204150
Hi Pascal, Ulf, and Georgi,

many thanks for your replies,  they brought me at least one step further :-)

Am Sonntag, 6. Januar 2019, 20:54:43 CET schrieb Ulf Volmer:
> On 06.01.19 20:47, Rainer Dorsch wrote:
> > does not do automatic stuff. But after a reboot I still get
> > 
> > root@mohot:~# ip -6 a show eth0
> > 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
> > 
> >     inet6 2a02:8070:898f:e400:d263:b4ff:fe00:4dd1/64 scope global dynamic
> > 
> > mngtmpaddr
> > 
> >        valid_lft 7126sec preferred_lft 3526sec
> >     
> >     inet6 2a02:8070:898f:e4f8:d263:b4ff:fe00:4dd1/64 scope global dynamic
> > 
> > mngtmpaddr
> > 
> >        valid_lft 14019sec preferred_lft 14019sec
> >     
> >     inet6 fe80::d263:b4ff:fe00:4dd1/64 scope link
> >     
> >        valid_lft forever preferred_lft forever
> > 
> > root@mohot:~# ip -6 r
> > 2a02:8070:898f:e400::/64 dev eth0 proto kernel metric 256 pref medium
> > 2a02:8070:898f:e4f8::/64 dev eth0 proto kernel metric 256 pref medium
> > fe80::/64 dev eth0 proto kernel metric 256 pref medium
> > default via fe80::d263:b4ff:fe00:325c dev eth0 proto ra metric 1024
> > hoplimit 64 pref medium
> > default via fe80::e228:6dff:fe43:5776 dev eth0 proto ra metric 1024
> > hoplimit 255 pref medium
> > root@mohot:~#
> > 
> > Is that expected? If yes, is that then dhcpv6 or slaac?
> 
> I think this is slaac. But you still get RAs from VLAN 1 and your ISP
> router. Are you should that your VLAN separating at layer 2 works as
> expected? tcpdump/wireshark is your friend.

I reviewed the VLAN config and indeed mohot still saw untagged traffic from 
VLAN1. 

After removing this, the upstream addresses and routings are gone.

But I (surprisingly) still get addresses and routes. Though they look now 
reasonable for me:

rd@mohot:~$ ip -6 a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 state UNKNOWN qlen 1000
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
    inet6 2a02:8070:898f:e4f8:d263:b4ff:fe00:4dd1/64 scope global dynamic 
mngtmpaddr 
       valid_lft 14303sec preferred_lft 14303sec
    inet6 fe80::d263:b4ff:fe00:4dd1/64 scope link 
       valid_lft forever preferred_lft forever
rd@mohot:~$ 

Still the problem is the same, I can ssh forth and back between home (stretch 
router) and mohot (endpoint). But I do not reach an internet server (outside 
of my home network):

rd@mohot:~$ ssh 2a03:4000:6:52b6::
ssh: connect to host 2a03:4000:6:52b6:: port 22: Connection refused
rd@mohot:~$ traceroute6 2a03:4000:6:52b6::
traceroute to 2a03:4000:6:52b6:: (2a03:4000:6:52b6::), 30 hops max, 80 byte 
packets
 1  2a02:8070:898f:e4f8:d263:b4ff:fe00:325c 
(2a02:8070:898f:e4f8:d263:b4ff:fe00:325c)  0.361 ms  0.295 ms  0.305 ms
 2  fritz.box (2a02:8070:898f:e400:e228:6dff:fe43:5776)  0.798 ms  0.821 ms  
0.813 ms
 3  2a02:8070:8900:0:30a0:caa7:42e0:93d2 
(2a02:8070:8900:0:30a0:caa7:42e0:93d2)  1.478 ms !X  1.459 ms !X  1.434 ms !X
rd@mohot:~$ 

Things are getting weirder now... :-/

Any hint is welcome
Rainer

-- 
Rainer Dorsch
http://bokomoko.de/

[toc] | [prev] | [next] | [standalone]

#204152

FromRainer Dorsch <ml@bokomoko.de>
Date2019-01-06 21:40 +0100
Message-ID<xdnnb-718-5@gated-at.bofh.it>
In reply to#204151
> 
> Things are getting weirder now... :-/

Just looking at the differences between the traceroutes from home (stretch 
router) and mohot (endpoint):

rd@home:~$ traceroute6 2a03:4000:6:52b6::
traceroute to 2a03:4000:6:52b6:: (2a03:4000:6:52b6::), 30 hops max, 80 byte 
packets
 1  fritz.box (2a02:8070:898f:e400:e228:6dff:fe43:5776)  0.723 ms  0.608 ms  
0.710 ms
 2  2a02:8070:8900::1 (2a02:8070:8900::1)  13.961 ms  22.110 ms  20.783 ms
 3  2a02:8070:80ff:24dc::1 (2a02:8070:80ff:24dc::1)  21.874 ms  21.751 ms  
21.733 ms
 4  de-fra01b-rc1-lo0-0.v6.aorta.net (2001:730:2d00::5474:8065)  25.916 ms  
29.503 ms  27.045 ms
 5  de-fra01b-ri1-lo0-0.v6.aorta.net (2001:730:2d00::5474:8042)  26.992 ms  
26.815 ms  26.774 ms
 6  2001:730:2d01:21::2 (2001:730:2d01:21::2)  31.834 ms  25.176 ms  26.755 ms
 7  mail.bokomoko.de (2a03:4000:6:52b6::)  26.708 ms  21.379 ms  20.737 ms
rd@home:~$

rd@mohot:~$ traceroute6 2a03:4000:6:52b6::
traceroute to 2a03:4000:6:52b6:: (2a03:4000:6:52b6::), 30 hops max, 80 byte 
packets
 1  2a02:8070:898f:e4f8:d263:b4ff:fe00:325c 
(2a02:8070:898f:e4f8:d263:b4ff:fe00:325c)  0.366 ms  0.306 ms  0.295 ms
 2  fritz.box (2a02:8070:898f:e400:e228:6dff:fe43:5776)  0.929 ms  1.063 ms  
1.182 ms
 3  2a02:8070:8900:0:30a0:caa7:42e0:93d2 
(2a02:8070:8900:0:30a0:caa7:42e0:93d2)  2.620 ms !X  2.584 ms !X  2.539 ms !X
rd@mohot:~$ 

I see after my ISP router (fritz.box) the routes differ.

I am still wondering what happens here....

Any hint is welcome
Rainer

Thanks
Rainer



-- 
Rainer Dorsch
http://bokomoko.de/

[toc] | [prev] | [next] | [standalone]

#204153

FromUlf Volmer <u.volmer@u-v.de>
Date2019-01-06 21:50 +0100
Message-ID<xdnwS-74I-7@gated-at.bofh.it>
In reply to#204152
On 06.01.19 21:29, Rainer Dorsch wrote:

> rd@home:~$ traceroute6 2a03:4000:6:52b6::
> traceroute to 2a03:4000:6:52b6:: (2a03:4000:6:52b6::), 30 hops max, 80 byte 
> packets
>  1  fritz.box (2a02:8070:898f:e400:e228:6dff:fe43:5776)  0.723 ms  0.608 ms  
> 0.710 ms
>  2  2a02:8070:8900::1 (2a02:8070:8900::1)  13.961 ms  22.110 ms  20.783 ms
>  3  2a02:8070:80ff:24dc::1 (2a02:8070:80ff:24dc::1)  21.874 ms  21.751 ms  
> 21.733 ms
>  4  de-fra01b-rc1-lo0-0.v6.aorta.net (2001:730:2d00::5474:8065)  25.916 ms  
> 29.503 ms  27.045 ms
>  5  de-fra01b-ri1-lo0-0.v6.aorta.net (2001:730:2d00::5474:8042)  26.992 ms  
> 26.815 ms  26.774 ms
>  6  2001:730:2d01:21::2 (2001:730:2d01:21::2)  31.834 ms  25.176 ms  26.755 ms
>  7  mail.bokomoko.de (2a03:4000:6:52b6::)  26.708 ms  21.379 ms  20.737 ms
> rd@home:~$
> 
> rd@mohot:~$ traceroute6 2a03:4000:6:52b6::
> traceroute to 2a03:4000:6:52b6:: (2a03:4000:6:52b6::), 30 hops max, 80 byte 
> packets
>  1  2a02:8070:898f:e4f8:d263:b4ff:fe00:325c 
> (2a02:8070:898f:e4f8:d263:b4ff:fe00:325c)  0.366 ms  0.306 ms  0.295 ms
>  2  fritz.box (2a02:8070:898f:e400:e228:6dff:fe43:5776)  0.929 ms  1.063 ms  
> 1.182 ms
>  3  2a02:8070:8900:0:30a0:caa7:42e0:93d2 
> (2a02:8070:8900:0:30a0:caa7:42e0:93d2)  2.620 ms !X  2.584 ms !X  2.539 ms !X
> rd@mohot:~$ 
> 
> I see after my ISP router (fritz.box) the routes differ.
> 
> I am still wondering what happens here....

Yes, looks weird. I have no idea why this happens.

Just one point, the response times of hop 3 in your second traceroute
are quite low (instead of hop 2 in the first one). Are you sure that hop
3 isn't somewhere in your LAN?

best regards
Ulf

[toc] | [prev] | [next] | [standalone]

#204202

FromRainer Dorsch <ml@bokomoko.de>
Date2019-01-07 23:10 +0100
Message-ID<xdLfP-536-13@gated-at.bofh.it>
In reply to#204153
Hi Ulf,

many thank again for your reply. 

Am Sonntag, 6. Januar 2019, 21:40:21 CET schrieb Ulf Volmer:
> Yes, looks weird. I have no idea why this happens.
>
> Just one point, the response times of hop 3 in your second traceroute
> are quite low (instead of hop 2 in the first one). Are you sure that hop
> 3 isn't somewhere in your LAN?

Good observation. I mirrored the traffic to the ISP router (fritz.box) to 
another unused switch port and used tcpdump to record it.

Looks unsuspicious to me. I see the traceroute packets outgoing (three per hop 
limit as from home). The only difference is that the I get an "administratively 
prohibited" type back (probably what traceroute marks with the X).

https://scw.bokomoko.de/~rd/icmpv6.png
shows a screenshot of the first ICPMv6 packet which returns from the "wrong" 
IPv6 address. 

>From your observation on the timings, I would conclude that the packet does 
not leave the IP router (fritz.box). The only surprising thing is that the 
source ipv6 address is not the router address but some other address 
2a02:8070:8900:0:30a0:caa7:42e0:93d2 .

To me it seems almost like an issue of the ISP router (fritz.box). The 
fritzbox documentation contains a section

"Configure the IPv6 router so that it requests its own prefix from the FRITZ!Box 
using IPv6 prefix delegation and that it announces its routing information to 
the FRITZ!Box via router advertisement."

https://en.avm.de/service/fritzbox/fritzbox-7390/knowledge-base/publication/
show/1239_Setting-up-an-IPv6-subnet-in-the-FRITZ-Box/

Which routing information does the fritzbox need? Do I need radvd for that?

Thanks
Rainer



-- 
Rainer Dorsch
http://bokomoko.de/

[toc] | [prev] | [next] | [standalone]

#204206

FromUlf Volmer <u.volmer@u-v.de>
Date2019-01-08 03:50 +0100
Message-ID<xdPCO-7Bx-9@gated-at.bofh.it>
In reply to#204202
On 07.01.19 23:06, Rainer Dorsch wrote:

> "Configure the IPv6 router so that it requests its own prefix from the FRITZ!Box 
> using IPv6 prefix delegation and that it announces its routing information to 
> the FRITZ!Box via router advertisement."
> 
> https://en.avm.de/service/fritzbox/fritzbox-7390/knowledge-base/publication/
> show/1239_Setting-up-an-IPv6-subnet-in-the-FRITZ-Box/
> 
> Which routing information does the fritzbox need? Do I need radvd for that?

radvd is just for announcing RA to your subnets. I think you can use
dnsmasq instead.

But you need something to get these prefixes from your fritzbox.

I personally use (on a CentOS router) wide-dhcpv6 for that.

best regards
Ulf

[toc] | [prev] | [next] | [standalone]

#204139

FromPascal Hambourg <pascal@plouf.fr.eu.org>
Date2019-01-06 19:50 +0100
Message-ID<xdlEK-5U5-11@gated-at.bofh.it>
In reply to#204134
Le 06/01/2019 à 18:36, Rainer Dorsch a écrit :
> 
> ISP router -> Stretch router (home) -> Endpoint (mohot)
> 
> I can ssh from the mohot to home (ssh 2a02:8070:898f:e4f8:d263:b4ff:fe00:325c)
> and I can ssh from the home to the internet (using IPv6), but I cannot ssh
> from the mohot (endpoint) to the internet:
(...)
> For me it seems that home is dropping the request, if that is the case, is
> there a good way to find out why that happens?

Did you run a packet capture on each involved interface ?

> Stretch Router:
> ---------------
> 
> The stretch router has an upstream interface
> 
> root@home:/etc# ip addr show dev eth0.1
> 4: eth0.1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state
> UP group default qlen 1000
>      link/ether d0:63:b4:00:32:5c brd ff:ff:ff:ff:ff:ff
>      inet 192.168.0.30/24 brd 192.168.0.255 scope global eth0.1
>         valid_lft forever preferred_lft forever
>      inet6 2a02:8070:898f:e400:d263:b4ff:fe00:325c/64 scope global mngtmpaddr
> dynamic
>         valid_lft 6968sec preferred_lft 3368sec
>      inet6 fe80::d263:b4ff:fe00:325c/64 scope link
>         valid_lft forever preferred_lft forever
> 
> and a downstream interface
> 
> root@home:/etc# ip addr show dev eth0.7
> 10: eth0.7@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
> state UP group default qlen 1000
>     link/ether d0:63:b4:00:32:5c brd ff:ff:ff:ff:ff:ff
>     inet 192.168.7.1/24 brd 192.168.7.255 scope global eth0.7
>        valid_lft forever preferred_lft forever
>     inet6 2a02:8070:898f:e4f8:d263:b4ff:fe00:325c/62 scope global

Why /62 instead of the standard /64 ?

>        valid_lft forever preferred_lft forever
>     inet6 fe80::d263:b4ff:fe00:325c/64 scope link
>        valid_lft forever preferred_lft forever
> root@home:/etc#
> 
> for prefix delegation, I followed the Debian wiki
> 
> https://wiki.debian.org/IPv6PrefixDelegation
> 
> And routing information:
> root@home:/etc# ip -6 r
> 2a02:8070:898f:e400::/64 dev eth0.1 proto kernel metric 256  expires 6974sec
> pref medium
> 2a02:8070:898f:e4f8::/62 dev eth0.7 proto kernel metric 256  pref medium
(...)
> IPv6 forwarding is enabled:
> 
> root@home:/etc# cat /proc/sys/net/ipv6/conf/all/forwarding
> 1
> root@home:/etc# cat /proc/sys/net/ipv6/conf/eth0.7/forwarding
> 1
> root@home:/etc#

What about eth0.1 ? Needed to forward incoming reply packets too.

> Endpoint:
> ---------
> rd@mohot:~$ ip addr show dev eth0
> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP
> group default qlen 1000
>     link/ether d0:63:b4:00:4d:d1 brd ff:ff:ff:ff:ff:ff
>     inet 192.168.7.31/24 brd 192.168.7.255 scope global eth0
>        valid_lft forever preferred_lft forever
>     inet6 2a02:8070:898f:e400:d263:b4ff:fe00:4dd1/64 scope global dynamic
> mngtmpaddr
>        valid_lft 7030sec preferred_lft 3430sec
>     inet6 2a02:8070:898f:e4f8:d263:b4ff:fe00:4dd1/64 scope global dynamic
> mngtmpaddr
>        valid_lft 14215sec preferred_lft 14215sec
>     inet6 fe80::d263:b4ff:fe00:4dd1/64 scope link
>        valid_lft forever preferred_lft forever

What about IPv6 routes ?

[toc] | [prev] | [next] | [standalone]

#204144

FromRainer Dorsch <ml@bokomoko.de>
Date2019-01-06 20:30 +0100
Message-ID<xdmhr-6nB-11@gated-at.bofh.it>
In reply to#204139
Hi Pascal,

Am Sonntag, 6. Januar 2019, 19:41:51 CET schrieb Pascal Hambourg:
> Le 06/01/2019 à 18:36, Rainer Dorsch a écrit :
> > ISP router -> Stretch router (home) -> Endpoint (mohot)
> > 
> > I can ssh from the mohot to home (ssh
> > 2a02:8070:898f:e4f8:d263:b4ff:fe00:325c) and I can ssh from the home to
> > the internet (using IPv6), but I cannot ssh
> > from the mohot (endpoint) to the internet:
> (...)
> 
> > For me it seems that home is dropping the request, if that is the case, is
> > there a good way to find out why that happens?
> 
> Did you run a packet capture on each involved interface ?

Since these are all vlans, I can run wireshark on each interface. That would 
be my next step...

> 
> > Stretch Router:
> > ---------------
> > 
> > The stretch router has an upstream interface
> > 
> > root@home:/etc# ip addr show dev eth0.1
> > 4: eth0.1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
> > state UP group default qlen 1000
> > 
> >      link/ether d0:63:b4:00:32:5c brd ff:ff:ff:ff:ff:ff
> >      inet 192.168.0.30/24 brd 192.168.0.255 scope global eth0.1
> >      
> >         valid_lft forever preferred_lft forever
> >      
> >      inet6 2a02:8070:898f:e400:d263:b4ff:fe00:325c/64 scope global
> >      mngtmpaddr
> > 
> > dynamic
> > 
> >         valid_lft 6968sec preferred_lft 3368sec
> >      
> >      inet6 fe80::d263:b4ff:fe00:325c/64 scope link
> >      
> >         valid_lft forever preferred_lft forever
> > 
> > and a downstream interface
> > 
> > root@home:/etc# ip addr show dev eth0.7
> > 10: eth0.7@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
> > state UP group default qlen 1000
> > 
> >     link/ether d0:63:b4:00:32:5c brd ff:ff:ff:ff:ff:ff
> >     inet 192.168.7.1/24 brd 192.168.7.255 scope global eth0.7
> >     
> >        valid_lft forever preferred_lft forever
> >     
> >     inet6 2a02:8070:898f:e4f8:d263:b4ff:fe00:325c/62 scope global
> 
> Why /62 instead of the standard /64 ?

I think that is the outcome of prefix delegation (I have /56 from the ISP).  
This is implemented by the script in

https://wiki.debian.org/IPv6PrefixDelegation

(at least I am not aware that I specify anywhere /62).

> 
> >        valid_lft forever preferred_lft forever
> >     
> >     inet6 fe80::d263:b4ff:fe00:325c/64 scope link
> >     
> >        valid_lft forever preferred_lft forever
> > 
> > root@home:/etc#
> > 
> > for prefix delegation, I followed the Debian wiki
> > 
> > https://wiki.debian.org/IPv6PrefixDelegation
> > 
> > And routing information:
> > root@home:/etc# ip -6 r
> > 2a02:8070:898f:e400::/64 dev eth0.1 proto kernel metric 256  expires
> > 6974sec pref medium
> > 2a02:8070:898f:e4f8::/62 dev eth0.7 proto kernel metric 256  pref medium
> 
> (...)
> 
> > IPv6 forwarding is enabled:
> > 
> > root@home:/etc# cat /proc/sys/net/ipv6/conf/all/forwarding
> > 1
> > root@home:/etc# cat /proc/sys/net/ipv6/conf/eth0.7/forwarding
> > 1
> > root@home:/etc#
> 
> What about eth0.1 ? Needed to forward incoming reply packets too.

They should be forwarded:

root@home:/etc# cat /proc/sys/net/ipv6/conf/eth0.1/forwarding 
1
root@home:/etc#
 
> > Endpoint:
> > ---------
> > rd@mohot:~$ ip addr show dev eth0
> > 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state
> > UP group default qlen 1000
> > 
> >     link/ether d0:63:b4:00:4d:d1 brd ff:ff:ff:ff:ff:ff
> >     inet 192.168.7.31/24 brd 192.168.7.255 scope global eth0
> >     
> >        valid_lft forever preferred_lft forever
> >     
> >     inet6 2a02:8070:898f:e400:d263:b4ff:fe00:4dd1/64 scope global dynamic
> > 
> > mngtmpaddr
> > 
> >        valid_lft 7030sec preferred_lft 3430sec
> >     
> >     inet6 2a02:8070:898f:e4f8:d263:b4ff:fe00:4dd1/64 scope global dynamic
> > 
> > mngtmpaddr
> > 
> >        valid_lft 14215sec preferred_lft 14215sec
> >     
> >     inet6 fe80::d263:b4ff:fe00:4dd1/64 scope link
> >     
> >        valid_lft forever preferred_lft forever
> 
> What about IPv6 routes ?

root@mohot:~# ip -6 r
2a02:8070:898f:e400::/64 dev eth0 proto kernel metric 256 pref medium
2a02:8070:898f:e4f8::/64 dev eth0 proto kernel metric 256 pref medium
2a02:8070:898f:e4f8::/62 dev eth0 proto kernel metric 256 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
default via fe80::d263:b4ff:fe00:325c dev eth0 proto ra metric 1024 hoplimit 64 
pref medium
default via fe80::e228:6dff:fe43:5776 dev eth0 proto ra metric 1024 hoplimit 
255 pref medium
root@mohot:~# 

I thought I better start with an empty ipv6 config on mohot, but even with

rd@mohot:~$ cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
allow-hotplug eth0
iface eth0 inet dhcp
# This is an autoconfigured IPv6 interface

#iface eth0 inet6 auto
iface eth0 inet6 manual
rd@mohot:~$ 

I get the same output. Not sure why these routes and addresses are configured 
on mohot.

Thanks
Rainer

-- 
Rainer Dorsch
http://bokomoko.de/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web