Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #66182 > unrolled thread

Bug#861115: Please consider increasing net.ipv6.route.max_size default value

Back to article view | Back to linux.debian.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Bug#861115: Please consider increasing net.ipv6.route.max_size default value Tim Bray <tim@provu.co.uk> - 2020-01-21 20:20 +0100
    Bug#861115: Please consider increasing net.ipv6.route.max_size default value Vincent Bernat <bernat@debian.org> - 2020-01-21 20:50 +0100
      Bug#861115: Please consider increasing net.ipv6.route.max_size default value Tim Bray <tim@provu.co.uk> - 2020-01-22 01:00 +0100
        Bug#861115: Please consider increasing net.ipv6.route.max_size default value Vincent Bernat <bernat@debian.org> - 2020-01-22 07:00 +0100
    Bug#861115: Please consider increasing net.ipv6.route.max_size default value Asbjørn Sloth Tønnesen <ast@fiberby.net> - 2020-02-27 15:50 +0100

#66182 — Bug#861115: Please consider increasing net.ipv6.route.max_size default value

I also agree.

I've just spent a month tracking down an issue where IPv6 networking 
connections hang.  Using Debian buster, 4.19.0-6-amd64.

Machine carries a full IPv6 table, received via BGP. About 80k routes.

Symptoms were BFD dropping out.  Connectivity hangs.

setting net.ipv6.route.max_size to 2147483647 fixed all issues.    
(aware this might be too big)

Later kernels  (5.4.11) cause a dmesg when full.

Tim

[toc] | [next] | [standalone]

#66183

[Multipart message — attachments visible in raw view] — view raw

 ❦ 21 janvier 2020 18:58 +00, Tim Bray <tim@provu.co.uk>:

> I also agree.
>
> I've just spent a month tracking down an issue where IPv6 networking
> connections hang.  Using Debian buster, 4.19.0-6-amd64.
>
> Machine carries a full IPv6 table, received via BGP. About 80k routes.
>
> Symptoms were BFD dropping out.  Connectivity hangs.
>
> setting net.ipv6.route.max_size to 2147483647 fixed all issues.   
> (aware this might be too big)
>
> Later kernels  (5.4.11) cause a dmesg when full.

This setting is for the size of the route cache. You can have far more
routes. In the past, there were bugs where each lookup will be put in
cache, but since 4.2, only the PMTU exceptions are stored in cache. See
<https://vincent.bernat.ch/en/blog/2017-ipv6-route-lookup-linux#caching>

Do you have a lot of entries in `ip -6 route show cache`?
-- 
Avoid multiple exits from loops.
            - The Elements of Programming Style (Kernighan & Plauger)

[toc] | [prev] | [next] | [standalone]

#66188

On 21/01/2020 19:37, Vincent Bernat wrote:
> This setting is for the size of the route cache. You can have far more
> routes. In the past, there were bugs where each lookup will be put in
> cache, but since 4.2, only the PMTU exceptions are stored in cache. See
> <https://vincent.bernat.ch/en/blog/2017-ipv6-route-lookup-linux#caching>
>
> Do you have a lot of entries in `ip -6 route show cache`?


No entries at all in
`ip -6 route show cache`

and none in `ip -6 route show table cache`

It is a quiet time of day, so I'll check again.

What prompted me to look at net.ipv6.route.max_size was because I built my own newer kernel (in desperation).

Linux version 5.4.11 (from kernel.org, .config from debian buster, make olddefconfig) gives out in dmesg:

Route cache is full: consider increasing sysctl net.ipv[4|6].route.max_size.


I've never seen this message on 4.19 Debian buster,  but increasing net.ipv6.route.max_size made all the network hangs go away, both on 5.4.11 and 4.19

Tim Bray

[toc] | [prev] | [next] | [standalone]

#66189

[Multipart message — attachments visible in raw view] — view raw

 ❦ 21 janvier 2020 23:51 +00, Tim Bray <tim@provu.co.uk>:

>> This setting is for the size of the route cache. You can have far more
>> routes. In the past, there were bugs where each lookup will be put in
>> cache, but since 4.2, only the PMTU exceptions are stored in cache. See
>> <https://vincent.bernat.ch/en/blog/2017-ipv6-route-lookup-linux#caching>
>>
>> Do you have a lot of entries in `ip -6 route show cache`?
>
>
> No entries at all in
> `ip -6 route show cache`
>
> and none in `ip -6 route show table cache`
>
> It is a quiet time of day, so I'll check again.
>
> What prompted me to look at net.ipv6.route.max_size was because I built my own newer kernel (in desperation).
>
> Linux version 5.4.11 (from kernel.org, .config from debian buster, make olddefconfig) gives out in dmesg:
>
> Route cache is full: consider increasing sysctl net.ipv[4|6].route.max_size.
>
>
> I've never seen this message on 4.19 Debian buster, but increasing
> net.ipv6.route.max_size made all the network hangs go away, both on
> 5.4.11 and 4.19

The message was addeed in 5.1. I'll test later today to check if "ip -6
route show cache" still works.
-- 
Parenthesise to avoid ambiguity.
            - The Elements of Programming Style (Kernighan & Plauger)

[toc] | [prev] | [next] | [standalone]

#66424

Hi Tim,

On 1/21/20 6:58 PM, Tim Bray wrote:
> I also agree.
> 
> I've just spent a month tracking down an issue where IPv6 networking connections hang.  Using Debian buster, 
> 4.19.0-6-amd64.

I also had fun finding a workaround for this bug, back in December, but I originally
noticed some weirdness in v4.9 (neighbor being gc'ed immediately, possibly due to route
falling out of cache).

The below messages seams related:

Basil Fillan <jack@basilfillan.uk> on the bird-users list[1]:
 > We've also experienced this after upgrading a few routers to Debian
 > Buster. With a kernel bisect we found that a bug was introduced in the
 > following commit:
 >
 > 3b6761d18bc11f2af2a6fc494e9026d39593f22c
 >
 > This bug was still present in master as of a few weeks ago.
 >
 > It appears entries are added to the IPv6 route cache which aren't
 > visible from "ip -6 route show cache", but are causing the route cache
 > garbage collection system to trigger extremely often (every packet?)
 > once it exceeds the value of net.ipv6.route.max_size. [...]

commit 3b6761d18bc11f2af2a6fc494e9026d39593f22c
Author: David Ahern <dsahern@gmail.com>
Date:   Tue Apr 17 17:33:20 2018 -0700

     net/ipv6: Move dst flags to booleans in fib entries

     Continuing to wean FIB paths off of dst_entry, use a bool to hold
     requests for certain dst settings. Add a helper to convert the
     flags to DST flags when a FIB entry is converted to a dst_entry.

     Signed-off-by: David Ahern <dsahern@gmail.com>
     Signed-off-by: David S. Miller <davem@davemloft.net>

Brian Rak <brak@gameservers.com> on the netdev list[2]:
> I have net.ipv6.route.max_size=4096, and the machine only has 376 routes 
> (calculated by `ip -6 route | wc -l`).  However, raising this sysctl to 
> 65k seems to instantly fix IPv6 (I'm not sure if this is a permanent fix 
> yet)

I also found an older thread[3] from 2014 on netdev, that revolves around
the same topic as this bug, but it's might only share the workaround.


[1] Re: IPv6 BGP & kernel 4.19
     https://bird.network.cz/pipermail/bird-users/2020-February/014270.html

[2] Re: IPv6 neighbor discovery issues on 4.18 (and now 4.19)
     https://lore.kernel.org/netdev/de54e925-9536-f2cc-7b89-7205b3fb2c18@gameservers.com/

[3] IPv6 routing table max_size badly dimensioned compared to IPv4
     https://lore.kernel.org/netdev/20140227192403.GA13472@xs.powerdns.com/

-- 
Best regards
Asbjørn Sloth Tønnesen
Network Engineer
Fiberby ApS - AS42541

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.kernel

csiph-web