Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #267974 > unrolled thread

systemd-resolved resolving fails sometimes on Debian12

Back to article view | Back to linux.debian.user

Contents

  systemd-resolved resolving fails sometimes on Debian12 Victor Sudakov <vas@sibptus.ru> - 2024-03-01 11:30 +0100
    Re: systemd-resolved resolving fails sometimes on Debian12 jeremy ardley <jeremy.ardley@gmail.com> - 2024-03-01 21:20 +0100
      Re: systemd-resolved resolving fails sometimes on Debian12 Victor Sudakov <vas@sibptus.ru> - 2024-03-02 16:30 +0100
        Re: systemd-resolved resolving fails sometimes on Debian12 jeremy ardley <jeremy.ardley@gmail.com> - 2024-03-03 01:30 +0100
          Re: systemd-resolved resolving fails sometimes on Debian12 Victor Sudakov <vas@sibptus.ru> - 2024-03-03 06:10 +0100
            Re: systemd-resolved resolving fails sometimes on Debian12 jeremy ardley <jeremy.ardley@gmail.com> - 2024-03-03 10:30 +0100
              Re: systemd-resolved resolving fails sometimes on Debian12 Victor Sudakov <vas@sibptus.ru> - 2024-03-03 16:00 +0100
                Re: systemd-resolved resolving fails sometimes on Debian12 jeremy ardley <jeremy.ardley@gmail.com> - 2024-03-04 07:30 +0100
                  Re: systemd-resolved resolving fails sometimes on Debian12 Noah Meyerhans <noahm@debian.org> - 2024-05-21 21:40 +0200
        Re: systemd-resolved resolving fails sometimes on Debian12 Marco Moock <mm@dorfdsl.de> - 2024-03-04 08:40 +0100

#267974 — systemd-resolved resolving fails sometimes on Debian12

[Multipart message — attachments visible in raw view] — view raw

Dear Colleagues,

Has anybody encountered this problem using systemd-resolved as a
resolver on Debian12? A DNS request via systemd-resolved fails, but
fails only occasionally. A failure can happen once per a hundred
successful requests or so. If I run:

while resolvectl query myredis.my.domain ; do sleep 1; done

This will eventually happen:

-- Information acquired via protocol DNS in 960us.
-- Data is authenticated: no; Data was acquired via local or encrypted transport: no
-- Data from: cache network
myredis.my.domain: x.x.44.189                -- link: ens5
                  (redis-cache2-002.tqma2d.0001.usw2.cache.amazonaws.com)

-- Information acquired via protocol DNS in 1.1ms.
-- Data is authenticated: no; Data was acquired via local or encrypted transport: no
-- Data from: cache network
myredis.my.domain: x.x.44.189                -- link: ens5
                  (redis-cache2-002.tqma2d.0001.usw2.cache.amazonaws.com)

-- Information acquired via protocol DNS in 2.2ms.
-- Data is authenticated: no; Data was acquired via local or encrypted transport: no
-- Data from: network
myredis.my.domain: resolve call failed: Lookup failed due to system error: Invalid argument

Then it works again for a hundred or so queries. Query monitoring
shows that systemd-resolved occasionally returns "EINVAL", but mostly
"success".

Any ideas please? It is very unpleasant because the AWS Debian AMI has
systemd-resolved as the default caching resolver and it will take some
effort to eradicate it and replace with unbound or something else.

I don't blame the parent DNS server (from AWS) because if I query it
directly, it always answers.

-- 
Victor Sudakov VAS4-RIPE
http://vas.tomsk.ru/
2:5005/49@fidonet

[toc] | [next] | [standalone]

#267987

On 1/3/24 17:47, Victor Sudakov wrote:
> Has anybody encountered this problem using systemd-resolved as a
> resolver on Debian12? A DNS request via systemd-resolved fails, but
> fails only occasionally. A failure can happen once per a hundred
> successful requests or so. If I run:


I recall a similar problem with systemd-resolved. I think it was related 
to DNSSEC.

I ended up not using systemd-resolved

Alternatives to systemd-resolved include dnsmasq  - which doesn't 
support DNSSEC - and bind9 which does.

[toc] | [prev] | [next] | [standalone]

#267992

[Multipart message — attachments visible in raw view] — view raw

jeremy ardley wrote:
> 
> On 1/3/24 17:47, Victor Sudakov wrote:
> > Has anybody encountered this problem using systemd-resolved as a
> > resolver on Debian12? A DNS request via systemd-resolved fails, but
> > fails only occasionally. A failure can happen once per a hundred
> > successful requests or so. If I run:
> 
> 
> I recall a similar problem with systemd-resolved. I think it was related to
> DNSSEC.

In my case the problem seems related to IPv6. That is, when I disable
IPv6 via "sysctl net.ipv6.conf.all.disable_ipv6=1" the problem
disappears.

I did not enable DNSSEC in systemd-networkd.

> 
> I ended up not using systemd-resolved
> 
> Alternatives to systemd-resolved include dnsmasq  - which doesn't support
> DNSSEC - and bind9 which does.

You know, the official Debian 12 AMI for AWS is built on
systemd-resolved and systemd-networkd. I'd prefer not to have to
modify the official AMI if I can help it, because this would probably
mean also replacing the systemd-networkd with some other network
manager.

Anyway, if there is a bug in systemd-resolved it should be reported, right? 

I have been able to google up similar (though not exactly the same)
issues with systemd-resolved and the caching of CNAME records which
give similar random resolution errors, but they are reported as fixed.

I tried enabling the debug messages in systemd-resolved and probably
(just probably) the random error happens when systemd-resolved's cache
for the particular entry expires, but I'm not sure. In fact the debug
was not very informative, or I lack the qualification to interpret it.

-- 
Victor Sudakov VAS4-RIPE
http://vas.tomsk.ru/
2:5005/49@fidonet

[toc] | [prev] | [next] | [standalone]

#267993

On 2/3/24 23:06, Victor Sudakov wrote:
> You know, the official Debian 12 AMI for AWS is built on
> systemd-resolved and systemd-networkd. I'd prefer not to have to
> modify the official AMI if I can help it, because this would probably
> mean also replacing the systemd-networkd with some other network
> manager.

systemd-networkd does not rely on systemd-resolved for name resolution.

There is a relationship where systemd-networkd can feed dns information 
to system-resolved that could be helpful in dynamic IP configurations 
like laptops. However this is not usual case in AWS deployments.

The Debian AWS AMI does not use the usual NetworkManager configuration 
because the usual AWS deployment does not required dynamic  DNS.

In my AWS deployments I remove systemd-resolved and use bind9 instead.

[toc] | [prev] | [next] | [standalone]

#267994

[Multipart message — attachments visible in raw view] — view raw

jeremy ardley wrote:
> 
> On 2/3/24 23:06, Victor Sudakov wrote:
> > You know, the official Debian 12 AMI for AWS is built on
> > systemd-resolved and systemd-networkd. I'd prefer not to have to
> > modify the official AMI if I can help it, because this would probably
> > mean also replacing the systemd-networkd with some other network
> > manager.
> 
> systemd-networkd does not rely on systemd-resolved for name resolution.
> 
> There is a relationship where systemd-networkd can feed dns information to
> system-resolved that could be helpful in dynamic IP configurations like
> laptops. However this is not usual case in AWS deployments.

How is it not usual when the Debian12 AMI for AWS works exactly this way?
The system-resolved config is empty (contains only comments) this means that it
obtains the upstream DNS address from systemd-networkd.

> 
> The Debian AWS AMI does not use the usual NetworkManager configuration
> because the usual AWS deployment does not required dynamic  DNS.

How does it not require dynamic DNS when an EC2 instance obtains the
upstream DNS server address from DHCP?

> 
> In my AWS deployments I remove systemd-resolved and use bind9 instead.
> 

Not that I would use bind9 as a caching resolver but still, how
do you pass the dynamically obtained AWS DNS server address from
systemd-networkd to bind9 ?

-- 
Victor Sudakov VAS4-RIPE
http://vas.tomsk.ru/
2:5005/49@fidonet

[toc] | [prev] | [next] | [standalone]

#267996

On 3/3/24 12:43, Victor Sudakov wrote:
> Not that I would use bind9 as a caching resolver but still, how
> do you pass the dynamically obtained AWS DNS server address from
> systemd-networkd to bind9 ?


The AWS DNS resolver IPs are static and are widely published.

It is permissible to not use AWS resolvers for upstream.

If you want to use AWS resolvers you may run into the problem that some 
RBL services reject queries from 'well known' free DNS servers; that may 
include AWS resolver addresses.

systemd-networkd without systemd-resolved maintains a list of DNS 
servers in /etc/resolv.conf that can be used by local services.

You can override dynamic setting of the dns resolvers in the 
systemd-networkd configuration to use a local caching resolver such as 
bind9, usually listening at 127.0.0.1:53

You can then configure bind 9 as a caching only DNS resolver and set 
appropriate upstream (forwarder) sites, or none at all defaulting to the 
root servers.

[toc] | [prev] | [next] | [standalone]

#268005

[Multipart message — attachments visible in raw view] — view raw

jeremy ardley wrote:
> 
> On 3/3/24 12:43, Victor Sudakov wrote:
> > Not that I would use bind9 as a caching resolver but still, how
> > do you pass the dynamically obtained AWS DNS server address from
> > systemd-networkd to bind9 ?
> 
> 
> The AWS DNS resolver IPs are static and are widely published.

Do you mean 169.254.169.253?
> 
> It is permissible to not use AWS resolvers for upstream.
> 
> If you want to use AWS resolvers you may run into the problem that some RBL
> services reject queries from 'well known' free DNS servers; that may include
> AWS resolver addresses.
> 
> systemd-networkd without systemd-resolved maintains a list of DNS servers in
> /etc/resolv.conf that can be used by local services.

Do you just disable the systemd-resolved service or do you remove the
systemd-resolved package completely?

If you disable it, you are also supposed to remove the "resolve"
service from nsswitch.conf, right?

> You can override dynamic setting of the dns resolvers in the
> systemd-networkd configuration to use a local caching resolver such as
> bind9, usually listening at 127.0.0.1:53

What would this be for? Sorry, I did not understand this step.
> 
> You can then configure bind 9 as a caching only DNS resolver and set
> appropriate upstream (forwarder) sites, or none at all defaulting to the
> root servers.
> 

Thank you for the ideas, I may use them but first I would like to do
something about the obvious bug in systemd-resolved.

-- 
Victor Sudakov VAS4-RIPE
http://vas.tomsk.ru/
2:5005/49@fidonet

[toc] | [prev] | [next] | [standalone]

#268017

On 3/3/24 22:39, Victor Sudakov wrote:
> jeremy ardley wrote:
>> On 3/3/24 12:43, Victor Sudakov wrote:
>>> Not that I would use bind9 as a caching resolver but still, how
>>> do you pass the dynamically obtained AWS DNS server address from
>>> systemd-networkd to bind9 ?
>>
>> The AWS DNS resolver IPs are static and are widely published.
> Do you mean 169.254.169.253?


That IP address is a non-routable AWS internal address for internal DNS 
services. It is not the public IP address

There is some convention that address 2 in non routable address ranges 
allocatted to customers is also a DNS server address. So 10.0.0.2 is a 
DNS server.

The actual public source addresses used by AWS DNS servers are not well 
defined and may vary by region

>> It is permissible to not use AWS resolvers for upstream.
>>
>> If you want to use AWS resolvers you may run into the problem that some RBL
>> services reject queries from 'well known' free DNS servers; that may include
>> AWS resolver addresses.
>>
>> systemd-networkd without systemd-resolved maintains a list of DNS servers in
>> /etc/resolv.conf that can be used by local services.
> Do you just disable the systemd-resolved service or do you remove the
> systemd-resolved package completely?

I completely removed system-resolved as when it is installed it changes 
the DNS configuration to be non-standard


>
> If you disable it, you are also supposed to remove the "resolve"
> service from nsswitch.conf, right?

I am not sure what you mean by resolve service. The current user manual 
manual for  nsswitch has

hosts:          dns [!UNAVAIL=return] files

which seems to be some new spin. It has always been the practice to use 
files and then dns if nothing found

hosts: files dns

in neither case is systemd-resolved required. The resolution uses the 
contents of /etc/resolv.conf to choose a resolver.


>
>> You can override dynamic setting of the dns resolvers in the
>> systemd-networkd configuration to use a local caching resolver such as
>> bind9, usually listening at 127.0.0.1:53
> What would this be for? Sorry, I did not understand this step.

I was in error stating that. You need to manually edit /etc/resolv.conf 
to contain a line

nameserver 127.0.0.1

and configure bind9 to listen to that

options {
     listen-on { 127.0.0.1; };
     // other options like directory, allow-query, etc.
     recursion yes;
     // Additional configuration to ensure it acts as a caching server
};


>> You can then configure bind 9 as a caching only DNS resolver and set
>> appropriate upstream (forwarder) sites, or none at all defaulting to the
>> root servers.
>>
> Thank you for the ideas, I may use them but first I would like to do
> something about the obvious bug in systemd-resolved.
>

[toc] | [prev] | [next] | [standalone]

#269478

On Mon, Mar 04, 2024 at 02:03:32PM +0800, jeremy ardley wrote:
> I completely removed system-resolved as when it is installed it changes the
> DNS configuration to be non-standard

The issues described in this thread are related to libnss-resolve, which
is no longer installed in the Debian 12 cloud images.

In most cases, I don't recommend removing systemd-resolved, as it is
responsible for managing the contents of /etc/resolv.conf based on the
information provided in the DHCP lease.  If you're managing
/etc/resolv.conf yourself, then you can remove systemd-resolved.

> > Thank you for the ideas, I may use them but first I would like to do
> > something about the obvious bug in systemd-resolved.

The name resolution issue with nss-resolve is tracked upstream at
https://github.com/systemd/systemd/issues/29069 and the cloud team's
response to it is discussed in the thread starting with
https://lists.debian.org/debian-cloud/2024/03/msg00017.html

The issue is not present in the current cloud images for AWS or other
environments.

Since the impacted configuration was common to the cloud images but not
the default installation, it might have been worth engaging directly
with the cloud community.  You can file bug reports against
cloud.debian.org or reach out to us on #debian-cloud or
debian-cloud@lists.debian.org.  Most of us aren't regular readers of
debian-user.

noah
(for the cloud team)

[toc] | [prev] | [next] | [standalone]

#268018

Am 02.03.2024 um 15:06:01 Uhr schrieb Victor Sudakov:

> In my case the problem seems related to IPv6. That is, when I disable
> IPv6 via "sysctl net.ipv6.conf.all.disable_ipv6=1" the problem
> disappears.

Please run 
resolvectl

that shows the resolvers available.

Check each of them with
dig example.org @<IP-of-resolver>

If that doesn't work for one of them, this must be fixed and is not a
problem of IPv6 nor of systemd-resolve.

-- 
Gruß
Marco

Send spam to 1709388361muell@cartoonies.org

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web