Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #60837 > unrolled thread

Bug#897917: Stretch kernel 4.9.88-1 breaks startup of RPC, KDC services

Back to article view | Back to linux.debian.kernel

Contents

  Bug#897917: Stretch kernel 4.9.88-1 breaks startup of RPC, KDC services "Michael J. Redd" <micredd@gmail.com> - 2018-05-04 20:30 +0200
    Bug#897917: Stretch kernel 4.9.88-1 breaks startup of RPC, KDC services Arne Nordmark <nordmark@mech.kth.se> - 2018-05-05 12:30 +0200
    Bug#897917: Stretch kernel 4.9.88-1 breaks startup of RPC, KDC services "Michael J. Redd" <micredd@gmail.com> - 2018-05-05 16:50 +0200
      Bug#897917: Stretch kernel 4.9.88-1 breaks startup of RPC, KDC services "Michael J. Redd" <micredd@gmail.com> - 2018-05-05 17:30 +0200

#60837 — Bug#897917: Stretch kernel 4.9.88-1 breaks startup of RPC, KDC services

Package: linux-image-4.9.0-6-amd64
Version: 4.9.88-1

Issue:
======

Kernel "linux-image-4.9.0-6-amd64," version 4.9.88-1, breaks systemd
startup of RPC, Kerberos KDC services. 

Description:
============

After upgrading to the latest Stretch kernel (4.9.88-1), RPC and KDC
services time out during the boot process. This issue is being seen on
a Kerberos KDC that is also an NFS client. Kerberos auth. and
encryption are being used with NFS in this environment, and this KDC
provides the Kerberos services for that to work.

Network is functional prior to these services starting, which is
proper.

After the server has booted completely, I can issue `service krb5-kdc
restart` and, after a short delay, the KDC service starts normally.

Not sure if this is a kernel bug, a systemd bug, or something else.
Since the kernel package was the only thing that was upgraded before
the issue started, I'm leaning toward the kernel.

Relevant output from /var/log/syslog:
-------------------------------------

May  4 09:03:17 <redacted> systemd[1]: rpc-svcgssd.service: Start
operation timed out. Terminating.
May  4 09:03:17 <redacted> systemd[1]: Failed to start RPC security
service for NFS server.
May  4 09:03:17 <redacted> systemd[1]: rpc-svcgssd.service: Unit
entered failed state.
May  4 09:03:17 <redacted> systemd[1]: rpc-svcgssd.service: Failed with
result 'timeout'.
May  4 09:03:17 <redacted> systemd[1]: rpc-gssd.service: Start
operation timed out. Terminating.
May  4 09:03:17 <redacted> systemd[1]: Failed to start RPC security
service for NFS client and server.
May  4 09:03:17 <redacted> systemd[1]: rpc-gssd.service: Unit entered
failed state.
May  4 09:03:17 <redacted> systemd[1]: rpc-gssd.service: Failed with
result 'timeout'.
May  4 09:03:20 <redacted> systemd[1]: krb5-kdc.service: Start
operation timed out. Terminating.
May  4 09:03:20 <redacted> systemd[1]: Failed to start Kerberos 5 Key
Distribution Center.
May  4 09:03:20 <redacted> systemd[1]: krb5-kdc.service: Unit entered
failed state.29s random time.
May  4 09:03:20 <redacted> systemd[1]: krb5-kdc.service: Failed with
result 'timeout'. random time.

Workaround:
===========

Rolling back to Stretch kernel 4.9.82-1+deb9u3 fixes the issue.

Setup:
======

1. KDC package: krb5-kdc 1.15-1+deb9u1
2. NFS package: nfs-common 1:1.3.4-2.1
3. Kernel: linux-image-4.9.0-6-amd64 4.9.88-1
4. Systemd version: 232-25+deb9u3
5. Server is a 64-bit Xen PV domU

[toc] | [next] | [standalone]

#60842

I have also seen this on a couple of SSD-only systems.

I think the problem is that the random number generator takes about two
minutes to initialize, long enough for systemd to give up on these
processes. Unbound is similar, but there unit file keeps trying until
the random numbers are available.

From the log:
May  5 10:19:02 ano2 kernel: [  126.436729] random: crng init done

Pressing the keyboard a few times (thus providing entropy) will allow
the boot to continue.

This definitely seems to be a kernel problem.

Arne

[toc] | [prev] | [next] | [standalone]

#60853

[Multipart message — attachments visible in raw view] — view raw

Interesting! I'd also noticed 'random: init done' being piped to console well after the server had booted, but I didn't mention it because I didn't think it was related. What you've said makes a lot of sense.

On Sat, 5 May 2018 11:54:54 +0200 Arne Nordmark <nordmark@mech.kth.se> wrote:
> I have also seen this on a couple of SSD-only systems.
> 
> I think the problem is that the random number generator takes about two
> minutes to initialize, long enough for systemd to give up on these
> processes. Unbound is similar, but there unit file keeps trying until
> the random numbers are available.
> 
> From the log:
> May  5 10:19:02 ano2 kernel: [  126.436729] random: crng init done
> 
> Pressing the keyboard a few times (thus providing entropy) will allow
> the boot to continue.
> 
> This definitely seems to be a kernel problem.
> 
> Arne
> 
> 

-- 
Sent from my Android device with K-9 Mail.

[toc] | [prev] | [next] | [standalone]

#60854

On further investigation, Arne's absolutely right. I upgraded the
kernel back to 4.9.88-1 from Debian Security and installed 'haveged'
(another random number generator). Everything started quickly and
normally after a reboot. Turns out I hadn't noticed this on any of my
other virtual servers because they're all running haveged anyway.

So,

Workarounds:
============
1. Roll back kernel to 4.9.82-1+deb9u3
OR
2. Install another RNG, such has 'haveged'

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.kernel

csiph-web