Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.debian.kernel > #60670 > unrolled thread
| Started by | superuser <kk@tatar.ru> |
|---|---|
| First post | 2018-04-11 10:10 +0200 |
| Last post | 2018-04-12 21:10 +0200 |
| Articles | 2 — 2 participants |
Back to article view | Back to linux.debian.kernel
Bug#895404: nfs-kernel-server: NFS server stops accepting mount request / mounted NFS directories became inaccecible on client superuser <kk@tatar.ru> - 2018-04-11 10:10 +0200
Bug#895404: NFS server stops accepting mount request / mounted NFS directories became inaccessible on client Sergio Gelato <Sergio.Gelato@astro.su.se> - 2018-04-12 21:10 +0200
| From | superuser <kk@tatar.ru> |
|---|---|
| Date | 2018-04-11 10:10 +0200 |
| Subject | Bug#895404: nfs-kernel-server: NFS server stops accepting mount request / mounted NFS directories became inaccecible on client |
| Message-ID | <vDiZj-2Kg-3@gated-at.bofh.it> |
Package: nfs-kernel-server
Version: 1:1.3.4-2.1
Severity: critical
Justification: causes serious data loss
Dear Maintainer,
NFS directory exported for local network host, and mounted by client.
After few days normal workload that share on client side became
inaccessible.
syslog on client side shows: nfs: server <server_ip> not responding,
still trying
Tried to mount that share from NFS-server itself but it hangs for
timeout value(5 min) and report about that:
$ mount.nfs: Connection timed out
Tried to restart nfs-kernel-server.service but it is not gives any
positive result. Tried to kill nfsd processed by -9 and PID but it does
not kills any of them. journalctl show nothing suspicious.
Problem get resolved only after reboot of the server each time.
Changing kernel to one from backport repository did not helped.
-- Package-specific info:
-- rpcinfo --
program vers proto port service
100000 4 tcp 111 portmapper
100000 3 tcp 111 portmapper
100000 2 tcp 111 portmapper
100000 4 udp 111 portmapper
100000 3 udp 111 portmapper
100000 2 udp 111 portmapper
100005 1 udp 42805 mountd
100005 1 tcp 42241 mountd
100005 2 udp 44844 mountd
100005 2 tcp 51819 mountd
100005 3 udp 52308 mountd
100005 3 tcp 54731 mountd
100003 3 tcp 2049 nfs
100227 3 tcp 2049
100003 3 udp 2049 nfs
100227 3 udp 2049
100021 1 udp 46342 nlockmgr
100021 3 udp 46342 nlockmgr
100021 4 udp 46342 nlockmgr
100021 1 tcp 41403 nlockmgr
100021 3 tcp 41403 nlockmgr
100021 4 tcp 41403 nlockmgr
-- /etc/default/nfs-kernel-server --
RPCNFSDCOUNT=8
RPCNFSDPRIORITY=0
RPCMOUNTDOPTS="--manage-gids --no-nfs-version 4"
RPCNFSDOPTS="--no-nfs-version 4"
NEED_SVCGSSD=""
RPCSVCGSSDOPTS=""
-- /etc/exports --
/srv/backups 10.196.1.224/27(rw,sync,no_subtree_check)
-- /proc/fs/nfs/exports --
# Version 1.1
# Path Client(Flags) # IPs
-- System Information:
Debian Release: 9.4
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'stable')
Architecture: amd64 (x86_64)
Kernel: Linux 4.9.0-6-amd64 (SMP w/4 CPU cores)
Locale: LANG=ru_RU.UTF-8, LC_CTYPE=ru_RU.UTF-8 (charmap=UTF-8), LANGUAGE=ru_RU.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
Versions of packages nfs-kernel-server depends on:
ii init-system-helpers 1.48
ii keyutils 1.5.9-9
ii libblkid1 2.29.2-1+deb9u1
ii libc6 2.24-11+deb9u3
ii libcap2 1:2.25-1
ii libsqlite3-0 3.16.2-5+deb9u1
ii libtirpc1 0.2.5-1.2
ii libwrap0 7.6.q-26
ii lsb-base 9.20161125
ii netbase 5.4
ii nfs-common 1:1.3.4-2.1
ii ucf 3.0036
nfs-kernel-server recommends no packages.
nfs-kernel-server suggests no packages.
-- no debconf information
[toc] | [next] | [standalone]
| From | Sergio Gelato <Sergio.Gelato@astro.su.se> |
|---|---|
| Date | 2018-04-12 21:10 +0200 |
| Subject | Bug#895404: NFS server stops accepting mount request / mounted NFS directories became inaccessible on client |
| Message-ID | <vDPLA-SI-9@gated-at.bofh.it> |
| In reply to | #60670 |
control: severity -1 normal control: tags -1 + moreinfo Dear reporter, I'm sorry to hear that you have lost data. However, it doesn't seem very constructive to make a bug release-critical without providing enough detail to make a fix possible. NFS is a complex network protocol, and the root cause of unexpected behaviour isn't always obvious at first glance. First of all, has this bug been filed against the right package? The nfsd processes are actually kernel threads (that's one reason "kill -9" doesn't work on them), the corresponding package is the kernel image. How many clients are accessing that NFS server when the problem occurs? I see that you have RPCNFSDCOUNT=8 but the address range for allowed clients is a /27. If you have 30 clients all trying to write at the same time, some of them are going to have to wait until a server thread becomes available. "server not responding, still trying" is a common symptom of this. Have you tried tuning the server? You can adjust the thread count without a reboot. I don't see sec=krb5p in your /etc/exports, so NFS traffic on the wire is probably unencrypted. Have you looked at it with tcpdump or a similar tool, particularly when the problem occurs? For example it would be nice to know whether that "Connection timed out" you get from mount.nfs is for the portmapper (unlikely), for mountd, or for nfsd itself. (strace may also tell you some of this.) Are you familiar with rpcdebug? If client traffic is coming in but the server isn't replying, you could set debugging flags and look at kernel log output. Other available debugging tools include the kernel's event tracing subsystem, as well as nfsstat, nfsiostat and mountstats from package nfs-common. (The last two are client-side, so maybe not so useful if your problem really is at the server end.) I can't help you much more than this: my own environment is NFSv4-only (and I feel no urge to look back) while yours is anything but. But if you manage to pinpoint more precisely what's wrong, someone else may be able to provide better hints (or you may figure it out yourself).
[toc] | [prev] | [standalone]
Back to top | Article view | linux.debian.kernel
csiph-web