Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #158163 > unrolled thread

NFS rename sometimes hangs for 15 seconds after upgrade to Debian 8

Back to article view | Back to linux.debian.user

Contents

  NFS rename sometimes hangs for 15 seconds after upgrade to Debian 8 Vincent Lefevre <vincent@vinc17.net> - 2015-09-22 15:20 +0200
    Re: NFS rename sometimes hangs for 15 seconds after upgrade to  Debian 8 Vincent Lefevre <vincent@vinc17.net> - 2015-09-23 14:00 +0200
    Re: NFS rename sometimes hangs for 15 seconds after upgrade to Debian  8 Peter Ludikovsky <peter@ludikovsky.name> - 2015-09-23 18:00 +0200
      Re: NFS rename sometimes hangs for 15 seconds after upgrade to  Debian 8 Vincent Lefevre <vincent@vinc17.net> - 2015-09-24 11:50 +0200
        Re: NFS rename sometimes hangs for 15 seconds after upgrade to Debian  8 Peter Ludikovsky <peter@ludikovsky.name> - 2015-09-24 14:40 +0200
          Re: NFS rename sometimes hangs for 15 seconds after upgrade to  Debian 8 Vincent Lefevre <vincent@vinc17.net> - 2015-09-25 09:40 +0200
            Re: NFS rename sometimes hangs for 15 seconds after upgrade to Debian  8 Peter Ludikovsky <peter@ludikovsky.name> - 2015-09-25 10:20 +0200
              Re: NFS rename sometimes hangs for 15 seconds after upgrade to  Debian 8 Vincent Lefevre <vincent@vinc17.net> - 2015-09-25 14:10 +0200
                Re: NFS rename sometimes hangs for 15 seconds after upgrade to  Debian 8 Vincent Lefevre <vincent@vinc17.net> - 2015-09-28 17:40 +0200

#158163 — NFS rename sometimes hangs for 15 seconds after upgrade to Debian 8

At my lab, with NFS accounts, some machines have been upgraded to
Debian 8, and I get regular hangs on these machines, while they
never occurred before on the same machine and still never occur
on a machine that is still under Debian 7.

This happens with one of my scripts, which does a lot of "mv",
and more often when the target has just been modified from another
machine.

A strace showed that it is the "rename" system call that hangs:

52026      0.000208 execve("/bin/mv", ["mv", "-v", "ssh/ssh-reconfigure", "/home/vlefevre/bin/ssh-reconfigu"...], [/* 112 vars */]) = 0
[...]
52026      0.000097 rename("ssh/ssh-reconfigure", "/home/vlefevre/bin/ssh-reconfigure") = 0
52026     15.057725 lseek(0, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek)
52026      0.000189 close(0)            = 0
52026      0.000175 close(1)            = 0
52026      0.000093 munmap(0x7fd5a610b000, 4096) = 0
52026      0.000079 close(2)            = 0
52026      0.000106 exit_group(0)       = ?
52026      0.000307 +++ exited with 0 +++

The NFS server isn't particularly loaded.

Has anyone seen a problem like that?
Where does it come from?
And why does it affect only Debian 8 machines? A new bug?

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

[toc] | [next] | [standalone]

#158179 — Re: NFS rename sometimes hangs for 15 seconds after upgrade to Debian 8

On 2015-09-22 15:19:10 +0200, Vincent Lefevre wrote:
> At my lab, with NFS accounts, some machines have been upgraded to
> Debian 8, and I get regular hangs on these machines, while they
> never occurred before on the same machine and still never occur
> on a machine that is still under Debian 7.
[...]

I've done a bug report:

  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=799838

with a simple way to reproduce the bug:

  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=799838#10

i.e. it suffices to read the target on some machine, then move a file
to the target on another (Debian 8) machine.

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

[toc] | [prev] | [next] | [standalone]

#158187 — Re: NFS rename sometimes hangs for 15 seconds after upgrade to Debian 8

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello,

I tried to replicate your problem, but couldn't.

When mounted as NFS3 over UDP:
- ----
root@lab1# echo foo > file1
root@lab1# cp file1 file2

root@lab2# cat file2
foo

root@lab1# time mv file1 file2
real	0m0.004s
user	0m0.000s
sys	0m0.000s
- ----

Same with NFS4 over TCP:
root@lab1# time mv file1 file2
real	0m0.005s
user	0m0.000s
sys	0m0.000s

Any special mount options or export options you might be using?

Regards,
/peter

Am 22.09.2015 um 15:19 schrieb Vincent Lefevre:
> At my lab, with NFS accounts, some machines have been upgraded to 
> Debian 8, and I get regular hangs on these machines, while they 
> never occurred before on the same machine and still never occur on 
> a machine that is still under Debian 7.
> 
> This happens with one of my scripts, which does a lot of "mv", and 
> more often when the target has just been modified from another 
> machine.
> 
> A strace showed that it is the "rename" system call that hangs:
> 
> 52026      0.000208 execve("/bin/mv", ["mv", "-v", 
> "ssh/ssh-reconfigure", "/home/vlefevre/bin/ssh-reconfigu"...], [/* 
> 112 vars */]) = 0 [...] 52026      0.000097 
> rename("ssh/ssh-reconfigure",
> "/home/vlefevre/bin/ssh-reconfigure") = 0 52026     15.057725
> lseek(0, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek) 52026
> 0.000189 close(0)            = 0 52026 0.000175 close(1)
> = 0 52026      0.000093 munmap(0x7fd5a610b000, 4096) = 0 52026
> 0.000079 close(2) = 0 52026      0.000106 exit_group(0)       = ?
> 52026      0.000307 +++ exited with 0 +++
> 
> The NFS server isn't particularly loaded.
> 
> Has anyone seen a problem like that? Where does it come from? And 
> why does it affect only Debian 8 machines? A new bug?
> 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (MingW32)

iQIcBAEBAgAGBQJWAsuuAAoJEM+6Ng5pbtyZGhMP/2yC/CgWhb5+yvA5jb/pd3sW
FwNw1mvH+FQikKVvvPgy5AMO3d2rQafh/fXQlfZRnC0Hu3jXn99K60mV//88KqGm
N+axMi5octq8s2zhF/K0/axsiPHAhgo9ggdlcdXE0ekQvgw5gzXJ6hDutnaFTaun
gay6HK6Ujp0JuvCxUQzq8uiv5VTPJiT0ftyBAF02dFqO0yBgGhDAt80NUtMGRwZB
nuya8pD7EhZH7x0vabnv3935+X0V1bpQhyKHWAbzCMY4Uce2wt0wtR9FEJVOyj4v
gQ7WHULo53fYOLdKw5REhgtEDqIFDz6MRsOup28gv7sZBW7/Ps3IbOi06/ovuR3W
WWZhYys+F/5Nmt8tKD7XWZfYQ7TnUN35Z1PS0xCK8+cEUWxSbI2OftyVBfI+iRCn
W3n0N2cvxyaRfWGwLs5ROO7/g+5xeNH5WiziaNZa5QfFae8j2J8lUIyG8aSZ/Axe
n0/HfwfA5vp9IIPVSOgsw+xmjkzCtIOtL+2cWBu5nTixD301hFRSuZCbUsz5f0Sq
tDUDzyYKYsUXJ5+YvBktgZ1dKD6nR3TuJdRl70gPJo6RuCxhvfRqHy47qrSOw6qL
HfKUZYnlgOMV1ufXLTI9sTq/YcDmJzKD+LDJUlumF88le58w5bq8KeR8uusGySSF
wQjgvRlwVkzvDsxm9b0z
=CCwk
-----END PGP SIGNATURE-----

[toc] | [prev] | [next] | [standalone]

#158205 — Re: NFS rename sometimes hangs for 15 seconds after upgrade to Debian 8

On 2015-09-23 17:56:30 +0200, Peter Ludikovsky wrote:
> When mounted as NFS3 over UDP:
> - ----
> root@lab1# echo foo > file1
> root@lab1# cp file1 file2
> 
> root@lab2# cat file2
> foo
> 
> root@lab1# time mv file1 file2
> real	0m0.004s
> user	0m0.000s
> sys	0m0.000s
> - ----
> 
> Same with NFS4 over TCP:
> root@lab1# time mv file1 file2
> real	0m0.005s
> user	0m0.000s
> sys	0m0.000s
> 
> Any special mount options or export options you might be using?

Here this is NFS4 over TCP:

filer.lip.ens-lyon.fr:/export/home/vlefevre on /home/vlefevre type nfs4 (rw,nosuid,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=140.77.14.11,local_lock=none,addr=140.77.14.5)

Is there any other information I could get?

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

[toc] | [prev] | [next] | [standalone]

#158209 — Re: NFS rename sometimes hangs for 15 seconds after upgrade to Debian 8

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Am 24.09.2015 um 11:46 schrieb Vincent Lefevre:
> On 2015-09-23 17:56:30 +0200, Peter Ludikovsky wrote:
>> When mounted as NFS3 over UDP: - ---- root@lab1# echo foo >
>> file1 root@lab1# cp file1 file2
>> 
>> root@lab2# cat file2 foo
>> 
>> root@lab1# time mv file1 file2 real	0m0.004s user	0m0.000s sys 
>> 0m0.000s - ----
>> 
>> Same with NFS4 over TCP: root@lab1# time mv file1 file2 real 
>> 0m0.005s user	0m0.000s sys	0m0.000s
>> 
>> Any special mount options or export options you might be using?
> 
> Here this is NFS4 over TCP:
> 
> filer.lip.ens-lyon.fr:/export/home/vlefevre on /home/vlefevre type
>  nfs4 
> (rw,nosuid,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=140.77.14.11,local_lock=none,addr=140.77.14.5)
>
>
>
> 
Is there any other information I could get?
> 

The most obvious thing: remove the public visible IPs and DNS names.

Other than that, what mount command did you use? Are you mounting the
share yourself, or is this an fstab or autofs mount? Do you have any
control over the NFS server, and if so what are its settings?

I did manage to replicate your issue. However, it disappears as soon
as I sync before the mv, as well as when both the rsize and wsize
parameters are reduced to 65536, suggesting this has something to do
with a local cache and/or lock.

Apparently the issue also disappears as soon as both sides have noac
added to the mount options.

Regards,
/peter
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (MingW32)

iQIcBAEBAgAGBQJWA+6pAAoJEM+6Ng5pbtyZvjEQALFrhCJHmF/JYmAvXncmbBdz
PYvSHTbAN9vNzCc4vqzkf4CmhgIVNSrP0WsPE/AHo2BNgpQQIQzfiiTgLGRBJnqT
jDctQE/bJYMhk1aw08b/c2l6a7Ykc9R1rXu516ftrBKe9f5fcxFLndjIrtgGP01K
3Tah6v8epDd2t3MDzhABZdn4QB5oWesgcnISJOBsrds8+vfYOKjEYiCNGm7wlduV
8968HXrCWJxNrkyhMmHTpPopFxzk1LZD270zxkUiidA4kDxjcADSCJhP6+Bivv1H
h+QhzNZ8NUEYBam+PuL3LhlBFalgvEZGfRF8gULEbjX7umA4b2tLWhgQAraYmiDO
arfCrUPkcqv4oB9mldrL1PfhKLzMtz4bQfg3/8u7gBBWnpkFq7JhE9CMQ7oddBBQ
rosT9sXsCRneYXfINJ57czof2kca3+5p/nZB7wgANKwD0nFWhFObduZJRnxqNXes
jdKNRrGg4oy0HJAMWXUvr/0Oy5iqoWwtGNhx4FHr2W4Lgie8g+YOciLsInsdytCu
x5x2EHcIOpVUBkya+xHxD02kk2loWKNh/sIV0RdgKabJ9YhY7srxh6uiPm/SNnvJ
67mpH7ekg5FOFc6njndeq1nXBzp/CX29GM8AnC5no5/Zmqf1qLiXxlTJFvYzZpNS
g4oRynIeq5Zd7sD2JoaW
=FojV
-----END PGP SIGNATURE-----

[toc] | [prev] | [next] | [standalone]

#158220 — Re: NFS rename sometimes hangs for 15 seconds after upgrade to Debian 8

On 2015-09-24 14:38:01 +0200, Peter Ludikovsky wrote:
> Other than that, what mount command did you use? Are you mounting the
> share yourself, or is this an fstab or autofs mount?

autofs

The only option used in /etc/auto.master is the obvious -o nosuid.

But the options seem to be the same on Debian 7 clients.

> Do you have any control over the NFS server, and if so what are its
> settings?

I could ask the admin, if they are useful for the bug.

> I did manage to replicate your issue. However, it disappears as soon
> as I sync before the mv, as well as when both the rsize and wsize
> parameters are reduced to 65536, suggesting this has something to do
> with a local cache and/or lock.
> 
> Apparently the issue also disappears as soon as both sides have noac
> added to the mount options.

Did you have to change the server settings to replicate the problem?

Note that if I do a "touch" on the target before the mv, the problem
doesn't occur.

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

[toc] | [prev] | [next] | [standalone]

#158221 — Re: NFS rename sometimes hangs for 15 seconds after upgrade to Debian 8

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



Am 25.09.2015 um 09:32 schrieb Vincent Lefevre:
> On 2015-09-24 14:38:01 +0200, Peter Ludikovsky wrote:
>> Other than that, what mount command did you use? Are you mounting
>> the share yourself, or is this an fstab or autofs mount?
> 
> autofs
> 
> The only option used in /etc/auto.master is the obvious -o nosuid.
> 
> But the options seem to be the same on Debian 7 clients.
> 
>> Do you have any control over the NFS server, and if so what are
>> its settings?
> 
> I could ask the admin, if they are useful for the bug.
> 
>> I did manage to replicate your issue. However, it disappears as
>> soon as I sync before the mv, as well as when both the rsize and
>> wsize parameters are reduced to 65536, suggesting this has
>> something to do with a local cache and/or lock.
>> 
>> Apparently the issue also disappears as soon as both sides have
>> noac added to the mount options.
> 
> Did you have to change the server settings to replicate the
> problem?
> 
> Note that if I do a "touch" on the target before the mv, the
> problem doesn't occur.
> 

No, I didn't have to change anything on the server side. noac is a
mount setting for NFS, which basically tells the client not to cache
any file/directory attributes, but to ask the server each time.

My guess is:
Due to the rather large wsize/rsize, the clients create a rather large
attribute cache. As a result, when you cat a file on the second
machine it updates the atime in the cache, but doesn't yet transfer
that information to the server. When you do the mv on the first
machine, the client tries to get the current attributes on the files
involved, which prompts the server to wait until the second client
runs into some kind of timeout and updates the attributes. Only then
is the second file "unlocked" to be overwritten.

Further proof of that: when setting noatime as a mount option, instead
of noac, there's also no delay on the mv.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (MingW32)

iQIcBAEBAgAGBQJWBQGeAAoJEM+6Ng5pbtyZel8QAItKpN7wTP527aF06YGYymwQ
Ema5V6cQ8dfYHT6dr2YKqCdG+I9TZeg4xeX5i6b8jtrbKqTHuOFK95SfN/dt3FTb
6clEgXndEZP022Mko7C7IGVBsQQIgnc57POywsglcBW3Vkq7FhBualg10S+KnhUw
XFeHhW9SFNwY6sjM+r77rvzilMjEnbry/ik8gSzEB9Y/Zak92DzHFVB/mQshgZKG
uDuwI3ZY9pRsW8QfxfX4/Ueo6IwtrP3K9eBObifbpiuFzfj8h13+hA4rCp+JYfhb
+2lvlngzQH6pArk/cXt/7RkIi7Rr4Baxyn4ZR5viO2HcDBtFszsvWJkwHMl6JO5v
7YeNL0ZuVQiHXZ5XjNKiVnMuyu0cBuzgYA/vOqwkMDbJj5GRZ9g2Mu3BIxHjCI5l
xqGJ2Lj3RJpzk5K0y0BYurxxEA2Fr4L2r3OqrChHXzYdU/+sCJ1FRvU+tHK++Vdn
CgGxnrWn7nzBo3l1MCmj1KeZgEVzCkBJhJ66AWlNL4EUYEDYowolaNOur1dsT9Ci
dxYxWru8YyifEW3zNlBkBJXqu4s2YkTY5J7pS4apYX4jvhrKpFkqtlu/gvTYzLrc
PU9W/12VsmBQqLhfIo/1We0iRX/wHNyMSHgI/vvWsuwtlgjcM4ECwNK/M96WLncr
0CuIVjGKpxXxn6FOC4Ey
=DNAx
-----END PGP SIGNATURE-----

[toc] | [prev] | [next] | [standalone]

#158230 — Re: NFS rename sometimes hangs for 15 seconds after upgrade to Debian 8

On 2015-09-25 10:11:10 +0200, Peter Ludikovsky wrote:
> My guess is:
> Due to the rather large wsize/rsize, the clients create a rather large
> attribute cache. As a result, when you cat a file on the second
> machine it updates the atime in the cache, but doesn't yet transfer
> that information to the server. When you do the mv on the first
> machine, the client tries to get the current attributes on the files
> involved, which prompts the server to wait until the second client
> runs into some kind of timeout and updates the attributes. Only then
> is the second file "unlocked" to be overwritten.
> 
> Further proof of that: when setting noatime as a mount option, instead
> of noac, there's also no delay on the mv.

Thanks for the explanations. But the second client, though it doesn't
update the attributes immediately, seems to do it quickly when asked
by the server, otherwise the problem would also be visible on Debian 7
machines. I think that on Debian 8, when the rename cannot be done
immediately, the client retries after 15 seconds instead of retrying
earlier.

Is this related to the following patch?

http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=8478eaa16e701ecfe054b62ec764bc1291b79e19

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

[toc] | [prev] | [next] | [standalone]

#158379 — Re: NFS rename sometimes hangs for 15 seconds after upgrade to Debian 8

On 2015-09-28 11:30:27 +0200, Peter Ludikovsky wrote:
> Maybe attach that information to your bug report as a point for
> investigation.

Thanks. I have pasted the message contents and added a link to it
(in the mailing-list archives).

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web