Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #59522 > unrolled thread

Bug#883217: linux: open on NFSv4 exported file on nfs server: "Resource temporarily unavailable" under reproducible conditions when client has granted read delegation on file

Back to article view | Back to linux.debian.kernel

Contents

  Bug#883217: linux: open on NFSv4 exported file on nfs server: "Resource temporarily unavailable" under reproducible conditions when client has granted read delegation on file Salvatore Bonaccorso <carnil@debian.org> - 2017-11-30 21:50 +0100
    Bug#883217: linux: open on NFSv4 exported file on nfs server: "Resource temporarily unavailable" under reproducible conditions when client has granted read delegation on file Stephen Dowdy <sdowdy@ucar.edu> - 2017-11-30 23:40 +0100
      Bug#855632: Bug#883217: linux: open on NFSv4 exported file on nfs server: "Resource temporarily unavailable" under reproducible conditions when client has granted read delegation on file Salvatore Bonaccorso <carnil@debian.org> - 2017-12-04 21:30 +0100
        Bug#855632: Bug#883217: linux: open on NFSv4 exported file on nfs server: "Resource temporarily unavailable" under reproducible conditions when client has granted read delegation on file Salvatore Bonaccorso <carnil@debian.org> - 2017-12-14 21:00 +0100
          Bug#855632: Bug#883217: linux: open on NFSv4 exported file on nfs server: "Resource temporarily unavailable" under reproducible conditions when client has granted read delegation on file Stephen Dowdy <sdowdy@ucar.edu> - 2017-12-15 02:00 +0100
            Bug#855632: Bug#883217: linux: open on NFSv4 exported file on nfs server: "Resource temporarily unavailable" under reproducible conditions when client has granted read delegation on file Salvatore Bonaccorso <carnil@debian.org> - 2017-12-15 09:40 +0100
            Bug#855632: Info received (Bug#883217: linux: open on NFSv4 exported file on nfs server: "Resource temporarily unavailable" under reproducible conditions when client has granted read delegation on file) Stephen Dowdy <sdowdy@ucar.edu> - 2018-07-05 22:10 +0200

#59522 — Bug#883217: linux: open on NFSv4 exported file on nfs server: "Resource temporarily unavailable" under reproducible conditions when client has granted read delegation on file

Source: linux
Version: 3.16.7-ckt25-2
Severity: important
Tags: upstream
Control: fixed -1 3.18-1~exp1

[the fixed version is assuming the bisect result is right]

Hi

This was originally reported in the debian-user mailinglist at [1],
followup [2] and [3] and it might be actually a duplicate of #855632.
It sounds very simliar, so we might merge the bugs if turns to be the
case.

On a NFSv4 exported filesystem, one might see regularly "Resource
temporarily unavailable" when trying to open a file for which a client
has been granted read delegation on the file. In my case it was noted
while regularly clients access and execute a script on such export and
on server side the script needed to be updated. But simlilary as well
as in #855632's case where the file was read by a client, whilest from
another system scp'ing a new version to the server.

Steps to reproduce with minimalized customized settings (at least
tried):

root@nfs-test:~# apt-get install nfs-kernel-server
root@nfs-test:~# mkdir -p /srv/test
root@nfs-test:~# echo '/srv/test *' >> /etc/exports
root@nfs-test:~# systemctl restart nfs-kernel-server.service
root@nfs-test:~# mount localhost:/srv/test /mnt
root@nfs-test:~# grep '/mnt' /proc/mounts
localhost:/srv/test /mnt nfs4 rw,relatime,vers=4.0,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp6,port=0,timeo=600,retrans=2,sec=sys,clientaddr=::1,local_lock=none,addr=::1 0 0
root@nfs-test:~#

on one terminal
root@nfs-test:~# while : ; do date >/srv/test/foo ; sleep 1 ; done

and on a second
root@nfs-test:~# while : ; do cat /mnt/foo ; sleep 1 ; done

within a couple of minutes

root@nfs-test:~# mount localhost:/srv/test /mnt
root@nfs-test:~# while : ; do date >/srv/test/foo ; sleep 1 ; done
-bash: /srv/test/foo: Resource temporarily unavailable

I tried to bisect, although in the "fixed" case it might not have been
correct that it's actually fixed, so this is more empirical result,
whereas when running a 'broken' kernel, that occured within a couple
of minutes.

----cut---------cut---------cut---------cut---------cut---------cut-----
git bisect start '--term-old' 'broken' '--term-new' 'fixed'
# broken: [754c780953397dd5ee5191b7b3ca67e09088ce7a] Merge branch 'for-v3.18' of git://git.linaro.org/people/mszyprowski/linux-dma-mapping
git bisect broken 754c780953397dd5ee5191b7b3ca67e09088ce7a
# fixed: [2d65a9f48fcdf7866aab6457bc707ca233e0c791] Merge branch 'drm-next' of git://people.freedesktop.org/~airlied/linux
git bisect fixed 2d65a9f48fcdf7866aab6457bc707ca233e0c791
# fixed: [5e40d331bd72447197f26525f21711c4a265b6a6] Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/jmorris/linux-security
git bisect fixed 5e40d331bd72447197f26525f21711c4a265b6a6
# broken: [a2ce35273c2f1aa0dcddd8822681d64ee5f31852] Merge tag 'sound-3.18-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound
git bisect broken a2ce35273c2f1aa0dcddd8822681d64ee5f31852
# fixed: [81ae31d78239318610d7c2acb3e2610d622a5aa4] Merge tag 'stable/for-linus-3.18-rc0-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip
git bisect fixed 81ae31d78239318610d7c2acb3e2610d622a5aa4
# broken: [ac0c49396d5ed9a33f08ce661635ac1bff80bb4f] Merge branch 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-fs
git bisect broken ac0c49396d5ed9a33f08ce661635ac1bff80bb4f
# broken: [e6c4efd87ab04e5ead363f24e6ac35ed3506d401] btrfs: Fix and enhance merge_extent_mapping() to insert best fitted extent map
git bisect broken e6c4efd87ab04e5ead363f24e6ac35ed3506d401
# broken: [90d0c376f5ee1927327b267faf15bf970476f09e] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs
git bisect broken 90d0c376f5ee1927327b267faf15bf970476f09e
# fixed: [6e129d00689c4d75253d1d428e82047b0aef5891] locks: flock_make_lock should return a struct file_lock (or PTR_ERR)
git bisect fixed 6e129d00689c4d75253d1d428e82047b0aef5891
# broken: [bfe8602436c803c6d5e271d52cd985d491a7470a] locks: close potential race in lease_get_mtime
git bisect broken bfe8602436c803c6d5e271d52cd985d491a7470a
# broken: [1c7dd2ff430fa14b45c9def54468e3a25ab8342b] locks: define a lm_setup handler for leases
git bisect broken 1c7dd2ff430fa14b45c9def54468e3a25ab8342b
# broken: [843c6b2f4cef384af8e0de6b7ac7191675030e3a] locks: remove i_have_this_lease check from __break_lease
git bisect broken 843c6b2f4cef384af8e0de6b7ac7191675030e3a
# fixed: [4d01b7f5e7576858b71cbaa72b541e17a229cb91] locks: give lm_break a return value
git bisect fixed 4d01b7f5e7576858b71cbaa72b541e17a229cb91
# fixed: [03d12ddf845a4eb874ffa558d65a548aee9b715b] locks: __break_lease cleanup in preparation of allowing direct removal of leases
git bisect fixed 03d12ddf845a4eb874ffa558d65a548aee9b715b
# first fixed commit: [03d12ddf845a4eb874ffa558d65a548aee9b715b] locks: __break_lease cleanup in preparation of allowing direct removal of leases
----cut---------cut---------cut---------cut---------cut---------cut-----

The above commit leads to 

https://git.kernel.org/linus/03d12ddf845a4eb874ffa558d65a548aee9b715b
as the first fixing commit, but it might be here as well that more of
the previous commits are needed.

Is this worth trying to be fixed for the jessie kernel?

Regards,
Salvatore

 [1] https://lists.debian.org/debian-user/2016/08/msg00022.html
 [2] https://lists.debian.org/debian-user/2016/08/msg00192.html
 [3] https://lists.debian.org/debian-user/2017/11/msg00920.html

[toc] | [next] | [standalone]

#59525

On 11/30/2017 01:39 PM, Salvatore Bonaccorso wrote:
> Is this worth trying to be fixed for the jessie kernel?

Salvatore,

I believe this is likely the reason for my bug report:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=855632

as that system has thrown EAGAIN errors since i installed it in April, 2015.
It's a 10 NIC NFS server for the department, and often throws the error when i update files that are likely being read/open by client systems.
(it doesn't have a huge resource consumption load ever and i get that failure)

So, i vote yeah ;)

--stephen

-- 
Stephen Dowdy  -  Systems Administrator  -  NCAR/RAL
303.497.2869   -  sdowdy@ucar.edu        -  http://www.ral.ucar.edu/~sdowdy/

[toc] | [prev] | [next] | [standalone]

#59555 — Bug#855632: Bug#883217: linux: open on NFSv4 exported file on nfs server: "Resource temporarily unavailable" under reproducible conditions when client has granted read delegation on file

[Multipart message — attachments visible in raw view] — view raw

Hi

On Thu, Nov 30, 2017 at 03:35:40PM -0700, Stephen Dowdy wrote:
> On 11/30/2017 01:39 PM, Salvatore Bonaccorso wrote:
> > Is this worth trying to be fixed for the jessie kernel?
> 
> Salvatore,
> 
> I believe this is likely the reason for my bug report:
> 
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=855632
> 
> as that system has thrown EAGAIN errors since i installed it in April, 2015.
> It's a 10 NIC NFS server for the department, and often throws the error when i update files that are likely being read/open by client systems.
> (it doesn't have a huge resource consumption load ever and i get that failure)
> 
> So, i vote yeah ;)

Okay.

I tried to track that further down, and attached 

0001-locks-remove-i_have_this_lease-check-from-__break_le.patch
0002-locks-__break_lease-cleanup-in-preparation-of-allowi.patch

to be applied on top of the current jessie branch in git.

Attached are the two individual patches:

locks-remove-i_have_this_lease-check-from-__break_le.patch
locks-__break_lease-cleanup-in-preparation-of-allowi.patch

With these two patches applied I was not able to reproduce the problem
now for a while, whereas previously it was relatively fast
triggerable.

Can you confirm the issue would be addressed as well for you?
See the kernel-handbook for the simple-patching guideline:
https://kernel-handbook.alioth.debian.org/ch-common-tasks.html#s4.2.2

Still since the patches were integrated in a bigger rewrite/touching
of fs/locks.c, fs/nfsd this might need a proper/deeper review if that
is complete and does not break anything.

Regards,
Salvatore

[toc] | [prev] | [next] | [standalone]

#59664 — Bug#855632: Bug#883217: linux: open on NFSv4 exported file on nfs server: "Resource temporarily unavailable" under reproducible conditions when client has granted read delegation on file

Hi Stephen,

On Mon, Dec 04, 2017 at 09:24:55PM +0100, Salvatore Bonaccorso wrote:
> Hi
> 
> On Thu, Nov 30, 2017 at 03:35:40PM -0700, Stephen Dowdy wrote:
> > On 11/30/2017 01:39 PM, Salvatore Bonaccorso wrote:
> > > Is this worth trying to be fixed for the jessie kernel?
> > 
> > Salvatore,
> > 
> > I believe this is likely the reason for my bug report:
> > 
> > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=855632
> > 
> > as that system has thrown EAGAIN errors since i installed it in April, 2015.
> > It's a 10 NIC NFS server for the department, and often throws the error when i update files that are likely being read/open by client systems.
> > (it doesn't have a huge resource consumption load ever and i get that failure)
> > 
> > So, i vote yeah ;)
> 
> Okay.

Did you got a chance to test this as well for your case of #855632?

Regards,
Salvatore

[toc] | [prev] | [next] | [standalone]

#59668 — Bug#855632: Bug#883217: linux: open on NFSv4 exported file on nfs server: "Resource temporarily unavailable" under reproducible conditions when client has granted read delegation on file

On 12/14/2017 12:51 PM, Salvatore Bonaccorso wrote:
> Hi Stephen,
> 
> On Mon, Dec 04, 2017 at 09:24:55PM +0100, Salvatore Bonaccorso wrote:
>> Hi
>>
>> On Thu, Nov 30, 2017 at 03:35:40PM -0700, Stephen Dowdy wrote:
>>> On 11/30/2017 01:39 PM, Salvatore Bonaccorso wrote:
>>>> Is this worth trying to be fixed for the jessie kernel?
>>>
>>> Salvatore,
>>>
>>> I believe this is likely the reason for my bug report:
>>>
>>> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=855632
>>>
>>> as that system has thrown EAGAIN errors since i installed it in April, 2015.
>>> It's a 10 NIC NFS server for the department, and often throws the error when i update files that are likely being read/open by client systems.
>>> (it doesn't have a huge resource consumption load ever and i get that failure)
>>>
>>> So, i vote yeah ;)
>>
>> Okay.
> 
> Did you got a chance to test this as well for your case of #855632?
> 
> Regards,
> Salvatore
> 
Salvatore,


Sorry i didn't respond.  things have been way crazy.  Unfortunately, i probably won't be able to test because:
   - problem is not reproducible easily sometimes
   - this machine services several hundred systems w/o any upcoming scheduled downtime.

I haven't noticed the problem on any other machines we have, though, so don't have any other candidates for testing.

I may just take the "upgrade to stretch" solution out of this when i have some scheduled downtime.

thanks,
--stephen

[toc] | [prev] | [next] | [standalone]

#59671 — Bug#855632: Bug#883217: linux: open on NFSv4 exported file on nfs server: "Resource temporarily unavailable" under reproducible conditions when client has granted read delegation on file

Hi Stephen,

On Thu, Dec 14, 2017 at 05:51:12PM -0700, Stephen Dowdy wrote:
> 
> 
> On 12/14/2017 12:51 PM, Salvatore Bonaccorso wrote:
> > Hi Stephen,
> > 
> > On Mon, Dec 04, 2017 at 09:24:55PM +0100, Salvatore Bonaccorso wrote:
> >> Hi
> >>
> >> On Thu, Nov 30, 2017 at 03:35:40PM -0700, Stephen Dowdy wrote:
> >>> On 11/30/2017 01:39 PM, Salvatore Bonaccorso wrote:
> >>>> Is this worth trying to be fixed for the jessie kernel?
> >>>
> >>> Salvatore,
> >>>
> >>> I believe this is likely the reason for my bug report:
> >>>
> >>> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=855632
> >>>
> >>> as that system has thrown EAGAIN errors since i installed it in April, 2015.
> >>> It's a 10 NIC NFS server for the department, and often throws the error when i update files that are likely being read/open by client systems.
> >>> (it doesn't have a huge resource consumption load ever and i get that failure)
> >>>
> >>> So, i vote yeah ;)
> >>
> >> Okay.
> > 
> > Did you got a chance to test this as well for your case of #855632?
> > 
> > Regards,
> > Salvatore
> > 
> Salvatore,
> 
> 
> Sorry i didn't respond.  things have been way crazy.  Unfortunately, i probably won't be able to test because:
>    - problem is not reproducible easily sometimes
>    - this machine services several hundred systems w/o any upcoming scheduled downtime.
> 
> I haven't noticed the problem on any other machines we have, though, so don't have any other candidates for testing.

Many thanks for the reply now, is much appreciated to see were we
stand. Yes I can perfectly understand the reasoning. the change is now
pending for 3.16.51-4 (or any later interation via a point release of
jessie), so if it happens to you to not have updated to stretch yet
and see your issue resolved as well we can close the second bug.

Regards,
Salvatore

[toc] | [prev] | [next] | [standalone]

#61354 — Bug#855632: Info received (Bug#883217: linux: open on NFSv4 exported file on nfs server: "Resource temporarily unavailable" under reproducible conditions when client has granted read delegation on file)

On 12/14/2017 05:54 PM, Debian Bug Tracking System wrote:
> Thank you for the additional information you have supplied regarding
> this Bug report.
> 
> This is an automatically generated reply to let you know your message
> has been received.
> 
> Your message is being forwarded to the package maintainers and other
> interested parties for their attention; they will reply in due course.
> 
> Your message has been sent to the package maintainer(s):
>   Debian Kernel Team <debian-kernel@lists.debian.org>
> 
> If you wish to submit further information on this problem, please
> send it to 855632@bugs.debian.org.
> 
> Please do not send mail to owner@bugs.debian.org unless you wish
> to report a problem with the Bug-tracking system.
> 


Salvatore,

Sorry i haven't gotten back, but just wanted to let you know that
at this point, i haven't seen the "resource temporarily unavailable" issues
on this server anymore.  It's presently running:

gizmo:/var/log# uname -a
Linux gizmo 3.16.0-6-amd64 #1 SMP Debian 3.16.56-1+deb8u1 (2018-05-08) x86_64 GNU/Linux

so, i think you can close this side of the bug.

Thanks for the feedback/follow-through!

--stephen

-- 
Stephen Dowdy  -  Systems Administrator  -  NCAR/RAL
303.497.2869   -  sdowdy@ucar.edu        -  http://www.ral.ucar.edu/~sdowdy/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.kernel

csiph-web