Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #85341 > unrolled thread

Bug#1049873: regression: linux-image-6.1.0-10-amd64: NFS4.1/pNFS mv hangs, but finishes after Ctrl-C

Back to article view | Back to linux.debian.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Bug#1049873: regression: linux-image-6.1.0-10-amd64: NFS4.1/pNFS mv hangs, but finishes after Ctrl-C Salvatore Bonaccorso <carnil@debian.org> - 2025-02-01 15:00 +0100
    Processed: Re: Bug#1049873: regression: linux-image-6.1.0-10-amd64:  NFS4.1/pNFS mv hangs, but finishes after Ctrl-C "Debian Bug Tracking System" <owner@bugs.debian.org> - 2025-02-01 15:00 +0100
    Bug#1049873: regression: linux-image-6.1.0-10-amd64: NFS4.1/pNFS mv hangs, but finishes after Ctrl-C Christoph Anton Mitterer <calestyo@scientia.org> - 2025-02-04 03:30 +0100
    Bug#1049873: regression: linux-image-6.1.0-10-amd64: NFS4.1/pNFS mv hangs, but finishes after Ctrl-C Christoph Anton Mitterer <calestyo@scientia.org> - 2025-02-04 03:30 +0100
      Bug#1049873: regression: linux-image-6.1.0-10-amd64: NFS4.1/pNFS mv hangs, but finishes after Ctrl-C Salvatore Bonaccorso <carnil@debian.org> - 2025-02-07 14:30 +0100
        Bug#1049873: regression: linux-image-6.1.0-10-amd64: NFS4.1/pNFS mv hangs, but finishes after Ctrl-C Christoph Anton Mitterer <calestyo@scientia.org> - 2025-02-10 20:50 +0100
          Bug#1049873: regression: linux-image-6.1.0-10-amd64: NFS4.1/pNFS mv hangs, but finishes after Ctrl-C Christoph Anton Mitterer <calestyo@scientia.org> - 2025-02-10 21:00 +0100

#85341 — Bug#1049873: regression: linux-image-6.1.0-10-amd64: NFS4.1/pNFS mv hangs, but finishes after Ctrl-C

Control: tags -1 + moreinfo

Hi Christoph,

On Wed, Aug 16, 2023 at 02:26:43PM +0200, Christoph Anton Mitterer wrote:
> Package: src:linux
> Version: 6.1.38-2
> Severity: normal
> 
> 
> Hey.
> 
> I'm seeing the following problem since upgrading from Debian bullseye
> to
> bookworm:
> 
> We run a Tier-2 for the LHC Computing Grid, where dCache is used as
> storage
> software.
> dCache in turn provides a NFS 4.1 / pNFS server.
> This means in specific, that there is one NFS "door" server and pool
> servers
> (which contain the actual data). The NFS client (from the Linux kernel)
> connects
> to the door, but when actual files are read/written the connection goes
> to one
> of the pools.
> 
> Now what fails is, when I try to mv a file on the NFS mountpoint.
> In specific:
> - the mv process seems to simply freeze
> - while it's frozen, when listing the directory, the file has still the
> old name
> - when I then Ctrl-C the mv it exits
> - when now listing the directory, the file has the new name
> 
> This worked properly with at least up to the 5.10.179-3 kerne from
> bullseye.
> 
> I should also note, that in the case where it fails, the server (the
> door) runs
> on the same host from where I also run the client... and that any
> loopback
> traffic is generally whitelisted for netfilter. Further, the pools are
> in the
> same subnet, and again any traffic within that subnet is whitelisted on
> all
> servers.
> 
> /etc/exports (which dCache uses as well) has:
>   / localhost(rw,no_root_squash,secure)
>   /pnfs localhost(rw,no_root_squash,secure)
> (with the /pnfs mountpoint being the one that's used)
> 
> 
> Next I tried the same from my laptop's Debian sid (kernel 6.4.4-3) from
> outside
> the subnet (but allowing NFS for my particular IP):
> There it also works.
> 
> 
> When mounting, kernel log shows:
> [Aug12 15:15] FS-Cache: Loaded
> [  +0,033084] RPC: Registered named UNIX socket transport module.
> [  +0,000005] RPC: Registered udp transport module.
> [  +0,000001] RPC: Registered tcp transport module.
> [  +0,000000] RPC: Registered tcp NFSv4.1 backchannel transport module.
> [  +0,136237] Key type dns_resolver registered
> [  +0,113564] NFS: Registering the id_resolver key type
> [  +0,000012] Key type id_resolver registered
> [  +0,000001] Key type id_legacy registered
> [Aug12 15:17] nfs4filelayout_init: NFSv4 File Layout Driver
> Registering...
> 
> but that's the same on both nodes (apart from times of course).
> 
> 
> Any ideas?

While looking at some NFS related bugs I noticed this one which was
unaswered, but reported against an old 6.1.y version.

I'm closing the bug in the sense of BTS housekeeping, but please do
the following: If you are able to reproduce the problem with a current
6.1.y version, then please reopen the bug and do remove the moreinfo
tag if you have an indepented reproducer of dCache, in which case it
might be considered a upstream problem, otherwise I would suggest you
first approach the dCache developers (it still could be a kernel
problem as dCache from a quick look is plain in userpace components?).

Please do attach the full boot log, after having triggered the
problem.

If you can, try please as well a more recent version ideally the one
from unstable to verify the problem is still present there.

Thanks for your understanding,

Regards,
Salvatore

[toc] | [next] | [standalone]

#85342 — Processed: Re: Bug#1049873: regression: linux-image-6.1.0-10-amd64: NFS4.1/pNFS mv hangs, but finishes after Ctrl-C

Processing control commands:

> tags -1 + moreinfo
Bug #1049873 [src:linux] regression: linux-image-6.1.0-10-amd64: NFS4.1/pNFS mv hangs, but finishes after Ctrl-C
Added tag(s) moreinfo.

-- 
1049873: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1049873
Debian Bug Tracking System
Contact owner@bugs.debian.org with problems

[toc] | [prev] | [next] | [standalone]

#85372

On Tue, 2025-02-04 at 03:12 +0100, Christoph Anton Mitterer wrote:
> When I just retried I also noticed that after Ctrl-C-ing the hanging
> mv
> it seems that dest file is kept, and the src file is gone (which I'd
> consider as data loss, caused by this issue).

Tried that several more times and couldn't reproduce this anymore. Now
it always seems to be the src file that remains after the mv (with the
new name).

[toc] | [prev] | [next] | [standalone]

#85373

[Multipart message — attachments visible in raw view] — view raw

Hey Salvatore.

On Sat, 2025-02-01 at 14:52 +0100, Salvatore Bonaccorso wrote:
> While looking at some NFS related bugs I noticed this one which was
> unaswered, but reported against an old 6.1.y version.

It still happens with 6.1.119-1.


> 6.1.y version, then please reopen the bug and do remove the moreinfo
> tag if you have an indepented reproducer of dCache, in which case it
> might be considered a upstream problem, otherwise I would suggest you
> first approach the dCache developers (it still could be a kernel
> problem as dCache from a quick look is plain in userpace
> components?).

I rather doubt it's a dCache bug, or well, at least it used to work in
older kernel versions as mentioned in the original report.
Also, as you say, dCache is purely in userspace.


> Please do attach the full boot log, after having triggered the
> problem.

There are no additional messages after the condition has been
triggered.
But I've attached the whole dmesg log.


> If you can, try please as well a more recent version ideally the one
> from unstable to verify the problem is still present there.

That's a bit difficult, as these are production nodes in use for WLCG,
I could only try doing so when we schedule the next downtime.


When I just retried I also noticed that after Ctrl-C-ing the hanging mv
it seems that dest file is kept, and the src file is gone (which I'd
consider as data loss, caused by this issue).


Cheers,
Chris.

[toc] | [prev] | [next] | [standalone]

#85414

Hi Christoph,

On Tue, Feb 04, 2025 at 03:12:42AM +0100, Christoph Anton Mitterer wrote:
> Hey Salvatore.
> 
> On Sat, 2025-02-01 at 14:52 +0100, Salvatore Bonaccorso wrote:
> > While looking at some NFS related bugs I noticed this one which was
> > unaswered, but reported against an old 6.1.y version.
> 
> It still happens with 6.1.119-1.

Ok. reopening the bug for now.

> 
> > 6.1.y version, then please reopen the bug and do remove the moreinfo
> > tag if you have an indepented reproducer of dCache, in which case it
> > might be considered a upstream problem, otherwise I would suggest you
> > first approach the dCache developers (it still could be a kernel
> > problem as dCache from a quick look is plain in userpace
> > components?).
> 
> I rather doubt it's a dCache bug, or well, at least it used to work in
> older kernel versions as mentioned in the original report.
> Also, as you say, dCache is purely in userspace.

Sure, but it is still a very specialized usecase. As I understand
there is for instance the PoolManager which takes action when a user
performs a reading or writing operation on a file. And I noticed you
filled earlier a (maybe related) https://github.com/dCache/dcache/issues/6592
> 
> 
> > Please do attach the full boot log, after having triggered the
> > problem.
> 
> There are no additional messages after the condition has been
> triggered.
> But I've attached the whole dmesg log.

Thanks!

I still supect that the dCache develpers might need to help here to
see if they can reproduce in lab conditions to narrow down the
problem.

Regards,
Salvatore

[toc] | [prev] | [next] | [standalone]

#85472

Hey Salvatore

On Fri, 2025-02-07 at 14:25 +0100, Salvatore Bonaccorso wrote:
> Sure, but it is still a very specialized usecase. As I understand
> there is for instance the PoolManager which takes action when a user
> performs a reading or writing operation on a file.

Yes. Though I would not expect the PoolManager to take part in this, as
it's just a rename in the virtual filesystem provided by dCache
("Chimera").

Only the PnfsManager (which is the interface to that) should be queried
here.

Also, the deletion of the old destination file that got unlinked by the
rename, doesn't happen immediately in dCache but is only scheduled to
happen (and this again should happen without PoolManager).


The smoking gun, why this may rather be a kernel issue is:
a) it worked in older kernels
b) it works[0] (again) in current sid kernels


>  And I noticed you
> filled earlier a (maybe related)
> https://github.com/dCache/dcache/issues/6592

Yeah,... it was actually about the issue and back then I hoped to get
some feedback.

But still, given that it works again (i.e. no hang) with 6.12.13 (and
when I tested during my original report, I had 6.4.x)... I'd rather say
that it sounds like a regression introduced somewhere after 5.10.179-3
but at least in 6.1.x.


> I still supect that the dCache develpers might need to help here to
> see if they can reproduce in lab conditions to narrow down the
> problem.

I'll report the [0] bug in a few minutes, maybe that triggers some more
feedback, but not sure.


Also, with trixie looming ahead,... don't put too much effort into
this.
Maybe just keep it open in case someone stumbles over the same issue.


Cheers,
Chris.


[0] Well there is another bug showing up (this time most likely being
actually a dCache bug, i.e. the "new" file (after the move) shows the
size of the old one, while it actually has the content of the new.
Probably a caching issue in dCache's NFS door.

[toc] | [prev] | [next] | [standalone]

#85473

On Mon, 2025-02-10 at 20:44 +0100, Christoph Anton Mitterer wrote:
> [0] Well there is another bug showing up (this time most likely being
> actually a dCache bug, i.e. the "new" file (after the move) shows the
> size of the old one, while it actually has the content of the new.
> Probably a caching issue in dCache's NFS door.

Just for the records: I think my eyes must have tricked me there and
dCache did indeed update the size of the file.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.kernel

csiph-web