Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #80179 > unrolled thread

Bug#1050446: nfs over rdma between debian12 client and debian11 server can cause data corruption

Back to article view | Back to linux.debian.kernel

Contents

  Bug#1050446: nfs over rdma between debian12 client and debian11 server can cause data corruption Alois Schlögl <alois.schloegl@gmail.com> - 2023-08-24 19:40 +0200
    Bug#1050446: nfs over rdma between debian12 client and debian11 server can cause data corruption Alois Schlögl <alois.schloegl@gmail.com> - 2023-11-09 19:20 +0100

#80179 — Bug#1050446: nfs over rdma between debian12 client and debian11 server can cause data corruption

Package: nfs-common,rdma-core

I've been testing the upgrade of a compute node from Debian11 to Debian12.
That node was connected through nfs with rdma protocol to a zfs-storage server running on Debian11.
The compute node and the storage server are part of a high-performance compute cluster, connected over infiniband.
Not sure whether this is important, but the storage server is using zfs.

After the upgrade of the compute node (node client) to Debian 12, this machine could not correctly read a few (small) files. The files were correctly shown with "ls", and the size matched as well.
However the content was corrupted (looked like random garbage). In one case the .ssh/authorized_keys was corrupted, in some other case the "version.lua" from the lmod system was affected, rendering lmod unusable.
Interestingly, only very few files seemed to be affected. Most files were correctly retrieved.

So this is a very subtle error, and not obvious.
When retrieving these files, no error was reported, but data of the expected size was retrieved.
Effectively, the retrieved data was corrupted, and could lead to potential data loss.

The compute node on Debian12 had

ii  libnfsidmap1:amd64                    1:2.6.2-4                               amd64        NFS idmapping library
ii  nfs-common                            1:2.6.2-4                               amd64        NFS support files common to client and server
ii  librdmacm1:amd64                      44.0-2                                  amd64        Library for managing RDMA connections
ii  rdma-core                             44.0-2                                  amd64        RDMA core userspace infrastructure and documentation
ii  rdmacm-utils                          44.0-2                                  amd64        Examples for the librdmacm library


The storage server on Debian11 had
ii  nfs-common                         1:1.3.4-6                      amd64        NFS support files common to client and server
ii  nfs-kernel-server                  1:1.3.4-6                      amd64        support for NFS kernel server
ii  librdmacm1:amd64                   33.2-1                         amd64        Library for managing RDMA connections


The problem went away, when changing nfs mount protocal from proto=rdma to proto=tcp.

I tried to learn about this incompatibility, but did not find any information.
I'm also curious whether an nfs 2.6 server would correctly talk to an nfs 1.3 client over rdma ?
Can anyone provide more information on that topic ?

[toc] | [next] | [standalone]

#80850

Severity: wishlist

We run more tests, and observed that pgrading the storage server to 
Debian12(/bookwork solves the issue.
The storage server can be accessed through rdma from debian11 and 12.

That means, the problem occurs only when the storage server is on 
debian11, the client is on debian11 mounted with proto=rdma.

This information is good enough for us, as we'll upgrade first the 
storage server.

Another workaround is to switch to proto=tcp

As this issue (proto=rdma from debian12 client to debian11 server) is 
difficult to fix and a workaround exists, I'll downgrade the severity.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.kernel

csiph-web