Bug#1050446: nfs over rdma between debian12 client and debian11 server can cause data corruption

Path	csiph.com!fu-berlin.de!bofh.it!news.nic.it!robomod
From	Alois Schlögl <alois.schloegl@gmail.com>
Newsgroups	linux.debian.bugs.dist, linux.debian.kernel
Subject	Bug#1050446: nfs over rdma between debian12 client and debian11 server can cause data corruption
Date	Thu, 24 Aug 2023 19:40:02 +0200
Message-ID	<H6mtk-47HC-9@gated-at.bofh.it> (permalink)
X-Original-To	submit@bugs.debian.org
X-Mailbox-Line	From debian-bugs-dist-request@lists.debian.org Thu Aug 24 17:36:15 2023
Old-Return-Path	<debbugs@buxtehude.debian.org>
X-Spam-Flag	NO
X-Spam-Score	-2.6
Reply-To	Alois Schlögl <alois.schloegl@gmail.com>, 1050446@bugs.debian.org
Resent-To	debian-bugs-dist@lists.debian.org
Resent-Cc	Debian kernel team <debian-kernel@lists.debian.org>, Benjamin Drung <bdrung@ubuntu.com>
X-Debian-Pr-Message	report 1050446
X-Debian-Pr-Package	nfs-common,rdma-core
X-Gm-Message-State	AOJu0YxU4H15aZ4ilr0ppAIClCrnM/85t1m9bDh9KjEQkGbeDYKYwX7J P3awC0tf2SlhGEFpfO24R6UbW8mEj/vsZw==
X-Google-SMTP-Source	AGHT+IFfrb0QJJlhdW/3yC/4O3nnNPlRtbqjYMt4SgNJbV/QBMxUyFftShmkfjmmfZzvzL1FXaICVg==
X-Received	by 2002:aa7:c414:0:b0:528:88e2:7fec with SMTP id j20-20020aa7c414000000b0052888e27fecmr12131215edq.2.1692898482783; Thu, 24 Aug 2023 10:34:42 -0700 (PDT)
MIME-Version	1.0
User-Agent	Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.14.0
Content-Language	en-US
Content-Type	text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding	7bit
X-Debian-Message	from BTS
X-Mailing-List	<debian-bugs-dist@lists.debian.org> archive/latest/1793637
List-ID	<debian-bugs-dist.lists.debian.org>
List-URL	<https://lists.debian.org/debian-bugs-dist/>
Approved	robomod@news.nic.it
Lines	35
Organization	linux.* mail to news gateway
Sender	robomod@news.nic.it
X-Original-Date	Thu, 24 Aug 2023 19:34:41 +0200
X-Original-Message-ID	<b9e3d214-be0c-c84a-3c25-4678dc34a4bd@gmail.com>
Xref	csiph.com linux.debian.bugs.dist:1165405 linux.debian.kernel:80179

Cross-posted to 2 groups.

Show key headers only | View raw

Package: nfs-common,rdma-core

I've been testing the upgrade of a compute node from Debian11 to Debian12.
That node was connected through nfs with rdma protocol to a zfs-storage server running on Debian11.
The compute node and the storage server are part of a high-performance compute cluster, connected over infiniband.
Not sure whether this is important, but the storage server is using zfs.

After the upgrade of the compute node (node client) to Debian 12, this machine could not correctly read a few (small) files. The files were correctly shown with "ls", and the size matched as well.
However the content was corrupted (looked like random garbage). In one case the .ssh/authorized_keys was corrupted, in some other case the "version.lua" from the lmod system was affected, rendering lmod unusable.
Interestingly, only very few files seemed to be affected. Most files were correctly retrieved.

So this is a very subtle error, and not obvious.
When retrieving these files, no error was reported, but data of the expected size was retrieved.
Effectively, the retrieved data was corrupted, and could lead to potential data loss.

The compute node on Debian12 had

ii libnfsidmap1:amd64 1:2.6.2-4 amd64 NFS idmapping library
ii nfs-common 1:2.6.2-4 amd64 NFS support files common to client and server
ii librdmacm1:amd64 44.0-2 amd64 Library for managing RDMA connections
ii rdma-core 44.0-2 amd64 RDMA core userspace infrastructure and documentation
ii rdmacm-utils 44.0-2 amd64 Examples for the librdmacm library

The storage server on Debian11 had
ii nfs-common 1:1.3.4-6 amd64 NFS support files common to client and server
ii nfs-kernel-server 1:1.3.4-6 amd64 support for NFS kernel server
ii librdmacm1:amd64 33.2-1 amd64 Library for managing RDMA connections

The problem went away, when changing nfs mount protocal from proto=rdma to proto=tcp.

I tried to learn about this incompatibility, but did not find any information.
I'm also curious whether an nfs 2.6 server would correctly talk to an nfs 1.3 client over rdma ?
Can anyone provide more information on that topic ?

Back to linux.debian.kernel | Previous | Next — Next in thread | Find similar | Unroll thread

Thread

Bug#1050446: nfs over rdma between debian12 client and debian11 server can cause data corruption Alois Schlögl <alois.schloegl@gmail.com> - 2023-08-24 19:40 +0200
  Bug#1050446: nfs over rdma between debian12 client and debian11 server can cause data corruption Alois Schlögl <alois.schloegl@gmail.com> - 2023-11-09 19:20 +0100

csiph-web