Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1716004 > unrolled thread

[Patch v2 00/19] CIFS: Implement SMBDirect

Back to article view | Back to linux.kernel

Contents

  [Patch v2 00/19] CIFS: Implement SMBDirect Long Li <longli@exchange.microsoft.com> - 2017-08-20 21:20 +0200
    [Patch v2 04/19] CIFS: SMBD: Add SMBDirect transport to SMB connection and Makefile Long Li <longli@exchange.microsoft.com> - 2017-08-20 21:20 +0200
    [Patch v2 06/19] CIFS: SMBD: Reconnect to SMBDirect session Long Li <longli@exchange.microsoft.com> - 2017-08-20 21:20 +0200
    [Patch v2 02/19] CIFS: SMBD: Add SMBDirect protocol and transport constants Long Li <longli@exchange.microsoft.com> - 2017-08-20 21:20 +0200
    RE: [Patch v2 00/19] CIFS: Implement SMBDirect "Steve Wise" <swise@opengridcomputing.com> - 2017-08-21 21:20 +0200
      RE: [Patch v2 00/19] CIFS: Implement SMBDirect "Steve Wise" <swise@opengridcomputing.com> - 2017-08-21 22:00 +0200
        RE: [Patch v2 00/19] CIFS: Implement SMBDirect Long Li <longli@microsoft.com> - 2017-08-21 22:30 +0200
      RE: [Patch v2 00/19] CIFS: Implement SMBDirect Long Li <longli@microsoft.com> - 2017-08-21 22:00 +0200

#1716004 — [Patch v2 00/19] CIFS: Implement SMBDirect

From: Long Li <longli@microsoft.com>

Starting with SMB2 dialect 3.0, Microsoft introduced SMBDirect transport protocol for transferring upper layer (SMB2) payload over RDMA via Infiniband, RoCE or iWARP. The prococol is published in [MS-SMBD] (https://msdn.microsoft.com/en-us/library/hh536346.aspx).

The patch v2 added RDMA read/write via memory registration, and addressed feedbacks on v1.

Long Li (19):
  CIFS: Add rdma mount option
  CIFS: SMBD: Add SMBDirect protocol and transport constants
  CIFS: SMBD: Implement SMBDirect transport
  CIFS: SMBD: Add SMBDirect transport to SMB connection and Makefile
  CIFS: SMBD: Connect to SMBDirect session
  CIFS: SMBD: Reconnect to SMBDirect session
  CIFS: SMBD: Destroy SMBDirect session on shutdown or umount
  CIFS: SMBD: Set SMBDirect maximum read or write size for I/O
  CIFS: SMBD: Read data from SMBDirect
  CIFS: SMBD: Send data through SMBDirect
  CIFS: SMBD: Define memory registration for I/O data
  CIFS: SMBD: Fix the definition for SMB2_CHANNEL_RDMA_V1_INVALIDATE
  CIFS: SMBD: Use registered memory RDMA read for SMB write
  CIFS: SMBD: Deregister memory when finishing SMB write
  CIFS: SMBD: Add parameter rdata to smb2_new_read_req
  CIFS: SMBD: Read correct returned data length for RDMA write (SMB
    READ) I/O
  CIFS: SMBD: Do not read from transport on registered memory RDMA write
    (SMB READ)
  CIFS: SMBD: Deregister memory when finishing SMB read
  CIFS: SMBD: Add SMBDirect debug counters

 fs/cifs/Makefile     |    2 +-
 fs/cifs/cifs_debug.c |   48 ++
 fs/cifs/cifsfs.c     |    2 +
 fs/cifs/cifsglob.h   |   17 +-
 fs/cifs/cifssmb.c    |    4 +-
 fs/cifs/connect.c    |   62 +-
 fs/cifs/file.c       |    5 +
 fs/cifs/smb1ops.c    |    2 +-
 fs/cifs/smb2ops.c    |   21 +-
 fs/cifs/smb2pdu.c    |  114 ++-
 fs/cifs/smb2pdu.h    |    2 +-
 fs/cifs/smbdirect.c  | 2328 ++++++++++++++++++++++++++++++++++++++++++++++++++
 fs/cifs/smbdirect.h  |  300 +++++++
 fs/cifs/transport.c  |    7 +
 14 files changed, 2895 insertions(+), 19 deletions(-)
 create mode 100644 fs/cifs/smbdirect.c
 create mode 100644 fs/cifs/smbdirect.h

-- 
2.7.4

[toc] | [next] | [standalone]

#1716005 — [Patch v2 04/19] CIFS: SMBD: Add SMBDirect transport to SMB connection and Makefile

From: Long Li <longli@microsoft.com>

Add SMBDirect as an optional connection to SMB session defined in CIFS. When connection is on SMBDirect, upper layer uses this connection to carry payloads.

With the transport hooked up, add SMBDirect code to Makefile.

Signed-off-by: Long Li <longli@microsoft.com>
---
 fs/cifs/Makefile   | 2 +-
 fs/cifs/cifsglob.h | 2 ++
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/fs/cifs/Makefile b/fs/cifs/Makefile
index eed7eb0..6bb9863 100644
--- a/fs/cifs/Makefile
+++ b/fs/cifs/Makefile
@@ -18,4 +18,4 @@ cifs-$(CONFIG_CIFS_DFS_UPCALL) += dns_resolve.o cifs_dfs_ref.o
 cifs-$(CONFIG_CIFS_FSCACHE) += fscache.o cache.o
 
 cifs-$(CONFIG_CIFS_SMB2) += smb2ops.o smb2maperror.o smb2transport.o \
-			    smb2misc.o smb2pdu.o smb2inode.o smb2file.o
+			    smb2misc.o smb2pdu.o smb2inode.o smb2file.o smbdirect.o
diff --git a/fs/cifs/cifsglob.h b/fs/cifs/cifsglob.h
index 703c2fb..dc5404d 100644
--- a/fs/cifs/cifsglob.h
+++ b/fs/cifs/cifsglob.h
@@ -652,6 +652,8 @@ struct TCP_Server_Info {
 	bool	large_buf;		/* is current buffer large? */
 	/* use SMBD connection instead of socket */
 	bool	rdma;
+	/* point to the SMBD connection if RDMA is used instead of socket */
+	struct smbd_connection *smbd_conn;
 	struct delayed_work	echo; /* echo ping workqueue job */
 	char	*smallbuf;	/* pointer to current "small" buffer */
 	char	*bigbuf;	/* pointer to current "big" buffer */
-- 
2.7.4

[toc] | [prev] | [next] | [standalone]

#1716006 — [Patch v2 06/19] CIFS: SMBD: Reconnect to SMBDirect session

From: Long Li <longli@microsoft.com>

Do a reconnect on SMBDirect when it is used as the connection. Reconnect can happen for many reasons and it's mostly the decision of upper layer SMB2 not SMBDirect.

Signed-off-by: Long Li <longli@microsoft.com>
---
 fs/cifs/connect.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/fs/cifs/connect.c b/fs/cifs/connect.c
index 309eba0..b337ca7 100644
--- a/fs/cifs/connect.c
+++ b/fs/cifs/connect.c
@@ -409,7 +409,11 @@ cifs_reconnect(struct TCP_Server_Info *server)
 
 		/* we should try only the port we connected to before */
 		mutex_lock(&server->srv_mutex);
-		rc = generic_ip_connect(server);
+		if (server->rdma)
+			rc = smbd_reconnect(server);
+		else
+			rc = generic_ip_connect(server);
+
 		if (rc) {
 			cifs_dbg(FYI, "reconnect error %d\n", rc);
 			mutex_unlock(&server->srv_mutex);
-- 
2.7.4

[toc] | [prev] | [next] | [standalone]

#1716007 — [Patch v2 02/19] CIFS: SMBD: Add SMBDirect protocol and transport constants

From: Long Li <longli@microsoft.com>

To prepare for protocol implementation, add constants and user-configurable values in the protocol.

Signed-off-by: Long Li <longli@microsoft.com>
---
 fs/cifs/smbdirect.c | 78 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 fs/cifs/smbdirect.h | 20 ++++++++++++++
 2 files changed, 98 insertions(+)
 create mode 100644 fs/cifs/smbdirect.c
 create mode 100644 fs/cifs/smbdirect.h

diff --git a/fs/cifs/smbdirect.c b/fs/cifs/smbdirect.c
new file mode 100644
index 0000000..d785bc1
--- /dev/null
+++ b/fs/cifs/smbdirect.c
@@ -0,0 +1,78 @@
+/*
+ *   Copyright (C) 2017, Microsoft Corporation.
+ *
+ *   Author(s): Long Li <longli@microsoft.com>
+ *
+ *   This program is free software;  you can redistribute it and/or modify
+ *   it under the terms of the GNU General Public License as published by
+ *   the Free Software Foundation; either version 2 of the License, or
+ *   (at your option) any later version.
+ *
+ *   This program is distributed in the hope that it will be useful,
+ *   but WITHOUT ANY WARRANTY;  without even the implied warranty of
+ *   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See
+ *   the GNU General Public License for more details.
+ */
+#include <linux/module.h>
+#include "smbdirect.h"
+#include "cifs_debug.h"
+
+/* SMBD version number */
+#define SMBD_V1	0x0100
+
+/* Port numbers for SMBD transport */
+#define SMB_PORT	445
+#define SMBD_PORT	5445
+
+/* Address lookup and resolve timeout in ms */
+#define RDMA_RESOLVE_TIMEOUT	5000
+
+/* SMBD negotiation timeout in seconds */
+#define SMBD_NEGOTIATE_TIMEOUT	120
+
+/* SMBD minimum receive size and fragmented sized defined in [MS-SMBD] */
+#define SMBD_MIN_RECEIVE_SIZE		128
+#define SMBD_MIN_FRAGMENTED_SIZE	131072
+
+/*
+ * Default maximum number of RDMA read/write outstanding on this connection
+ * This value is possibly decreased during QP creation on hardware limit
+ */
+#define SMBD_CM_RESPONDER_RESOURCES	32
+
+/* Maximum number of retries on data transfer operations */
+#define SMBD_CM_RETRY			6
+/* No need to retry on Receiver Not Ready since SMBD manages credits */
+#define SMBD_CM_RNR_RETRY		0
+
+/*
+ * User configurable initial values per SMBD transport connection
+ * as defined in [MS-SMBD] 3.1.1.1
+ * Those may change after a SMBD negotiation
+ */
+/* The local peer's maximum number of credits to grant to the peer */
+static int receive_credit_max = 255;
+/* The remote peer's credit request of local peer */
+static int send_credit_target = 255;
+/* The maximum single message size can be sent to remote peer */
+static int max_send_size = 1364;
+/*  The maximum fragmented upper-layer payload receive size supported */
+static int max_fragmented_recv_size = 1024 * 1024;
+/*  The maximum single-message size which can be received */
+static int max_receive_size = 8192;
+
+/* The timeout to initiate send of a keepalive message on idle */
+static int keep_alive_interval = 120;
+
+/*
+ * User configurable initial values for RDMA transport
+ * The actual values used may be lower and are limited to hardware capabilities
+ */
+/* Default maximum number of SGEs in a RDMA send/recv */
+static int max_send_sge = SMBDIRECT_MAX_SGE;
+static int max_recv_sge = SMBDIRECT_MAX_SGE;
+/* Default maximum number of SGEs in a RDMA write/read */
+static int max_frmr_depth = 2048;
+
+/* If payload is less than this byte, use RDMA send/recv not read/write */
+static int rdma_readwrite_threshold = 4096;
diff --git a/fs/cifs/smbdirect.h b/fs/cifs/smbdirect.h
new file mode 100644
index 0000000..06eeb0b
--- /dev/null
+++ b/fs/cifs/smbdirect.h
@@ -0,0 +1,20 @@
+/*
+ *   Copyright (C) 2017, Microsoft Corporation.
+ *
+ *   Author(s): Long Li <longli@microsoft.com>
+ *
+ *   This program is free software;  you can redistribute it and/or modify
+ *   it under the terms of the GNU General Public License as published by
+ *   the Free Software Foundation; either version 2 of the License, or
+ *   (at your option) any later version.
+ *
+ *   This program is distributed in the hope that it will be useful,
+ *   but WITHOUT ANY WARRANTY;  without even the implied warranty of
+ *   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See
+ *   the GNU General Public License for more details.
+ */
+#ifndef _SMBDIRECT_H
+#define _SMBDIRECT_H
+
+#define SMBDIRECT_MAX_SGE	16
+#endif
-- 
2.7.4

[toc] | [prev] | [next] | [standalone]

#1716788

> 
> From: Long Li <longli@microsoft.com>
> 
> Starting with SMB2 dialect 3.0, Microsoft introduced SMBDirect transport
protocol
> for transferring upper layer (SMB2) payload over RDMA via Infiniband, RoCE or
> iWARP. The prococol is published in [MS-SMBD] (https://msdn.microsoft.com/en-
> us/library/hh536346.aspx).
> 
> The patch v2 added RDMA read/write via memory registration, and addressed
> feedbacks on v1.
> 

Hey Long,

What testing have you done with this on the various rdma transports?  Does it
work over IB, RoCE, and iWARP providers?

Thanks,

Steve.

[toc] | [prev] | [next] | [standalone]

#1716825

> >
> > Hey Long,
> >
> > What testing have you done with this on the various rdma transports?  Does
> > it work over IB, RoCE, and iWARP providers?
> 
> Hi Steve,
> 
> Currently all the tests have been done over Infiniband. We haven't tested on
RoCE
> or iWARP, but planned to do it in the following weeks.
> 
> Long

Ok, good.

Is this series available on github or somewhere so we can clone it and review it
as it is applied to the kernel src?

Thanks,

Steve.

[toc] | [prev] | [next] | [standalone]

#1716840

> > > Hey Long,
> > >
> > > What testing have you done with this on the various rdma transports?
> > > Does it work over IB, RoCE, and iWARP providers?
> >
> > Hi Steve,
> >
> > Currently all the tests have been done over Infiniband. We haven't
> > tested on
> RoCE
> > or iWARP, but planned to do it in the following weeks.
> >
> > Long
> 
> Ok, good.
> 
> Is this series available on github or somewhere so we can clone it and review
> it as it is applied to the kernel src?

Unfortunately they are not on github. I will look into putting them there for review. Will update soon.

Thanks for helping out!

> 
> Thanks,
> 
> Steve.

[toc] | [prev] | [next] | [standalone]

#1716827

> -----Original Message-----
> From: Steve Wise [mailto:swise@opengridcomputing.com]
> Sent: Monday, August 21, 2017 12:15 PM
> To: Long Li <longli@microsoft.com>; 'Steve French' <sfrench@samba.org>;
> linux-cifs@vger.kernel.org; samba-technical@lists.samba.org; linux-
> kernel@vger.kernel.org; linux-rdma@vger.kernel.org; 'Christoph Hellwig'
> <hch@infradead.org>; Tom Talpey <ttalpey@microsoft.com>; Matthew
> Wilcox <mawilcox@microsoft.com>
> Cc: Long Li <longli@microsoft.com>
> Subject: RE: [Patch v2 00/19] CIFS: Implement SMBDirect
> 
> [You don't often get email from SWISE@OPENGRIDCOMPUTING.COM. Learn
> why this is important at http://aka.ms/LearnAboutSenderIdentification.]
> 
> >
> > From: Long Li <longli@microsoft.com>
> >
> > Starting with SMB2 dialect 3.0, Microsoft introduced SMBDirect
> > transport
> protocol
> > for transferring upper layer (SMB2) payload over RDMA via Infiniband,
> > RoCE or iWARP. The prococol is published in [MS-SMBD]
> >
> (https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmsdn
> > .microsoft.com%2Fen-
> &data=02%7C01%7Clongli%40microsoft.com%7C6082b57a9
> >
> 13844901b5b08d4e8c8ec1d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7
> C0%7C6
> >
> 36389397044862458&sdata=MnF3ljWT9cTEfFZWmj7zVADgNSYdMFROK%2B
> WXAdfG%2FB
> > I%3D&reserved=0
> > us/library/hh536346.aspx).
> >
> > The patch v2 added RDMA read/write via memory registration, and
> > addressed feedbacks on v1.
> >
> 
> Hey Long,
> 
> What testing have you done with this on the various rdma transports?  Does
> it work over IB, RoCE, and iWARP providers?

Hi Steve,

Currently all the tests have been done over Infiniband. We haven't tested on RoCE or iWARP, but planned to do it in the following weeks.

Long

> 
> Thanks,
> 
> Steve.
> 

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web