Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1722740 > unrolled thread

[Patch v3 00/19] CIFS: Implement SMB Direct

Back to article view | Back to linux.kernel

Contents

  [Patch v3 00/19] CIFS: Implement SMB Direct Long Li <longli@exchange.microsoft.com> - 2017-08-29 21:40 +0200
    [Patch v3 07/19] CIFS: SMBD: Destroy SMBDirect session on shutdown or umount Long Li <longli@exchange.microsoft.com> - 2017-08-29 21:40 +0200
    [Patch v3 04/19] CIFS: SMBD: Add SMBDirect transport to SMB connection and Makefile Long Li <longli@exchange.microsoft.com> - 2017-08-29 21:40 +0200
      Re: [Patch v3 04/19] CIFS: SMBD: Add SMBDirect transport to SMB  connection and Makefile Pavel Shilovsky <piastryyy@gmail.com> - 2017-08-30 01:20 +0200
        RE: [Patch v3 04/19] CIFS: SMBD: Add SMBDirect transport to SMB  connection and Makefile Long Li <longli@microsoft.com> - 2017-08-30 01:20 +0200
    [Patch v3 09/19] CIFS: SMBD: Receive data from SMBDirect Long Li <longli@exchange.microsoft.com> - 2017-08-29 21:40 +0200
    [Patch v3 01/19] CIFS: Add rdma mount option Long Li <longli@exchange.microsoft.com> - 2017-08-29 21:40 +0200
      Re: [Patch v3 01/19] CIFS: Add rdma mount option Pavel Shilovsky <piastryyy@gmail.com> - 2017-08-30 00:40 +0200
        RE: [Patch v3 01/19] CIFS: Add rdma mount option Long Li <longli@microsoft.com> - 2017-08-30 01:00 +0200
    [Patch v3 11/19] CIFS: SMBD: Define memory registration for I/O data Long Li <longli@exchange.microsoft.com> - 2017-08-29 21:40 +0200
      Re: [Patch v3 11/19] CIFS: SMBD: Define memory registration for I/O data Pavel Shilovsky <piastryyy@gmail.com> - 2017-08-30 02:50 +0200
        RE: [Patch v3 11/19] CIFS: SMBD: Define memory registration for I/O  data Long Li <longli@microsoft.com> - 2017-08-30 04:20 +0200
    [Patch v3 12/19] CIFS: SMBD: Fix the definition for SMB2_CHANNEL_RDMA_V1_INVALIDATE Long Li <longli@exchange.microsoft.com> - 2017-08-29 21:40 +0200
      Re: [Patch v3 12/19] CIFS: SMBD: Fix the definition for SMB2_CHANNEL_RDMA_V1_INVALIDATE Pavel Shilovsky <piastryyy@gmail.com> - 2017-08-30 03:00 +0200
        RE: [Patch v3 12/19] CIFS: SMBD: Fix the definition for  SMB2_CHANNEL_RDMA_V1_INVALIDATE Long Li <longli@microsoft.com> - 2017-08-30 04:20 +0200
    [Patch v3 10/19] CIFS: SMBD: Send data through SMBDirect Long Li <longli@exchange.microsoft.com> - 2017-08-29 21:40 +0200
    [Patch v3 19/19] CIFS: SMBD: Add SMBDirect debug counters Long Li <longli@exchange.microsoft.com> - 2017-08-29 21:40 +0200
    [Patch v3 17/19] CIFS: SMBD: Implement SMB read via RDMA write through memory registration Long Li <longli@exchange.microsoft.com> - 2017-08-29 21:40 +0200
    [Patch v3 14/19] CIFS: SMBD: Deregister memory on finishing SMB write Long Li <longli@exchange.microsoft.com> - 2017-08-29 21:40 +0200
    [Patch v3 08/19] CIFS: SMBD: Set SMBDirect maximum read or write size for I/O Long Li <longli@exchange.microsoft.com> - 2017-08-29 21:40 +0200
    [Patch v3 18/19] CIFS: SMBD: Deregister memory when finishing SMB read Long Li <longli@exchange.microsoft.com> - 2017-08-29 21:40 +0200
    [Patch v3 06/19] CIFS: SMBD: Reconnect to SMBDirect session Long Li <longli@exchange.microsoft.com> - 2017-08-29 21:40 +0200
      Re: [Patch v3 06/19] CIFS: SMBD: Reconnect to SMBDirect session Pavel Shilovsky <piastryyy@gmail.com> - 2017-08-30 01:30 +0200
        RE: [Patch v3 06/19] CIFS: SMBD: Reconnect to SMBDirect session Long Li <longli@microsoft.com> - 2017-08-30 01:40 +0200
    [Patch v3 15/19] CIFS: SMBD: Add parameter rdata to smb2_new_read_req Long Li <longli@exchange.microsoft.com> - 2017-08-29 21:40 +0200
    [Patch v3 13/19] CIFS: SMBD: Implement SMB write via RDMA read through memory registration Long Li <longli@exchange.microsoft.com> - 2017-08-29 21:40 +0200
    [Patch v3 02/19] CIFS: SMBD: Add SMBDirect protocol and transport constants Long Li <longli@exchange.microsoft.com> - 2017-08-29 21:40 +0200
      Re: [Patch v3 02/19] CIFS: SMBD: Add SMBDirect protocol and transport constants Pavel Shilovsky <piastryyy@gmail.com> - 2017-08-30 00:50 +0200
        RE: [Patch v3 02/19] CIFS: SMBD: Add SMBDirect protocol and transport  constants Long Li <longli@microsoft.com> - 2017-08-30 01:10 +0200
          Re: [Patch v3 02/19] CIFS: SMBD: Add SMBDirect protocol and transport constants Pavel Shilovsky <piastryyy@gmail.com> - 2017-08-30 20:20 +0200
            RE: [Patch v3 02/19] CIFS: SMBD: Add SMBDirect protocol and transport  constants Long Li <longli@microsoft.com> - 2017-08-30 20:30 +0200
              Re: [Patch v3 02/19] CIFS: SMBD: Add SMBDirect protocol and transport constants Pavel Shilovsky <piastryyy@gmail.com> - 2017-08-30 20:40 +0200
                Re: [Patch v3 02/19] CIFS: SMBD: Add SMBDirect protocol and transport constants Steve French <smfrench@gmail.com> - 2017-09-01 17:50 +0200
    RE: [Patch v3 00/19] CIFS: Implement SMB Direct Long Li <longli@microsoft.com> - 2017-08-30 20:50 +0200

Page 2 of 2 — ← Prev page 1 [2]

#1722754 — [Patch v3 18/19] CIFS: SMBD: Deregister memory when finishing SMB read

From: Long Li <longli@microsoft.com>

When SMB read is finished, deregister the memory regions if RDMA write is used
for this SMB read. smbd_deregister_mr may need to do local invalidation and
sleep, if server remote invalidation is not used.

There are situations where the MID may not be created on I/O failure, under
which memory region is deregistered when read data context is released.

Signed-off-by: Long Li <longli@microsoft.com>
---
 fs/cifs/file.c    |  5 +++++
 fs/cifs/smb2pdu.c | 10 ++++++++++
 2 files changed, 15 insertions(+)

diff --git a/fs/cifs/file.c b/fs/cifs/file.c
index 41460a5..5a6df25 100644
--- a/fs/cifs/file.c
+++ b/fs/cifs/file.c
@@ -2910,6 +2910,11 @@ cifs_readdata_release(struct kref *refcount)
 	struct cifs_readdata *rdata = container_of(refcount,
 					struct cifs_readdata, refcount);
 
+	if (rdata->mr) {
+		smbd_deregister_mr(rdata->mr);
+		rdata->mr = NULL;
+	}
+
 	if (rdata->cfile)
 		cifsFileInfo_put(rdata->cfile);
 
diff --git a/fs/cifs/smb2pdu.c b/fs/cifs/smb2pdu.c
index 1f08c75..43a7b60 100644
--- a/fs/cifs/smb2pdu.c
+++ b/fs/cifs/smb2pdu.c
@@ -2504,6 +2504,16 @@ smb2_readv_callback(struct mid_q_entry *mid)
 			rdata->result = -EIO;
 	}
 
+	/*
+	 * If this rdata has a memmory registered, the MR can be freed
+	 * MR needs to be freed as soon as I/O finishes to prevent deadlock
+	 * because they have limited number and are used for future I/Os
+	 */
+	if (rdata->mr) {
+		smbd_deregister_mr(rdata->mr);
+		rdata->mr = NULL;
+	}
+
 	if (rdata->result)
 		cifs_stats_fail_inc(tcon, SMB2_READ_HE);
 
-- 
2.7.4

[toc] | [prev] | [next] | [standalone]

#1722756 — [Patch v3 06/19] CIFS: SMBD: Reconnect to SMBDirect session

From: Long Li <longli@microsoft.com>

Do a reconnect on SMBDirect when it is used as the connection. Reconnect can
happen for many reasons and it's mostly the decision of upper layer SMB2 not
SMBDirect.

Signed-off-by: Long Li <longli@microsoft.com>
---
 fs/cifs/connect.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/fs/cifs/connect.c b/fs/cifs/connect.c
index 43b4d54..341a3fd 100644
--- a/fs/cifs/connect.c
+++ b/fs/cifs/connect.c
@@ -409,7 +409,11 @@ cifs_reconnect(struct TCP_Server_Info *server)
 
 		/* we should try only the port we connected to before */
 		mutex_lock(&server->srv_mutex);
-		rc = generic_ip_connect(server);
+		if (server->rdma)
+			rc = smbd_reconnect(server);
+		else
+			rc = generic_ip_connect(server);
+
 		if (rc) {
 			cifs_dbg(FYI, "reconnect error %d\n", rc);
 			mutex_unlock(&server->srv_mutex);
-- 
2.7.4

[toc] | [prev] | [next] | [standalone]

#1722909 — Re: [Patch v3 06/19] CIFS: SMBD: Reconnect to SMBDirect session

2017-08-29 12:29 GMT-07:00 Long Li <longli@exchange.microsoft.com>:
> From: Long Li <longli@microsoft.com>
>
> Do a reconnect on SMBDirect when it is used as the connection. Reconnect can
> happen for many reasons and it's mostly the decision of upper layer SMB2 not
> SMBDirect.
>
> Signed-off-by: Long Li <longli@microsoft.com>
> ---
>  fs/cifs/connect.c | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
>
> diff --git a/fs/cifs/connect.c b/fs/cifs/connect.c
> index 43b4d54..341a3fd 100644
> --- a/fs/cifs/connect.c
> +++ b/fs/cifs/connect.c
> @@ -409,7 +409,11 @@ cifs_reconnect(struct TCP_Server_Info *server)
>
>                 /* we should try only the port we connected to before */
>                 mutex_lock(&server->srv_mutex);
> -               rc = generic_ip_connect(server);
> +               if (server->rdma)
> +                       rc = smbd_reconnect(server);
> +               else
> +                       rc = generic_ip_connect(server);
> +
>                 if (rc) {
>                         cifs_dbg(FYI, "reconnect error %d\n", rc);
>                         mutex_unlock(&server->srv_mutex);
> --
> 2.7.4
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-cifs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

It is a easier to follow the logic if the definition is introduced
together with using of this function in the existing code. I suggest
to add smbd_reconnect() definition in this patch.
--
Best regards,
Pavel Shilovsky

[toc] | [prev] | [next] | [standalone]

#1722911 — RE: [Patch v3 06/19] CIFS: SMBD: Reconnect to SMBDirect session

> -----Original Message-----
> From: Pavel Shilovsky [mailto:piastryyy@gmail.com]
> Sent: Tuesday, August 29, 2017 4:15 PM
> To: Long Li <longli@microsoft.com>
> Cc: Steve French <sfrench@samba.org>; linux-cifs <linux-
> cifs@vger.kernel.org>; samba-technical <samba-technical@lists.samba.org>;
> Kernel Mailing List <linux-kernel@vger.kernel.org>; linux-
> rdma@vger.kernel.org; Christoph Hellwig <hch@infradead.org>; Tom Talpey
> <ttalpey@microsoft.com>; Matthew Wilcox <mawilcox@microsoft.com>;
> Long Li <longli@microsoft.com>
> Subject: Re: [Patch v3 06/19] CIFS: SMBD: Reconnect to SMBDirect session
> 
> 2017-08-29 12:29 GMT-07:00 Long Li <longli@exchange.microsoft.com>:
> > From: Long Li <longli@microsoft.com>
> >
> > Do a reconnect on SMBDirect when it is used as the connection.
> > Reconnect can happen for many reasons and it's mostly the decision of
> > upper layer SMB2 not SMBDirect.
> >
> > Signed-off-by: Long Li <longli@microsoft.com>
> > ---
> >  fs/cifs/connect.c | 6 +++++-
> >  1 file changed, 5 insertions(+), 1 deletion(-)
> >
> > diff --git a/fs/cifs/connect.c b/fs/cifs/connect.c index
> > 43b4d54..341a3fd 100644
> > --- a/fs/cifs/connect.c
> > +++ b/fs/cifs/connect.c
> > @@ -409,7 +409,11 @@ cifs_reconnect(struct TCP_Server_Info *server)
> >
> >                 /* we should try only the port we connected to before */
> >                 mutex_lock(&server->srv_mutex);
> > -               rc = generic_ip_connect(server);
> > +               if (server->rdma)
> > +                       rc = smbd_reconnect(server);
> > +               else
> > +                       rc = generic_ip_connect(server);
> > +
> >                 if (rc) {
> >                         cifs_dbg(FYI, "reconnect error %d\n", rc);
> >                         mutex_unlock(&server->srv_mutex);
> > --
> > 2.7.4
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-cifs"
> > in the body of a message to majordomo@vger.kernel.org More
> majordomo
> > info at
> >
> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fvger.k
> > ernel.org%2Fmajordomo-
> info.html&data=02%7C01%7Clongli%40microsoft.com%
> >
> 7C192678645752449a875b08d4ef33d874%7C72f988bf86f141af91ab2d7cd011d
> b47%
> >
> 7C1%7C0%7C636396453334398562&sdata=i5dZUoUU91foFGipTsgQBnmYJeU
> aeWww2QI
> > xoP9BSO4%3D&reserved=0
> 
> It is a easier to follow the logic if the definition is introduced together with
> using of this function in the existing code. I suggest to add smbd_reconnect()
> definition in this patch.

I will do this in v4.

> --
> Best regards,
> Pavel Shilovsky

[toc] | [prev] | [next] | [standalone]

#1722759 — [Patch v3 15/19] CIFS: SMBD: Add parameter rdata to smb2_new_read_req

From: Long Li <longli@microsoft.com>

When we assemble the SMB read packet header, we need to know the I/O layout
if this request is to use a RDMA write. rdata has all the information we need
for memory registration. Add rdata to smb2_new_read_req.

Signed-off-by: Long Li <longli@microsoft.com>
---
 fs/cifs/smb2pdu.c | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

diff --git a/fs/cifs/smb2pdu.c b/fs/cifs/smb2pdu.c
index 5551053..fbad987 100644
--- a/fs/cifs/smb2pdu.c
+++ b/fs/cifs/smb2pdu.c
@@ -2363,18 +2363,21 @@ SMB2_flush(const unsigned int xid, struct cifs_tcon *tcon, u64 persistent_fid,
  */
 static int
 smb2_new_read_req(void **buf, unsigned int *total_len,
-		  struct cifs_io_parms *io_parms, unsigned int remaining_bytes,
-		  int request_type)
+	struct cifs_io_parms *io_parms, struct cifs_readdata *rdata,
+	unsigned int remaining_bytes, int request_type)
 {
 	int rc = -EACCES;
 	struct smb2_read_plain_req *req = NULL;
 	struct smb2_sync_hdr *shdr;
+	struct TCP_Server_Info *server;
 
 	rc = smb2_plain_req_init(SMB2_READ, io_parms->tcon, (void **) &req,
 				 total_len);
 	if (rc)
 		return rc;
-	if (io_parms->tcon->ses->server == NULL)
+
+	server = io_parms->tcon->ses->server;
+	if (server == NULL)
 		return -ECONNABORTED;
 
 	shdr = &req->sync_hdr;
@@ -2502,7 +2505,8 @@ smb2_async_readv(struct cifs_readdata *rdata)
 
 	server = io_parms.tcon->ses->server;
 
-	rc = smb2_new_read_req((void **) &buf, &total_len, &io_parms, 0, 0);
+	rc = smb2_new_read_req(
+		(void **) &buf, &total_len, &io_parms, rdata, 0, 0);
 	if (rc) {
 		if (rc == -EAGAIN && rdata->credits) {
 			/* credits was reset by reconnect */
@@ -2570,7 +2574,7 @@ SMB2_read(const unsigned int xid, struct cifs_io_parms *io_parms,
 	struct cifs_ses *ses = io_parms->tcon->ses;
 
 	*nbytes = 0;
-	rc = smb2_new_read_req((void **)&req, &total_len, io_parms, 0, 0);
+	rc = smb2_new_read_req((void **)&req, &total_len, io_parms, NULL, 0, 0);
 	if (rc)
 		return rc;
 
-- 
2.7.4

[toc] | [prev] | [next] | [standalone]

#1722760 — [Patch v3 13/19] CIFS: SMBD: Implement SMB write via RDMA read through memory registration

From: Long Li <longli@microsoft.com>

When sending I/O, if size is larger than rdma_readwrite_threshold we prepare
to send SMB write packet for a RDMA read via memory registration. The actual
I/O is done by remote peer through local RDMA hardware. Modify the relevant
fields in the packet accordingly, and append a smbd_buffer_descriptor_v1 to
the end of the SMB write packet.

Signed-off-by: Long Li <longli@microsoft.com>
---
 fs/cifs/smb2pdu.c | 45 ++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 44 insertions(+), 1 deletion(-)

diff --git a/fs/cifs/smb2pdu.c b/fs/cifs/smb2pdu.c
index 5cc5f6c..5581afd 100644
--- a/fs/cifs/smb2pdu.c
+++ b/fs/cifs/smb2pdu.c
@@ -48,6 +48,7 @@
 #include "smb2glob.h"
 #include "cifspdu.h"
 #include "cifs_spnego.h"
+#include "smbdirect.h"
 
 /*
  *  The following table defines the expected "StructureSize" of SMB2 requests
@@ -2716,6 +2717,41 @@ smb2_async_writev(struct cifs_writedata *wdata,
 				offsetof(struct smb2_write_req, Buffer) - 4);
 	req->RemainingBytes = 0;
 
+	/*
+	 * If we want to do a server RDMA read, fill in and append
+	 * smbd_buffer_descriptor_v1 to the end of write request
+	 */
+	if (server->rdma && wdata->bytes >
+		server->smbd_conn->rdma_readwrite_threshold) {
+
+		struct smbd_buffer_descriptor_v1 *v1;
+		bool need_invalidate = server->dialect == SMB30_PROT_ID;
+
+		wdata->mr = smbd_register_mr(
+				server->smbd_conn, wdata->pages,
+				wdata->nr_pages, wdata->tailsz,
+				false, need_invalidate);
+		if (!wdata->mr) {
+			rc = -ENOBUFS;
+			goto async_writev_out;
+		}
+		req->Length = 0;
+		req->DataOffset = 0;
+		req->RemainingBytes =
+			(wdata->nr_pages-1)*PAGE_SIZE + wdata->tailsz;
+		req->Channel = SMB2_CHANNEL_RDMA_V1_INVALIDATE;
+		if (need_invalidate)
+			req->Channel = SMB2_CHANNEL_RDMA_V1;
+		req->WriteChannelInfoOffset =
+			offsetof(struct smb2_write_req, Buffer) - 4;
+		req->WriteChannelInfoLength =
+			sizeof(struct smbd_buffer_descriptor_v1);
+		v1 = (struct smbd_buffer_descriptor_v1 *) &req->Buffer[0];
+		v1->offset = wdata->mr->mr->iova;
+		v1->token = wdata->mr->mr->rkey;
+		v1->length = wdata->mr->mr->length;
+	}
+
 	/* 4 for rfc1002 length field and 1 for Buffer */
 	iov[0].iov_len = 4;
 	iov[0].iov_base = req;
@@ -2729,10 +2765,17 @@ smb2_async_writev(struct cifs_writedata *wdata,
 	rqst.rq_pagesz = wdata->pagesz;
 	rqst.rq_tailsz = wdata->tailsz;
 
+	if (wdata->mr) {
+		iov[1].iov_len += sizeof(struct smbd_buffer_descriptor_v1);
+		rqst.rq_npages = 0;
+	}
+
 	cifs_dbg(FYI, "async write at %llu %u bytes\n",
 		 wdata->offset, wdata->bytes);
 
-	req->Length = cpu_to_le32(wdata->bytes);
+	/* For RDMA read, I/O size is in RemainingBytes not in Length */
+	if (!wdata->mr)
+		req->Length = cpu_to_le32(wdata->bytes);
 
 	inc_rfc1001_len(&req->hdr, wdata->bytes - 1 /* Buffer */);
 
-- 
2.7.4

[toc] | [prev] | [next] | [standalone]

#1722761 — [Patch v3 02/19] CIFS: SMBD: Add SMBDirect protocol and transport constants

From: Long Li <longli@microsoft.com>

To prepare for protocol implementation, add constants and user-configurable
values in the SMBDirect protocol.

Signed-off-by: Long Li <longli@microsoft.com>
---
 fs/cifs/smbdirect.c | 78 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 fs/cifs/smbdirect.h | 20 ++++++++++++++
 2 files changed, 98 insertions(+)
 create mode 100644 fs/cifs/smbdirect.c
 create mode 100644 fs/cifs/smbdirect.h

diff --git a/fs/cifs/smbdirect.c b/fs/cifs/smbdirect.c
new file mode 100644
index 0000000..d785bc1
--- /dev/null
+++ b/fs/cifs/smbdirect.c
@@ -0,0 +1,78 @@
+/*
+ *   Copyright (C) 2017, Microsoft Corporation.
+ *
+ *   Author(s): Long Li <longli@microsoft.com>
+ *
+ *   This program is free software;  you can redistribute it and/or modify
+ *   it under the terms of the GNU General Public License as published by
+ *   the Free Software Foundation; either version 2 of the License, or
+ *   (at your option) any later version.
+ *
+ *   This program is distributed in the hope that it will be useful,
+ *   but WITHOUT ANY WARRANTY;  without even the implied warranty of
+ *   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See
+ *   the GNU General Public License for more details.
+ */
+#include <linux/module.h>
+#include "smbdirect.h"
+#include "cifs_debug.h"
+
+/* SMBD version number */
+#define SMBD_V1	0x0100
+
+/* Port numbers for SMBD transport */
+#define SMB_PORT	445
+#define SMBD_PORT	5445
+
+/* Address lookup and resolve timeout in ms */
+#define RDMA_RESOLVE_TIMEOUT	5000
+
+/* SMBD negotiation timeout in seconds */
+#define SMBD_NEGOTIATE_TIMEOUT	120
+
+/* SMBD minimum receive size and fragmented sized defined in [MS-SMBD] */
+#define SMBD_MIN_RECEIVE_SIZE		128
+#define SMBD_MIN_FRAGMENTED_SIZE	131072
+
+/*
+ * Default maximum number of RDMA read/write outstanding on this connection
+ * This value is possibly decreased during QP creation on hardware limit
+ */
+#define SMBD_CM_RESPONDER_RESOURCES	32
+
+/* Maximum number of retries on data transfer operations */
+#define SMBD_CM_RETRY			6
+/* No need to retry on Receiver Not Ready since SMBD manages credits */
+#define SMBD_CM_RNR_RETRY		0
+
+/*
+ * User configurable initial values per SMBD transport connection
+ * as defined in [MS-SMBD] 3.1.1.1
+ * Those may change after a SMBD negotiation
+ */
+/* The local peer's maximum number of credits to grant to the peer */
+static int receive_credit_max = 255;
+/* The remote peer's credit request of local peer */
+static int send_credit_target = 255;
+/* The maximum single message size can be sent to remote peer */
+static int max_send_size = 1364;
+/*  The maximum fragmented upper-layer payload receive size supported */
+static int max_fragmented_recv_size = 1024 * 1024;
+/*  The maximum single-message size which can be received */
+static int max_receive_size = 8192;
+
+/* The timeout to initiate send of a keepalive message on idle */
+static int keep_alive_interval = 120;
+
+/*
+ * User configurable initial values for RDMA transport
+ * The actual values used may be lower and are limited to hardware capabilities
+ */
+/* Default maximum number of SGEs in a RDMA send/recv */
+static int max_send_sge = SMBDIRECT_MAX_SGE;
+static int max_recv_sge = SMBDIRECT_MAX_SGE;
+/* Default maximum number of SGEs in a RDMA write/read */
+static int max_frmr_depth = 2048;
+
+/* If payload is less than this byte, use RDMA send/recv not read/write */
+static int rdma_readwrite_threshold = 4096;
diff --git a/fs/cifs/smbdirect.h b/fs/cifs/smbdirect.h
new file mode 100644
index 0000000..06eeb0b
--- /dev/null
+++ b/fs/cifs/smbdirect.h
@@ -0,0 +1,20 @@
+/*
+ *   Copyright (C) 2017, Microsoft Corporation.
+ *
+ *   Author(s): Long Li <longli@microsoft.com>
+ *
+ *   This program is free software;  you can redistribute it and/or modify
+ *   it under the terms of the GNU General Public License as published by
+ *   the Free Software Foundation; either version 2 of the License, or
+ *   (at your option) any later version.
+ *
+ *   This program is distributed in the hope that it will be useful,
+ *   but WITHOUT ANY WARRANTY;  without even the implied warranty of
+ *   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See
+ *   the GNU General Public License for more details.
+ */
+#ifndef _SMBDIRECT_H
+#define _SMBDIRECT_H
+
+#define SMBDIRECT_MAX_SGE	16
+#endif
-- 
2.7.4

[toc] | [prev] | [next] | [standalone]

#1722897 — Re: [Patch v3 02/19] CIFS: SMBD: Add SMBDirect protocol and transport constants

2017-08-29 12:28 GMT-07:00 Long Li <longli@exchange.microsoft.com>:
> From: Long Li <longli@microsoft.com>
>
> To prepare for protocol implementation, add constants and user-configurable
> values in the SMBDirect protocol.
>
> Signed-off-by: Long Li <longli@microsoft.com>
> ---
>  fs/cifs/smbdirect.c | 78 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>  fs/cifs/smbdirect.h | 20 ++++++++++++++
>  2 files changed, 98 insertions(+)
>  create mode 100644 fs/cifs/smbdirect.c
>  create mode 100644 fs/cifs/smbdirect.h
>
> diff --git a/fs/cifs/smbdirect.c b/fs/cifs/smbdirect.c
> new file mode 100644
> index 0000000..d785bc1
> --- /dev/null
> +++ b/fs/cifs/smbdirect.c
> @@ -0,0 +1,78 @@
> +/*
> + *   Copyright (C) 2017, Microsoft Corporation.
> + *
> + *   Author(s): Long Li <longli@microsoft.com>
> + *
> + *   This program is free software;  you can redistribute it and/or modify
> + *   it under the terms of the GNU General Public License as published by
> + *   the Free Software Foundation; either version 2 of the License, or
> + *   (at your option) any later version.
> + *
> + *   This program is distributed in the hope that it will be useful,
> + *   but WITHOUT ANY WARRANTY;  without even the implied warranty of
> + *   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See
> + *   the GNU General Public License for more details.
> + */
> +#include <linux/module.h>
> +#include "smbdirect.h"
> +#include "cifs_debug.h"
> +
> +/* SMBD version number */
> +#define SMBD_V1        0x0100
> +
> +/* Port numbers for SMBD transport */
> +#define SMB_PORT       445
> +#define SMBD_PORT      5445
> +
> +/* Address lookup and resolve timeout in ms */
> +#define RDMA_RESOLVE_TIMEOUT   5000
> +
> +/* SMBD negotiation timeout in seconds */
> +#define SMBD_NEGOTIATE_TIMEOUT 120
> +
> +/* SMBD minimum receive size and fragmented sized defined in [MS-SMBD] */
> +#define SMBD_MIN_RECEIVE_SIZE          128
> +#define SMBD_MIN_FRAGMENTED_SIZE       131072
> +
> +/*
> + * Default maximum number of RDMA read/write outstanding on this connection
> + * This value is possibly decreased during QP creation on hardware limit
> + */
> +#define SMBD_CM_RESPONDER_RESOURCES    32
> +
> +/* Maximum number of retries on data transfer operations */
> +#define SMBD_CM_RETRY                  6
> +/* No need to retry on Receiver Not Ready since SMBD manages credits */
> +#define SMBD_CM_RNR_RETRY              0
> +
> +/*
> + * User configurable initial values per SMBD transport connection
> + * as defined in [MS-SMBD] 3.1.1.1
> + * Those may change after a SMBD negotiation
> + */

Since these value are per transport connection, why they are global?
Shouldn't they be inside a some structure that is created for a
particular connection? Also the constants below should be defines.

> +/* The local peer's maximum number of credits to grant to the peer */
> +static int receive_credit_max = 255;
> +/* The remote peer's credit request of local peer */
> +static int send_credit_target = 255;
> +/* The maximum single message size can be sent to remote peer */
> +static int max_send_size = 1364;
> +/*  The maximum fragmented upper-layer payload receive size supported */
> +static int max_fragmented_recv_size = 1024 * 1024;
> +/*  The maximum single-message size which can be received */
> +static int max_receive_size = 8192;
> +
> +/* The timeout to initiate send of a keepalive message on idle */
> +static int keep_alive_interval = 120;
> +
> +/*
> + * User configurable initial values for RDMA transport
> + * The actual values used may be lower and are limited to hardware capabilities
> + */
> +/* Default maximum number of SGEs in a RDMA send/recv */
> +static int max_send_sge = SMBDIRECT_MAX_SGE;
> +static int max_recv_sge = SMBDIRECT_MAX_SGE;
> +/* Default maximum number of SGEs in a RDMA write/read */
> +static int max_frmr_depth = 2048;
> +
> +/* If payload is less than this byte, use RDMA send/recv not read/write */
> +static int rdma_readwrite_threshold = 4096;
> diff --git a/fs/cifs/smbdirect.h b/fs/cifs/smbdirect.h
> new file mode 100644
> index 0000000..06eeb0b
> --- /dev/null
> +++ b/fs/cifs/smbdirect.h
> @@ -0,0 +1,20 @@
> +/*
> + *   Copyright (C) 2017, Microsoft Corporation.
> + *
> + *   Author(s): Long Li <longli@microsoft.com>
> + *
> + *   This program is free software;  you can redistribute it and/or modify
> + *   it under the terms of the GNU General Public License as published by
> + *   the Free Software Foundation; either version 2 of the License, or
> + *   (at your option) any later version.
> + *
> + *   This program is distributed in the hope that it will be useful,
> + *   but WITHOUT ANY WARRANTY;  without even the implied warranty of
> + *   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See
> + *   the GNU General Public License for more details.
> + */
> +#ifndef _SMBDIRECT_H
> +#define _SMBDIRECT_H
> +
> +#define SMBDIRECT_MAX_SGE      16
> +#endif
> --
> 2.7.4
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-cifs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



--
Best regards,
Pavel Shilovsky

[toc] | [prev] | [next] | [standalone]

#1722901 — RE: [Patch v3 02/19] CIFS: SMBD: Add SMBDirect protocol and transport constants

> -----Original Message-----
> From: Pavel Shilovsky [mailto:piastryyy@gmail.com]
> Sent: Tuesday, August 29, 2017 3:45 PM
> To: Long Li <longli@microsoft.com>
> Cc: Steve French <sfrench@samba.org>; linux-cifs <linux-
> cifs@vger.kernel.org>; samba-technical <samba-technical@lists.samba.org>;
> Kernel Mailing List <linux-kernel@vger.kernel.org>; linux-
> rdma@vger.kernel.org; Christoph Hellwig <hch@infradead.org>; Tom Talpey
> <ttalpey@microsoft.com>; Matthew Wilcox <mawilcox@microsoft.com>;
> Long Li <longli@microsoft.com>
> Subject: Re: [Patch v3 02/19] CIFS: SMBD: Add SMBDirect protocol and
> transport constants
> 
> 2017-08-29 12:28 GMT-07:00 Long Li <longli@exchange.microsoft.com>:
> > From: Long Li <longli@microsoft.com>
> >
> > To prepare for protocol implementation, add constants and
> > user-configurable values in the SMBDirect protocol.
> >
> > Signed-off-by: Long Li <longli@microsoft.com>
> > ---
> >  fs/cifs/smbdirect.c | 78
> > +++++++++++++++++++++++++++++++++++++++++++++++++++++
> >  fs/cifs/smbdirect.h | 20 ++++++++++++++
> >  2 files changed, 98 insertions(+)
> >  create mode 100644 fs/cifs/smbdirect.c  create mode 100644
> > fs/cifs/smbdirect.h
> >
> > diff --git a/fs/cifs/smbdirect.c b/fs/cifs/smbdirect.c new file mode
> > 100644 index 0000000..d785bc1
> > --- /dev/null
> > +++ b/fs/cifs/smbdirect.c
> > @@ -0,0 +1,78 @@
> > +/*
> > + *   Copyright (C) 2017, Microsoft Corporation.
> > + *
> > + *   Author(s): Long Li <longli@microsoft.com>
> > + *
> > + *   This program is free software;  you can redistribute it and/or modify
> > + *   it under the terms of the GNU General Public License as published by
> > + *   the Free Software Foundation; either version 2 of the License, or
> > + *   (at your option) any later version.
> > + *
> > + *   This program is distributed in the hope that it will be useful,
> > + *   but WITHOUT ANY WARRANTY;  without even the implied warranty of
> > + *   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See
> > + *   the GNU General Public License for more details.
> > + */
> > +#include <linux/module.h>
> > +#include "smbdirect.h"
> > +#include "cifs_debug.h"
> > +
> > +/* SMBD version number */
> > +#define SMBD_V1        0x0100
> > +
> > +/* Port numbers for SMBD transport */
> > +#define SMB_PORT       445
> > +#define SMBD_PORT      5445
> > +
> > +/* Address lookup and resolve timeout in ms */
> > +#define RDMA_RESOLVE_TIMEOUT   5000
> > +
> > +/* SMBD negotiation timeout in seconds */ #define
> > +SMBD_NEGOTIATE_TIMEOUT 120
> > +
> > +/* SMBD minimum receive size and fragmented sized defined in [MS-
> SMBD] */
> > +#define SMBD_MIN_RECEIVE_SIZE          128
> > +#define SMBD_MIN_FRAGMENTED_SIZE       131072
> > +
> > +/*
> > + * Default maximum number of RDMA read/write outstanding on this
> > +connection
> > + * This value is possibly decreased during QP creation on hardware
> > +limit  */
> > +#define SMBD_CM_RESPONDER_RESOURCES    32
> > +
> > +/* Maximum number of retries on data transfer operations */
> > +#define SMBD_CM_RETRY                  6
> > +/* No need to retry on Receiver Not Ready since SMBD manages credits
> */
> > +#define SMBD_CM_RNR_RETRY              0
> > +
> > +/*
> > + * User configurable initial values per SMBD transport connection
> > + * as defined in [MS-SMBD] 3.1.1.1
> > + * Those may change after a SMBD negotiation  */
> 
> Since these value are per transport connection, why they are global?
> Shouldn't they be inside a some structure that is created for a particular
> connection? Also the constants below should be defines.

Those are configurable initial values (default values) for all connections.

Each connection has its own values based on those initial values. But connection-based values can change after negotiation is done, or after RDMA hardware capabilities are probed.

> 
> > +/* The local peer's maximum number of credits to grant to the peer */
> > +static int receive_credit_max = 255;
> > +/* The remote peer's credit request of local peer */ static int
> > +send_credit_target = 255;
> > +/* The maximum single message size can be sent to remote peer */
> > +static int max_send_size = 1364;
> > +/*  The maximum fragmented upper-layer payload receive size supported
> > +*/ static int max_fragmented_recv_size = 1024 * 1024;
> > +/*  The maximum single-message size which can be received */ static
> > +int max_receive_size = 8192;
> > +
> > +/* The timeout to initiate send of a keepalive message on idle */
> > +static int keep_alive_interval = 120;
> > +
> > +/*
> > + * User configurable initial values for RDMA transport
> > + * The actual values used may be lower and are limited to hardware
> > +capabilities  */
> > +/* Default maximum number of SGEs in a RDMA send/recv */ static int
> > +max_send_sge = SMBDIRECT_MAX_SGE; static int max_recv_sge =
> > +SMBDIRECT_MAX_SGE;
> > +/* Default maximum number of SGEs in a RDMA write/read */ static int
> > +max_frmr_depth = 2048;
> > +
> > +/* If payload is less than this byte, use RDMA send/recv not
> > +read/write */ static int rdma_readwrite_threshold = 4096;
> > diff --git a/fs/cifs/smbdirect.h b/fs/cifs/smbdirect.h new file mode
> > 100644 index 0000000..06eeb0b
> > --- /dev/null
> > +++ b/fs/cifs/smbdirect.h
> > @@ -0,0 +1,20 @@
> > +/*
> > + *   Copyright (C) 2017, Microsoft Corporation.
> > + *
> > + *   Author(s): Long Li <longli@microsoft.com>
> > + *
> > + *   This program is free software;  you can redistribute it and/or modify
> > + *   it under the terms of the GNU General Public License as published by
> > + *   the Free Software Foundation; either version 2 of the License, or
> > + *   (at your option) any later version.
> > + *
> > + *   This program is distributed in the hope that it will be useful,
> > + *   but WITHOUT ANY WARRANTY;  without even the implied warranty of
> > + *   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See
> > + *   the GNU General Public License for more details.
> > + */
> > +#ifndef _SMBDIRECT_H
> > +#define _SMBDIRECT_H
> > +
> > +#define SMBDIRECT_MAX_SGE      16
> > +#endif
> > --
> > 2.7.4
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-cifs"
> > in the body of a message to majordomo@vger.kernel.org More
> majordomo
> > info at
> >
> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fvger.k
> > ernel.org%2Fmajordomo-
> info.html&data=02%7C01%7Clongli%40microsoft.com%
> >
> 7C048aaa90c05748ee36a808d4ef2f9564%7C72f988bf86f141af91ab2d7cd011d
> b47%
> >
> 7C1%7C0%7C636396435035660594&sdata=qdY4CwjZ74W7AdqTVklpo8XlWFI6
> fxZ9y3i
> > fPyPQ%2Fcg%3D&reserved=0
> 
> 
> 
> --
> Best regards,
> Pavel Shilovsky

[toc] | [prev] | [next] | [standalone]

#1723575 — Re: [Patch v3 02/19] CIFS: SMBD: Add SMBDirect protocol and transport constants

2017-08-29 16:00 GMT-07:00 Long Li <longli@microsoft.com>:
>> -----Original Message-----
>> From: Pavel Shilovsky [mailto:piastryyy@gmail.com]
>> Sent: Tuesday, August 29, 2017 3:45 PM
>> To: Long Li <longli@microsoft.com>
>> Cc: Steve French <sfrench@samba.org>; linux-cifs <linux-
>> cifs@vger.kernel.org>; samba-technical <samba-technical@lists.samba.org>;
>> Kernel Mailing List <linux-kernel@vger.kernel.org>; linux-
>> rdma@vger.kernel.org; Christoph Hellwig <hch@infradead.org>; Tom Talpey
>> <ttalpey@microsoft.com>; Matthew Wilcox <mawilcox@microsoft.com>;
>> Long Li <longli@microsoft.com>
>> Subject: Re: [Patch v3 02/19] CIFS: SMBD: Add SMBDirect protocol and
>> transport constants
>>
>> 2017-08-29 12:28 GMT-07:00 Long Li <longli@exchange.microsoft.com>:
>> > From: Long Li <longli@microsoft.com>
>> >
>> > To prepare for protocol implementation, add constants and
>> > user-configurable values in the SMBDirect protocol.
>> >
>> > Signed-off-by: Long Li <longli@microsoft.com>
>> > ---
>> >  fs/cifs/smbdirect.c | 78
>> > +++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >  fs/cifs/smbdirect.h | 20 ++++++++++++++
>> >  2 files changed, 98 insertions(+)
>> >  create mode 100644 fs/cifs/smbdirect.c  create mode 100644
>> > fs/cifs/smbdirect.h
>> >
>> > diff --git a/fs/cifs/smbdirect.c b/fs/cifs/smbdirect.c new file mode
>> > 100644 index 0000000..d785bc1
>> > --- /dev/null
>> > +++ b/fs/cifs/smbdirect.c
>> > @@ -0,0 +1,78 @@
>> > +/*
>> > + *   Copyright (C) 2017, Microsoft Corporation.
>> > + *
>> > + *   Author(s): Long Li <longli@microsoft.com>
>> > + *
>> > + *   This program is free software;  you can redistribute it and/or modify
>> > + *   it under the terms of the GNU General Public License as published by
>> > + *   the Free Software Foundation; either version 2 of the License, or
>> > + *   (at your option) any later version.
>> > + *
>> > + *   This program is distributed in the hope that it will be useful,
>> > + *   but WITHOUT ANY WARRANTY;  without even the implied warranty of
>> > + *   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See
>> > + *   the GNU General Public License for more details.
>> > + */
>> > +#include <linux/module.h>
>> > +#include "smbdirect.h"
>> > +#include "cifs_debug.h"
>> > +
>> > +/* SMBD version number */
>> > +#define SMBD_V1        0x0100
>> > +
>> > +/* Port numbers for SMBD transport */
>> > +#define SMB_PORT       445
>> > +#define SMBD_PORT      5445
>> > +
>> > +/* Address lookup and resolve timeout in ms */
>> > +#define RDMA_RESOLVE_TIMEOUT   5000
>> > +
>> > +/* SMBD negotiation timeout in seconds */ #define
>> > +SMBD_NEGOTIATE_TIMEOUT 120
>> > +
>> > +/* SMBD minimum receive size and fragmented sized defined in [MS-
>> SMBD] */
>> > +#define SMBD_MIN_RECEIVE_SIZE          128
>> > +#define SMBD_MIN_FRAGMENTED_SIZE       131072
>> > +
>> > +/*
>> > + * Default maximum number of RDMA read/write outstanding on this
>> > +connection
>> > + * This value is possibly decreased during QP creation on hardware
>> > +limit  */
>> > +#define SMBD_CM_RESPONDER_RESOURCES    32
>> > +
>> > +/* Maximum number of retries on data transfer operations */
>> > +#define SMBD_CM_RETRY                  6
>> > +/* No need to retry on Receiver Not Ready since SMBD manages credits
>> */
>> > +#define SMBD_CM_RNR_RETRY              0
>> > +
>> > +/*
>> > + * User configurable initial values per SMBD transport connection
>> > + * as defined in [MS-SMBD] 3.1.1.1
>> > + * Those may change after a SMBD negotiation  */
>>
>> Since these value are per transport connection, why they are global?
>> Shouldn't they be inside a some structure that is created for a particular
>> connection? Also the constants below should be defines.
>
> Those are configurable initial values (default values) for all connections.
>
> Each connection has its own values based on those initial values. But connection-based values can change after negotiation is done, or after RDMA hardware capabilities are probed.

If these are configurable values, let's add a way to actually
configure them. Through /proc?

--
Best regards,
Pavel Shilovsky

[toc] | [prev] | [next] | [standalone]

#1723581 — RE: [Patch v3 02/19] CIFS: SMBD: Add SMBDirect protocol and transport constants

> -----Original Message-----
> From: Pavel Shilovsky [mailto:piastryyy@gmail.com]
> Sent: Wednesday, August 30, 2017 11:19 AM
> To: Long Li <longli@microsoft.com>
> Cc: Steve French <sfrench@samba.org>; linux-cifs <linux-
> cifs@vger.kernel.org>; samba-technical <samba-technical@lists.samba.org>;
> Kernel Mailing List <linux-kernel@vger.kernel.org>; linux-
> rdma@vger.kernel.org; Christoph Hellwig <hch@infradead.org>; Tom Talpey
> <ttalpey@microsoft.com>; Matthew Wilcox <mawilcox@microsoft.com>
> Subject: Re: [Patch v3 02/19] CIFS: SMBD: Add SMBDirect protocol and
> transport constants
> 
> 2017-08-29 16:00 GMT-07:00 Long Li <longli@microsoft.com>:
> >> -----Original Message-----
> >> From: Pavel Shilovsky [mailto:piastryyy@gmail.com]
> >> Sent: Tuesday, August 29, 2017 3:45 PM
> >> To: Long Li <longli@microsoft.com>
> >> Cc: Steve French <sfrench@samba.org>; linux-cifs <linux-
> >> cifs@vger.kernel.org>; samba-technical
> >> <samba-technical@lists.samba.org>;
> >> Kernel Mailing List <linux-kernel@vger.kernel.org>; linux-
> >> rdma@vger.kernel.org; Christoph Hellwig <hch@infradead.org>; Tom
> >> Talpey <ttalpey@microsoft.com>; Matthew Wilcox
> >> <mawilcox@microsoft.com>; Long Li <longli@microsoft.com>
> >> Subject: Re: [Patch v3 02/19] CIFS: SMBD: Add SMBDirect protocol and
> >> transport constants
> >>
> >> 2017-08-29 12:28 GMT-07:00 Long Li <longli@exchange.microsoft.com>:
> >> > From: Long Li <longli@microsoft.com>
> >> >
> >> > To prepare for protocol implementation, add constants and
> >> > user-configurable values in the SMBDirect protocol.
> >> >
> >> > Signed-off-by: Long Li <longli@microsoft.com>
> >> > ---
> >> >  fs/cifs/smbdirect.c | 78
> >> > +++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> >  fs/cifs/smbdirect.h | 20 ++++++++++++++
> >> >  2 files changed, 98 insertions(+)
> >> >  create mode 100644 fs/cifs/smbdirect.c  create mode 100644
> >> > fs/cifs/smbdirect.h
> >> >
> >> > diff --git a/fs/cifs/smbdirect.c b/fs/cifs/smbdirect.c new file
> >> > mode
> >> > 100644 index 0000000..d785bc1
> >> > --- /dev/null
> >> > +++ b/fs/cifs/smbdirect.c
> >> > @@ -0,0 +1,78 @@
> >> > +/*
> >> > + *   Copyright (C) 2017, Microsoft Corporation.
> >> > + *
> >> > + *   Author(s): Long Li <longli@microsoft.com>
> >> > + *
> >> > + *   This program is free software;  you can redistribute it and/or modify
> >> > + *   it under the terms of the GNU General Public License as published
> by
> >> > + *   the Free Software Foundation; either version 2 of the License, or
> >> > + *   (at your option) any later version.
> >> > + *
> >> > + *   This program is distributed in the hope that it will be useful,
> >> > + *   but WITHOUT ANY WARRANTY;  without even the implied warranty
> of
> >> > + *   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See
> >> > + *   the GNU General Public License for more details.
> >> > + */
> >> > +#include <linux/module.h>
> >> > +#include "smbdirect.h"
> >> > +#include "cifs_debug.h"
> >> > +
> >> > +/* SMBD version number */
> >> > +#define SMBD_V1        0x0100
> >> > +
> >> > +/* Port numbers for SMBD transport */
> >> > +#define SMB_PORT       445
> >> > +#define SMBD_PORT      5445
> >> > +
> >> > +/* Address lookup and resolve timeout in ms */
> >> > +#define RDMA_RESOLVE_TIMEOUT   5000
> >> > +
> >> > +/* SMBD negotiation timeout in seconds */ #define
> >> > +SMBD_NEGOTIATE_TIMEOUT 120
> >> > +
> >> > +/* SMBD minimum receive size and fragmented sized defined in [MS-
> >> SMBD] */
> >> > +#define SMBD_MIN_RECEIVE_SIZE          128
> >> > +#define SMBD_MIN_FRAGMENTED_SIZE       131072
> >> > +
> >> > +/*
> >> > + * Default maximum number of RDMA read/write outstanding on this
> >> > +connection
> >> > + * This value is possibly decreased during QP creation on hardware
> >> > +limit  */
> >> > +#define SMBD_CM_RESPONDER_RESOURCES    32
> >> > +
> >> > +/* Maximum number of retries on data transfer operations */
> >> > +#define SMBD_CM_RETRY                  6
> >> > +/* No need to retry on Receiver Not Ready since SMBD manages
> >> > +credits
> >> */
> >> > +#define SMBD_CM_RNR_RETRY              0
> >> > +
> >> > +/*
> >> > + * User configurable initial values per SMBD transport connection
> >> > + * as defined in [MS-SMBD] 3.1.1.1
> >> > + * Those may change after a SMBD negotiation  */
> >>
> >> Since these value are per transport connection, why they are global?
> >> Shouldn't they be inside a some structure that is created for a
> >> particular connection? Also the constants below should be defines.
> >
> > Those are configurable initial values (default values) for all connections.
> >
> > Each connection has its own values based on those initial values. But
> connection-based values can change after negotiation is done, or after
> RDMA hardware capabilities are probed.
> 
> If these are configurable values, let's add a way to actually configure them.
> Through /proc?

You mean putting them in /proc/fs/cifs?

Or how about module_param?

> 
> --
> Best regards,
> Pavel Shilovsky

[toc] | [prev] | [next] | [standalone]

#1723584 — Re: [Patch v3 02/19] CIFS: SMBD: Add SMBDirect protocol and transport constants

2017-08-30 11:24 GMT-07:00 Long Li <longli@microsoft.com>:
>> -----Original Message-----
>> From: Pavel Shilovsky [mailto:piastryyy@gmail.com]
>> Sent: Wednesday, August 30, 2017 11:19 AM
>> To: Long Li <longli@microsoft.com>
>> Cc: Steve French <sfrench@samba.org>; linux-cifs <linux-
>> cifs@vger.kernel.org>; samba-technical <samba-technical@lists.samba.org>;
>> Kernel Mailing List <linux-kernel@vger.kernel.org>; linux-
>> rdma@vger.kernel.org; Christoph Hellwig <hch@infradead.org>; Tom Talpey
>> <ttalpey@microsoft.com>; Matthew Wilcox <mawilcox@microsoft.com>
>> Subject: Re: [Patch v3 02/19] CIFS: SMBD: Add SMBDirect protocol and
>> transport constants
>>
>> 2017-08-29 16:00 GMT-07:00 Long Li <longli@microsoft.com>:
>> >> -----Original Message-----
>> >> From: Pavel Shilovsky [mailto:piastryyy@gmail.com]
>> >> Sent: Tuesday, August 29, 2017 3:45 PM
>> >> To: Long Li <longli@microsoft.com>
>> >> Cc: Steve French <sfrench@samba.org>; linux-cifs <linux-
>> >> cifs@vger.kernel.org>; samba-technical
>> >> <samba-technical@lists.samba.org>;
>> >> Kernel Mailing List <linux-kernel@vger.kernel.org>; linux-
>> >> rdma@vger.kernel.org; Christoph Hellwig <hch@infradead.org>; Tom
>> >> Talpey <ttalpey@microsoft.com>; Matthew Wilcox
>> >> <mawilcox@microsoft.com>; Long Li <longli@microsoft.com>
>> >> Subject: Re: [Patch v3 02/19] CIFS: SMBD: Add SMBDirect protocol and
>> >> transport constants
>> >>
>> >> 2017-08-29 12:28 GMT-07:00 Long Li <longli@exchange.microsoft.com>:
>> >> > From: Long Li <longli@microsoft.com>
>> >> >
>> >> > To prepare for protocol implementation, add constants and
>> >> > user-configurable values in the SMBDirect protocol.
>> >> >
>> >> > Signed-off-by: Long Li <longli@microsoft.com>
>> >> > ---
>> >> >  fs/cifs/smbdirect.c | 78
>> >> > +++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >> >  fs/cifs/smbdirect.h | 20 ++++++++++++++
>> >> >  2 files changed, 98 insertions(+)
>> >> >  create mode 100644 fs/cifs/smbdirect.c  create mode 100644
>> >> > fs/cifs/smbdirect.h
>> >> >
>> >> > diff --git a/fs/cifs/smbdirect.c b/fs/cifs/smbdirect.c new file
>> >> > mode
>> >> > 100644 index 0000000..d785bc1
>> >> > --- /dev/null
>> >> > +++ b/fs/cifs/smbdirect.c
>> >> > @@ -0,0 +1,78 @@
>> >> > +/*
>> >> > + *   Copyright (C) 2017, Microsoft Corporation.
>> >> > + *
>> >> > + *   Author(s): Long Li <longli@microsoft.com>
>> >> > + *
>> >> > + *   This program is free software;  you can redistribute it and/or modify
>> >> > + *   it under the terms of the GNU General Public License as published
>> by
>> >> > + *   the Free Software Foundation; either version 2 of the License, or
>> >> > + *   (at your option) any later version.
>> >> > + *
>> >> > + *   This program is distributed in the hope that it will be useful,
>> >> > + *   but WITHOUT ANY WARRANTY;  without even the implied warranty
>> of
>> >> > + *   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See
>> >> > + *   the GNU General Public License for more details.
>> >> > + */
>> >> > +#include <linux/module.h>
>> >> > +#include "smbdirect.h"
>> >> > +#include "cifs_debug.h"
>> >> > +
>> >> > +/* SMBD version number */
>> >> > +#define SMBD_V1        0x0100
>> >> > +
>> >> > +/* Port numbers for SMBD transport */
>> >> > +#define SMB_PORT       445
>> >> > +#define SMBD_PORT      5445
>> >> > +
>> >> > +/* Address lookup and resolve timeout in ms */
>> >> > +#define RDMA_RESOLVE_TIMEOUT   5000
>> >> > +
>> >> > +/* SMBD negotiation timeout in seconds */ #define
>> >> > +SMBD_NEGOTIATE_TIMEOUT 120
>> >> > +
>> >> > +/* SMBD minimum receive size and fragmented sized defined in [MS-
>> >> SMBD] */
>> >> > +#define SMBD_MIN_RECEIVE_SIZE          128
>> >> > +#define SMBD_MIN_FRAGMENTED_SIZE       131072
>> >> > +
>> >> > +/*
>> >> > + * Default maximum number of RDMA read/write outstanding on this
>> >> > +connection
>> >> > + * This value is possibly decreased during QP creation on hardware
>> >> > +limit  */
>> >> > +#define SMBD_CM_RESPONDER_RESOURCES    32
>> >> > +
>> >> > +/* Maximum number of retries on data transfer operations */
>> >> > +#define SMBD_CM_RETRY                  6
>> >> > +/* No need to retry on Receiver Not Ready since SMBD manages
>> >> > +credits
>> >> */
>> >> > +#define SMBD_CM_RNR_RETRY              0
>> >> > +
>> >> > +/*
>> >> > + * User configurable initial values per SMBD transport connection
>> >> > + * as defined in [MS-SMBD] 3.1.1.1
>> >> > + * Those may change after a SMBD negotiation  */
>> >>
>> >> Since these value are per transport connection, why they are global?
>> >> Shouldn't they be inside a some structure that is created for a
>> >> particular connection? Also the constants below should be defines.
>> >
>> > Those are configurable initial values (default values) for all connections.
>> >
>> > Each connection has its own values based on those initial values. But
>> connection-based values can change after negotiation is done, or after
>> RDMA hardware capabilities are probed.
>>
>> If these are configurable values, let's add a way to actually configure them.
>> Through /proc?
>
> You mean putting them in /proc/fs/cifs?
>
> Or how about module_param?

I am ok with either way. If it is safe to change them without
reloading the module let's do /proc/fs/cifs which would make it easier
to use. Otherwise - module_param is also good.

--
Best regards,
Pavel Shilovsky

[toc] | [prev] | [next] | [standalone]

#1725172 — Re: [Patch v3 02/19] CIFS: SMBD: Add SMBDirect protocol and transport constants

Probably easier to configure in /proc/fs/cifs than as module param

On Wed, Aug 30, 2017 at 1:30 PM, Pavel Shilovsky via samba-technical
<samba-technical@lists.samba.org> wrote:
> 2017-08-30 11:24 GMT-07:00 Long Li <longli@microsoft.com>:
>>> -----Original Message-----
>>> From: Pavel Shilovsky [mailto:piastryyy@gmail.com]
>>> Sent: Wednesday, August 30, 2017 11:19 AM
>>> To: Long Li <longli@microsoft.com>
>>> Cc: Steve French <sfrench@samba.org>; linux-cifs <linux-
>>> cifs@vger.kernel.org>; samba-technical <samba-technical@lists.samba.org>;
>>> Kernel Mailing List <linux-kernel@vger.kernel.org>; linux-
>>> rdma@vger.kernel.org; Christoph Hellwig <hch@infradead.org>; Tom Talpey
>>> <ttalpey@microsoft.com>; Matthew Wilcox <mawilcox@microsoft.com>
>>> Subject: Re: [Patch v3 02/19] CIFS: SMBD: Add SMBDirect protocol and
>>> transport constants
>>>
>>> 2017-08-29 16:00 GMT-07:00 Long Li <longli@microsoft.com>:
>>> >> -----Original Message-----
>>> >> From: Pavel Shilovsky [mailto:piastryyy@gmail.com]
>>> >> Sent: Tuesday, August 29, 2017 3:45 PM
>>> >> To: Long Li <longli@microsoft.com>
>>> >> Cc: Steve French <sfrench@samba.org>; linux-cifs <linux-
>>> >> cifs@vger.kernel.org>; samba-technical
>>> >> <samba-technical@lists.samba.org>;
>>> >> Kernel Mailing List <linux-kernel@vger.kernel.org>; linux-
>>> >> rdma@vger.kernel.org; Christoph Hellwig <hch@infradead.org>; Tom
>>> >> Talpey <ttalpey@microsoft.com>; Matthew Wilcox
>>> >> <mawilcox@microsoft.com>; Long Li <longli@microsoft.com>
>>> >> Subject: Re: [Patch v3 02/19] CIFS: SMBD: Add SMBDirect protocol and
>>> >> transport constants
>>> >>
>>> >> 2017-08-29 12:28 GMT-07:00 Long Li <longli@exchange.microsoft.com>:
>>> >> > From: Long Li <longli@microsoft.com>
>>> >> >
>>> >> > To prepare for protocol implementation, add constants and
>>> >> > user-configurable values in the SMBDirect protocol.
>>> >> >
>>> >> > Signed-off-by: Long Li <longli@microsoft.com>
>>> >> > ---
>>> >> >  fs/cifs/smbdirect.c | 78
>>> >> > +++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >> >  fs/cifs/smbdirect.h | 20 ++++++++++++++
>>> >> >  2 files changed, 98 insertions(+)
>>> >> >  create mode 100644 fs/cifs/smbdirect.c  create mode 100644
>>> >> > fs/cifs/smbdirect.h
>>> >> >
>>> >> > diff --git a/fs/cifs/smbdirect.c b/fs/cifs/smbdirect.c new file
>>> >> > mode
>>> >> > 100644 index 0000000..d785bc1
>>> >> > --- /dev/null
>>> >> > +++ b/fs/cifs/smbdirect.c
>>> >> > @@ -0,0 +1,78 @@
>>> >> > +/*
>>> >> > + *   Copyright (C) 2017, Microsoft Corporation.
>>> >> > + *
>>> >> > + *   Author(s): Long Li <longli@microsoft.com>
>>> >> > + *
>>> >> > + *   This program is free software;  you can redistribute it and/or modify
>>> >> > + *   it under the terms of the GNU General Public License as published
>>> by
>>> >> > + *   the Free Software Foundation; either version 2 of the License, or
>>> >> > + *   (at your option) any later version.
>>> >> > + *
>>> >> > + *   This program is distributed in the hope that it will be useful,
>>> >> > + *   but WITHOUT ANY WARRANTY;  without even the implied warranty
>>> of
>>> >> > + *   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See
>>> >> > + *   the GNU General Public License for more details.
>>> >> > + */
>>> >> > +#include <linux/module.h>
>>> >> > +#include "smbdirect.h"
>>> >> > +#include "cifs_debug.h"
>>> >> > +
>>> >> > +/* SMBD version number */
>>> >> > +#define SMBD_V1        0x0100
>>> >> > +
>>> >> > +/* Port numbers for SMBD transport */
>>> >> > +#define SMB_PORT       445
>>> >> > +#define SMBD_PORT      5445
>>> >> > +
>>> >> > +/* Address lookup and resolve timeout in ms */
>>> >> > +#define RDMA_RESOLVE_TIMEOUT   5000
>>> >> > +
>>> >> > +/* SMBD negotiation timeout in seconds */ #define
>>> >> > +SMBD_NEGOTIATE_TIMEOUT 120
>>> >> > +
>>> >> > +/* SMBD minimum receive size and fragmented sized defined in [MS-
>>> >> SMBD] */
>>> >> > +#define SMBD_MIN_RECEIVE_SIZE          128
>>> >> > +#define SMBD_MIN_FRAGMENTED_SIZE       131072
>>> >> > +
>>> >> > +/*
>>> >> > + * Default maximum number of RDMA read/write outstanding on this
>>> >> > +connection
>>> >> > + * This value is possibly decreased during QP creation on hardware
>>> >> > +limit  */
>>> >> > +#define SMBD_CM_RESPONDER_RESOURCES    32
>>> >> > +
>>> >> > +/* Maximum number of retries on data transfer operations */
>>> >> > +#define SMBD_CM_RETRY                  6
>>> >> > +/* No need to retry on Receiver Not Ready since SMBD manages
>>> >> > +credits
>>> >> */
>>> >> > +#define SMBD_CM_RNR_RETRY              0
>>> >> > +
>>> >> > +/*
>>> >> > + * User configurable initial values per SMBD transport connection
>>> >> > + * as defined in [MS-SMBD] 3.1.1.1
>>> >> > + * Those may change after a SMBD negotiation  */
>>> >>
>>> >> Since these value are per transport connection, why they are global?
>>> >> Shouldn't they be inside a some structure that is created for a
>>> >> particular connection? Also the constants below should be defines.
>>> >
>>> > Those are configurable initial values (default values) for all connections.
>>> >
>>> > Each connection has its own values based on those initial values. But
>>> connection-based values can change after negotiation is done, or after
>>> RDMA hardware capabilities are probed.
>>>
>>> If these are configurable values, let's add a way to actually configure them.
>>> Through /proc?
>>
>> You mean putting them in /proc/fs/cifs?
>>
>> Or how about module_param?
>
> I am ok with either way. If it is safe to change them without
> reloading the module let's do /proc/fs/cifs which would make it easier
> to use. Otherwise - module_param is also good.
>
> --
> Best regards,
> Pavel Shilovsky
>



-- 
Thanks,

Steve

[toc] | [prev] | [next] | [standalone]

#1723586

> -----Original Message-----
> From: Long Li [mailto:longli@exchange.microsoft.com]
> Sent: Tuesday, August 29, 2017 12:29 PM
> To: Steve French <sfrench@samba.org>; linux-cifs@vger.kernel.org; samba-
> technical@lists.samba.org; linux-kernel@vger.kernel.org; linux-
> rdma@vger.kernel.org; Christoph Hellwig <hch@infradead.org>; Tom Talpey
> <ttalpey@microsoft.com>; Matthew Wilcox <mawilcox@microsoft.com>
> Cc: Long Li <longli@microsoft.com>
> Subject: [Patch v3 00/19] CIFS: Implement SMB Direct
> 
> From: Long Li <longli@microsoft.com>
> 
> Starting with SMB2 dialect 3.0, Microsoft introduced SMBDirect transport
> protocol for transferring upper layer (SMB2) payload over RDMA via
> Infiniband, RoCE or iWARP. The prococol is published in [MS-SMBD]
> (https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmsdn
> .microsoft.com%2Fen-
> us%2Flibrary%2Fhh536346.aspx&data=02%7C01%7Clongli%40microsoft.com
> %7C73e721d48ac0402f414208d4ef1470f2%7C72f988bf86f141af91ab2d7cd011
> db47%7C1%7C0%7C636396318489728526&sdata=3ES4XY3qXqUVEJ37YUHlMJ
> QWuxFeljzOEMhGXhmO2i0%3D&reserved=0).
> 
> Patch v2 added RDMA read/write via memory registration, and addressed
> feedbacks on v1.
> 
> Patch v3 improved performance by introducing an additional queue for
> handling empty packets and reducing lock contention on IRQ path. Also
> added light weight profiling by reading TSC and addressed feedbacks on v2.

I received some requests about testing the patch. Thank you for help testing.

A snapshot of the development kernel with the patch set is at:
https://github.com/longlimsft/linux-next/tree/patch_v3

During development, it is tested on Infiniband with Mellanox ConnectX-3.

> 
> Long Li (19):
>   CIFS: Add rdma mount option
>   CIFS: SMBD: Add SMBDirect protocol and transport constants
>   CIFS: SMBD: Implement SMBDirect transport
>   CIFS: SMBD: Add SMBDirect transport to SMB connection and Makefile
>   CIFS: SMBD: Connect to SMBDirect session
>   CIFS: SMBD: Reconnect to SMBDirect session
>   CIFS: SMBD: Destroy SMBDirect session on shutdown or umount
>   CIFS: SMBD: Set SMBDirect maximum read or write size for I/O
>   CIFS: SMBD: Receive data from SMBDirect
>   CIFS: SMBD: Send data through SMBDirect
>   CIFS: SMBD: Define memory registration for I/O data
>   CIFS: SMBD: Fix the definition for SMB2_CHANNEL_RDMA_V1_INVALIDATE
>   CIFS: SMBD: Implement SMB write via RDMA read through memory
>     registration
>   CIFS: SMBD: Deregister memory on finishing SMB write
>   CIFS: SMBD: Add parameter rdata to smb2_new_read_req
>   CIFS: SMBD: Read correct returned data length for RDMA write (SMB
>     read) I/O
>   CIFS: SMBD: Implement SMB read via RDMA write through memory
>     registration
>   CIFS: SMBD: Deregister memory when finishing SMB read
>   CIFS: SMBD: Add SMBDirect debug counters
> 
>  fs/cifs/Makefile     |    2 +-
>  fs/cifs/cifs_debug.c |   84 ++
>  fs/cifs/cifsfs.c     |    2 +
>  fs/cifs/cifsglob.h   |   17 +-
>  fs/cifs/cifssmb.c    |   10 +-
>  fs/cifs/connect.c    |   46 +-
>  fs/cifs/file.c       |   10 +
>  fs/cifs/smb1ops.c    |    2 +-
>  fs/cifs/smb2ops.c    |   21 +-
>  fs/cifs/smb2pdu.c    |  114 ++-
>  fs/cifs/smb2pdu.h    |    2 +-
>  fs/cifs/smbdirect.c  | 2639
> ++++++++++++++++++++++++++++++++++++++++++++++++++
>  fs/cifs/smbdirect.h  |  324 +++++++
>  fs/cifs/transport.c  |    7 +
>  14 files changed, 3260 insertions(+), 20 deletions(-)  create mode 100644
> fs/cifs/smbdirect.c  create mode 100644 fs/cifs/smbdirect.h
> 
> --
> 2.7.4

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | linux.kernel

csiph-web