Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1415463 > unrolled thread

general preparation for NVMe over Fabrics support

Back to article view | Back to linux.kernel

Contents

  general preparation for NVMe over Fabrics support Christoph Hellwig <hch@lst.de> - 2016-06-06 23:30 +0200
    [PATCH 03/10] nvme.h: add NVM command set SQE/CQE size defines Christoph Hellwig <hch@lst.de> - 2016-06-06 23:30 +0200
    [PATCH 02/10] nvme.h: Add get_log_page command strucure Christoph Hellwig <hch@lst.de> - 2016-06-06 23:30 +0200
    [PATCH 09/10] nvme: update nvme_cancel_io() a bit Christoph Hellwig <hch@lst.de> - 2016-06-06 23:30 +0200
    [PATCH 04/10] nvme.h: add AER constants Christoph Hellwig <hch@lst.de> - 2016-06-06 23:30 +0200
    [PATCH 08/10] nvme: move the workaround for I/O queue-less controllers from PCIe to core Christoph Hellwig <hch@lst.de> - 2016-06-06 23:30 +0200
    [PATCH 01/10] nvme.h: add RTD3R, RTD3E and OAES fields Christoph Hellwig <hch@lst.de> - 2016-06-06 23:30 +0200
      Re: [PATCH 01/10] nvme.h: add RTD3R, RTD3E and OAES fields Johannes Thumshirn <jthumshirn@suse.de> - 2016-06-07 10:20 +0200
        Re: [PATCH 01/10] nvme.h: add RTD3R, RTD3E and OAES fields Johannes Thumshirn <jthumshirn@suse.de> - 2016-06-07 10:40 +0200
        Re: [PATCH 01/10] nvme.h: add RTD3R, RTD3E and OAES fields Christoph Hellwig <hch@lst.de> - 2016-06-07 12:50 +0200
    [PATCH 06/10] nvme: allow for size limitations from transport drivers Christoph Hellwig <hch@lst.de> - 2016-06-06 23:30 +0200
    [PATCH 07/10] nvme: factor out a add nvme_is_write helper Christoph Hellwig <hch@lst.de> - 2016-06-06 23:30 +0200
    Re: general preparation for NVMe over Fabrics support Keith Busch <keith.busch@intel.com> - 2016-06-09 18:40 +0200
      Re: general preparation for NVMe over Fabrics support Christoph Hellwig <hch@lst.de> - 2016-06-09 19:20 +0200

#1415463 — general preparation for NVMe over Fabrics support

This patch set adds some needed preparations for the upcoming NVMe over
Fabrics support.

Contains:
- Allow transfer size limitations for NVMe transports
- Add the get_log_page command definition required by the NVMe target
- more helpers in core code that can be used by various transports
- add some missing constants and identify attributes

[toc] | [next] | [standalone]

#1415471 — [PATCH 03/10] nvme.h: add NVM command set SQE/CQE size defines

Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
 include/linux/nvme.h | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/include/linux/nvme.h b/include/linux/nvme.h
index 9925b85..9807d98 100644
--- a/include/linux/nvme.h
+++ b/include/linux/nvme.h
@@ -50,6 +50,13 @@ enum {
 #define NVME_CMB_CQS(cmbsz)	((cmbsz) & 0x2)
 #define NVME_CMB_SQS(cmbsz)	((cmbsz) & 0x1)
 
+/*
+ * Submission and Completion Queue Entry Sizes for the NVM command set.
+ * (In bytes and specified as a power of two (2^n)).
+ */
+#define NVME_NVM_IOSQES		6
+#define NVME_NVM_IOCQES		4
+
 enum {
 	NVME_CC_ENABLE		= 1 << 0,
 	NVME_CC_CSS_NVM		= 0 << 4,
@@ -61,8 +68,8 @@ enum {
 	NVME_CC_SHN_NORMAL	= 1 << 14,
 	NVME_CC_SHN_ABRUPT	= 2 << 14,
 	NVME_CC_SHN_MASK	= 3 << 14,
-	NVME_CC_IOSQES		= 6 << 16,
-	NVME_CC_IOCQES		= 4 << 20,
+	NVME_CC_IOSQES		= NVME_NVM_IOSQES << 16,
+	NVME_CC_IOCQES		= NVME_NVM_IOCQES << 20,
 	NVME_CSTS_RDY		= 1 << 0,
 	NVME_CSTS_CFS		= 1 << 1,
 	NVME_CSTS_NSSRO		= 1 << 4,
-- 
2.1.4

[toc] | [prev] | [next] | [standalone]

#1415472 — [PATCH 02/10] nvme.h: Add get_log_page command strucure

From: Armen Baloyan <armenx.baloyan@intel.com>

Add get_log_page command structure and a corresponding entry in
nvme_command union

Signed-off-by: Armen Baloyan <armenx.baloyan@intel.com>
Reviewed-by: Jay Freyensee <james.p.freyensee@intel.com>
Reviewed--by: Sagi Grimberg <sagi@grimberg.me>
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
 include/linux/nvme.h | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/include/linux/nvme.h b/include/linux/nvme.h
index ff5ebc3..9925b85 100644
--- a/include/linux/nvme.h
+++ b/include/linux/nvme.h
@@ -519,6 +519,24 @@ struct nvme_format_cmd {
 	__u32			rsvd11[5];
 };
 
+struct nvme_get_log_page_command {
+	__u8			opcode;
+	__u8			flags;
+	__u16			command_id;
+	__le32			nsid;
+	__u64			rsvd2[2];
+	__le64			prp1;
+	__le64			prp2;
+	__u8			lid;
+	__u8			rsvd10;
+	__le16			numdl;
+	__le16			numdu;
+	__u16			rsvd11;
+	__le32			lpol;
+	__le32			lpou;
+	__u32			rsvd14[2];
+};
+
 struct nvme_command {
 	union {
 		struct nvme_common_command common;
@@ -532,6 +550,7 @@ struct nvme_command {
 		struct nvme_format_cmd format;
 		struct nvme_dsm_cmd dsm;
 		struct nvme_abort_cmd abort;
+		struct nvme_get_log_page_command get_log_page;
 	};
 };
 
-- 
2.1.4

[toc] | [prev] | [next] | [standalone]

#1415473 — [PATCH 09/10] nvme: update nvme_cancel_io() a bit

From: Ming Lin <ming.l@samsung.com>

Pass in nvme_ctrl, so it can be used by the fabrics drivers also.
Also update to upstream version and rename it to nvme_cancel_request.

Signed-off-by: Ming Lin <ming.l@samsung.com>
Signed-off-by: Sagi Grimberg <sagi@grimberg.me>
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
 drivers/nvme/host/pci.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index cf7ea73..5ae5804 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -919,14 +919,14 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req, bool reserved)
 	return BLK_EH_RESET_TIMER;
 }
 
-static void nvme_cancel_io(struct request *req, void *data, bool reserved)
+static void nvme_cancel_request(struct request *req, void *data, bool reserved)
 {
 	int status;
 
 	if (!blk_mq_request_started(req))
 		return;
 
-	dev_dbg_ratelimited(((struct nvme_dev *) data)->ctrl.device,
+	dev_dbg_ratelimited(((struct nvme_ctrl *) data)->device,
 				"Cancelling I/O %d", req->tag);
 
 	status = NVME_SC_ABORT_REQ;
@@ -1714,8 +1714,9 @@ static void nvme_dev_disable(struct nvme_dev *dev, bool shutdown)
 	}
 	nvme_pci_disable(dev);
 
-	blk_mq_tagset_busy_iter(&dev->tagset, nvme_cancel_io, dev);
-	blk_mq_tagset_busy_iter(&dev->admin_tagset, nvme_cancel_io, dev);
+	blk_mq_tagset_busy_iter(&dev->tagset, nvme_cancel_request, &dev->ctrl);
+	blk_mq_tagset_busy_iter(&dev->admin_tagset,
+				nvme_cancel_request, &dev->ctrl);
 	mutex_unlock(&dev->shutdown_lock);
 }
 
-- 
2.1.4

[toc] | [prev] | [next] | [standalone]

#1415476 — [PATCH 04/10] nvme.h: add AER constants

Reviewed-by: Jay Freyensee <james.p.freyensee@intel.com>
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
Reviewed-by: Ming Lin <ming.l@ssi.samsung.com>
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
 include/linux/nvme.h | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/include/linux/nvme.h b/include/linux/nvme.h
index 9807d98..a9b8c7b 100644
--- a/include/linux/nvme.h
+++ b/include/linux/nvme.h
@@ -284,6 +284,12 @@ struct nvme_reservation_status {
 	} regctl_ds[];
 };
 
+enum nvme_async_event_type {
+	NVME_AER_TYPE_ERROR	= 0,
+	NVME_AER_TYPE_SMART	= 1,
+	NVME_AER_TYPE_NOTICE	= 2,
+};
+
 /* I/O commands */
 
 enum nvme_opcode {
-- 
2.1.4

[toc] | [prev] | [next] | [standalone]

#1415480 — [PATCH 08/10] nvme: move the workaround for I/O queue-less controllers from PCIe to core

We want to apply this to Fabrics drivers as well, so move it to common
code.

Reviewed-by: Jay Freyensee <james.p.freyensee@intel.com>
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
Tested-by: Ming Lin <ming.l@ssi.samsung.com>
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
 drivers/nvme/host/core.c | 17 ++++++++++++++---
 drivers/nvme/host/pci.c  | 10 +---------
 2 files changed, 15 insertions(+), 12 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index ffd9910..972029a 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -557,11 +557,22 @@ int nvme_set_queue_count(struct nvme_ctrl *ctrl, int *count)
 
 	status = nvme_set_features(ctrl, NVME_FEAT_NUM_QUEUES, q_count, 0,
 			&result);
-	if (status)
+	if (status < 0)
 		return status;
 
-	nr_io_queues = min(result & 0xffff, result >> 16) + 1;
-	*count = min(*count, nr_io_queues);
+	/*
+	 * Degraded controllers might return an error when setting the queue
+	 * count.  We still want to be able to bring them online and offer
+	 * access to the admin queue, as that might be only way to fix them up.
+	 */
+	if (status > 0) {
+		dev_err(ctrl->dev, "Could not set queue count (%d)\n", status);
+		*count = 0;
+	} else {
+		nr_io_queues = min(result & 0xffff, result >> 16) + 1;
+		*count = min(*count, nr_io_queues);
+	}
+
 	return 0;
 }
 EXPORT_SYMBOL_GPL(nvme_set_queue_count);
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 78dca31..cf7ea73 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -1399,16 +1399,8 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)
 	if (result < 0)
 		return result;
 
-	/*
-	 * Degraded controllers might return an error when setting the queue
-	 * count.  We still want to be able to bring them online and offer
-	 * access to the admin queue, as that might be only way to fix them up.
-	 */
-	if (result > 0) {
-		dev_err(dev->ctrl.device,
-			"Could not set queue count (%d)\n", result);
+	if (nr_io_queues == 0)
 		return 0;
-	}
 
 	if (dev->cmb && NVME_CMB_SQS(dev->cmbsz)) {
 		result = nvme_cmb_qdepth(dev, nr_io_queues,
-- 
2.1.4

[toc] | [prev] | [next] | [standalone]

#1415483 — [PATCH 01/10] nvme.h: add RTD3R, RTD3E and OAES fields

These have been added in NVMe 1.2 and we'll need at least oaes for the
NVMe target driver.

Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
Reviewed-by: Jay Freyensee <james.p.freyensee@intel.com>
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
 include/linux/nvme.h | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/include/linux/nvme.h b/include/linux/nvme.h
index 7d51b29..ff5ebc3 100644
--- a/include/linux/nvme.h
+++ b/include/linux/nvme.h
@@ -107,7 +107,10 @@ struct nvme_id_ctrl {
 	__u8			mdts;
 	__le16			cntlid;
 	__le32			ver;
-	__u8			rsvd84[172];
+	__le32			rtd3r;
+	__le32			rtd3e;
+	__le32			oaes;
+	__u8			rsvd96[160];
 	__le16			oacs;
 	__u8			acl;
 	__u8			aerl;
-- 
2.1.4

[toc] | [prev] | [next] | [standalone]

#1415830 — Re: [PATCH 01/10] nvme.h: add RTD3R, RTD3E and OAES fields

On Mon, Jun 06, 2016 at 11:20:43PM +0200, Christoph Hellwig wrote:
> These have been added in NVMe 1.2 and we'll need at least oaes for the
> NVMe target driver.
> 
> Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
> Reviewed-by: Jay Freyensee <james.p.freyensee@intel.com>
> Signed-off-by: Christoph Hellwig <hch@lst.de>
> ---
>  include/linux/nvme.h | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/include/linux/nvme.h b/include/linux/nvme.h
> index 7d51b29..ff5ebc3 100644
> --- a/include/linux/nvme.h
> +++ b/include/linux/nvme.h
> @@ -107,7 +107,10 @@ struct nvme_id_ctrl {
>  	__u8			mdts;
>  	__le16			cntlid;
>  	__le32			ver;
> -	__u8			rsvd84[172];
> +	__le32			rtd3r;
> +	__le32			rtd3e;
> +	__le32			oaes;

Just out of curiosity, why not CTRATT as well?

> +	__u8			rsvd96[160];
>  	__le16			oacs;
>  	__u8			acl;
>  	__u8			aerl;
> -- 
> 2.1.4
> 
> 
> _______________________________________________
> Linux-nvme mailing list
> Linux-nvme@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-nvme

-- 
Johannes Thumshirn                                          Storage
jthumshirn@suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Felix Imendörffer, Jane Smithard, Graham Norton
HRB 21284 (AG Nürnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850

[toc] | [prev] | [next] | [standalone]

#1415861 — Re: [PATCH 01/10] nvme.h: add RTD3R, RTD3E and OAES fields

On Tue, Jun 07, 2016 at 10:10:09AM +0200, Johannes Thumshirn wrote:
> On Mon, Jun 06, 2016 at 11:20:43PM +0200, Christoph Hellwig wrote:
> > These have been added in NVMe 1.2 and we'll need at least oaes for the
> > NVMe target driver.
> > 
> > Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
> > Reviewed-by: Jay Freyensee <james.p.freyensee@intel.com>
> > Signed-off-by: Christoph Hellwig <hch@lst.de>
> > ---
> >  include/linux/nvme.h | 5 ++++-
> >  1 file changed, 4 insertions(+), 1 deletion(-)
> > 
> > diff --git a/include/linux/nvme.h b/include/linux/nvme.h
> > index 7d51b29..ff5ebc3 100644
> > --- a/include/linux/nvme.h
> > +++ b/include/linux/nvme.h
> > @@ -107,7 +107,10 @@ struct nvme_id_ctrl {
> >  	__u8			mdts;
> >  	__le16			cntlid;
> >  	__le32			ver;
> > -	__u8			rsvd84[172];
> > +	__le32			rtd3r;
> > +	__le32			rtd3e;
> > +	__le32			oaes;
> 
> Just out of curiosity, why not CTRATT as well?

OK found the patch adding it myself.

Sorry,
	Johannes
-- 
Johannes Thumshirn                                          Storage
jthumshirn@suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Felix Imendörffer, Jane Smithard, Graham Norton
HRB 21284 (AG Nürnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850

[toc] | [prev] | [next] | [standalone]

#1415981 — Re: [PATCH 01/10] nvme.h: add RTD3R, RTD3E and OAES fields

On Tue, Jun 07, 2016 at 10:10:09AM +0200, Johannes Thumshirn wrote:
> Just out of curiosity, why not CTRATT as well?

ctratt is part of the fabrics spec and not the NVMe 1.2 spec, and we tried
to keep those separate.

[toc] | [prev] | [next] | [standalone]

#1415487 — [PATCH 06/10] nvme: allow for size limitations from transport drivers

Some transport drivers may have a lower transfer size than
the controller. So allow the transport to set it in the
controller max_hw_sectors.

Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
 drivers/nvme/host/core.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 1a51584..b582c11 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -1088,6 +1088,7 @@ int nvme_init_identify(struct nvme_ctrl *ctrl)
 	struct nvme_id_ctrl *id;
 	u64 cap;
 	int ret, page_shift;
+	u32 max_hw_sectors;
 
 	ret = ctrl->ops->reg_read32(ctrl, NVME_REG_VS, &ctrl->vs);
 	if (ret) {
@@ -1120,9 +1121,11 @@ int nvme_init_identify(struct nvme_ctrl *ctrl)
 	memcpy(ctrl->model, id->mn, sizeof(id->mn));
 	memcpy(ctrl->firmware_rev, id->fr, sizeof(id->fr));
 	if (id->mdts)
-		ctrl->max_hw_sectors = 1 << (id->mdts + page_shift - 9);
+		max_hw_sectors = 1 << (id->mdts + page_shift - 9);
 	else
-		ctrl->max_hw_sectors = UINT_MAX;
+		max_hw_sectors = UINT_MAX;
+	ctrl->max_hw_sectors =
+		min_not_zero(ctrl->max_hw_sectors, max_hw_sectors);
 
 	if ((ctrl->quirks & NVME_QUIRK_STRIPE_SIZE) && id->vs[3]) {
 		unsigned int max_hw_sectors;
-- 
2.1.4

[toc] | [prev] | [next] | [standalone]

#1415488 — [PATCH 07/10] nvme: factor out a add nvme_is_write helper

Centralize the check if a given NVMe command reads or writes data.

Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
Reviewed-by: Jay Freyensee <james.p.freyensee@intel.com>
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
 drivers/nvme/host/core.c | 5 ++---
 include/linux/nvme.h     | 5 +++++
 2 files changed, 7 insertions(+), 3 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index b582c11..ffd9910 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -176,10 +176,9 @@ EXPORT_SYMBOL_GPL(nvme_requeue_req);
 struct request *nvme_alloc_request(struct request_queue *q,
 		struct nvme_command *cmd, unsigned int flags)
 {
-	bool write = cmd->common.opcode & 1;
 	struct request *req;
 
-	req = blk_mq_alloc_request(q, write, flags);
+	req = blk_mq_alloc_request(q, nvme_is_write(cmd), flags);
 	if (IS_ERR(req))
 		return req;
 
@@ -344,7 +343,7 @@ int __nvme_submit_user_cmd(struct request_queue *q, struct nvme_command *cmd,
 		void __user *meta_buffer, unsigned meta_len, u32 meta_seed,
 		u32 *result, unsigned timeout)
 {
-	bool write = cmd->common.opcode & 1;
+	bool write = nvme_is_write(cmd);
 	struct nvme_completion cqe;
 	struct nvme_ns *ns = q->queuedata;
 	struct gendisk *disk = ns ? ns->disk : NULL;
diff --git a/include/linux/nvme.h b/include/linux/nvme.h
index 2b82f05..dc815cc 100644
--- a/include/linux/nvme.h
+++ b/include/linux/nvme.h
@@ -590,6 +590,11 @@ struct nvme_command {
 	};
 };
 
+static inline bool nvme_is_write(struct nvme_command *cmd)
+{
+	return cmd->common.opcode & 1;
+}
+
 enum {
 	NVME_SC_SUCCESS			= 0x0,
 	NVME_SC_INVALID_OPCODE		= 0x1,
-- 
2.1.4

[toc] | [prev] | [next] | [standalone]

#1418477

On Mon, Jun 06, 2016 at 11:20:42PM +0200, Christoph Hellwig wrote:
> This patch set adds some needed preparations for the upcoming NVMe over
> Fabrics support.
> 
> Contains:
> - Allow transfer size limitations for NVMe transports
> - Add the get_log_page command definition required by the NVMe target
> - more helpers in core code that can be used by various transports
> - add some missing constants and identify attributes

This series looks good.

Reviewed-by: Keith Busch <keith.busch@inte.com>

For patch ordering, I had thought there were no issues with these
I submitted:

  http://lists.infradead.org/pipermail/linux-nvme/2016-May/004704.html
  http://lists.infradead.org/pipermail/linux-nvme/2016-May/004709.html

It's the ones adding the SCHED_RESET state to fixup synchronizing
continuous user initiated resets, and think it should still go in for 4.7.
There will be a minor conflict with 10/10 in this set it does go first.

Anyway, moving on to the Fabrics library support set!

[toc] | [prev] | [next] | [standalone]

#1418506

On Thu, Jun 09, 2016 at 12:38:55PM -0400, Keith Busch wrote:
> For patch ordering, I had thought there were no issues with these
> I submitted:
> 
>   http://lists.infradead.org/pipermail/linux-nvme/2016-May/004704.html
>   http://lists.infradead.org/pipermail/linux-nvme/2016-May/004709.html
> 
> It's the ones adding the SCHED_RESET state to fixup synchronizing
> continuous user initiated resets, and think it should still go in for 4.7.
> There will be a minor conflict with 10/10 in this set it does go first.

The 4.8 tree has forked already. I'm fine with adding these to 4.7,
but we'll also need them in 4.8 going forward.

Jens: are you fine with merging them for 4.7 for now, and merging them
for 4.8 separately once I've finished the fabrics bits?

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web