Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1556968 > unrolled thread

[PATCHSET v6] blk-mq scheduling framework

Started byJens Axboe <axboe@fb.com>
First post2017-01-11 22:50 +0100
Last post2017-01-15 11:20 +0100
Articles 20 on this page of 21 — 5 participants

Back to article view | Back to linux.kernel

Contents

  [PATCHSET v6] blk-mq scheduling framework Jens Axboe <axboe@fb.com> - 2017-01-11 22:50 +0100
    [PATCH 10/10] blk-mq-sched: allow setting of default IO scheduler Jens Axboe <axboe@fb.com> - 2017-01-11 22:50 +0100
    Re: [PATCH 08/10] blk-mq-sched: add framework for MQ capable IO  schedulers Jens Axboe <axboe@fb.com> - 2017-01-12 23:10 +0100
    Re: [PATCHSET v6] blk-mq scheduling framework Hannes Reinecke <hare@suse.de> - 2017-01-13 09:20 +0100
      Re: [PATCHSET v6] blk-mq scheduling framework Hannes Reinecke <hare@suse.de> - 2017-01-13 12:10 +0100
        Re: [PATCHSET v6] blk-mq scheduling framework Hannes Reinecke <hare@suse.de> - 2017-01-13 13:20 +0100
          Re: [PATCHSET v6] blk-mq scheduling framework Jens Axboe <axboe@fb.com> - 2017-01-13 16:10 +0100
        Re: [PATCHSET v6] blk-mq scheduling framework Jens Axboe <axboe@fb.com> - 2017-01-13 16:10 +0100
        Re: [PATCHSET v6] blk-mq scheduling framework Hannes Reinecke <hare@suse.de> - 2017-01-13 16:40 +0100
          Re: [PATCHSET v6] blk-mq scheduling framework Hannes Reinecke <hare@suse.de> - 2017-01-13 17:00 +0100
            Re: [PATCHSET v6] blk-mq scheduling framework Jens Axboe <axboe@kernel.dk> - 2017-01-13 17:10 +0100
              Re: [PATCHSET v6] blk-mq scheduling framework Jens Axboe <axboe@kernel.dk> - 2017-01-13 22:50 +0100
              Re: [PATCHSET v6] blk-mq scheduling framework Hannes Reinecke <hare@suse.de> - 2017-01-16 09:20 +0100
                Re: [PATCHSET v6] blk-mq scheduling framework Jens Axboe <axboe@kernel.dk> - 2017-01-16 16:20 +0100
                  Re: [PATCHSET v6] blk-mq scheduling framework Jens Axboe <axboe@kernel.dk> - 2017-01-16 16:50 +0100
                Re: [PATCHSET v6] blk-mq scheduling framework Jens Axboe <axboe@kernel.dk> - 2017-01-16 16:20 +0100
    Re: [PATCHSET v6] blk-mq scheduling framework Hannes Reinecke <hare@suse.de> - 2017-01-13 11:20 +0100
    Re: [PATCH 08/10] blk-mq-sched: add framework for MQ capable IO  schedulers Hannes Reinecke <hare@suse.de> - 2017-01-13 12:20 +0100
      Re: [PATCH 08/10] blk-mq-sched: add framework for MQ capable IO  schedulers Omar Sandoval <osandov@osandov.com> - 2017-01-13 17:50 +0100
        Re: [PATCH 08/10] blk-mq-sched: add framework for MQ capable IO  schedulers Hannes Reinecke <hare@suse.de> - 2017-01-13 18:50 +0100
    Re: [PATCHSET v6] blk-mq scheduling framework Paolo Valente <paolo.valente@linaro.org> - 2017-01-15 11:20 +0100

Page 1 of 2  [1] 2  Next page →

#1556968 — [PATCHSET v6] blk-mq scheduling framework

FromJens Axboe <axboe@fb.com>
Date2017-01-11 22:50 +0100
Subject[PATCHSET v6] blk-mq scheduling framework
Message-ID<sYyWl-3GV-7@gated-at.bofh.it>
Another year, another posting of this patchset. The previous posting
was here:

https://www.spinics.net/lists/kernel/msg2406106.html

(yes, I've skipped v5, it was fixes on top of v4, not the rework).

I've reworked bits of this to get rid of the shadow requests, thanks
to Bart for the inspiration. The missing piece, for me, was the fact
that we have the tags->rqs[] indirection array already. I've done this
somewhat differently, though, by having the internal scheduler tag
map be allocated/torn down when an IO scheduler is attached or
detached. This also means that when we run without a scheduler, we
don't have to do double tag allocations, it'll work like before.

The patchset applies on top of 4.10-rc3, or can be pulled here:

git://git.kernel.dk/linux-block blk-mq-sched.6


 block/Kconfig.iosched    |   50 ++++
 block/Makefile           |    3 
 block/blk-core.c         |   19 -
 block/blk-exec.c         |    3 
 block/blk-flush.c        |   15 -
 block/blk-ioc.c          |   12 
 block/blk-merge.c        |    4 
 block/blk-mq-sched.c     |  354 +++++++++++++++++++++++++++++
 block/blk-mq-sched.h     |  157 ++++++++++++
 block/blk-mq-sysfs.c     |   13 +
 block/blk-mq-tag.c       |   58 ++--
 block/blk-mq-tag.h       |    4 
 block/blk-mq.c           |  413 +++++++++++++++++++---------------
 block/blk-mq.h           |   40 +++
 block/blk-tag.c          |    1 
 block/blk.h              |   26 +-
 block/cfq-iosched.c      |    2 
 block/deadline-iosched.c |    2 
 block/elevator.c         |  247 +++++++++++++++-----
 block/mq-deadline.c      |  569 +++++++++++++++++++++++++++++++++++++++++++++++
 block/noop-iosched.c     |    2 
 drivers/nvme/host/pci.c  |    1 
 include/linux/blk-mq.h   |    9 
 include/linux/blkdev.h   |    6 
 include/linux/elevator.h |   36 ++
 25 files changed, 1732 insertions(+), 314 deletions(-)

-- 
Jens Axboe

[toc] | [next] | [standalone]

#1556969 — [PATCH 10/10] blk-mq-sched: allow setting of default IO scheduler

FromJens Axboe <axboe@fb.com>
Date2017-01-11 22:50 +0100
Subject[PATCH 10/10] blk-mq-sched: allow setting of default IO scheduler
Message-ID<sYyWm-3GV-35@gated-at.bofh.it>
In reply to#1556968
Add Kconfig entries to manage what devices get assigned an MQ
scheduler, and add a blk-mq flag for drivers to opt out of scheduling.
The latter is useful for admin type queues that still allocate a blk-mq
queue and tag set, but aren't use for normal IO.

Signed-off-by: Jens Axboe <axboe@fb.com>
---
 block/Kconfig.iosched   | 56 +++++++++++++++++++++++++++++++++++++++++++------
 block/blk-mq-sched.c    | 20 ++++++++++++++++++
 block/blk-mq-sched.h    |  2 ++
 block/blk-mq.c          |  8 +++++++
 block/elevator.c        |  8 ++++++-
 drivers/nvme/host/pci.c |  1 +
 include/linux/blk-mq.h  |  1 +
 7 files changed, 89 insertions(+), 7 deletions(-)

diff --git a/block/Kconfig.iosched b/block/Kconfig.iosched
index 490ef2850fae..0715ce93daef 100644
--- a/block/Kconfig.iosched
+++ b/block/Kconfig.iosched
@@ -32,12 +32,6 @@ config IOSCHED_CFQ
 
 	  This is the default I/O scheduler.
 
-config MQ_IOSCHED_DEADLINE
-	tristate "MQ deadline I/O scheduler"
-	default y
-	---help---
-	  MQ version of the deadline IO scheduler.
-
 config CFQ_GROUP_IOSCHED
 	bool "CFQ Group Scheduling support"
 	depends on IOSCHED_CFQ && BLK_CGROUP
@@ -69,6 +63,56 @@ config DEFAULT_IOSCHED
 	default "cfq" if DEFAULT_CFQ
 	default "noop" if DEFAULT_NOOP
 
+config MQ_IOSCHED_DEADLINE
+	tristate "MQ deadline I/O scheduler"
+	default y
+	---help---
+	  MQ version of the deadline IO scheduler.
+
+config MQ_IOSCHED_NONE
+	bool
+	default y
+
+choice
+	prompt "Default single-queue blk-mq I/O scheduler"
+	default DEFAULT_SQ_NONE
+	help
+	  Select the I/O scheduler which will be used by default for blk-mq
+	  managed block devices with a single queue.
+
+	config DEFAULT_SQ_DEADLINE
+		bool "MQ Deadline" if MQ_IOSCHED_DEADLINE=y
+
+	config DEFAULT_SQ_NONE
+		bool "None"
+
+endchoice
+
+config DEFAULT_SQ_IOSCHED
+	string
+	default "mq-deadline" if DEFAULT_SQ_DEADLINE
+	default "none" if DEFAULT_SQ_NONE
+
+choice
+	prompt "Default multi-queue blk-mq I/O scheduler"
+	default DEFAULT_MQ_NONE
+	help
+	  Select the I/O scheduler which will be used by default for blk-mq
+	  managed block devices with multiple queues.
+
+	config DEFAULT_MQ_DEADLINE
+		bool "MQ Deadline" if MQ_IOSCHED_DEADLINE=y
+
+	config DEFAULT_MQ_NONE
+		bool "None"
+
+endchoice
+
+config DEFAULT_MQ_IOSCHED
+	string
+	default "mq-deadline" if DEFAULT_MQ_DEADLINE
+	default "none" if DEFAULT_MQ_NONE
+
 endmenu
 
 endif
diff --git a/block/blk-mq-sched.c b/block/blk-mq-sched.c
index 44cf30eb1589..26e9e20f67ce 100644
--- a/block/blk-mq-sched.c
+++ b/block/blk-mq-sched.c
@@ -332,3 +332,23 @@ int blk_mq_sched_teardown(struct request_queue *q)
 
 	return 0;
 }
+
+int blk_mq_sched_init(struct request_queue *q)
+{
+	int ret;
+
+#if defined(CONFIG_DEFAULT_SQ_NONE)
+	if (q->nr_hw_queues == 1)
+		return 0;
+#endif
+#if defined(CONFIG_DEFAULT_MQ_NONE)
+	if (q->nr_hw_queues > 1)
+		return 0;
+#endif
+
+	mutex_lock(&q->sysfs_lock);
+	ret = elevator_init(q, NULL);
+	mutex_unlock(&q->sysfs_lock);
+
+	return ret;
+}
diff --git a/block/blk-mq-sched.h b/block/blk-mq-sched.h
index 68d6a202b827..77859eae19c9 100644
--- a/block/blk-mq-sched.h
+++ b/block/blk-mq-sched.h
@@ -25,6 +25,8 @@ void blk_mq_sched_dispatch_requests(struct blk_mq_hw_ctx *hctx);
 int blk_mq_sched_setup(struct request_queue *q);
 int blk_mq_sched_teardown(struct request_queue *q);
 
+int blk_mq_sched_init(struct request_queue *q);
+
 static inline bool
 blk_mq_sched_bio_merge(struct request_queue *q, struct bio *bio)
 {
diff --git a/block/blk-mq.c b/block/blk-mq.c
index 3180b5fac88c..0dcd593e4ddd 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -2227,6 +2227,14 @@ struct request_queue *blk_mq_init_allocated_queue(struct blk_mq_tag_set *set,
 	mutex_unlock(&all_q_mutex);
 	put_online_cpus();
 
+	if (!(set->flags & BLK_MQ_F_NO_SCHED)) {
+		int ret;
+
+		ret = blk_mq_sched_init(q);
+		if (ret)
+			return ERR_PTR(ret);
+	}
+
 	return q;
 
 err_hctxs:
diff --git a/block/elevator.c b/block/elevator.c
index 79e74da26343..b3ea721e51b4 100644
--- a/block/elevator.c
+++ b/block/elevator.c
@@ -219,7 +219,13 @@ int elevator_init(struct request_queue *q, char *name)
 	}
 
 	if (!e) {
-		e = elevator_get(CONFIG_DEFAULT_IOSCHED, false);
+		if (q->mq_ops && q->nr_hw_queues == 1)
+			e = elevator_get(CONFIG_DEFAULT_SQ_IOSCHED, false);
+		else if (q->mq_ops)
+			e = elevator_get(CONFIG_DEFAULT_MQ_IOSCHED, false);
+		else
+			e = elevator_get(CONFIG_DEFAULT_IOSCHED, false);
+
 		if (!e) {
 			printk(KERN_ERR
 				"Default I/O scheduler not found. " \
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 19beeb7b2ac2..e1b4e603b1cf 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -1181,6 +1181,7 @@ static int nvme_alloc_admin_tags(struct nvme_dev *dev)
 		dev->admin_tagset.timeout = ADMIN_TIMEOUT;
 		dev->admin_tagset.numa_node = dev_to_node(dev->dev);
 		dev->admin_tagset.cmd_size = nvme_cmd_size(dev);
+		dev->admin_tagset.flags = BLK_MQ_F_NO_SCHED;
 		dev->admin_tagset.driver_data = dev;
 
 		if (blk_mq_alloc_tag_set(&dev->admin_tagset))
diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h
index 63569eb46d15..8e4df3d6c8cd 100644
--- a/include/linux/blk-mq.h
+++ b/include/linux/blk-mq.h
@@ -153,6 +153,7 @@ enum {
 	BLK_MQ_F_SG_MERGE	= 1 << 2,
 	BLK_MQ_F_DEFER_ISSUE	= 1 << 4,
 	BLK_MQ_F_BLOCKING	= 1 << 5,
+	BLK_MQ_F_NO_SCHED	= 1 << 6,
 	BLK_MQ_F_ALLOC_POLICY_START_BIT = 8,
 	BLK_MQ_F_ALLOC_POLICY_BITS = 1,
 
-- 
2.7.4

[toc] | [prev] | [next] | [standalone]

#1557856 — Re: [PATCH 08/10] blk-mq-sched: add framework for MQ capable IO schedulers

FromJens Axboe <axboe@fb.com>
Date2017-01-12 23:10 +0100
SubjectRe: [PATCH 08/10] blk-mq-sched: add framework for MQ capable IO schedulers
Message-ID<sYVJg-Zh-23@gated-at.bofh.it>
In reply to#1556968
On Thu, Jan 12 2017, Bart Van Assche wrote:
> On Wed, 2017-01-11 at 14:40 -0700, Jens Axboe wrote:
> > @@ -451,11 +456,11 @@ void blk_insert_flush(struct request *rq)
> >  	 * processed directly without going through flush machinery.  Queue
> >  	 * for normal execution.
> >  	 */
> > -	if ((policy & REQ_FSEQ_DATA) &&
> > -	    !(policy & (REQ_FSEQ_PREFLUSH | REQ_FSEQ_POSTFLUSH))) {
> > -		if (q->mq_ops) {
> > -			blk_mq_insert_request(rq, false, true, false);
> > -		} else
> > +	if (((policy & REQ_FSEQ_DATA) &&
> > +	     !(policy & (REQ_FSEQ_PREFLUSH | REQ_FSEQ_POSTFLUSH)))) {
> > +		if (q->mq_ops)
> > +			blk_mq_sched_insert_request(rq, false, true, false);
> > +		else
> >  			list_add_tail(&rq->queuelist, &q->queue_head);
> >  		return;
> >  	}
> 
> Not that it really matters, but this change adds a pair of parentheses --
> "if (e)" is changed into "if ((e))". Is this necessary?

I fixed that up earlier today, as I noticed the same. So that's gone in
the current -git tree.

> > +void blk_mq_sched_free_hctx_data(struct request_queue *q,
> > +				 void (*exit)(struct blk_mq_hw_ctx *))
> > +{
> > +	struct blk_mq_hw_ctx *hctx;
> > +	int i;
> > +
> > +	queue_for_each_hw_ctx(q, hctx, i) {
> > +		if (exit)
> > +			exit(hctx);
> > +		kfree(hctx->sched_data);
> > +		hctx->sched_data = NULL;
> > +	}
> > +}
> > +EXPORT_SYMBOL_GPL(blk_mq_sched_free_hctx_data);
> > +
> > +int blk_mq_sched_init_hctx_data(struct request_queue *q, size_t size,
> > +				int (*init)(struct blk_mq_hw_ctx *),
> > +				void (*exit)(struct blk_mq_hw_ctx *))
> > +{
> > +	struct blk_mq_hw_ctx *hctx;
> > +	int ret;
> > +	int i;
> > +
> > +	queue_for_each_hw_ctx(q, hctx, i) {
> > +		hctx->sched_data = kmalloc_node(size, GFP_KERNEL, hctx->numa_node);
> > +		if (!hctx->sched_data) {
> > +			ret = -ENOMEM;
> > +			goto error;
> > +		}
> > +
> > +		if (init) {
> > +			ret = init(hctx);
> > +			if (ret) {
> > +				/*
> > +				 * We don't want to give exit() a partially
> > +				 * initialized sched_data. init() must clean up
> > +				 * if it fails.
> > +				 */
> > +				kfree(hctx->sched_data);
> > +				hctx->sched_data = NULL;
> > +				goto error;
> > +			}
> > +		}
> > +	}
> > +
> > +	return 0;
> > +error:
> > +	blk_mq_sched_free_hctx_data(q, exit);
> > +	return ret;
> > +}
> 
> If one of the init() calls by blk_mq_sched_init_hctx_data() fails then
> blk_mq_sched_free_hctx_data() will call exit() even for hctx's for which
> init() has not been called. How about changing "if (exit)" into "if (exit &&
> hctx->sched_data)" such that exit() is only called for hctx's for which
> init() has been called?

Good point, I'll make that change to the exit function.

> > +struct request *blk_mq_sched_get_request(struct request_queue *q,
> > +					 struct bio *bio,
> > +					 unsigned int op,
> > +					 struct blk_mq_alloc_data *data)
> > +{
> > +	struct elevator_queue *e = q->elevator;
> > +	struct blk_mq_hw_ctx *hctx;
> > +	struct blk_mq_ctx *ctx;
> > +	struct request *rq;
> > +
> > +	blk_queue_enter_live(q);
> > +	ctx = blk_mq_get_ctx(q);
> > +	hctx = blk_mq_map_queue(q, ctx->cpu);
> > +
> > +	blk_mq_set_alloc_data(data, q, 0, ctx, hctx);
> > +
> > +	if (e) {
> > +		data->flags |= BLK_MQ_REQ_INTERNAL;
> > +		if (e->type->ops.mq.get_request)
> > +			rq = e->type->ops.mq.get_request(q, op, data);
> > +		else
> > +			rq = __blk_mq_alloc_request(data, op);
> > +	} else {
> > +		rq = __blk_mq_alloc_request(data, op);
> > +		if (rq) {
> > +			rq->tag = rq->internal_tag;
> > +			rq->internal_tag = -1;
> > +		}
> > +	}
> > +
> > +	if (rq) {
> > +		rq->elv.icq = NULL;
> > +		if (e && e->type->icq_cache)
> > +			blk_mq_sched_assign_ioc(q, rq, bio);
> > +		data->hctx->queued++;
> > +		return rq;
> > +	}
> > +
> > +	blk_queue_exit(q);
> > +	return NULL;
> > +}
> 
> The "rq->tag = rq->internal_tag; rq->internal_tag = -1;" occurs not only
> here but also in blk_mq_alloc_request_hctx(). Has it been considered to move
> that code into __blk_mq_alloc_request()?

Yes, it's in two locations. I wanted to keep it out of
__blk_mq_alloc_request(), so we can still use that for normal tag
allocations. But maybe it's better for __blk_mq_alloc_request() to just
do:

        if (flags & BLK_MQ_REQ_INTERNAL) {
                rq->tag = -1;
                rq->internal_tag = tag;
        } else {
                rq->tag = tag;
                rq->internal_tag = -1;
        }

and handle it directly in there. What do you think?

> @@ -223,14 +225,17 @@ struct request *__blk_mq_alloc_request(struct blk_mq_alloc_data *data,
> >  
> >  	tag = blk_mq_get_tag(data);
> >  	if (tag != BLK_MQ_TAG_FAIL) {
> > -		rq = data->hctx->tags->rqs[tag];
> > +		struct blk_mq_tags *tags = blk_mq_tags_from_data(data);
> > +
> > +		rq = tags->rqs[tag];
> >  
> >  		if (blk_mq_tag_busy(data->hctx)) {
> >  			rq->rq_flags = RQF_MQ_INFLIGHT;
> >  			atomic_inc(&data->hctx->nr_active);
> >  		}
> >  
> > -		rq->tag = tag;
> > +		rq->tag = -1;
> > +		rq->internal_tag = tag;
> >  		blk_mq_rq_ctx_init(data->q, data->ctx, rq, op);
> >  		return rq;
> >  	}
> 
> How about using the following code for tag assignment instead of "rq->tag =
> -1; rq->internal_tag = tag"?
> 
> 		if (data->flags & BLK_MQ_REQ_INTERNAL) {
> 			rq->tag = -1;
> 			rq->internal_tag = tag;
> 		} else {
> 			rq->tag = tag;
> 			rq->internal_tag = -1;
> 		}

Hah, nevermind, I should have read further. I guess we agree, I'll make
that change.

> > @@ -313,6 +313,9 @@ struct request *blk_mq_alloc_request_hctx(struct request_queue *q, int rw,
> >  		goto out_queue_exit;
> >  	}
> >  
> > +	rq->tag = rq->internal_tag;
> > +	rq->internal_tag = -1;
> > +
> >  	return rq;
> >  
> >  out_queue_exit:
> > @@ -321,10 +324,10 @@ struct request *blk_mq_alloc_request_hctx(struct request_queue *q, int rw,
> >  }
> >  EXPORT_SYMBOL_GPL(blk_mq_alloc_request_hctx);
> 
> Should something like "WARN_ON_ONCE(flags & BLK_MQ_REQ_INTERNAL)" be added
> at the start of this function to avoid that BLK_MQ_REQ_INTERNAL is passed in
> from outside the block layer?

Yes, seems like a prudent safety check. I'll add it, thanks.

-- 
Jens Axboe

[toc] | [prev] | [next] | [standalone]

#1558101

FromHannes Reinecke <hare@suse.de>
Date2017-01-13 09:20 +0100
Message-ID<sZ5fA-6Wf-13@gated-at.bofh.it>
In reply to#1556968
On 01/11/2017 10:39 PM, Jens Axboe wrote:
> Another year, another posting of this patchset. The previous posting
> was here:
> 
> https://www.spinics.net/lists/kernel/msg2406106.html
> 
> (yes, I've skipped v5, it was fixes on top of v4, not the rework).
> 
> I've reworked bits of this to get rid of the shadow requests, thanks
> to Bart for the inspiration. The missing piece, for me, was the fact
> that we have the tags->rqs[] indirection array already. I've done this
> somewhat differently, though, by having the internal scheduler tag
> map be allocated/torn down when an IO scheduler is attached or
> detached. This also means that when we run without a scheduler, we
> don't have to do double tag allocations, it'll work like before.
> 
> The patchset applies on top of 4.10-rc3, or can be pulled here:
> 
> git://git.kernel.dk/linux-block blk-mq-sched.6
> 
Well ... something's wrong here on my machine:

[   39.886886] ------------[ cut here ]------------
[   39.886895] WARNING: CPU: 9 PID: 62 at block/blk-mq.c:342
__blk_mq_finish_request+0x124/0x140
[   39.886895] Modules linked in: sd_mod ahci uhci_hcd ehci_pci
mpt3sas(+) libahci ehci_hcd serio_raw crc32c_intel raid_class drm libata
usbcore hpsa usb_common scsi_transport_sas sg dm_multipath dm_mod
scsi_dh_rdac scsi_dh_emc scsi_dh_alua autofs4
[   39.886910] CPU: 9 PID: 62 Comm: kworker/u130:0 Not tainted
4.10.0-rc3+ #528
[   39.886911] Hardware name: HP ProLiant ML350p Gen8, BIOS P72 09/08/2013
[   39.886917] Workqueue: events_unbound async_run_entry_fn
[   39.886918] Call Trace:
[   39.886923]  dump_stack+0x85/0xc9
[   39.886927]  __warn+0xd1/0xf0
[   39.886928]  warn_slowpath_null+0x1d/0x20
[   39.886930]  __blk_mq_finish_request+0x124/0x140
[   39.886932]  blk_mq_finish_request+0x55/0x60
[   39.886934]  blk_mq_sched_put_request+0x78/0x80
[   39.886936]  blk_mq_free_request+0xe/0x10
[   39.886938]  blk_put_request+0x25/0x60
[   39.886944]  __scsi_execute.isra.24+0x104/0x160
[   39.886946]  scsi_execute_req_flags+0x94/0x100
[   39.886948]  scsi_report_opcode+0xab/0x100

checking ...

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare@suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)

[toc] | [prev] | [next] | [standalone]

#1558233

FromHannes Reinecke <hare@suse.de>
Date2017-01-13 12:10 +0100
Message-ID<sZ7U6-7f-35@gated-at.bofh.it>
In reply to#1558101
On 01/13/2017 09:15 AM, Hannes Reinecke wrote:
> On 01/11/2017 10:39 PM, Jens Axboe wrote:
>> Another year, another posting of this patchset. The previous posting
>> was here:
>>
>> https://www.spinics.net/lists/kernel/msg2406106.html
>>
>> (yes, I've skipped v5, it was fixes on top of v4, not the rework).
>>
>> I've reworked bits of this to get rid of the shadow requests, thanks
>> to Bart for the inspiration. The missing piece, for me, was the fact
>> that we have the tags->rqs[] indirection array already. I've done this
>> somewhat differently, though, by having the internal scheduler tag
>> map be allocated/torn down when an IO scheduler is attached or
>> detached. This also means that when we run without a scheduler, we
>> don't have to do double tag allocations, it'll work like before.
>>
>> The patchset applies on top of 4.10-rc3, or can be pulled here:
>>
>> git://git.kernel.dk/linux-block blk-mq-sched.6
>>
> Well ... something's wrong here on my machine:
> 
> [   39.886886] ------------[ cut here ]------------
> [   39.886895] WARNING: CPU: 9 PID: 62 at block/blk-mq.c:342
> __blk_mq_finish_request+0x124/0x140
> [   39.886895] Modules linked in: sd_mod ahci uhci_hcd ehci_pci
> mpt3sas(+) libahci ehci_hcd serio_raw crc32c_intel raid_class drm libata
> usbcore hpsa usb_common scsi_transport_sas sg dm_multipath dm_mod
> scsi_dh_rdac scsi_dh_emc scsi_dh_alua autofs4
> [   39.886910] CPU: 9 PID: 62 Comm: kworker/u130:0 Not tainted
> 4.10.0-rc3+ #528
> [   39.886911] Hardware name: HP ProLiant ML350p Gen8, BIOS P72 09/08/2013
> [   39.886917] Workqueue: events_unbound async_run_entry_fn
> [   39.886918] Call Trace:
> [   39.886923]  dump_stack+0x85/0xc9
> [   39.886927]  __warn+0xd1/0xf0
> [   39.886928]  warn_slowpath_null+0x1d/0x20
> [   39.886930]  __blk_mq_finish_request+0x124/0x140
> [   39.886932]  blk_mq_finish_request+0x55/0x60
> [   39.886934]  blk_mq_sched_put_request+0x78/0x80
> [   39.886936]  blk_mq_free_request+0xe/0x10
> [   39.886938]  blk_put_request+0x25/0x60
> [   39.886944]  __scsi_execute.isra.24+0x104/0x160
> [   39.886946]  scsi_execute_req_flags+0x94/0x100
> [   39.886948]  scsi_report_opcode+0xab/0x100
> 
> checking ...
> 
Ah.
Seems like the elevator switch races with device setup:

 1188.490326] ------------[ cut here ]------------
[ 1188.490334] WARNING: CPU: 9 PID: 30155 at block/blk-mq.c:342
__blk_mq_finish_request+0x172/0x180
[ 1188.490335] Modules linked in: mpt3sas(+) raid_class rpcsec_gss_krb5
auth_rpcgss nfsv4 nfs lockd grace fscache ebtable_filt
er ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables
af_packet br_netfilter bridge stp llc iscsi_ibft iscs
i_boot_sysfs sb_edac edac_core x86_pkg_temp_thermal intel_powerclamp
coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_p
clmul tg3 ixgbe ghash_clmulni_intel pcbc ptp aesni_intel pps_core
aes_x86_64 ipmi_ssif hpilo hpwdt mdio libphy pcc_cpufreq cry
pto_simd glue_helper iTCO_wdt iTCO_vendor_support acpi_cpufreq tpm_tis
ipmi_si ipmi_devintf cryptd lpc_ich pcspkr ioatdma tpm_
tis_core thermal wmi shpchp dca ipmi_msghandler tpm fjes button sunrpc
btrfs xor sr_mod raid6_pq cdrom ehci_pci mgag200 i2c_al
go_bit drm_kms_helper syscopyarea sysfillrect uhci_hcd
[ 1188.490399]  sysimgblt fb_sys_fops sd_mod ahci ehci_hcd ttm libahci
crc32c_intel serio_raw drm libata usbcore usb_common hp
sa scsi_transport_sas sg dm_multipath dm_mod scsi_dh_rdac scsi_dh_emc
scsi_dh_alua autofs4
[ 1188.490411] CPU: 9 PID: 30155 Comm: kworker/u130:6 Not tainted
4.10.0-rc3+ #535
[ 1188.490411] Hardware name: HP ProLiant ML350p Gen8, BIOS P72 09/08/2013
[ 1188.490425] Workqueue: events_unbound async_run_entry_fn
[ 1188.490427] Call Trace:
[ 1188.490433]  dump_stack+0x85/0xc9
[ 1188.490436]  __warn+0xd1/0xf0
[ 1188.490438]  warn_slowpath_null+0x1d/0x20
[ 1188.490440]  __blk_mq_finish_request+0x172/0x180
[ 1188.490442]  blk_mq_finish_request+0x55/0x60
[ 1188.490443]  blk_mq_sched_put_request+0x78/0x80
[ 1188.490445]  blk_mq_free_request+0xe/0x10
[ 1188.490448]  blk_put_request+0x25/0x60
[ 1188.490453]  __scsi_execute.isra.24+0x104/0x160
[ 1188.490455]  scsi_execute_req_flags+0x94/0x100
[ 1188.490457]  scsi_report_opcode+0xab/0x100
[ 1188.490461]  sd_revalidate_disk+0xaef/0x1450 [sd_mod]
[ 1188.490464]  sd_probe_async+0xd1/0x1d0 [sd_mod]
[ 1188.490466]  async_run_entry_fn+0x37/0x150
[ 1188.490470]  process_one_work+0x1d0/0x660
[ 1188.490472]  ? process_one_work+0x151/0x660
[ 1188.490474]  worker_thread+0x12b/0x4a0
[ 1188.490475]  kthread+0x10c/0x140
[ 1188.490477]  ? process_one_work+0x660/0x660
[ 1188.490478]  ? kthread_create_on_node+0x40/0x40
[ 1188.490483]  ret_from_fork+0x2a/0x40
[ 1188.490484] ---[ end trace d5e3a32ac269fc2a ]---
[ 1188.490485] rq (487/52) rqs (-1/-1)
[ 1188.523518] sd 7:0:0:0: [sdb] Attached SCSI disk
[ 1188.540954] elevator: switch to deadline failed

(The 'rqs' line is a debug output from me:

			struct request *rqs_rq =
				hctx->tags->rqs[rq->tag];

			printk(KERN_WARNING "rq (%d/%d) rqs (%d/%d)\n",
			       rq->tag, sched_tag,
			       rqs_rq ? rqs_rq->tag : -1,
			       rqs_rq ? rqs_rq->internal_tag : -1);

telling us that request with 'tag' 487 is _not_ at position
487 in the rqs array; rather that position is empty.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare@suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)

[toc] | [prev] | [next] | [standalone]

#1558349

FromHannes Reinecke <hare@suse.de>
Date2017-01-13 13:20 +0100
Message-ID<sZ8ZQ-L2-15@gated-at.bofh.it>
In reply to#1558233
On 01/13/2017 12:04 PM, Hannes Reinecke wrote:
> On 01/13/2017 09:15 AM, Hannes Reinecke wrote:
>> On 01/11/2017 10:39 PM, Jens Axboe wrote:
>>> Another year, another posting of this patchset. The previous posting
>>> was here:
>>>
>>> https://www.spinics.net/lists/kernel/msg2406106.html
>>>
>>> (yes, I've skipped v5, it was fixes on top of v4, not the rework).
>>>
>>> I've reworked bits of this to get rid of the shadow requests, thanks
>>> to Bart for the inspiration. The missing piece, for me, was the fact
>>> that we have the tags->rqs[] indirection array already. I've done this
>>> somewhat differently, though, by having the internal scheduler tag
>>> map be allocated/torn down when an IO scheduler is attached or
>>> detached. This also means that when we run without a scheduler, we
>>> don't have to do double tag allocations, it'll work like before.
>>>
>>> The patchset applies on top of 4.10-rc3, or can be pulled here:
>>>
>>> git://git.kernel.dk/linux-block blk-mq-sched.6
>>>
>> Well ... something's wrong here on my machine:
>>
[ .. ]

Turns out that selecting CONFIG_DEFAULT_MQ_DEADLINE is the culprit;
switching to CONFIG_DEFAULT_MQ_NONE and selecting mq-deadline after
booting manually makes the problem go away.

So there is a race condition during device init and switching the I/O
scheduler.

But the results from using mq-deadline are promising; the performance
drop I've seen on older hardware seems to be resolved:

mq iosched:
 seq read : io=13383MB, bw=228349KB/s, iops=57087
 rand read : io=12876MB, bw=219709KB/s, iops=54927
 seq write: io=14532MB, bw=247987KB/s, iops=61996
 rand write: io=13779MB, bw=235127KB/s, iops=58781
mq default:
 seq read : io=13056MB, bw=222588KB/s, iops=55647
 rand read : io=12908MB, bw=220069KB/s, iops=55017
 seq write: io=13986MB, bw=238444KB/s, iops=59611
 rand write: io=13733MB, bw=234128KB/s, iops=58532
sq default:
 seq read : io=10240MB, bw=194787KB/s, iops=48696
 rand read : io=10240MB, bw=191374KB/s, iops=47843
 seq write: io=10240MB, bw=245333KB/s, iops=61333
 rand write: io=10240MB, bw=228239KB/s, iops=57059

measured on mpt2sas with SSD devices.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare@suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)

[toc] | [prev] | [next] | [standalone]

#1558463

FromJens Axboe <axboe@fb.com>
Date2017-01-13 16:10 +0100
Message-ID<sZbEo-2pl-71@gated-at.bofh.it>
In reply to#1558349
On Fri, Jan 13 2017, Hannes Reinecke wrote:
> On 01/13/2017 12:04 PM, Hannes Reinecke wrote:
> > On 01/13/2017 09:15 AM, Hannes Reinecke wrote:
> >> On 01/11/2017 10:39 PM, Jens Axboe wrote:
> >>> Another year, another posting of this patchset. The previous posting
> >>> was here:
> >>>
> >>> https://www.spinics.net/lists/kernel/msg2406106.html
> >>>
> >>> (yes, I've skipped v5, it was fixes on top of v4, not the rework).
> >>>
> >>> I've reworked bits of this to get rid of the shadow requests, thanks
> >>> to Bart for the inspiration. The missing piece, for me, was the fact
> >>> that we have the tags->rqs[] indirection array already. I've done this
> >>> somewhat differently, though, by having the internal scheduler tag
> >>> map be allocated/torn down when an IO scheduler is attached or
> >>> detached. This also means that when we run without a scheduler, we
> >>> don't have to do double tag allocations, it'll work like before.
> >>>
> >>> The patchset applies on top of 4.10-rc3, or can be pulled here:
> >>>
> >>> git://git.kernel.dk/linux-block blk-mq-sched.6
> >>>
> >> Well ... something's wrong here on my machine:
> >>
> [ .. ]
> 
> Turns out that selecting CONFIG_DEFAULT_MQ_DEADLINE is the culprit;
> switching to CONFIG_DEFAULT_MQ_NONE and selecting mq-deadline after
> booting manually makes the problem go away.
> 
> So there is a race condition during device init and switching the I/O
> scheduler.
> 
> But the results from using mq-deadline are promising; the performance
> drop I've seen on older hardware seems to be resolved:
> 
> mq iosched:
>  seq read : io=13383MB, bw=228349KB/s, iops=57087
>  rand read : io=12876MB, bw=219709KB/s, iops=54927
>  seq write: io=14532MB, bw=247987KB/s, iops=61996
>  rand write: io=13779MB, bw=235127KB/s, iops=58781
> mq default:
>  seq read : io=13056MB, bw=222588KB/s, iops=55647
>  rand read : io=12908MB, bw=220069KB/s, iops=55017
>  seq write: io=13986MB, bw=238444KB/s, iops=59611
>  rand write: io=13733MB, bw=234128KB/s, iops=58532
> sq default:
>  seq read : io=10240MB, bw=194787KB/s, iops=48696
>  rand read : io=10240MB, bw=191374KB/s, iops=47843
>  seq write: io=10240MB, bw=245333KB/s, iops=61333
>  rand write: io=10240MB, bw=228239KB/s, iops=57059
> 
> measured on mpt2sas with SSD devices.

Perfect! Straight on the path of kill of non scsi-mq, then.

I'll fix up the async scan issue. The new mq schedulers don't really
behave differently in this regard, so I'm a bit puzzled. Hopefully it
reproduces here.

-- 
Jens Axboe

[toc] | [prev] | [next] | [standalone]

#1558457

FromJens Axboe <axboe@fb.com>
Date2017-01-13 16:10 +0100
Message-ID<sZbEo-2pl-57@gated-at.bofh.it>
In reply to#1558233
On Fri, Jan 13 2017, Hannes Reinecke wrote:
> On 01/13/2017 09:15 AM, Hannes Reinecke wrote:
> > On 01/11/2017 10:39 PM, Jens Axboe wrote:
> >> Another year, another posting of this patchset. The previous posting
> >> was here:
> >>
> >> https://www.spinics.net/lists/kernel/msg2406106.html
> >>
> >> (yes, I've skipped v5, it was fixes on top of v4, not the rework).
> >>
> >> I've reworked bits of this to get rid of the shadow requests, thanks
> >> to Bart for the inspiration. The missing piece, for me, was the fact
> >> that we have the tags->rqs[] indirection array already. I've done this
> >> somewhat differently, though, by having the internal scheduler tag
> >> map be allocated/torn down when an IO scheduler is attached or
> >> detached. This also means that when we run without a scheduler, we
> >> don't have to do double tag allocations, it'll work like before.
> >>
> >> The patchset applies on top of 4.10-rc3, or can be pulled here:
> >>
> >> git://git.kernel.dk/linux-block blk-mq-sched.6
> >>
> > Well ... something's wrong here on my machine:
> > 
> > [   39.886886] ------------[ cut here ]------------
> > [   39.886895] WARNING: CPU: 9 PID: 62 at block/blk-mq.c:342
> > __blk_mq_finish_request+0x124/0x140
> > [   39.886895] Modules linked in: sd_mod ahci uhci_hcd ehci_pci
> > mpt3sas(+) libahci ehci_hcd serio_raw crc32c_intel raid_class drm libata
> > usbcore hpsa usb_common scsi_transport_sas sg dm_multipath dm_mod
> > scsi_dh_rdac scsi_dh_emc scsi_dh_alua autofs4
> > [   39.886910] CPU: 9 PID: 62 Comm: kworker/u130:0 Not tainted
> > 4.10.0-rc3+ #528
> > [   39.886911] Hardware name: HP ProLiant ML350p Gen8, BIOS P72 09/08/2013
> > [   39.886917] Workqueue: events_unbound async_run_entry_fn
> > [   39.886918] Call Trace:
> > [   39.886923]  dump_stack+0x85/0xc9
> > [   39.886927]  __warn+0xd1/0xf0
> > [   39.886928]  warn_slowpath_null+0x1d/0x20
> > [   39.886930]  __blk_mq_finish_request+0x124/0x140
> > [   39.886932]  blk_mq_finish_request+0x55/0x60
> > [   39.886934]  blk_mq_sched_put_request+0x78/0x80
> > [   39.886936]  blk_mq_free_request+0xe/0x10
> > [   39.886938]  blk_put_request+0x25/0x60
> > [   39.886944]  __scsi_execute.isra.24+0x104/0x160
> > [   39.886946]  scsi_execute_req_flags+0x94/0x100
> > [   39.886948]  scsi_report_opcode+0xab/0x100
> > 
> > checking ...
> > 
> Ah.
> Seems like the elevator switch races with device setup:

Huh, funky, haven't seen that. I'll see if I can reproduce it here. I
don't have SCAN_ASYNC turned on, on my test box.

-- 
Jens Axboe

[toc] | [prev] | [next] | [standalone]

#1558501

FromHannes Reinecke <hare@suse.de>
Date2017-01-13 16:40 +0100
Message-ID<sZc7t-2zn-31@gated-at.bofh.it>
In reply to#1558233
On 01/13/2017 04:23 PM, Jens Axboe wrote:
> On 01/13/2017 04:04 AM, Hannes Reinecke wrote:
>> On 01/13/2017 09:15 AM, Hannes Reinecke wrote:
>>> On 01/11/2017 10:39 PM, Jens Axboe wrote:
>>>> Another year, another posting of this patchset. The previous posting
>>>> was here:
>>>>
>>>> https://www.spinics.net/lists/kernel/msg2406106.html
>>>>
>>>> (yes, I've skipped v5, it was fixes on top of v4, not the rework).
>>>>
>>>> I've reworked bits of this to get rid of the shadow requests, thanks
>>>> to Bart for the inspiration. The missing piece, for me, was the fact
>>>> that we have the tags->rqs[] indirection array already. I've done this
>>>> somewhat differently, though, by having the internal scheduler tag
>>>> map be allocated/torn down when an IO scheduler is attached or
>>>> detached. This also means that when we run without a scheduler, we
>>>> don't have to do double tag allocations, it'll work like before.
>>>>
>>>> The patchset applies on top of 4.10-rc3, or can be pulled here:
>>>>
>>>> git://git.kernel.dk/linux-block blk-mq-sched.6
>>>>
>>> Well ... something's wrong here on my machine:
>>>
>>> [   39.886886] ------------[ cut here ]------------
>>> [   39.886895] WARNING: CPU: 9 PID: 62 at block/blk-mq.c:342
>>> __blk_mq_finish_request+0x124/0x140
>>> [   39.886895] Modules linked in: sd_mod ahci uhci_hcd ehci_pci
>>> mpt3sas(+) libahci ehci_hcd serio_raw crc32c_intel raid_class drm libata
>>> usbcore hpsa usb_common scsi_transport_sas sg dm_multipath dm_mod
>>> scsi_dh_rdac scsi_dh_emc scsi_dh_alua autofs4
>>> [   39.886910] CPU: 9 PID: 62 Comm: kworker/u130:0 Not tainted
>>> 4.10.0-rc3+ #528
>>> [   39.886911] Hardware name: HP ProLiant ML350p Gen8, BIOS P72 09/08/2013
>>> [   39.886917] Workqueue: events_unbound async_run_entry_fn
>>> [   39.886918] Call Trace:
>>> [   39.886923]  dump_stack+0x85/0xc9
>>> [   39.886927]  __warn+0xd1/0xf0
>>> [   39.886928]  warn_slowpath_null+0x1d/0x20
>>> [   39.886930]  __blk_mq_finish_request+0x124/0x140
>>> [   39.886932]  blk_mq_finish_request+0x55/0x60
>>> [   39.886934]  blk_mq_sched_put_request+0x78/0x80
>>> [   39.886936]  blk_mq_free_request+0xe/0x10
>>> [   39.886938]  blk_put_request+0x25/0x60
>>> [   39.886944]  __scsi_execute.isra.24+0x104/0x160
>>> [   39.886946]  scsi_execute_req_flags+0x94/0x100
>>> [   39.886948]  scsi_report_opcode+0xab/0x100
>>>
>>> checking ...
>>>
>> Ah.
>> Seems like the elevator switch races with device setup:
>>
>>  1188.490326] ------------[ cut here ]------------
>> [ 1188.490334] WARNING: CPU: 9 PID: 30155 at block/blk-mq.c:342
>> __blk_mq_finish_request+0x172/0x180
>> [ 1188.490335] Modules linked in: mpt3sas(+) raid_class rpcsec_gss_krb5
>> auth_rpcgss nfsv4 nfs lockd grace fscache ebtable_filt
>> er ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables
>> af_packet br_netfilter bridge stp llc iscsi_ibft iscs
>> i_boot_sysfs sb_edac edac_core x86_pkg_temp_thermal intel_powerclamp
>> coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_p
>> clmul tg3 ixgbe ghash_clmulni_intel pcbc ptp aesni_intel pps_core
>> aes_x86_64 ipmi_ssif hpilo hpwdt mdio libphy pcc_cpufreq cry
>> pto_simd glue_helper iTCO_wdt iTCO_vendor_support acpi_cpufreq tpm_tis
>> ipmi_si ipmi_devintf cryptd lpc_ich pcspkr ioatdma tpm_
>> tis_core thermal wmi shpchp dca ipmi_msghandler tpm fjes button sunrpc
>> btrfs xor sr_mod raid6_pq cdrom ehci_pci mgag200 i2c_al
>> go_bit drm_kms_helper syscopyarea sysfillrect uhci_hcd
>> [ 1188.490399]  sysimgblt fb_sys_fops sd_mod ahci ehci_hcd ttm libahci
>> crc32c_intel serio_raw drm libata usbcore usb_common hp
>> sa scsi_transport_sas sg dm_multipath dm_mod scsi_dh_rdac scsi_dh_emc
>> scsi_dh_alua autofs4
>> [ 1188.490411] CPU: 9 PID: 30155 Comm: kworker/u130:6 Not tainted
>> 4.10.0-rc3+ #535
>> [ 1188.490411] Hardware name: HP ProLiant ML350p Gen8, BIOS P72 09/08/2013
>> [ 1188.490425] Workqueue: events_unbound async_run_entry_fn
>> [ 1188.490427] Call Trace:
>> [ 1188.490433]  dump_stack+0x85/0xc9
>> [ 1188.490436]  __warn+0xd1/0xf0
>> [ 1188.490438]  warn_slowpath_null+0x1d/0x20
>> [ 1188.490440]  __blk_mq_finish_request+0x172/0x180
>> [ 1188.490442]  blk_mq_finish_request+0x55/0x60
>> [ 1188.490443]  blk_mq_sched_put_request+0x78/0x80
>> [ 1188.490445]  blk_mq_free_request+0xe/0x10
>> [ 1188.490448]  blk_put_request+0x25/0x60
>> [ 1188.490453]  __scsi_execute.isra.24+0x104/0x160
>> [ 1188.490455]  scsi_execute_req_flags+0x94/0x100
>> [ 1188.490457]  scsi_report_opcode+0xab/0x100
>> [ 1188.490461]  sd_revalidate_disk+0xaef/0x1450 [sd_mod]
>> [ 1188.490464]  sd_probe_async+0xd1/0x1d0 [sd_mod]
>> [ 1188.490466]  async_run_entry_fn+0x37/0x150
>> [ 1188.490470]  process_one_work+0x1d0/0x660
>> [ 1188.490472]  ? process_one_work+0x151/0x660
>> [ 1188.490474]  worker_thread+0x12b/0x4a0
>> [ 1188.490475]  kthread+0x10c/0x140
>> [ 1188.490477]  ? process_one_work+0x660/0x660
>> [ 1188.490478]  ? kthread_create_on_node+0x40/0x40
>> [ 1188.490483]  ret_from_fork+0x2a/0x40
>> [ 1188.490484] ---[ end trace d5e3a32ac269fc2a ]---
>> [ 1188.490485] rq (487/52) rqs (-1/-1)
>> [ 1188.523518] sd 7:0:0:0: [sdb] Attached SCSI disk
>> [ 1188.540954] elevator: switch to deadline failed
> 
> Where did this last line come from? Do you have a udev rule or something
> that attempts to switch to deadline, which will fail with scsi-mq?
> 
> Trying to get closer to reproducing this, no luck just setting the async
> scan option, mine still boots fine.
> 
Ah, indeed.
There is an ominous udev rule here, trying to switch to 'deadline'.

# cat 60-ssd-scheduler.rules
# do not edit this file, it will be overwritten on update

ACTION!="add", GOTO="ssd_scheduler_end"
SUBSYSTEM!="block", GOTO="ssd_scheduler_end"

IMPORT{cmdline}="elevator"
ENV{elevator}=="*?", GOTO="ssd_scheduler_end"

KERNEL=="sd*[!0-9]", ATTR{queue/rotational}=="0",
ATTR{queue/scheduler}="deadline"

LABEL="ssd_scheduler_end"

Still shouldn't crash the kernel, though ...

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare@suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)

[toc] | [prev] | [next] | [standalone]

#1558519

FromHannes Reinecke <hare@suse.de>
Date2017-01-13 17:00 +0100
Message-ID<sZcqJ-2FK-7@gated-at.bofh.it>
In reply to#1558501
On 01/13/2017 04:34 PM, Jens Axboe wrote:
> On 01/13/2017 08:33 AM, Hannes Reinecke wrote:
[ .. ]
>> Ah, indeed.
>> There is an ominous udev rule here, trying to switch to 'deadline'.
>>
>> # cat 60-ssd-scheduler.rules
>> # do not edit this file, it will be overwritten on update
>>
>> ACTION!="add", GOTO="ssd_scheduler_end"
>> SUBSYSTEM!="block", GOTO="ssd_scheduler_end"
>>
>> IMPORT{cmdline}="elevator"
>> ENV{elevator}=="*?", GOTO="ssd_scheduler_end"
>>
>> KERNEL=="sd*[!0-9]", ATTR{queue/rotational}=="0",
>> ATTR{queue/scheduler}="deadline"
>>
>> LABEL="ssd_scheduler_end"
>>
>> Still shouldn't crash the kernel, though ...
> 
> Of course not, and it's not a given that it does, it could just be
> triggering after the device load and failing like expected. But just in
> case, can you try and disable that rule and see if it still crashes with
> MQ_DEADLINE set as the default?
> 
Yes, it does.
Same stacktrace as before.

Cheers

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare@suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)

[toc] | [prev] | [next] | [standalone]

#1558544

FromJens Axboe <axboe@kernel.dk>
Date2017-01-13 17:10 +0100
Message-ID<sZcAq-2YI-41@gated-at.bofh.it>
In reply to#1558519
On 01/13/2017 09:00 AM, Jens Axboe wrote:
> On 01/13/2017 08:59 AM, Hannes Reinecke wrote:
>> On 01/13/2017 04:34 PM, Jens Axboe wrote:
>>> On 01/13/2017 08:33 AM, Hannes Reinecke wrote:
>> [ .. ]
>>>> Ah, indeed.
>>>> There is an ominous udev rule here, trying to switch to 'deadline'.
>>>>
>>>> # cat 60-ssd-scheduler.rules
>>>> # do not edit this file, it will be overwritten on update
>>>>
>>>> ACTION!="add", GOTO="ssd_scheduler_end"
>>>> SUBSYSTEM!="block", GOTO="ssd_scheduler_end"
>>>>
>>>> IMPORT{cmdline}="elevator"
>>>> ENV{elevator}=="*?", GOTO="ssd_scheduler_end"
>>>>
>>>> KERNEL=="sd*[!0-9]", ATTR{queue/rotational}=="0",
>>>> ATTR{queue/scheduler}="deadline"
>>>>
>>>> LABEL="ssd_scheduler_end"
>>>>
>>>> Still shouldn't crash the kernel, though ...
>>>
>>> Of course not, and it's not a given that it does, it could just be
>>> triggering after the device load and failing like expected. But just in
>>> case, can you try and disable that rule and see if it still crashes with
>>> MQ_DEADLINE set as the default?
>>>
>> Yes, it does.
>> Same stacktrace as before.
> 
> Alright, that's as expected. I've tried with your rule and making
> everything modular, but it still boots fine for me. Very odd. Can you
> send me your .config? And are all the SCSI disks hanging off ahci? Or
> sdb specifically, is that ahci or something else?

Also, would be great if you could pull:

git://git.kernel.dk/linux-block blk-mq-sched

into current 'master' and see if it still reproduces. I expect that it
will, but just want to ensure that it's a problem in the current code
base as well.

-- 
Jens Axboe

[toc] | [prev] | [next] | [standalone]

#1558762

FromJens Axboe <axboe@kernel.dk>
Date2017-01-13 22:50 +0100
Message-ID<sZhTs-5ZM-13@gated-at.bofh.it>
In reply to#1558544
On 01/13/2017 09:02 AM, Jens Axboe wrote:
> Also, would be great if you could pull:
> 
> git://git.kernel.dk/linux-block blk-mq-sched
> 
> into current 'master' and see if it still reproduces. I expect that it
> will, but just want to ensure that it's a problem in the current code
> base as well.

Hannes, can you try the current branch? I believe your problem should be
fixed now, would be great if you could verify.

-- 
Jens Axboe

[toc] | [prev] | [next] | [standalone]

#1559511

FromHannes Reinecke <hare@suse.de>
Date2017-01-16 09:20 +0100
Message-ID<t0aGd-5VO-3@gated-at.bofh.it>
In reply to#1558544
On 01/13/2017 05:02 PM, Jens Axboe wrote:
> On 01/13/2017 09:00 AM, Jens Axboe wrote:
>> On 01/13/2017 08:59 AM, Hannes Reinecke wrote:
>>> On 01/13/2017 04:34 PM, Jens Axboe wrote:
>>>> On 01/13/2017 08:33 AM, Hannes Reinecke wrote:
>>> [ .. ]
>>>>> Ah, indeed.
>>>>> There is an ominous udev rule here, trying to switch to 'deadline'.
>>>>>
>>>>> # cat 60-ssd-scheduler.rules
>>>>> # do not edit this file, it will be overwritten on update
>>>>>
>>>>> ACTION!="add", GOTO="ssd_scheduler_end"
>>>>> SUBSYSTEM!="block", GOTO="ssd_scheduler_end"
>>>>>
>>>>> IMPORT{cmdline}="elevator"
>>>>> ENV{elevator}=="*?", GOTO="ssd_scheduler_end"
>>>>>
>>>>> KERNEL=="sd*[!0-9]", ATTR{queue/rotational}=="0",
>>>>> ATTR{queue/scheduler}="deadline"
>>>>>
>>>>> LABEL="ssd_scheduler_end"
>>>>>
>>>>> Still shouldn't crash the kernel, though ...
>>>>
>>>> Of course not, and it's not a given that it does, it could just be
>>>> triggering after the device load and failing like expected. But just in
>>>> case, can you try and disable that rule and see if it still crashes with
>>>> MQ_DEADLINE set as the default?
>>>>
>>> Yes, it does.
>>> Same stacktrace as before.
>>
>> Alright, that's as expected. I've tried with your rule and making
>> everything modular, but it still boots fine for me. Very odd. Can you
>> send me your .config? And are all the SCSI disks hanging off ahci? Or
>> sdb specifically, is that ahci or something else?
> 
> Also, would be great if you could pull:
> 
> git://git.kernel.dk/linux-block blk-mq-sched
> 
> into current 'master' and see if it still reproduces. I expect that it
> will, but just want to ensure that it's a problem in the current code
> base as well.
> 
Actually, it doesn't. Seems to have resolved itself with the latest drop.

However, not I've got a lockdep splat:

Jan 16 09:05:02 lammermuir kernel: ------------[ cut here ]------------
Jan 16 09:05:02 lammermuir kernel: WARNING: CPU: 29 PID: 5860 at
kernel/locking/lockdep.c:3514 lock_release+0x2a7/0x490
Jan 16 09:05:02 lammermuir kernel: DEBUG_LOCKS_WARN_ON(depth <= 0)
Jan 16 09:05:02 lammermuir kernel: Modules linked in: raid0 mpt3sas
raid_class rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace fscache e
Jan 16 09:05:02 lammermuir kernel:  fb_sys_fops ahci uhci_hcd ttm
ehci_pci libahci ehci_hcd serio_raw crc32c_intel drm libata usbcore hpsa
Jan 16 09:05:02 lammermuir kernel: CPU: 29 PID: 5860 Comm: fio Not
tainted 4.10.0-rc3+ #540
Jan 16 09:05:02 lammermuir kernel: Hardware name: HP ProLiant ML350p
Gen8, BIOS P72 09/08/2013
Jan 16 09:05:02 lammermuir kernel: Call Trace:
Jan 16 09:05:02 lammermuir kernel:  dump_stack+0x85/0xc9
Jan 16 09:05:02 lammermuir kernel:  __warn+0xd1/0xf0
Jan 16 09:05:02 lammermuir kernel:  ? aio_write+0x118/0x170
Jan 16 09:05:02 lammermuir kernel:  warn_slowpath_fmt+0x4f/0x60
Jan 16 09:05:02 lammermuir kernel:  lock_release+0x2a7/0x490
Jan 16 09:05:02 lammermuir kernel:  ? blkdev_write_iter+0x89/0xd0
Jan 16 09:05:02 lammermuir kernel:  aio_write+0x138/0x170
Jan 16 09:05:02 lammermuir kernel:  do_io_submit+0x4d2/0x8f0
Jan 16 09:05:02 lammermuir kernel:  ? do_io_submit+0x413/0x8f0
Jan 16 09:05:02 lammermuir kernel:  SyS_io_submit+0x10/0x20
Jan 16 09:05:02 lammermuir kernel:  entry_SYSCALL_64_fastpath+0x23/0xc6

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare@suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)

[toc] | [prev] | [next] | [standalone]

#1559824

FromJens Axboe <axboe@kernel.dk>
Date2017-01-16 16:20 +0100
Message-ID<t0heG-271-3@gated-at.bofh.it>
In reply to#1559511
On 01/16/2017 08:12 AM, Jens Axboe wrote:
> On 01/16/2017 01:11 AM, Hannes Reinecke wrote:
>> On 01/13/2017 05:02 PM, Jens Axboe wrote:
>>> On 01/13/2017 09:00 AM, Jens Axboe wrote:
>>>> On 01/13/2017 08:59 AM, Hannes Reinecke wrote:
>>>>> On 01/13/2017 04:34 PM, Jens Axboe wrote:
>>>>>> On 01/13/2017 08:33 AM, Hannes Reinecke wrote:
>>>>> [ .. ]
>>>>>>> Ah, indeed.
>>>>>>> There is an ominous udev rule here, trying to switch to 'deadline'.
>>>>>>>
>>>>>>> # cat 60-ssd-scheduler.rules
>>>>>>> # do not edit this file, it will be overwritten on update
>>>>>>>
>>>>>>> ACTION!="add", GOTO="ssd_scheduler_end"
>>>>>>> SUBSYSTEM!="block", GOTO="ssd_scheduler_end"
>>>>>>>
>>>>>>> IMPORT{cmdline}="elevator"
>>>>>>> ENV{elevator}=="*?", GOTO="ssd_scheduler_end"
>>>>>>>
>>>>>>> KERNEL=="sd*[!0-9]", ATTR{queue/rotational}=="0",
>>>>>>> ATTR{queue/scheduler}="deadline"
>>>>>>>
>>>>>>> LABEL="ssd_scheduler_end"
>>>>>>>
>>>>>>> Still shouldn't crash the kernel, though ...
>>>>>>
>>>>>> Of course not, and it's not a given that it does, it could just be
>>>>>> triggering after the device load and failing like expected. But just in
>>>>>> case, can you try and disable that rule and see if it still crashes with
>>>>>> MQ_DEADLINE set as the default?
>>>>>>
>>>>> Yes, it does.
>>>>> Same stacktrace as before.
>>>>
>>>> Alright, that's as expected. I've tried with your rule and making
>>>> everything modular, but it still boots fine for me. Very odd. Can you
>>>> send me your .config? And are all the SCSI disks hanging off ahci? Or
>>>> sdb specifically, is that ahci or something else?
>>>
>>> Also, would be great if you could pull:
>>>
>>> git://git.kernel.dk/linux-block blk-mq-sched
>>>
>>> into current 'master' and see if it still reproduces. I expect that it
>>> will, but just want to ensure that it's a problem in the current code
>>> base as well.
>>>
>> Actually, it doesn't. Seems to have resolved itself with the latest drop.
>>
>> However, not I've got a lockdep splat:
>>
>> Jan 16 09:05:02 lammermuir kernel: ------------[ cut here ]------------
>> Jan 16 09:05:02 lammermuir kernel: WARNING: CPU: 29 PID: 5860 at
>> kernel/locking/lockdep.c:3514 lock_release+0x2a7/0x490
>> Jan 16 09:05:02 lammermuir kernel: DEBUG_LOCKS_WARN_ON(depth <= 0)
>> Jan 16 09:05:02 lammermuir kernel: Modules linked in: raid0 mpt3sas
>> raid_class rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace fscache e
>> Jan 16 09:05:02 lammermuir kernel:  fb_sys_fops ahci uhci_hcd ttm
>> ehci_pci libahci ehci_hcd serio_raw crc32c_intel drm libata usbcore hpsa
>> Jan 16 09:05:02 lammermuir kernel: CPU: 29 PID: 5860 Comm: fio Not
>> tainted 4.10.0-rc3+ #540
>> Jan 16 09:05:02 lammermuir kernel: Hardware name: HP ProLiant ML350p
>> Gen8, BIOS P72 09/08/2013
>> Jan 16 09:05:02 lammermuir kernel: Call Trace:
>> Jan 16 09:05:02 lammermuir kernel:  dump_stack+0x85/0xc9
>> Jan 16 09:05:02 lammermuir kernel:  __warn+0xd1/0xf0
>> Jan 16 09:05:02 lammermuir kernel:  ? aio_write+0x118/0x170
>> Jan 16 09:05:02 lammermuir kernel:  warn_slowpath_fmt+0x4f/0x60
>> Jan 16 09:05:02 lammermuir kernel:  lock_release+0x2a7/0x490
>> Jan 16 09:05:02 lammermuir kernel:  ? blkdev_write_iter+0x89/0xd0
>> Jan 16 09:05:02 lammermuir kernel:  aio_write+0x138/0x170
>> Jan 16 09:05:02 lammermuir kernel:  do_io_submit+0x4d2/0x8f0
>> Jan 16 09:05:02 lammermuir kernel:  ? do_io_submit+0x413/0x8f0
>> Jan 16 09:05:02 lammermuir kernel:  SyS_io_submit+0x10/0x20
>> Jan 16 09:05:02 lammermuir kernel:  entry_SYSCALL_64_fastpath+0x23/0xc6
> 
> Odd, not sure that's me. What did you pull my branch into? And what is the
> sha of the stuff you pulled in?

Forgot to ask, please send me the fio job you ran here.

-- 
Jens Axboe

[toc] | [prev] | [next] | [standalone]

#1559850

FromJens Axboe <axboe@kernel.dk>
Date2017-01-16 16:50 +0100
Message-ID<t0hHI-2nd-31@gated-at.bofh.it>
In reply to#1559824
On 01/16/2017 08:16 AM, Jens Axboe wrote:
> On 01/16/2017 08:12 AM, Jens Axboe wrote:
>> On 01/16/2017 01:11 AM, Hannes Reinecke wrote:
>>> On 01/13/2017 05:02 PM, Jens Axboe wrote:
>>>> On 01/13/2017 09:00 AM, Jens Axboe wrote:
>>>>> On 01/13/2017 08:59 AM, Hannes Reinecke wrote:
>>>>>> On 01/13/2017 04:34 PM, Jens Axboe wrote:
>>>>>>> On 01/13/2017 08:33 AM, Hannes Reinecke wrote:
>>>>>> [ .. ]
>>>>>>>> Ah, indeed.
>>>>>>>> There is an ominous udev rule here, trying to switch to 'deadline'.
>>>>>>>>
>>>>>>>> # cat 60-ssd-scheduler.rules
>>>>>>>> # do not edit this file, it will be overwritten on update
>>>>>>>>
>>>>>>>> ACTION!="add", GOTO="ssd_scheduler_end"
>>>>>>>> SUBSYSTEM!="block", GOTO="ssd_scheduler_end"
>>>>>>>>
>>>>>>>> IMPORT{cmdline}="elevator"
>>>>>>>> ENV{elevator}=="*?", GOTO="ssd_scheduler_end"
>>>>>>>>
>>>>>>>> KERNEL=="sd*[!0-9]", ATTR{queue/rotational}=="0",
>>>>>>>> ATTR{queue/scheduler}="deadline"
>>>>>>>>
>>>>>>>> LABEL="ssd_scheduler_end"
>>>>>>>>
>>>>>>>> Still shouldn't crash the kernel, though ...
>>>>>>>
>>>>>>> Of course not, and it's not a given that it does, it could just be
>>>>>>> triggering after the device load and failing like expected. But just in
>>>>>>> case, can you try and disable that rule and see if it still crashes with
>>>>>>> MQ_DEADLINE set as the default?
>>>>>>>
>>>>>> Yes, it does.
>>>>>> Same stacktrace as before.
>>>>>
>>>>> Alright, that's as expected. I've tried with your rule and making
>>>>> everything modular, but it still boots fine for me. Very odd. Can you
>>>>> send me your .config? And are all the SCSI disks hanging off ahci? Or
>>>>> sdb specifically, is that ahci or something else?
>>>>
>>>> Also, would be great if you could pull:
>>>>
>>>> git://git.kernel.dk/linux-block blk-mq-sched
>>>>
>>>> into current 'master' and see if it still reproduces. I expect that it
>>>> will, but just want to ensure that it's a problem in the current code
>>>> base as well.
>>>>
>>> Actually, it doesn't. Seems to have resolved itself with the latest drop.
>>>
>>> However, not I've got a lockdep splat:
>>>
>>> Jan 16 09:05:02 lammermuir kernel: ------------[ cut here ]------------
>>> Jan 16 09:05:02 lammermuir kernel: WARNING: CPU: 29 PID: 5860 at
>>> kernel/locking/lockdep.c:3514 lock_release+0x2a7/0x490
>>> Jan 16 09:05:02 lammermuir kernel: DEBUG_LOCKS_WARN_ON(depth <= 0)
>>> Jan 16 09:05:02 lammermuir kernel: Modules linked in: raid0 mpt3sas
>>> raid_class rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace fscache e
>>> Jan 16 09:05:02 lammermuir kernel:  fb_sys_fops ahci uhci_hcd ttm
>>> ehci_pci libahci ehci_hcd serio_raw crc32c_intel drm libata usbcore hpsa
>>> Jan 16 09:05:02 lammermuir kernel: CPU: 29 PID: 5860 Comm: fio Not
>>> tainted 4.10.0-rc3+ #540
>>> Jan 16 09:05:02 lammermuir kernel: Hardware name: HP ProLiant ML350p
>>> Gen8, BIOS P72 09/08/2013
>>> Jan 16 09:05:02 lammermuir kernel: Call Trace:
>>> Jan 16 09:05:02 lammermuir kernel:  dump_stack+0x85/0xc9
>>> Jan 16 09:05:02 lammermuir kernel:  __warn+0xd1/0xf0
>>> Jan 16 09:05:02 lammermuir kernel:  ? aio_write+0x118/0x170
>>> Jan 16 09:05:02 lammermuir kernel:  warn_slowpath_fmt+0x4f/0x60
>>> Jan 16 09:05:02 lammermuir kernel:  lock_release+0x2a7/0x490
>>> Jan 16 09:05:02 lammermuir kernel:  ? blkdev_write_iter+0x89/0xd0
>>> Jan 16 09:05:02 lammermuir kernel:  aio_write+0x138/0x170
>>> Jan 16 09:05:02 lammermuir kernel:  do_io_submit+0x4d2/0x8f0
>>> Jan 16 09:05:02 lammermuir kernel:  ? do_io_submit+0x413/0x8f0
>>> Jan 16 09:05:02 lammermuir kernel:  SyS_io_submit+0x10/0x20
>>> Jan 16 09:05:02 lammermuir kernel:  entry_SYSCALL_64_fastpath+0x23/0xc6
>>
>> Odd, not sure that's me. What did you pull my branch into? And what is the
>> sha of the stuff you pulled in?
> 
> Forgot to ask, please send me the fio job you ran here.

Nevermind, it's a mainline bug that's fixed in -rc4:

commit a12f1ae61c489076a9aeb90bddca7722bf330df3
Author: Shaohua Li <shli@fb.com>
Date:   Tue Dec 13 12:09:56 2016 -0800

    aio: fix lock dep warning

-- 
Jens Axboe

[toc] | [prev] | [next] | [standalone]

#1559831

FromJens Axboe <axboe@kernel.dk>
Date2017-01-16 16:20 +0100
Message-ID<t0heG-271-5@gated-at.bofh.it>
In reply to#1559511
On 01/16/2017 01:11 AM, Hannes Reinecke wrote:
> On 01/13/2017 05:02 PM, Jens Axboe wrote:
>> On 01/13/2017 09:00 AM, Jens Axboe wrote:
>>> On 01/13/2017 08:59 AM, Hannes Reinecke wrote:
>>>> On 01/13/2017 04:34 PM, Jens Axboe wrote:
>>>>> On 01/13/2017 08:33 AM, Hannes Reinecke wrote:
>>>> [ .. ]
>>>>>> Ah, indeed.
>>>>>> There is an ominous udev rule here, trying to switch to 'deadline'.
>>>>>>
>>>>>> # cat 60-ssd-scheduler.rules
>>>>>> # do not edit this file, it will be overwritten on update
>>>>>>
>>>>>> ACTION!="add", GOTO="ssd_scheduler_end"
>>>>>> SUBSYSTEM!="block", GOTO="ssd_scheduler_end"
>>>>>>
>>>>>> IMPORT{cmdline}="elevator"
>>>>>> ENV{elevator}=="*?", GOTO="ssd_scheduler_end"
>>>>>>
>>>>>> KERNEL=="sd*[!0-9]", ATTR{queue/rotational}=="0",
>>>>>> ATTR{queue/scheduler}="deadline"
>>>>>>
>>>>>> LABEL="ssd_scheduler_end"
>>>>>>
>>>>>> Still shouldn't crash the kernel, though ...
>>>>>
>>>>> Of course not, and it's not a given that it does, it could just be
>>>>> triggering after the device load and failing like expected. But just in
>>>>> case, can you try and disable that rule and see if it still crashes with
>>>>> MQ_DEADLINE set as the default?
>>>>>
>>>> Yes, it does.
>>>> Same stacktrace as before.
>>>
>>> Alright, that's as expected. I've tried with your rule and making
>>> everything modular, but it still boots fine for me. Very odd. Can you
>>> send me your .config? And are all the SCSI disks hanging off ahci? Or
>>> sdb specifically, is that ahci or something else?
>>
>> Also, would be great if you could pull:
>>
>> git://git.kernel.dk/linux-block blk-mq-sched
>>
>> into current 'master' and see if it still reproduces. I expect that it
>> will, but just want to ensure that it's a problem in the current code
>> base as well.
>>
> Actually, it doesn't. Seems to have resolved itself with the latest drop.
> 
> However, not I've got a lockdep splat:
> 
> Jan 16 09:05:02 lammermuir kernel: ------------[ cut here ]------------
> Jan 16 09:05:02 lammermuir kernel: WARNING: CPU: 29 PID: 5860 at
> kernel/locking/lockdep.c:3514 lock_release+0x2a7/0x490
> Jan 16 09:05:02 lammermuir kernel: DEBUG_LOCKS_WARN_ON(depth <= 0)
> Jan 16 09:05:02 lammermuir kernel: Modules linked in: raid0 mpt3sas
> raid_class rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace fscache e
> Jan 16 09:05:02 lammermuir kernel:  fb_sys_fops ahci uhci_hcd ttm
> ehci_pci libahci ehci_hcd serio_raw crc32c_intel drm libata usbcore hpsa
> Jan 16 09:05:02 lammermuir kernel: CPU: 29 PID: 5860 Comm: fio Not
> tainted 4.10.0-rc3+ #540
> Jan 16 09:05:02 lammermuir kernel: Hardware name: HP ProLiant ML350p
> Gen8, BIOS P72 09/08/2013
> Jan 16 09:05:02 lammermuir kernel: Call Trace:
> Jan 16 09:05:02 lammermuir kernel:  dump_stack+0x85/0xc9
> Jan 16 09:05:02 lammermuir kernel:  __warn+0xd1/0xf0
> Jan 16 09:05:02 lammermuir kernel:  ? aio_write+0x118/0x170
> Jan 16 09:05:02 lammermuir kernel:  warn_slowpath_fmt+0x4f/0x60
> Jan 16 09:05:02 lammermuir kernel:  lock_release+0x2a7/0x490
> Jan 16 09:05:02 lammermuir kernel:  ? blkdev_write_iter+0x89/0xd0
> Jan 16 09:05:02 lammermuir kernel:  aio_write+0x138/0x170
> Jan 16 09:05:02 lammermuir kernel:  do_io_submit+0x4d2/0x8f0
> Jan 16 09:05:02 lammermuir kernel:  ? do_io_submit+0x413/0x8f0
> Jan 16 09:05:02 lammermuir kernel:  SyS_io_submit+0x10/0x20
> Jan 16 09:05:02 lammermuir kernel:  entry_SYSCALL_64_fastpath+0x23/0xc6

Odd, not sure that's me. What did you pull my branch into? And what is the
sha of the stuff you pulled in?

-- 
Jens Axboe

[toc] | [prev] | [next] | [standalone]

#1558190

FromHannes Reinecke <hare@suse.de>
Date2017-01-13 11:20 +0100
Message-ID<sZ77J-83Q-33@gated-at.bofh.it>
In reply to#1556968
On 01/11/2017 10:39 PM, Jens Axboe wrote:
> Another year, another posting of this patchset. The previous posting
> was here:
> 
> https://www.spinics.net/lists/kernel/msg2406106.html
> 
> (yes, I've skipped v5, it was fixes on top of v4, not the rework).
> 
> I've reworked bits of this to get rid of the shadow requests, thanks
> to Bart for the inspiration. The missing piece, for me, was the fact
> that we have the tags->rqs[] indirection array already. I've done this
> somewhat differently, though, by having the internal scheduler tag
> map be allocated/torn down when an IO scheduler is attached or
> detached. This also means that when we run without a scheduler, we
> don't have to do double tag allocations, it'll work like before.
> 
> The patchset applies on top of 4.10-rc3, or can be pulled here:
> 
> git://git.kernel.dk/linux-block blk-mq-sched.6
> 
Fun continues:

[   28.976708] ata3.00: configured for UDMA/100
[   28.987625] BUG: unable to handle kernel NULL pointer dereference at
0000000000000048
[   28.987632] IP: deadline_add_request+0x15/0x70
[   28.987633] PGD 0
[   28.987634]
[   28.987636] Oops: 0000 [#1] SMP
[   28.987638] Modules linked in: ahci libahci libata uhci_hcd(+)
mgag200(+) i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt
fb_sys_fops ttm drm tg3 libphy ehci_pci ehci_hcd usbcore usb_common
ixgbe mdio hpsa(+) dca ptp pps_core scsi_transport_sas fjes(+) sg
dm_multipath dm_mod scsi_dh_rdac scsi_dh_emc scsi_dh_alua autofs4
[   28.987654] CPU: 0 PID: 268 Comm: kworker/u2:2 Not tainted
4.10.0-rc3+ #535
[   28.987655] Hardware name: HP ProLiant ML350p Gen8, BIOS P72 09/08/2013
[   28.987660] Workqueue: events_unbound async_run_entry_fn
[   28.987661] task: ffff880029391600 task.stack: ffffc9000038c000
[   28.987663] RIP: 0010:deadline_add_request+0x15/0x70
[   28.987664] RSP: 0018:ffffc9000038fb00 EFLAGS: 00010286
[   28.987665] RAX: ffff88003260c400 RBX: 0000000000000000 RCX:
0000000000000000
[   28.987666] RDX: ffffc9000038fb68 RSI: 0000000000000000 RDI:
ffff8800293b9040
[   28.987666] RBP: ffffc9000038fb18 R08: 000000000087668] R13:
ffff88003260c400 R14: 0000000000000000 R15: 0000000000000000
[   28.987670] FS:  0000000000000000(0000) GS:ffff880035c00000(0000)
knlGS:0000000000000000
[   28.987670] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   28.987671] CR2: 0000000000000048 CR3: 0000000032b64000 CR4:
00000000000406f0
[   28.987672] Call Trace:
[   28.987677]  blk_mq_sched_get_request+0x12e/0x310
[   28.987678]  ? blk_mq_sched_get_request+0x5/0x310
[   28.987681]  blk_mq_alloc_request+0x40/0x90
[   28.987684]  blk_get_request+0x35/0x110
[   28.987689]  __scsi_execute.isra.24+0x3c/0x160
[   28.987691]  scsi_execute_req_flags+0x94/0x100
[   28.987694]  scsi_probe_and_add_lun+0x207/0xd60
[   28.987699]  ? __pm_rme_resume+0x5c/0x80
[   28.987701]  __scsi_add_device+0x103/0x120
[   28.987709]  ata_scsi_scan_host+0xa3/0x1d0 [libata]
[   28.987716]  async_port_probe+0x43/0x60 [libata]
[   28.987718]  async_run_entry_fn+0x37/0x150
[   28.987722]  process_one_work+0x1d0/0x660

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare@suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)

[toc] | [prev] | [next] | [standalone]

#1558240 — Re: [PATCH 08/10] blk-mq-sched: add framework for MQ capable IO schedulers

FromHannes Reinecke <hare@suse.de>
Date2017-01-13 12:20 +0100
SubjectRe: [PATCH 08/10] blk-mq-sched: add framework for MQ capable IO schedulers
Message-ID<sZ83L-ay-9@gated-at.bofh.it>
In reply to#1556968
On 01/11/2017 10:40 PM, Jens Axboe wrote:
> This adds a set of hooks that intercepts the blk-mq path of
> allocating/inserting/issuing/completing requests, allowing
> us to develop a scheduler within that framework.
> 
> We reuse the existing elevator scheduler API on the registration
> side, but augment that with the scheduler flagging support for
> the blk-mq interfce, and with a separate set of ops hooks for MQ
> devices.
> 
> We split driver and scheduler tags, so we can run the scheduling
> independent of device queue depth.
> 
> Signed-off-by: Jens Axboe <axboe@fb.com>
[ .. ]
> @@ -823,6 +847,35 @@ static inline unsigned int queued_to_index(unsigned int queued)
>  	return min(BLK_MQ_MAX_DISPATCH_ORDER - 1, ilog2(queued) + 1);
>  }
>  
> +static bool blk_mq_get_driver_tag(struct request *rq,
> +				  struct blk_mq_hw_ctx **hctx, bool wait)
> +{
> +	struct blk_mq_alloc_data data = {
> +		.q = rq->q,
> +		.ctx = rq->mq_ctx,
> +		.hctx = blk_mq_map_queue(rq->q, rq->mq_ctx->cpu),
> +		.flags = wait ? 0 : BLK_MQ_REQ_NOWAIT,
> +	};
> +
> +	if (blk_mq_hctx_stopped(data.hctx))
> +		return false;
> +
> +	if (rq->tag != -1) {
> +done:
> +		if (hctx)
> +			*hctx = data.hctx;
> +		return true;
> +	}
> +
> +	rq->tag = blk_mq_get_tag(&data);
> +	if (rq->tag >= 0) {
> +		data.hctx->tags->rqs[rq->tag] = rq;
> +		goto done;
> +	}
> +
> +	return false;
> +}
> +
What happens with the existing request at 'rqs[rq->tag]' ?
Surely there is one already, right?
Things like '->init_request' assume a fully populated array, so moving
one entry to another location is ... interesting.

I would have thought we need to do a request cloning here,
otherwise this would introduce a memory leak, right?
(Not to mention a potential double completion, as the request is now at
two positions in the array)

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare@suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)

[toc] | [prev] | [next] | [standalone]

#1558576 — Re: [PATCH 08/10] blk-mq-sched: add framework for MQ capable IO schedulers

FromOmar Sandoval <osandov@osandov.com>
Date2017-01-13 17:50 +0100
SubjectRe: [PATCH 08/10] blk-mq-sched: add framework for MQ capable IO schedulers
Message-ID<sZdd9-3bn-21@gated-at.bofh.it>
In reply to#1558240
On Fri, Jan 13, 2017 at 12:15:17PM +0100, Hannes Reinecke wrote:
> On 01/11/2017 10:40 PM, Jens Axboe wrote:
> > This adds a set of hooks that intercepts the blk-mq path of
> > allocating/inserting/issuing/completing requests, allowing
> > us to develop a scheduler within that framework.
> > 
> > We reuse the existing elevator scheduler API on the registration
> > side, but augment that with the scheduler flagging support for
> > the blk-mq interfce, and with a separate set of ops hooks for MQ
> > devices.
> > 
> > We split driver and scheduler tags, so we can run the scheduling
> > independent of device queue depth.
> > 
> > Signed-off-by: Jens Axboe <axboe@fb.com>
> [ .. ]
> > @@ -823,6 +847,35 @@ static inline unsigned int queued_to_index(unsigned int queued)
> >  	return min(BLK_MQ_MAX_DISPATCH_ORDER - 1, ilog2(queued) + 1);
> >  }
> >  
> > +static bool blk_mq_get_driver_tag(struct request *rq,
> > +				  struct blk_mq_hw_ctx **hctx, bool wait)
> > +{
> > +	struct blk_mq_alloc_data data = {
> > +		.q = rq->q,
> > +		.ctx = rq->mq_ctx,
> > +		.hctx = blk_mq_map_queue(rq->q, rq->mq_ctx->cpu),
> > +		.flags = wait ? 0 : BLK_MQ_REQ_NOWAIT,
> > +	};
> > +
> > +	if (blk_mq_hctx_stopped(data.hctx))
> > +		return false;
> > +
> > +	if (rq->tag != -1) {
> > +done:
> > +		if (hctx)
> > +			*hctx = data.hctx;
> > +		return true;
> > +	}
> > +
> > +	rq->tag = blk_mq_get_tag(&data);
> > +	if (rq->tag >= 0) {
> > +		data.hctx->tags->rqs[rq->tag] = rq;
> > +		goto done;
> > +	}
> > +
> > +	return false;
> > +}
> > +
> What happens with the existing request at 'rqs[rq->tag]' ?
> Surely there is one already, right?
> Things like '->init_request' assume a fully populated array, so moving
> one entry to another location is ... interesting.
> 
> I would have thought we need to do a request cloning here,
> otherwise this would introduce a memory leak, right?
> (Not to mention a potential double completion, as the request is now at
> two positions in the array)
> 
> Cheers,
> 
> Hannes

The entries in tags->rqs aren't slab objects, they're pointers into
pages allocated separately and tracked on tags->page_list. See
blk_mq_alloc_rqs(). In blk_mq_free_rqs(), we free all of the pages on
tags->page_list, so there shouldn't be a memory leak.

As for hctx->tags->rqs, entries are only overwritten when a scheduler is
enabled. In that case, the rqs array is storing pointers to requests
actually from hctx->sched_tags, so overwriting/leaking isn't an issue.

[toc] | [prev] | [next] | [standalone]

#1558638 — Re: [PATCH 08/10] blk-mq-sched: add framework for MQ capable IO schedulers

FromHannes Reinecke <hare@suse.de>
Date2017-01-13 18:50 +0100
SubjectRe: [PATCH 08/10] blk-mq-sched: add framework for MQ capable IO schedulers
Message-ID<sZe9b-3Kc-13@gated-at.bofh.it>
In reply to#1558576
On 01/13/2017 05:41 PM, Omar Sandoval wrote:
> On Fri, Jan 13, 2017 at 12:15:17PM +0100, Hannes Reinecke wrote:
>> On 01/11/2017 10:40 PM, Jens Axboe wrote:
>>> This adds a set of hooks that intercepts the blk-mq path of
>>> allocating/inserting/issuing/completing requests, allowing
>>> us to develop a scheduler within that framework.
>>>
>>> We reuse the existing elevator scheduler API on the registration
>>> side, but augment that with the scheduler flagging support for
>>> the blk-mq interfce, and with a separate set of ops hooks for MQ
>>> devices.
>>>
>>> We split driver and scheduler tags, so we can run the scheduling
>>> independent of device queue depth.
>>>
>>> Signed-off-by: Jens Axboe <axboe@fb.com>
>> [ .. ]
>>> @@ -823,6 +847,35 @@ static inline unsigned int queued_to_index(unsigned int queued)
>>>  	return min(BLK_MQ_MAX_DISPATCH_ORDER - 1, ilog2(queued) + 1);
>>>  }
>>>  
>>> +static bool blk_mq_get_driver_tag(struct request *rq,
>>> +				  struct blk_mq_hw_ctx **hctx, bool wait)
>>> +{
>>> +	struct blk_mq_alloc_data data = {
>>> +		.q = rq->q,
>>> +		.ctx = rq->mq_ctx,
>>> +		.hctx = blk_mq_map_queue(rq->q, rq->mq_ctx->cpu),
>>> +		.flags = wait ? 0 : BLK_MQ_REQ_NOWAIT,
>>> +	};
>>> +
>>> +	if (blk_mq_hctx_stopped(data.hctx))
>>> +		return false;
>>> +
>>> +	if (rq->tag != -1) {
>>> +done:
>>> +		if (hctx)
>>> +			*hctx = data.hctx;
>>> +		return true;
>>> +	}
>>> +
>>> +	rq->tag = blk_mq_get_tag(&data);
>>> +	if (rq->tag >= 0) {
>>> +		data.hctx->tags->rqs[rq->tag] = rq;
>>> +		goto done;
>>> +	}
>>> +
>>> +	return false;
>>> +}
>>> +
>> What happens with the existing request at 'rqs[rq->tag]' ?
>> Surely there is one already, right?
>> Things like '->init_request' assume a fully populated array, so moving
>> one entry to another location is ... interesting.
>>
>> I would have thought we need to do a request cloning here,
>> otherwise this would introduce a memory leak, right?
>> (Not to mention a potential double completion, as the request is now at
>> two positions in the array)
>>
>> Cheers,
>>
>> Hannes
> 
> The entries in tags->rqs aren't slab objects, they're pointers into
> pages allocated separately and tracked on tags->page_list. See
> blk_mq_alloc_rqs(). In blk_mq_free_rqs(), we free all of the pages on
> tags->page_list, so there shouldn't be a memory leak.
> 
> As for hctx->tags->rqs, entries are only overwritten when a scheduler is
> enabled. In that case, the rqs array is storing pointers to requests
> actually from hctx->sched_tags, so overwriting/leaking isn't an issue.

Ah. Thanks.
That explains it.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | linux.kernel

csiph-web