Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1339971 > unrolled thread

[PATCH V2 00/13] block-throttle: proportional throttle

Back to article view | Back to linux.kernel

Contents

  [PATCH V2 00/13] block-throttle: proportional throttle Shaohua Li <shli@fb.com> - 2016-02-22 23:10 +0100
    [PATCH V2 08/13] blk-throttle: add cgroup2 interface Shaohua Li <shli@fb.com> - 2016-02-22 23:10 +0100
    [PATCH V2 06/13] blk-throttle: add per-cgroup data Shaohua Li <shli@fb.com> - 2016-02-22 23:10 +0100
    Re: [PATCH V2 00/13] block-throttle: proportional throttle Pavel Machek <pavel@ucw.cz> - 2016-02-28 16:10 +0100
      Re: [PATCH V2 00/13] block-throttle: proportional throttle Shaohua Li <shli@fb.com> - 2016-03-01 06:30 +0100

#1339971 — [PATCH V2 00/13] block-throttle: proportional throttle

Hi,

Currently we have 2 iocontrollers. blk-throttling is bandwidth/iops based. CFQ
is weight based. It would be great there is a unified iocontroller for the two.
And blk-mq doesn't support ioscheduler, leaving blk-throttling the only option
for blk-mq. It's time to have a scalable iocontroller supporting both
bandwidth/weight based control and working with blk-mq.

blk-throttling is a good candidate, it works for both blk-mq and legacy queue.
It has a global lock which is scaring for scalability, but it's not terrible in
practice. In my test, the NVMe IOPS can reach 1M/s and I have all CPU run IO. Enabling
blk-throttle has around 2~3% IOPS and 10% cpu utilization impact. I'd expect
this isn't a big problem for today's workload. This patchset then try to make a
unified iocontroller with blk-throttling.

The idea is pretty simple. If we know disk total capability, we can divide the
capability to cgroups according to their weight. blk-throttling can account IO
cost and use the calculated capability to throttle cgroup. The problem is how
to estimate the disk total capability and IO cost. We can use bandwidth
(capability is bandwidth, IO cost is request size), IOPS (capability is IOPS,
IO cost is 1), or anything better. There are a lot of disscussions about this
topic, but we can't find the best option right now. Bandwidth/IOPS isn't
optimal but the only option at hand and should work generally. This patch set
tries to create a framework and make bandwidth/IOPS based proportional
throttling work. Later we can add other options for capability/cost
measurement. I'll focus on bandwidth based approach here. The first 9 patches
demonstrate the ideas and should be pretty straightforward.

The problem is we don't know the max bandwidth a disk can provide for a
specific workload, which depends on the device and IO pattern. The estimated
bandwidth by patch 1 will be always not accurate unless the disk is already in
max bandwidth. To solve this issue, we always over estimate the bandwidth. Over
esitmate bandwidth, workload dispatchs more IO, estimated bandwidth becomes
higher, dispatches even more IO. The loop will run till we enter a stable
state, in which the disk gets max bandwidth. The 'slightly adjust and run into
stable state' is the core algorithm the patch series use. We also use it to
detect inactive cgroup.

Over estimate bandwidth can introduce fairness issue, because it's possible
some cgroups can use the extra bandwidth but others not, the cgroup using the
extra bandwidth gets more share than expected. On the other hand, smaller extra
bandwidth means disk gets to max bandwidth more slowly. We assign 1/8 extra
bandwidth in the patch.

The tricky part is cgroup might not use its share fully. The cgroup can run
into this because it is not willing to (for example, a fio job with '--rate'
option) or not able to (for example, a specific IO depth limits the bandwidth)
dispatch enough IO. Let's have some examples. Assume cg1 has 20% share, cg2 has
80% share.

1. disk bandwidth 200M/s. cgroup hasn't limitation
cg1 bps = 40M/s, cg2 bps = 160M/s

2. disk bandwidth 200M/s. cg2 has rate limit 10M/s
cg1 bps = 190M/s, cg2 bps = 10M/s
In this case, cg1's bps isn't 10/4 = 2.5M/s.

3. disk bandwidth 200M/s. cg2 has rate limit 10M/s, cg1 has limit 100M/s
cg1 bps = 100M/s, cg2 bps = 10M/s

We should detect cgroup which has big share but can't use its share. Otherwise,
we can't drive disk to max bandwidth. To solve the issue, if a cgroup doesn't
use its share, we adjust its weight/share. For example, cg2 in example 2 will
get 5% share even user sets its share to 80%. The adjustment is slight to avoid
spike, but we will reach a stable state eventually.

It's possible the adjustment is wrong. If a cgroup with shrinked share hits
bandwidth limit, we recover its share. There is another tricky adjustment case

4. disk bandwidth 100M/s. each cgroup has 80M/s.
cg1 bps = 20M/s, cg2 bps = 80M/s

This is the ideal state. cg1 uses its share fully. But since we assign 1/8
extra bandwidth, cg2 doesn't use its share fully. The adjustment algorithm will
adjust cg2's share, say 5M/s. cg1 gets 25M/s then. cg2 can only get 75M/s
because the disk max bandwidth is 100M/s. If this adjustment continues, both
cg1 and cg2 will get 50M/s bandwidth. To mitigate this issue, if a cgroup's
bandwidth drops after its share is shrinked, we restore its share. This is
controversial sometimes. In above case, when cg2 gets 75M/s, cg1 might get
40M/s, because IO pattern changes and max bandwidth changes. Somebody might
think 40M/75M is better, but this patch series choose 20M/80M.

We don't bias read/sync IO in the patch set yet. Idealy we should divide a
cgroup's share for read/write IO and give read IO more share. The problem is
some cgroups might only do read or write. A fixed read/write ratio will make
such cgroups waste their share. This issue can be fixed if we introduce a sub
service queue for read and write of a cgroup in the future.

I have been tested the patches in different setups. Test setup, scripts and
results are uploaded at:
https://github.com/shligit/iocontroller-test

There are still some tests we don't get optimal performance yet and some
calculations must be revised, but I think the code is good enough to
demonstrate the idea and different issues. Comments and benchmarks are warmly
welcome!

-----------------------------------------------------------------

Shaohua Li (13):
  block: estimate disk performance
  blk-throttle: cleanup io cost related stuff
  blk-throttle: add abstract to index data
  blk-throttle: weight based throttling
  blk-throttling: detect inactive cgroup
  blk-throttle: add per-cgroup data
  blk-throttle: add interface for proporation based throttle
  blk-throttle: add cgroup2 interface
  blk-throttle: add trace for new proporation throttle
  blk-throttle: over estimate bandwidth
  blk-throttle: shrink cgroup share if its target is overestimated
  blk-throttle: restore shrinked cgroup share
  blk-throttle: detect wrong shrink

 block/blk-core.c           |   56 ++
 block/blk-sysfs.c          |   13 +
 block/blk-throttle.c       | 1217 ++++++++++++++++++++++++++++++++++++++------
 include/linux/blk-cgroup.h |   10 +
 include/linux/blkdev.h     |    7 +
 5 files changed, 1158 insertions(+), 145 deletions(-)

-- 
2.6.5

[toc] | [next] | [standalone]

#1339973 — [PATCH V2 08/13] blk-throttle: add cgroup2 interface

Example usage of the cgroup2 interface:
$echo "+io" > /sys/fs/cgroup/cgroup.subtree_control
set bandwidth based proporation mode
$echo "8:0 weight_bw" > /sys/fs/cgroup/io.throttle.mode_device
$mkdir /sys/fs/cgroup/test
set cgroup weight
$echo "8:0 200" > /sys/fs/cgroup/test/io.throttle.weight
$echo $$ > /sys/fs/cgroup/test/cgroup.procs

Signed-off-by: Shaohua Li <shli@fb.com>
---
 block/blk-throttle.c | 42 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 42 insertions(+)

diff --git a/block/blk-throttle.c b/block/blk-throttle.c
index a594000..01ca04e 100644
--- a/block/blk-throttle.c
+++ b/block/blk-throttle.c
@@ -1912,6 +1912,35 @@ static ssize_t tg_set_max(struct kernfs_open_file *of,
 	return ret ?: nbytes;
 }
 
+static int tg_print_cg2_weight(struct seq_file *sf, void *v)
+{
+	struct blkcg *blkcg = css_to_blkcg(seq_css(sf));
+	struct throtl_group_data *tgd = blkcg_to_tgd(blkcg);
+
+	seq_printf(sf, "default %u\n", tgd->weight);
+	return tg_print_weight_device(sf, v);
+}
+
+static ssize_t tg_set_cg2_weight(struct kernfs_open_file *of,
+	char *buf, size_t nbytes, loff_t off)
+{
+	char *endp;
+	int ret;
+	u64 v;
+
+	buf = strim(buf);
+
+	/* "WEIGHT" or "default WEIGHT" sets the default weight */
+	v = simple_strtoull(buf, &endp, 0);
+	if (*endp == '\0' || sscanf(buf, "default %llu", &v) == 1) {
+		ret = tg_set_weight(of_css(of), of_cft(of), v);
+		return ret ?: nbytes;
+	}
+
+	/* "MAJ:MIN WEIGHT" */
+	return tg_set_weight_device(of, buf, nbytes, off);
+}
+
 static struct cftype throtl_files[] = {
 	{
 		.name = "max",
@@ -1919,6 +1948,19 @@ static struct cftype throtl_files[] = {
 		.seq_show = tg_print_max,
 		.write = tg_set_max,
 	},
+	{
+		.name = "throttle.weight",
+		.flags = CFTYPE_NOT_ON_ROOT,
+		.private = offsetof(struct throtl_grp, service_queue.weight),
+		.seq_show = tg_print_cg2_weight,
+		.write = tg_set_cg2_weight,
+	},
+	{
+		.name = "throttle.mode_device",
+		.flags = CFTYPE_ONLY_ON_ROOT,
+		.seq_show = throtl_print_mode_device,
+		.write = tg_set_mode_device,
+	},
 	{ }	/* terminate */
 };
 
-- 
2.6.5

[toc] | [prev] | [next] | [standalone]

#1339975 — [PATCH V2 06/13] blk-throttle: add per-cgroup data

Currently we only per-cgroup per-queue data. This adds per-cgroup data
(cgroup weight). Changing the per-cgroup weight will change all
per-cgroup per-queue weight.

Signed-off-by: Shaohua Li <shli@fb.com>
---
 block/blk-throttle.c | 47 ++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 46 insertions(+), 1 deletion(-)

diff --git a/block/blk-throttle.c b/block/blk-throttle.c
index 43de1dc..a0fd33e 100644
--- a/block/blk-throttle.c
+++ b/block/blk-throttle.c
@@ -112,6 +112,7 @@ struct throtl_io_cost {
 	unsigned int io_disp[2];
 };
 
+/* per cgroup per device data */
 struct throtl_grp {
 	/* must be the first member */
 	struct blkg_policy_data pd;
@@ -155,6 +156,14 @@ struct throtl_grp {
 	unsigned long slice_end[2];
 };
 
+/* per-cgroup data */
+struct throtl_group_data {
+	/* must be the first member */
+	struct blkcg_policy_data cpd;
+
+	unsigned int weight;
+};
+
 enum run_mode {
 	MODE_NONE = 0,
 	MODE_THROTTLE = 1, /* bandwidth/iops based throttle */
@@ -246,6 +255,16 @@ static struct throtl_data *sq_to_td(struct throtl_service_queue *sq)
 		return container_of(sq, struct throtl_data, service_queue);
 }
 
+static inline struct throtl_group_data *cpd_to_tgd(struct blkcg_policy_data *cpd)
+{
+	return cpd ? container_of(cpd, struct throtl_group_data, cpd) : NULL;
+}
+
+static inline struct throtl_group_data *blkcg_to_tgd(struct blkcg *blkcg)
+{
+	return cpd_to_tgd(blkcg_to_cpd(blkcg, &blkcg_policy_throtl));
+}
+
 static inline int tg_data_index(struct throtl_grp *tg, bool rw)
 {
 	if (td_weight_based(tg->td))
@@ -385,6 +404,28 @@ static struct bio *throtl_pop_queued(struct list_head *queued,
 	return bio;
 }
 
+static struct blkcg_policy_data *throtl_cpd_alloc(gfp_t gfp)
+{
+	struct throtl_group_data *tgd;
+
+	tgd = kzalloc(sizeof(*tgd), gfp);
+	if (!tgd)
+		return NULL;
+	return &tgd->cpd;
+}
+
+static void throtl_cpd_init(struct blkcg_policy_data *cpd)
+{
+	struct throtl_group_data *tgd = cpd_to_tgd(cpd);
+
+	tgd->weight = DFT_WEIGHT;
+}
+
+static void throtl_cpd_free(struct blkcg_policy_data *cpd)
+{
+	kfree(cpd_to_tgd(cpd));
+}
+
 /* init a service_queue, assumes the caller zeroed it */
 static void throtl_service_queue_init(struct throtl_service_queue *sq)
 {
@@ -449,7 +490,7 @@ static void throtl_pd_init(struct blkg_policy_data *pd)
 	sq->parent_sq = &td->service_queue;
 	if (cgroup_subsys_on_dfl(io_cgrp_subsys) && blkg->parent)
 		sq->parent_sq = &blkg_to_tg(blkg->parent)->service_queue;
-	sq->weight = DFT_WEIGHT;
+	sq->weight = blkcg_to_tgd(blkg->blkcg)->weight;
 	sq->acting_weight = 0;
 	tg->td = td;
 }
@@ -1677,6 +1718,10 @@ static struct blkcg_policy blkcg_policy_throtl = {
 	.dfl_cftypes		= throtl_files,
 	.legacy_cftypes		= throtl_legacy_files,
 
+	.cpd_alloc_fn		= throtl_cpd_alloc,
+	.cpd_init_fn		= throtl_cpd_init,
+	.cpd_free_fn		= throtl_cpd_free,
+
 	.pd_alloc_fn		= throtl_pd_alloc,
 	.pd_init_fn		= throtl_pd_init,
 	.pd_online_fn		= throtl_pd_online,
-- 
2.6.5

[toc] | [prev] | [next] | [standalone]

#1345304

Hi!

> The problem is we don't know the max bandwidth a disk can provide for a
> specific workload, which depends on the device and IO pattern. The estimated
> bandwidth by patch 1 will be always not accurate unless the disk is already in
> max bandwidth. To solve this issue, we always over estimate the bandwidth. Over
> esitmate bandwidth, workload dispatchs more IO, estimated bandwidth becomes
> higher, dispatches even more IO. The loop will run till we enter a stable
> state, in which the disk gets max bandwidth. The 'slightly adjust and run into
> stable state' is the core algorithm the patch series use. We also use it to
> detect inactive cgroup.

Ok, so you want to reach a steady state, but what if workloads varies
a lot?

Lets say random writes for ten minutes, then linear write.

Will the linear write be severely throttled because of the previous
seeks?

Can a task get bigger bandwidth by doing some additional (useless)
work?

Like "I do bigger reads in the random read phase, so that I'm not
throttled that badly when I do the linear read"?
								Pavel

[toc] | [prev] | [next] | [standalone]

#1346373

On Sun, Feb 28, 2016 at 04:02:51PM +0100, Pavel Machek wrote:
> Hi!
> 
> > The problem is we don't know the max bandwidth a disk can provide for a
> > specific workload, which depends on the device and IO pattern. The estimated
> > bandwidth by patch 1 will be always not accurate unless the disk is already in
> > max bandwidth. To solve this issue, we always over estimate the bandwidth. Over
> > esitmate bandwidth, workload dispatchs more IO, estimated bandwidth becomes
> > higher, dispatches even more IO. The loop will run till we enter a stable
> > state, in which the disk gets max bandwidth. The 'slightly adjust and run into
> > stable state' is the core algorithm the patch series use. We also use it to
> > detect inactive cgroup.
> 
> Ok, so you want to reach a steady state, but what if workloads varies
> a lot?
> 
> Lets say random writes for ten minutes, then linear write.
> 
> Will the linear write be severely throttled because of the previous
> seeks?

If the workload vary a lot, it's possible there is fairness issue or
performance issue when the workload is changing. The fairness or
performance issue will depend on the changing interval. If the changing
interval is short, say, several milliseconds, the issue would be big.
For the case above, the linear write will get throttled initially as
previously estimated bandwidth is low. The workload will get less
throttled soon as estimated bandwidth will get bigger. Within some time,
the workload will enter stable state and get highest bandwidth.

> Can a task get bigger bandwidth by doing some additional (useless)
> work?
> 
> Like "I do bigger reads in the random read phase, so that I'm not
> throttled that badly when I do the linear read"?

Yes, it's possible, but only at the stage approaching to stable state.

Thanks,
Shaohua

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web