Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1199305 > unrolled thread

[PATCHSET sched,cgroup] sched: Implement interface for cgroup unified hierarchy

Back to article view | Back to linux.kernel

Contents

  [PATCHSET sched,cgroup] sched: Implement interface for cgroup unified hierarchy Tejun Heo <tj@kernel.org> - 2015-08-04 00:50 +0200
    [PATCH 3/3] sched: Implement interface for cgroup unified hierarchy Tejun Heo <tj@kernel.org> - 2015-08-04 00:50 +0200
      Re: [PATCH 3/3] sched: Implement interface for cgroup unified  hierarchy Peter Zijlstra <peterz@infradead.org> - 2015-08-04 11:10 +0200
        Re: [PATCH 3/3] sched: Implement interface for cgroup unified  hierarchy Tejun Heo <tj@kernel.org> - 2015-08-04 17:20 +0200
          Re: [PATCH 3/3] sched: Implement interface for cgroup unified  hierarchy Peter Zijlstra <peterz@infradead.org> - 2015-08-05 11:20 +0200
            Re: [PATCH 3/3] sched: Implement interface for cgroup unified  hierarchy Tejun Heo <tj@kernel.org> - 2015-08-05 16:40 +0200
              Re: [PATCH 3/3] sched: Implement interface for cgroup unified  hierarchy Tejun Heo <tj@kernel.org> - 2015-08-17 22:40 +0200
              Re: [PATCH 3/3] sched: Implement interface for cgroup unified hierarchy Paul Turner <pjt@google.com> - 2015-08-18 06:10 +0200
                Re: [PATCH 3/3] sched: Implement interface for cgroup unified  hierarchy Tejun Heo <tj@kernel.org> - 2015-08-18 22:40 +0200
                  Re: [PATCH 3/3] sched: Implement interface for cgroup unified  hierarchy Kamezawa Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> - 2015-08-19 01:50 +0200
                    Re: [PATCH 3/3] sched: Implement interface for cgroup unified  hierarchy Tejun Heo <tj@kernel.org> - 2015-08-19 18:30 +0200
                  Re: [PATCH 3/3] sched: Implement interface for cgroup unified  hierarchy Mike Galbraith <umgwanakikbuti@gmail.com> - 2015-08-19 05:30 +0200
                    Re: [PATCH 3/3] sched: Implement interface for cgroup unified  hierarchy Tejun Heo <tj@kernel.org> - 2015-08-19 18:50 +0200
      [PATCH v2 3/3] sched: Implement interface for cgroup unified  hierarchy Tejun Heo <tj@kernel.org> - 2015-08-04 21:40 +0200
    [PATCH 1/3] cgroup: define controller file conventions Tejun Heo <tj@kernel.org> - 2015-08-04 00:50 +0200
      Re: [PATCH 1/3] cgroup: define controller file conventions Peter Zijlstra <peterz@infradead.org> - 2015-08-04 10:50 +0200
        Re: [PATCH 1/3] cgroup: define controller file conventions Tejun Heo <tj@kernel.org> - 2015-08-04 17:00 +0200
      Re: [PATCH 1/3] cgroup: define controller file conventions Peter Zijlstra <peterz@infradead.org> - 2015-08-04 10:50 +0200
        Re: [PATCH 1/3] cgroup: define controller file conventions Tejun Heo <tj@kernel.org> - 2015-08-04 17:00 +0200
      [PATCH v2 1/3] cgroup: define controller file conventions Tejun Heo <tj@kernel.org> - 2015-08-04 21:40 +0200
        Re: [PATCH v2 1/3] cgroup: define controller file conventions Kamezawa Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> - 2015-08-05 03:00 +0200
          Re: [PATCH v2 1/3] cgroup: define controller file conventions Michal Hocko <mhocko@kernel.org> - 2015-08-05 09:50 +0200
            Re: [PATCH v2 1/3] cgroup: define controller file conventions Kamezawa Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> - 2015-08-06 04:40 +0200
              Re: [PATCH v2 1/3] cgroup: define controller file conventions Michal Hocko <mhocko@kernel.org> - 2015-08-07 20:20 +0200
                Re: [PATCH v2 1/3] cgroup: define controller file conventions Johannes Weiner <hannes@cmpxchg.org> - 2015-08-18 00:10 +0200
        Re: [PATCH v2 1/3] cgroup: define controller file conventions Johannes Weiner <hannes@cmpxchg.org> - 2015-08-17 23:40 +0200
    [PATCH 2/3] sched: Misc preps for cgroup unified hierarchy interface Tejun Heo <tj@kernel.org> - 2015-08-04 00:50 +0200

Page 1 of 2  [1] 2  Next page →

#1199305 — [PATCHSET sched,cgroup] sched: Implement interface for cgroup unified hierarchy

Hello,

This patchset implements cpu controller's interface for unified
hierarchy.  While cpu controller didn't have structural issues that
memcg and blkcg had, there still are minor issues such as cpuacct and
use of different time units and its interface can be made consistent
with other controllers so that cgroup as a whole presents uniform ways
to achieve similar things with different resources.

This patchset contains the following three patches.

 0001-cgroup-define-controller-file-conventions.patch
 0002-sched-Misc-preps-for-cgroup-unified-hierarchy-interf.patch
 0003-sched-Implement-interface-for-cgroup-unified-hierarc.patch

The "Controller file conventions" section in
Documentation/cgroups/unified-hierarchy.txt which is added by the
first patch codifies the syntax and semantics for controller knobs and
the next two patches implement the new interface for the cpu
controller.

The first patch is needed by blkcg too, so once the changes get acked
I'll set up a branch containing the patch so that it can be pulled
from both sched and blkcg.

This patchset is on top of v4.2-rc1 and also available in the
following git branch.

 git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup.git review-sched-unified-intf

diffstat follows, thanks.

 Documentation/cgroups/unified-hierarchy.txt |  128 +++++++++++++++++++-
 include/linux/cgroup.h                      |    9 +
 kernel/sched/core.c                         |  173 +++++++++++++++++++++++++++-
 kernel/sched/cpuacct.c                      |   57 ++++++---
 kernel/sched/cpuacct.h                      |    5 
 5 files changed, 342 insertions(+), 30 deletions(-)

--
tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1199306 — [PATCH 3/3] sched: Implement interface for cgroup unified hierarchy

While the cpu controller doesn't have any functional problems, there
are a couple interface issues which can be addressed in the v2
interface.

* cpuacct being a separate controller.  This separation is artificial
  and rather pointless as demonstrated by most use cases co-mounting
  the two controllers.  It also forces certain information to be
  accounted twice.

* Use of different time units.  Writable control knobs use
  microseconds, some stat fields use nanoseconds while other cpuacct
  stat fields use centiseconds.

* Control knobs which can't be used in the root cgroup still show up
  in the root.

* Control knob names and semantics aren't consistent with other
  controllers.

This patchset implements cpu controller's interface on the unified
hierarchy which adheres to the controller file conventions described
in Documentation/cgroups/unified-hierarchy.txt.  Overall, the
following changes are made.

* cpuacct is implictly enabled and disabled by cpu and its information
  is reported through "cpu.stat" which now uses microseconds for all
  time durations.  All time duration fields now have "_usec" appended
  to them for clarity.  While this doesn't solve the double accounting
  immediately, once majority of users switch to v2, cpu can directly
  account and report the relevant stats and cpuacct can be disabled on
  the unified hierarchy.

  Note that cpuacct.usage_percpu is currently not included in
  "cpu.stat".  If this information is actually called for, it can be
  added later.

* "cpu.shares" is replaced with "cpu.weight" and operates on the
  standard scale defined by CGROUP_WEIGHT_MIN/DFL/MAX (1, 100, 10000).
  The weight is scaled to scheduler weight so that 100 maps to 1024
  and the ratio relationship is preserved - if weight is W and its
  scaled value is S, W / 100 == S / 1024.  While the mapped range is a
  bit smaller than the orignal scheduler weight range, the dead zones
  on both sides are relatively small and covers wider range than the
  nice value mappings.  This file doesn't make sense in the root
  cgroup and isn't create on root.

* "cpu.cfs_quota_us" and "cpu.cfs_period_us" are replaced by "cpu.max"
  which contains both quota and period.

* "cpu.rt_runtime_us" and "cpu.rt_period_us" are replaced by
  "cpu.rt.max" which contains both runtime and period.

Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Li Zefan <lizefan@huawei.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
---
 Documentation/cgroups/unified-hierarchy.txt |  53 +++++++++
 kernel/sched/core.c                         | 165 ++++++++++++++++++++++++++++
 kernel/sched/cpuacct.c                      |  24 ++++
 kernel/sched/cpuacct.h                      |   5 +
 4 files changed, 247 insertions(+)

diff --git a/Documentation/cgroups/unified-hierarchy.txt b/Documentation/cgroups/unified-hierarchy.txt
index fc372b8..24c3e89 100644
--- a/Documentation/cgroups/unified-hierarchy.txt
+++ b/Documentation/cgroups/unified-hierarchy.txt
@@ -30,6 +30,7 @@ CONTENTS
     5-4-1. blkio
     5-4-2. cpuset
     5-4-3. memory
+    5-4-4. cpu, cpuacct
 6. Planned Changes
   6-1. CAP for resource control
 
@@ -532,6 +533,58 @@ may be specified in any order and not all pairs have to be specified.
   memory.low, memory.high, and memory.max will use the string "max" to
   indicate and set the highest possible value.
 
+5-4-4. cpu, cpuacct
+
+- cpuacct is no longer an independent controller.  It's implicitly
+  enabled by cpu and its information is reported in cpu.stat.
+
+- All time durations, including all stats, are now in microseconds.
+
+- The interface is updated as follows.
+
+  cpu.stat
+
+	Currently reports the following six stats.  All time stats are
+	in microseconds.
+
+	  usage_usec
+	  user_usec
+	  system_usec
+	  nr_periods
+	  nr_throttled
+	  throttled_usec
+
+  cpu.weight
+
+	The weight setting.  The weight is between 1 and 10000 and
+	defaults to 100.
+
+	This file is available only on non-root cgroups.
+
+  cpu.max
+
+	The maximum bandwidth setting.  It's in the following format.
+
+	  $MAX $PERIOD
+
+	which indicates that the group may consume upto $MAX in each
+	$PERIOD duration.  "max" for $MAX indicates no limit.  If only
+	one number is written, $MAX is updated.
+
+	This file is available only on non-root cgroups.
+
+  cpu.rt.max
+
+	The maximum realtime runtime setting.  It's in the following
+	format.
+
+	  $MAX $PERIOD
+
+	which indicates that the group may consume upto $MAX in each
+	$PERIOD duration.  If only one number is written, $MAX is
+	updated.
+
+
 6. Planned Changes
 
 6-1. CAP for resource control
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 6137037..0fb1dd7 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -8438,6 +8438,163 @@ static struct cftype cpu_legacy_files[] = {
 	{ }	/* terminate */
 };
 
+static int cpu_stats_show(struct seq_file *sf, void *v)
+{
+	cpuacct_cpu_stats_show(sf);
+
+#ifdef CONFIG_FAIR_GROUP_SCHED
+	{
+		struct task_group *tg = css_tg(seq_css(sf));
+		struct cfs_bandwidth *cfs_b = &tg->cfs_bandwidth;
+
+		seq_printf(sf, "nr_periods %d\n"
+			   "nr_throttled %d\n"
+			   "throttled_usec %llu\n",
+			   cfs_b->nr_periods, cfs_b->nr_throttled,
+			   cfs_b->throttled_time / NSEC_PER_USEC);
+	}
+#endif
+	return 0;
+}
+
+#ifdef CONFIG_FAIR_GROUP_SCHED
+static u64 cpu_weight_read_u64(struct cgroup_subsys_state *css,
+			       struct cftype *cft)
+{
+	struct task_group *tg = css_tg(css);
+	u64 weight = scale_load_down(tg->shares);
+
+	return DIV_ROUND_CLOSEST_ULL(weight * CGROUP_WEIGHT_DFL, 1024);
+}
+
+static int cpu_weight_write_u64(struct cgroup_subsys_state *css,
+				struct cftype *cftype, u64 weight)
+{
+	/*
+	 * cgroup weight knobs should use the common MIN, DFL and MAX
+	 * values which are 1, 100 and 10000 respectively.  While it loses
+	 * a bit of range on both ends, it maps pretty well onto the shares
+	 * value used by scheduler and the round-trip conversions preserve
+	 * the original value over the entire range.
+	 */
+	if (weight < CGROUP_WEIGHT_MIN || weight > CGROUP_WEIGHT_MAX)
+		return -ERANGE;
+
+	weight = DIV_ROUND_CLOSEST_ULL(weight * 1024, CGROUP_WEIGHT_DFL);
+
+	return sched_group_set_shares(css_tg(css), scale_load(weight));
+}
+#endif
+
+/* caller should put the current value in *@periodp before calling */
+static int __maybe_unused cpu_max_parse(char *buf, u64 *periodp, u64 *quotap)
+{
+	char tok[21];	/* U64_MAX */
+
+	if (!sscanf(buf, "%s %llu", tok, periodp))
+		return -EINVAL;
+
+	*periodp *= NSEC_PER_USEC;
+
+	if (sscanf(tok, "%llu", quotap))
+		*quotap *= NSEC_PER_USEC;
+	else if (!strcmp(tok, "max"))
+		*quotap = RUNTIME_INF;
+	else
+		return -EINVAL;
+
+	return 0;
+}
+
+static void __maybe_unused cpu_max_print(struct seq_file *sf, long period,
+					 long quota)
+{
+	if (quota < 0)
+		seq_puts(sf, "max");
+	else
+		seq_printf(sf, "%ld", quota);
+
+	seq_printf(sf, " %ld\n", period);
+}
+
+#ifdef CONFIG_CFS_BANDWIDTH
+static int cpu_max_show(struct seq_file *sf, void *v)
+{
+	struct task_group *tg = css_tg(seq_css(sf));
+
+	cpu_max_print(sf, tg_get_cfs_period(tg), tg_get_cfs_quota(tg));
+	return 0;
+}
+
+static ssize_t cpu_max_write(struct kernfs_open_file *of,
+			     char *buf, size_t nbytes, loff_t off)
+{
+	struct task_group *tg = css_tg(of_css(of));
+	u64 period = tg_get_cfs_period(tg);
+	u64 quota;
+	int ret;
+
+	ret = cpu_max_parse(buf, &period, &quota);
+	if (!ret)
+		ret = tg_set_cfs_bandwidth(tg, period, quota);
+	return ret ?: nbytes;
+}
+#endif
+#ifdef CONFIG_RT_GROUP_SCHED
+static int cpu_rt_max_show(struct seq_file *sf, void *v)
+{
+	struct task_group *tg = css_tg(seq_css(sf));
+
+	cpu_max_print(sf, sched_group_rt_period(tg), sched_group_rt_runtime(tg));
+	return 0;
+}
+
+static ssize_t cpu_rt_max_write(struct kernfs_open_file *of,
+				char *buf, size_t nbytes, loff_t off)
+{
+	struct task_group *tg = css_tg(of_css(of));
+	u64 period = sched_group_rt_period(tg);
+	u64 runtime;
+	int ret;
+
+	ret = cpu_max_parse(buf, &period, &runtime);
+	if (!ret)
+		ret = tg_set_rt_bandwidth(tg, period, runtime);
+	return ret ?: nbytes;
+}
+#endif
+
+static struct cftype cpu_files[] = {
+	{
+		.name = "stat",
+		.seq_show = cpu_stats_show,
+	},
+#ifdef CONFIG_FAIR_GROUP_SCHED
+	{
+		.name = "weight",
+		.flags = CFTYPE_NOT_ON_ROOT,
+		.read_u64 = cpu_weight_read_u64,
+		.write_u64 = cpu_weight_write_u64,
+	},
+#endif
+#ifdef CONFIG_CFS_BANDWIDTH
+	{
+		.name = "max",
+		.flags = CFTYPE_NOT_ON_ROOT,
+		.seq_show = cpu_max_show,
+		.write = cpu_max_write,
+	},
+#endif
+#ifdef CONFIG_RT_GROUP_SCHED
+	{
+		.name = "rt.max",
+		.seq_show = cpu_rt_max_show,
+		.write = cpu_rt_max_write,
+	},
+#endif
+	{ }	/* terminate */
+};
+
 struct cgroup_subsys cpu_cgrp_subsys = {
 	.css_alloc	= cpu_cgroup_css_alloc,
 	.css_free	= cpu_cgroup_css_free,
@@ -8448,7 +8605,15 @@ struct cgroup_subsys cpu_cgrp_subsys = {
 	.attach		= cpu_cgroup_attach,
 	.exit		= cpu_cgroup_exit,
 	.legacy_cftypes	= cpu_legacy_files,
+	.dfl_cftypes	= cpu_files,
 	.early_init	= 1,
+#ifdef CONFIG_CGROUP_CPUACCT
+	/*
+	 * cpuacct is enabled together with cpu on the unified hierarchy
+	 * and its stats are reported through "cpu.stat".
+	 */
+	.depends_on	= 1 << cpuacct_cgrp_id,
+#endif
 };
 
 #endif	/* CONFIG_CGROUP_SCHED */
diff --git a/kernel/sched/cpuacct.c b/kernel/sched/cpuacct.c
index 42b2dd5..b4d32a6 100644
--- a/kernel/sched/cpuacct.c
+++ b/kernel/sched/cpuacct.c
@@ -224,6 +224,30 @@ static struct cftype files[] = {
 	{ }	/* terminate */
 };
 
+/* used to print cpuacct stats in cpu.stat on the unified hierarchy */
+void cpuacct_cpu_stats_show(struct seq_file *sf)
+{
+	struct cgroup_subsys_state *css;
+	u64 usage, user, sys;
+
+	css = cgroup_get_e_css(seq_css(sf)->cgroup, &cpuacct_cgrp_subsys);
+
+	usage = cpuusage_read(css, seq_cft(sf));
+	cpuacct_stats_read(css_ca(css), &user, &sys);
+
+	user *= TICK_NSEC;
+	sys *= TICK_NSEC;
+	do_div(usage, NSEC_PER_USEC);
+	do_div(user, NSEC_PER_USEC);
+	do_div(sys, NSEC_PER_USEC);
+
+	seq_printf(sf, "usage_usec %llu\n"
+		   "user_usec %llu\n"
+		   "system_usec %llu\n", usage, user, sys);
+
+	css_put(css);
+}
+
 /*
  * charge this task's execution time to its accounting group.
  *
diff --git a/kernel/sched/cpuacct.h b/kernel/sched/cpuacct.h
index ed60562..44eace9 100644
--- a/kernel/sched/cpuacct.h
+++ b/kernel/sched/cpuacct.h
@@ -2,6 +2,7 @@
 
 extern void cpuacct_charge(struct task_struct *tsk, u64 cputime);
 extern void cpuacct_account_field(struct task_struct *p, int index, u64 val);
+extern void cpuacct_cpu_stats_show(struct seq_file *sf);
 
 #else
 
@@ -14,4 +15,8 @@ cpuacct_account_field(struct task_struct *p, int index, u64 val)
 {
 }
 
+static inline void cpuacct_cpu_stats_show(struct seq_file *sf)
+{
+}
+
 #endif
-- 
2.4.3

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1199580 — Re: [PATCH 3/3] sched: Implement interface for cgroup unified hierarchy

On Mon, Aug 03, 2015 at 06:41:29PM -0400, Tejun Heo wrote:
> While the cpu controller doesn't have any functional problems, there
> are a couple interface issues which can be addressed in the v2
> interface.
> 
> * cpuacct being a separate controller.  This separation is artificial
>   and rather pointless as demonstrated by most use cases co-mounting
>   the two controllers.  It also forces certain information to be
>   accounted twice.
> 
> * Use of different time units.  Writable control knobs use
>   microseconds, some stat fields use nanoseconds while other cpuacct
>   stat fields use centiseconds.
> 
> * Control knobs which can't be used in the root cgroup still show up
>   in the root.
> 
> * Control knob names and semantics aren't consistent with other
>   controllers.

What about the unified hierarchy stuff cannot deal with per-task
controllers?

_That_ was the biggest problem from what I can remember, and I see no
proposed resolution for that here.

> This patchset implements cpu controller's interface on the unified
> hierarchy which adheres to the controller file conventions described
> in Documentation/cgroups/unified-hierarchy.txt.  Overall, the
> following changes are made.
> 
> * cpuacct is implictly enabled and disabled by cpu and its information
>   is reported through "cpu.stat" which now uses microseconds for all
>   time durations.  All time duration fields now have "_usec" appended
>   to them for clarity.  While this doesn't solve the double accounting
>   immediately, once majority of users switch to v2, cpu can directly
>   account and report the relevant stats and cpuacct can be disabled on
>   the unified hierarchy.
> 
>   Note that cpuacct.usage_percpu is currently not included in
>   "cpu.stat".  If this information is actually called for, it can be
>   added later.

Since you're rev'ing the interface, can't we simply kill the old cpuacct
and implement the missing pieces in cpu directly ?

> * "cpu.shares" is replaced with "cpu.weight" and operates on the
>   standard scale defined by CGROUP_WEIGHT_MIN/DFL/MAX (1, 100, 10000).
>   The weight is scaled to scheduler weight so that 100 maps to 1024
>   and the ratio relationship is preserved - if weight is W and its
>   scaled value is S, W / 100 == S / 1024.  While the mapped range is a
>   bit smaller than the orignal scheduler weight range, the dead zones
>   on both sides are relatively small and covers wider range than the
>   nice value mappings.  This file doesn't make sense in the root
>   cgroup and isn't create on root.

Not too thrilled about this, but if people can live with the reduced
resolution then I suppose we can do.

> * "cpu.cfs_quota_us" and "cpu.cfs_period_us" are replaced by "cpu.max"
>   which contains both quota and period.

This is indeed a maximum limit, however

> * "cpu.rt_runtime_us" and "cpu.rt_period_us" are replaced by
>   "cpu.rt.max" which contains both runtime and period.

the RT thing is conceptually more of a minimum guarantee, than a
maximum, even though the current implementation is both, there are plans
to allow (controlled) relaxation of the maximum part.

Also, if you're going to rev the interface, there's more changes we
should make. I'll have to go dig them out.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1200057 — Re: [PATCH 3/3] sched: Implement interface for cgroup unified hierarchy

Hello, Peter.

On Tue, Aug 04, 2015 at 11:07:11AM +0200, Peter Zijlstra wrote:
> What about the unified hierarchy stuff cannot deal with per-task
> controllers?
> 
> _That_ was the biggest problem from what I can remember, and I see no
> proposed resolution for that here.

I've been thinking about it and I'm now convinced that cgroups just is
the wrong interface to require each application to be programming
against.  I wrote this in the CAT thread too but cgroups may be an
okay management / administration interface but is a horrible
programming interface to be used by individual applications.

For things which don't require hierarchy, the obvious thing to do is
implementing a usual syscall-like interface be it a separate syscall,
an prctl command, an ioctl or whatever.  For things which require
building a hierarchy of member threads, the right thing to do is
making it a part of the usual process hierarchy - this is *the*
hierarchy that applications are familiar with and have the facilities
to deal with, so we can, for example, add a clone or unshare flag
which puts the calling threads in a new child group and then let that
use the fore-mentioned syscall-like interface to configure whatever it
wants to configure.  In the long term, this is *way* better than
letting individual applications fumble with cgroup hierarchy
delegation and pseudo filesystem access.

If hierarchical weight and/or bandwidth limiting for thread hierarchy
is absolutely necessary, doing this shouldn't be too difficult and I
suspect it wouldn't be all that different from autogroup.

> > * cpuacct is implictly enabled and disabled by cpu and its information
> >   is reported through "cpu.stat" which now uses microseconds for all
> >   time durations.  All time duration fields now have "_usec" appended
> >   to them for clarity.  While this doesn't solve the double accounting
> >   immediately, once majority of users switch to v2, cpu can directly
> >   account and report the relevant stats and cpuacct can be disabled on
> >   the unified hierarchy.
> > 
> >   Note that cpuacct.usage_percpu is currently not included in
> >   "cpu.stat".  If this information is actually called for, it can be
> >   added later.
> 
> Since you're rev'ing the interface, can't we simply kill the old cpuacct
> and implement the missing pieces in cpu directly ?

Yeah, that's the plan.  For the transitional period however, we'd have
a lot more usages where cpuacct is mounted in a legacy hierarchy so I
didn't want to incur the overhead of duplicate accounting for those
cases and the dependency mechanism is already there making it trivial.

> > * "cpu.cfs_quota_us" and "cpu.cfs_period_us" are replaced by "cpu.max"
> >   which contains both quota and period.
> 
> This is indeed a maximum limit, however
> 
> > * "cpu.rt_runtime_us" and "cpu.rt_period_us" are replaced by
> >   "cpu.rt.max" which contains both runtime and period.
> 
> the RT thing is conceptually more of a minimum guarantee, than a
> maximum, even though the current implementation is both, there are plans
> to allow (controlled) relaxation of the maximum part.

Ah, I see.  Yeah, then it should be cpu.rt.min.  I'll just remove the
file until the relaxation part is determined.

> Also, if you're going to rev the interface, there's more changes we
> should make. I'll have to go dig them out.

Great, please let me know what you have on mind.

Thanks.

-- 
tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1200568 — Re: [PATCH 3/3] sched: Implement interface for cgroup unified hierarchy

On Tue, Aug 04, 2015 at 11:10:17AM -0400, Tejun Heo wrote:
> Hello, Peter.
> 
> On Tue, Aug 04, 2015 at 11:07:11AM +0200, Peter Zijlstra wrote:
> > What about the unified hierarchy stuff cannot deal with per-task
> > controllers?
> > 
> > _That_ was the biggest problem from what I can remember, and I see no
> > proposed resolution for that here.
> 
> I've been thinking about it and I'm now convinced that cgroups just is
> the wrong interface to require each application to be programming
> against.

But people are doing it. So you must give them something. You cannot
just tell them to go away.

So where are the people doing this in this discussion? Or are you
one-sidedly forcing things? IIRC Google was doing this.

The whole libvirt trainwreck also does this (the programming against
cgroups, not the per task thing afaik).

You also cannot mandate system-disease, not everybody will want to run
that monster. From what I understood last time, Google has no interest
what so ever of using it.

> I wrote this in the CAT thread too but cgroups may be an
> okay management / administration interface but is a horrible
> programming interface to be used by individual applications.

Yeah, I need to catch up on that CAT thread, but the reality is, people
use it as a programming interface, whether you like it or not.

> For things which don't require hierarchy, the obvious thing to do is
> implementing a usual syscall-like interface be it a separate syscall,
> an prctl command, an ioctl or whatever.

And then you get /proc extensions to observe them, then people make
those /proc extensions writable and before you know it you've got an
equal or bigger mess back than you started out with :-(

> For things which require
> building a hierarchy of member threads, the right thing to do is
> making it a part of the usual process hierarchy - this is *the*
> hierarchy that applications are familiar with and have the facilities
> to deal with, so we can, for example, add a clone or unshare flag
> which puts the calling threads in a new child group and then let that
> use the fore-mentioned syscall-like interface to configure whatever it
> wants to configure.

And then you get to add support to cgroups to migrate hierarchies, is
that complexity you're waiting for?

Not to mention that its an unwieldy interface because then you get spawn
spawning threads etc.. Seeing how its impossible for the main thread to
create N tasks in one subgroup and another M tasks in another subgroup.

Instead they get to spawn a thread A, with which they then need to
communicate to spawn a further N tasks, then spawn a thread B, and again
communicate for another M tasks.

That's a rather awkward change to how people usually spawn threads.

Also, what to do when a thread changes profile? I can imagine a
situation where a task accepts a connection and depending on the kind of
request it gets, gets placed into a certain sub-group.

But there's no migration facility, so you get to go hand the work
around, which is expensive.

If there would be a migration facility, you've just lost naming, so how
are you going to denote the subgroups?

> In the long term, this is *way* better than
> letting individual applications fumble with cgroup hierarchy
> delegation and pseudo filesystem access.

You're worried about the intersection between what a task does and what
the administrator does, and that's a valid worry. But I'm really not
convinced this is going to make it better.

We already have relative file ops (openat(), mkdirat(), unlinkat()
etc..) can't we make sure they do the right thing in the face of a
process (hierarchy) getting migrated by the administrator.

That way, things at least _can_ work right, and I think being able to do
the right thing trumps not being able to make a mess -- people are
people, they'll always make a mess.

> If hierarchical weight and/or bandwidth limiting for thread hierarchy
> is absolutely necessary, doing this shouldn't be too difficult and I
> suspect it wouldn't be all that different from autogroup.

Autogroups are a bit icky and have the 'advantage' of not intersecting
with regular cgroups (much). The above has intricate intersection with
the cgroup stuff.

As said, your migrate process becomes a move hierarchy. You further get
more 'hidden' cgroups. /proc files that report what cgroup a task is in
will report a cgroup that's not actually present in the filesystem
(autogroups already does this, it confuses people). And as stated you
take away a lot of things that are now possible.



--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1200833 — Re: [PATCH 3/3] sched: Implement interface for cgroup unified hierarchy

Hello,

On Wed, Aug 05, 2015 at 11:10:36AM +0200, Peter Zijlstra wrote:
> > I've been thinking about it and I'm now convinced that cgroups just is
> > the wrong interface to require each application to be programming
> > against.
> 
> But people are doing it. So you must give them something. You cannot
> just tell them to go away.

Sure, more on specifics later, but, first of all, the transition to v2
is a gradual process.  The new and old hierarchies can co-exist, so
nothing forces abrupt transitions.  Also, we do want to start as
restricted as possible and then widen it gradually as necessary.

> So where are the people doing this in this discussion? Or are you
> one-sidedly forcing things? IIRC Google was doing this.

We've been having those discussions for years in person and on the
cgroup mailing list.  IIRC, the google case was for blkcg where they
have an IO proxy process which wanna issue IOs as different cgroups
depending on who's the original issuer.  They created multiple
threads, put them in different cgroups and bounce the IOs to the
matching one; however, this is already pretty silly as they have to
bounce IOs to different threads.  What makes a lot more sense here is
the ability to tag an IO as coming from a specific cgroup (or a
process's cgroup) and there was discussion of using an extra field in
aio request to indicate this, which is an a lot better solution for
the problem, can also express different IO priority and pretty easy to
implement.

> The whole libvirt trainwreck also does this (the programming against
> cgroups, not the per task thing afaik).

AFAIK, libvirt is doing multiple backends anyway and as long as the
delegation rules are clear, libvirt managing its own subhierarchy is
not a problem.  It's an administration software stack which requires
fairly close integration with the userland part of operating system.

> You also cannot mandate system-disease, not everybody will want to run
> that monster. From what I understood last time, Google has no interest
> what so ever of using it.

But what would require tight coupling of individual applications and
something like systemd is the kernel failing to set up a reasonable
boundary between management and application interfaces.  If the kernel
provides a useable API for individual applications to use, they'll
program against it and the management part can be whatever.  If we
fail to do that, individual applications will have to talk to external
agent to coordinate access to management interface and that's what'll
end up creating hard dependency on specific system agents from
applications like apache or mysql or whatever.  We really don't want
that.  The kernel *NEEDS* to clearly distinguish those two to prevent
that from happening.

> > I wrote this in the CAT thread too but cgroups may be an
> > okay management / administration interface but is a horrible
> > programming interface to be used by individual applications.
> 
> Yeah, I need to catch up on that CAT thread, but the reality is, people
> use it as a programming interface, whether you like it or not.

And that's one of the major fuck ups on cgroup's part that must be
rectified.  Look at the interface being proposed there.  It's exposing
direct hardware details w/o much abstraction which is fine for a
system management interface but at the same time it's intended to be
exposed to individual applications.  This lack of distinction makes
people skip the attention that they should be paying when they're
designing interface exposed to individual programs.  Worse, this makes
these things fly under the review scrutiny that public API accessible
to applications usually receives.  Yet, that's what these things end
up to be.  This just has to stop.  cgroups can't continue to be this
ghetto shortcut to implementing half-assed APIs.

> > For things which don't require hierarchy, the obvious thing to do is
> > implementing a usual syscall-like interface be it a separate syscall,
> > an prctl command, an ioctl or whatever.
> 
> And then you get /proc extensions to observe them, then people make
> those /proc extensions writable and before you know it you've got an
> equal or bigger mess back than you started out with :-(

What we should be doing is pushing them into the same arena as any
other publicly accessible API.  I don't think there can be a shortcut
to this.

> > For things which require
> > building a hierarchy of member threads, the right thing to do is
> > making it a part of the usual process hierarchy - this is *the*
> > hierarchy that applications are familiar with and have the facilities
> > to deal with, so we can, for example, add a clone or unshare flag
> > which puts the calling threads in a new child group and then let that
> > use the fore-mentioned syscall-like interface to configure whatever it
> > wants to configure.
> 
> And then you get to add support to cgroups to migrate hierarchies, is
> that complexity you're waiting for?

Absolutely, if it comes to that, that's what we should do.  The only
other option is spilling and getting locked into half-baked interface
to applications which not only harm userland but also kernel.

> Not to mention that its an unwieldy interface because then you get spawn
> spawning threads etc.. Seeing how its impossible for the main thread to
> create N tasks in one subgroup and another M tasks in another subgroup.
> 
> Instead they get to spawn a thread A, with which they then need to
> communicate to spawn a further N tasks, then spawn a thread B, and again
> communicate for another M tasks.
> 
> That's a rather awkward change to how people usually spawn threads.

It is within the usual purview of how userland deals with hierarchies
of processes / threads and I don't think it's necessarily bad and more
importantly I don't think the use case or the perceived awkwardness
justifies introducing a wholely new mechanism.

> Also, what to do when a thread changes profile? I can imagine a
> situation where a task accepts a connection and depending on the kind of
> request it gets, gets placed into a certain sub-group.

Migration is a very expensive operation.  The obvious thing to do for
such cases is having pools of workers for different profiles.  Also,
as mentioned before, for more specific cases like IO, it makes a lot
more sense to override things per operation rather than moving threads
around.

> But there's no migration facility, so you get to go hand the work
> around, which is expensive.

That's a lot cheaper than migrating.

> If there would be a migration facility, you've just lost naming, so how
> are you going to denote the subgroups?

I don't think we want migration in sub-process hierarchy but in the
off chance we do the naming can follow the same pid/program
group/session id scheme, which, again, is a lot easier to deal with
from applications.

> > In the long term, this is *way* better than
> > letting individual applications fumble with cgroup hierarchy
> > delegation and pseudo filesystem access.
> 
> You're worried about the intersection between what a task does and what
> the administrator does, and that's a valid worry. But I'm really not
> convinced this is going to make it better.
> 
> We already have relative file ops (openat(), mkdirat(), unlinkat()
> etc..) can't we make sure they do the right thing in the face of a
> process (hierarchy) getting migrated by the administrator.

But those are relative to the current directory per operation and
there's no way to define a transaction across multiple file
operations.  There's no way to prevent a process from being migrated
inbetween openat() and subsequent write().

> That way, things at least _can_ work right, and I think being able to do
> the right thing trumps not being able to make a mess -- people are
> people, they'll always make a mess.

It can't, at least not in the usual manner that file system operations
are defined.  This is an interface which requires central coordination
(even for delegation) and a horrible one to expose to individual
applications.

> > If hierarchical weight and/or bandwidth limiting for thread hierarchy
> > is absolutely necessary, doing this shouldn't be too difficult and I
> > suspect it wouldn't be all that different from autogroup.
> 
> Autogroups are a bit icky and have the 'advantage' of not intersecting
> with regular cgroups (much). The above has intricate intersection with
> the cgroup stuff.
>
> As said, your migrate process becomes a move hierarchy. You further get
> more 'hidden' cgroups. /proc files that report what cgroup a task is in
> will report a cgroup that's not actually present in the filesystem
> (autogroups already does this, it confuses people). And as stated you
> take away a lot of things that are now possible.

I don't think it's a lot that per-process is gonna take away.
Per-thread use cases are pretty niche to begin with and most can and
should be implemented better using a more fitting mechanism.  As for
having to deal with more complexity in cgroup core, that's fine.  If
it comes to that, we'll have to bite the bullet and do it.  Sure, we
want to be simpler but not at the cost of messing up userland API and
please note that what we lost with cgroups is this tension.

This tension between the difficulty and complexity of implementing
something which can be used by applications and the necessity or
desirability of the proposed use cases is crucial in steering kernel
development and the APIs it exposes.  Abusing cgroups like we've been
doing bypasses that tension and we of course end up locked into an
extremely crappy interfaces and mechanisms which could never be
justified in the first place.  This is about time we stopped this
disaster train.

Thanks.

-- 
tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1208862 — Re: [PATCH 3/3] sched: Implement interface for cgroup unified hierarchy

Hello, Peter.

Do we have an agreement on the sched changes?

Thanks a lot.

-- 
tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1209027 — Re: [PATCH 3/3] sched: Implement interface for cgroup unified hierarchy

Apologies for the repeat.  Gmail ate its plain text setting for some
reason.  Shame bells.

On Mon, Aug 17, 2015 at 9:02 PM, Paul Turner <pjt@google.com> wrote:
>
>
> On Wed, Aug 5, 2015 at 7:31 AM, Tejun Heo <tj@kernel.org> wrote:
>>
>> Hello,
>>
>> On Wed, Aug 05, 2015 at 11:10:36AM +0200, Peter Zijlstra wrote:
>> > > I've been thinking about it and I'm now convinced that cgroups just is
>> > > the wrong interface to require each application to be programming
>> > > against.
>> >
>> > But people are doing it. So you must give them something. You cannot
>> > just tell them to go away.
>>
>> Sure, more on specifics later, but, first of all, the transition to v2
>> is a gradual process.  The new and old hierarchies can co-exist, so
>> nothing forces abrupt transitions.  Also, we do want to start as
>> restricted as possible and then widen it gradually as necessary.
>>
>> > So where are the people doing this in this discussion? Or are you
>> > one-sidedly forcing things? IIRC Google was doing this.
>>
>> We've been having those discussions for years in person and on the
>> cgroup mailing list.  IIRC, the google case was for blkcg where they
>> have an IO proxy process which wanna issue IOs as different cgroups
>> depending on who's the original issuer.  They created multiple
>> threads, put them in different cgroups and bounce the IOs to the
>> matching one; however, this is already pretty silly as they have to
>> bounce IOs to different threads.  What makes a lot more sense here is
>> the ability to tag an IO as coming from a specific cgroup (or a
>> process's cgroup) and there was discussion of using an extra field in
>> aio request to indicate this, which is an a lot better solution for
>> the problem, can also express different IO priority and pretty easy to
>> implement.
>>
>
> So we have two major types of use that are relevant to this interface:
>
> 1) Proxy agents.  When a control systems want to perform work on behalf of a
> container, they will sometimes move the acting thread into the relevant
> control groups so that it can be accounted on that container's behalf.
> [This is more relevant for non-persistent resources such as CPU time or I/O
> priorities than charges that will outlive the work such as memory
> allocations.]
>
> I agree (1) is at best a bit of a hack and can be worked around on the type
> of time-frame these interfaces move at.
>
> 2) Control within an address-space.  For subsystems with fungible resources,
> e.g. CPU, it can be useful for an address space to partition its own
> threads.  Losing the capability to do this against the CPU controller would
> be a large set-back for instance.  Occasionally, it is useful to share these
> groupings between address spaces when processes are cooperative, but this is
> less of a requirement.
>
> This is important to us.
>
>
>> > The whole libvirt trainwreck also does this (the programming against
>> > cgroups, not the per task thing afaik).
>>
>> AFAIK, libvirt is doing multiple backends anyway and as long as the
>> delegation rules are clear, libvirt managing its own subhierarchy is
>> not a problem.  It's an administration software stack which requires
>> fairly close integration with the userland part of operating system.
>>
>> > You also cannot mandate system-disease, not everybody will want to run
>> > that monster. From what I understood last time, Google has no interest
>> > what so ever of using it.
>>
>> But what would require tight coupling of individual applications and
>> something like systemd is the kernel failing to set up a reasonable
>> boundary between management and application interfaces.  If the kernel
>> provides a useable API for individual applications to use, they'll
>> program against it and the management part can be whatever.  If we
>> fail to do that, individual applications will have to talk to external
>> agent to coordinate access to management interface
>
>
> It's notable here that for a managed system, the agent coordinating access
> *must* be external
>
>>
>> and that's what'll
>> end up creating hard dependency on specific system agents from
>> applications like apache or mysql or whatever.  We really don't want
>> that.  The kernel *NEEDS* to clearly distinguish those two to prevent
>> that from happening.
>>
>> > > I wrote this in the CAT thread too but cgroups may be an
>> > > okay management / administration interface but is a horrible
>> > > programming interface to be used by individual applications.
>> >
>> > Yeah, I need to catch up on that CAT thread, but the reality is, people
>> > use it as a programming interface, whether you like it or not.
>>
>> And that's one of the major fuck ups on cgroup's part that must be
>> rectified.  Look at the interface being proposed there.  It's exposing
>> direct hardware details w/o much abstraction which is fine for a
>> system management interface but at the same time it's intended to be
>> exposed to individual applications.
>
>
> FWIW this is something we've had no significant problems managing with
> separate mount mounts and file system protections.  Yes, there are some
> potential warts around atomicity; but we've not found them too onerous.
>
> What I don't quite follow here is the assumption that CAT should would be
> necessarily exposed to individual applications? What's wrong with subsystems
> that are primarily intended only for system management agents, we already
> have several of these.
>
>
>>
>> This lack of distinction makes
>> people skip the attention that they should be paying when they're
>> designing interface exposed to individual programs.  Worse, this makes
>> these things fly under the review scrutiny that public API accessible
>> to applications usually receives.  Yet, that's what these things end
>> up to be.  This just has to stop.  cgroups can't continue to be this
>> ghetto shortcut to implementing half-assed APIs.
>
>
> I certainly don't disagree on this point :).  But as above, I don't quite
> follow why an API being in cgroups must mean it's accessible to an
> application controlled by that group.  This has certainly not been a
> requirement for our use.
>
>>
>>
>> > > For things which don't require hierarchy, the obvious thing to do is
>> > > implementing a usual syscall-like interface be it a separate syscall,
>> > > an prctl command, an ioctl or whatever.
>> >
>> > And then you get /proc extensions to observe them, then people make
>> > those /proc extensions writable and before you know it you've got an
>> > equal or bigger mess back than you started out with :-(
>>
>> What we should be doing is pushing them into the same arena as any
>> other publicly accessible API.  I don't think there can be a shortcut
>> to this.
>>
>
> Are you explicitly opposed to non-hierarchical partitions, however?  Cpuset
> is [typically] an example of this, where the interface wants to control
> unified properties across a set of processes.  Without necessarily being
> usefully hierarchical.  (This is just to understand your core position, I'm
> not proposing cpuset should shape *anything*.)
>
>>
>> > > For things which require
>> > > building a hierarchy of member threads, the right thing to do is
>> > > making it a part of the usual process hierarchy - this is *the*
>> > > hierarchy that applications are familiar with and have the facilities
>> > > to deal with, so we can, for example, add a clone or unshare flag
>> > > which puts the calling threads in a new child group and then let that
>> > > use the fore-mentioned syscall-like interface to configure whatever it
>> > > wants to configure.
>> >
>> > And then you get to add support to cgroups to migrate hierarchies, is
>> > that complexity you're waiting for?
>>
>> Absolutely, if it comes to that, that's what we should do.  The only
>> other option is spilling and getting locked into half-baked interface
>> to applications which not only harm userland but also kernel.
>>
>> > Not to mention that its an unwieldy interface because then you get spawn
>> > spawning threads etc.. Seeing how its impossible for the main thread to
>> > create N tasks in one subgroup and another M tasks in another subgroup.
>> >
>> > Instead they get to spawn a thread A, with which they then need to
>> > communicate to spawn a further N tasks, then spawn a thread B, and again
>> > communicate for another M tasks.
>> >
>> > That's a rather awkward change to how people usually spawn threads.
>>
>> It is within the usual purview of how userland deals with hierarchies
>> of processes / threads and I don't think it's necessarily bad and more
>> importantly I don't think the use case or the perceived awkwardness
>> justifies introducing a wholely new mechanism.
>>
>> > Also, what to do when a thread changes profile? I can imagine a
>> > situation where a task accepts a connection and depending on the kind of
>> > request it gets, gets placed into a certain sub-group.
>>
>> Migration is a very expensive operation.  The obvious thing to do for
>> such cases is having pools of workers for different profiles.  Also,
>> as mentioned before, for more specific cases like IO, it makes a lot
>> more sense to override things per operation rather than moving threads
>> around.
>>
>> > But there's no migration facility, so you get to go hand the work
>> > around, which is expensive.
>>
>> That's a lot cheaper than migrating.
>>
>> > If there would be a migration facility, you've just lost naming, so how
>> > are you going to denote the subgroups?
>>
>> I don't think we want migration in sub-process hierarchy but in the
>> off chance we do the naming can follow the same pid/program
>> group/session id scheme, which, again, is a lot easier to deal with
>> from applications.
>
>
> I don't have many objections with hand-off versus migration above, however,
> I think that this is a big drawback.  Threads are expensive to create and
> are often cached rather than released.  While migration may be expensive,
> creating a more thread is more so.  The important to reconfigure a thread's
> personality at run-time is important.
>
>>
>> > > In the long term, this is *way* better than
>> > > letting individual applications fumble with cgroup hierarchy
>> > > delegation and pseudo filesystem access.
>> >
>> > You're worried about the intersection between what a task does and what
>> > the administrator does, and that's a valid worry. But I'm really not
>> > convinced this is going to make it better.
>> >
>> > We already have relative file ops (openat(), mkdirat(), unlinkat()
>> > etc..) can't we make sure they do the right thing in the face of a
>> > process (hierarchy) getting migrated by the administrator.
>>
>> But those are relative to the current directory per operation and
>> there's no way to define a transaction across multiple file
>> operations.  There's no way to prevent a process from being migrated
>> inbetween openat() and subsequent write().
>
>
> A forwarding /proc/thread_self/cgroup accessor, or similar, would be another
> way to address some of these issues.
>
>>
>>
>> > That way, things at least _can_ work right, and I think being able to do
>> > the right thing trumps not being able to make a mess -- people are
>> > people, they'll always make a mess.
>>
>> It can't, at least not in the usual manner that file system operations
>> are defined.  This is an interface which requires central coordination
>> (even for delegation) and a horrible one to expose to individual
>> applications.
>>
>> > > If hierarchical weight and/or bandwidth limiting for thread hierarchy
>> > > is absolutely necessary, doing this shouldn't be too difficult and I
>> > > suspect it wouldn't be all that different from autogroup.
>> >
>> > Autogroups are a bit icky and have the 'advantage' of not intersecting
>> > with regular cgroups (much). The above has intricate intersection with
>> > the cgroup stuff.
>> >
>> > As said, your migrate process becomes a move hierarchy. You further get
>> > more 'hidden' cgroups. /proc files that report what cgroup a task is in
>> > will report a cgroup that's not actually present in the filesystem
>> > (autogroups already does this, it confuses people). And as stated you
>> > take away a lot of things that are now possible.
>>
>> I don't think it's a lot that per-process is gonna take away.
>> Per-thread use cases are pretty niche to begin with and most can and
>> should be implemented better using a more fitting mechanism.  As for
>> having to deal with more complexity in cgroup core, that's fine.  If
>> it comes to that, we'll have to bite the bullet and do it.  Sure, we
>> want to be simpler but not at the cost of messing up userland API and
>> please note that what we lost with cgroups is this tension.
>
>
> I don't quite agree here.  Losing per-thread control within the cpu
> controller is likely going to mean that much of it ends up being
> reimplemented as some duplicate-in-appearance interface that gets us back to
> where we are today.  I recognize that these controllers (cpu, cpuacct) are
> square pegs in that per-process makes sense for most other sub-systems; but
> unfortunately, their needs and use-cases are real / dependent on their
> present form.
>
>>
>> This tension between the difficulty and complexity of implementing
>> something which can be used by applications and the necessity or
>> desirability of the proposed use cases is crucial in steering kernel
>> development and the APIs it exposes.  Abusing cgroups like we've been
>> doing bypasses that tension and we of course end up locked into an
>> extremely crappy interfaces and mechanisms which could never be
>> justified in the first place.  This is about time we stopped this
>> disaster train.
>>
>> Thanks.
>>
>> --
>> tejun
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1209470 — Re: [PATCH 3/3] sched: Implement interface for cgroup unified hierarchy

Hello, Paul.

On Mon, Aug 17, 2015 at 09:03:30PM -0700, Paul Turner wrote:
> > 2) Control within an address-space.  For subsystems with fungible resources,
> > e.g. CPU, it can be useful for an address space to partition its own
> > threads.  Losing the capability to do this against the CPU controller would
> > be a large set-back for instance.  Occasionally, it is useful to share these
> > groupings between address spaces when processes are cooperative, but this is
> > less of a requirement.
> >
> > This is important to us.

Sure, let's build a proper interface for that.  Do you actually need
sub-hierarchy inside a process?  Can you describe your use case in
detail and why having hierarchical CPU cycle distribution is essential
for your use case?

> >> And that's one of the major fuck ups on cgroup's part that must be
> >> rectified.  Look at the interface being proposed there.  It's exposing
> >> direct hardware details w/o much abstraction which is fine for a
> >> system management interface but at the same time it's intended to be
> >> exposed to individual applications.
> >
> > FWIW this is something we've had no significant problems managing with
> > separate mount mounts and file system protections.  Yes, there are some
> > potential warts around atomicity; but we've not found them too onerous.

You guys control the whole stack.  Of course, you can get away with an
interface which are pretty messed up in terms of layering and
isolation; however, generic kernel interface cannot be designed
according to that standard.

> > What I don't quite follow here is the assumption that CAT should would be
> > necessarily exposed to individual applications? What's wrong with subsystems
> > that are primarily intended only for system management agents, we already
> > have several of these.

Why would you assume that threads of a process wouldn't want to
configure it ever?  How is this different from CPU affinity?

> >> This lack of distinction makes
> >> people skip the attention that they should be paying when they're
> >> designing interface exposed to individual programs.  Worse, this makes
> >> these things fly under the review scrutiny that public API accessible
> >> to applications usually receives.  Yet, that's what these things end
> >> up to be.  This just has to stop.  cgroups can't continue to be this
> >> ghetto shortcut to implementing half-assed APIs.
> >
> > I certainly don't disagree on this point :).  But as above, I don't quite
> > follow why an API being in cgroups must mean it's accessible to an
> > application controlled by that group.  This has certainly not been a
> > requirement for our use.

I don't follow what you're trying to way with the above paragraph.
Are you still talking about CAT?  If so, that use case isn't the only
one.  I'm pretty sure there are people who would want to configure
cache allocation at thread level.

> >> What we should be doing is pushing them into the same arena as any
> >> other publicly accessible API.  I don't think there can be a shortcut
> >> to this.
> >
> > Are you explicitly opposed to non-hierarchical partitions, however?  Cpuset
> > is [typically] an example of this, where the interface wants to control
> > unified properties across a set of processes.  Without necessarily being
> > usefully hierarchical.  (This is just to understand your core position, I'm
> > not proposing cpuset should shape *anything*.)

I'm having trouble following what you're trying to say.  FWIW, cpuset
is fully hierarchical.

> >> I don't think we want migration in sub-process hierarchy but in the
> >> off chance we do the naming can follow the same pid/program
> >> group/session id scheme, which, again, is a lot easier to deal with
> >> from applications.
> >
> > I don't have many objections with hand-off versus migration above, however,
> > I think that this is a big drawback.  Threads are expensive to create and
> > are often cached rather than released.  While migration may be expensive,
> > creating a more thread is more so.  The important to reconfigure a thread's
> > personality at run-time is important.

The core problem here is picking the hot path.  If cgroups as a whole
doesn't pick a position here, controllers have to assume that
migration might not be a very cold path which naturally leads to
overall designs and synchronization schemes which concede hot path
performance to accomodate migration.  We simply can't afford to do
that - we end up losing way more in way hotter paths for something
which may be marginally useful in some corner cases.

So, this is a trade-off we're consciously making.  If there are
common-enough use cases which require jumping across different cgroup
domains, we'll try to figure out a way to accomodate those but by
default migration is a very cold and expensive path.

> >> But those are relative to the current directory per operation and
> >> there's no way to define a transaction across multiple file
> >> operations.  There's no way to prevent a process from being migrated
> >> inbetween openat() and subsequent write().
> >
> > A forwarding /proc/thread_self/cgroup accessor, or similar, would be another
> > way to address some of these issues.

That sounds horrible to me.  What if the process wants to do RMW a
config?  What if the permissions are different after an intervening
migration?  What if the sub-hierarchy no longer exists or has been
replaced by a hierarchy with the same topology but actualy is a
different one?

> > I don't quite agree here.  Losing per-thread control within the cpu
> > controller is likely going to mean that much of it ends up being
> > reimplemented as some duplicate-in-appearance interface that gets us back to
> > where we are today.  I recognize that these controllers (cpu, cpuacct) are
> > square pegs in that per-process makes sense for most other sub-systems; but
> > unfortunately, their needs and use-cases are real / dependent on their
> > present form.

Let's build an API which actually looks and behaves like an API which
is properly isolated from what external agents may do to the process.
I can't see how that would be "back to where we are today".  All of
those are pretty critical attributes for a public kernel API and
utterly broken right now.

Thanks.

-- 
tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1209544 — Re: [PATCH 3/3] sched: Implement interface for cgroup unified hierarchy

On 2015/08/19 5:31, Tejun Heo wrote:
> Hello, Paul.
>
> On Mon, Aug 17, 2015 at 09:03:30PM -0700, Paul Turner wrote:
>>> 2) Control within an address-space.  For subsystems with fungible resources,
>>> e.g. CPU, it can be useful for an address space to partition its own
>>> threads.  Losing the capability to do this against the CPU controller would
>>> be a large set-back for instance.  Occasionally, it is useful to share these
>>> groupings between address spaces when processes are cooperative, but this is
>>> less of a requirement.
>>>
>>> This is important to us.
>
> Sure, let's build a proper interface for that.  Do you actually need
> sub-hierarchy inside a process?  Can you describe your use case in
> detail and why having hierarchical CPU cycle distribution is essential
> for your use case?

An actual per-thread use case in our customers is qemu-kvm + cpuset.
customers pin each vcpus and qemu-kvm's worker threads to cpus.
For example, pinning 4 vcpus to cpu 2-6 and pinning qemu main thread and
others(vhost) to cpu 0-1.

This is an actual kvm tuning on our customers for performance guarantee.

In another case, cpu cgroup's throttling feature is used per vcpu for vm cpu sizing.

Thanks,
-Kame

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1209937 — Re: [PATCH 3/3] sched: Implement interface for cgroup unified hierarchy

Hello, Kame.

On Wed, Aug 19, 2015 at 08:39:43AM +0900, Kamezawa Hiroyuki wrote:
> An actual per-thread use case in our customers is qemu-kvm + cpuset.
> customers pin each vcpus and qemu-kvm's worker threads to cpus.
> For example, pinning 4 vcpus to cpu 2-6 and pinning qemu main thread and
> others(vhost) to cpu 0-1.

taskset and/or teach qemu how to confiure its worker threads?

> This is an actual kvm tuning on our customers for performance guarantee.
> 
> In another case, cpu cgroup's throttling feature is used per vcpu for vm cpu sizing.

Yeap, this is something we likely want to implement in an accessible
way.  For kvm, per-thread throttling configuration is enough, right?

Thanks.

-- 
tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1209593 — Re: [PATCH 3/3] sched: Implement interface for cgroup unified hierarchy

On Tue, 2015-08-18 at 13:31 -0700, Tejun Heo wrote:

> So, this is a trade-off we're consciously making.  If there are
> common-enough use cases which require jumping across different cgroup
> domains, we'll try to figure out a way to accomodate those but by
> default migration is a very cold and expensive path.

Hm.  I know of a big data outfit to which attach/detach performance was
important enough for them to have plucked an old experimental overhead
reduction hack (mine) off lkml, and shipped it.  It must have mattered a
LOT for them (not suicidal crash test dummies) to have done that.

	-Mike

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1209952 — Re: [PATCH 3/3] sched: Implement interface for cgroup unified hierarchy

Hello, Mike.

On Wed, Aug 19, 2015 at 05:23:40AM +0200, Mike Galbraith wrote:
> Hm.  I know of a big data outfit to which attach/detach performance was
> important enough for them to have plucked an old experimental overhead
> reduction hack (mine) off lkml, and shipped it.  It must have mattered a
> LOT for them (not suicidal crash test dummies) to have done that.

There haven't been any guidelines on cgroup usage.  Of course people
have been developing in all directions.  It's a natural learning
process and there are use cases which can be served by migrating
processes back and forth.  Nobody is trying to prevent that; however,
if one examines how resources and their associations need to be
tracked for accounting and control, it's evident that there are
inherent trade-offs between migration and the stuff which happens
while not migrating and it's clear which side is more important.

Most problems can be solved in different ways and I'm doubtful that
e.g. bouncing jobs to worker threads would be more expensive than
migrating the worker back and forth in a lot of cases.  If migrating
threads around floats somebody's boat, that's fine but that has never
been and can't be the focus of design and optimization, not at the
cost of the actual hot paths.

Thanks.

-- 
tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1200241 — [PATCH v2 3/3] sched: Implement interface for cgroup unified hierarchy

From f85c07ea11a52068c45cdd5f5528ed7c842c833a Mon Sep 17 00:00:00 2001
From: Tejun Heo <tj@kernel.org>
Date: Tue, 4 Aug 2015 15:24:08 -0400

While the cpu controller doesn't have any functional problems, there
are a couple interface issues which can be addressed in the v2
interface.

* cpuacct being a separate controller.  This separation is artificial
  and rather pointless as demonstrated by most use cases co-mounting
  the two controllers.  It also forces certain information to be
  accounted twice.

* Use of different time units.  Writable control knobs use
  microseconds, some stat fields use nanoseconds while other cpuacct
  stat fields use centiseconds.

* Control knobs which can't be used in the root cgroup still show up
  in the root.

* Control knob names and semantics aren't consistent with other
  controllers.

This patchset implements cpu controller's interface on the unified
hierarchy which adheres to the controller file conventions described
in Documentation/cgroups/unified-hierarchy.txt.  Overall, the
following changes are made.

* cpuacct is implictly enabled and disabled by cpu and its information
  is reported through "cpu.stat" which now uses microseconds for all
  time durations.  All time duration fields now have "_usec" appended
  to them for clarity.  While this doesn't solve the double accounting
  immediately, once majority of users switch to v2, cpu can directly
  account and report the relevant stats and cpuacct can be disabled on
  the unified hierarchy.

  Note that cpuacct.usage_percpu is currently not included in
  "cpu.stat".  If this information is actually called for, it can be
  added later.

* "cpu.shares" is replaced with "cpu.weight" and operates on the
  standard scale defined by CGROUP_WEIGHT_MIN/DFL/MAX (1, 100, 10000).
  The weight is scaled to scheduler weight so that 100 maps to 1024
  and the ratio relationship is preserved - if weight is W and its
  scaled value is S, W / 100 == S / 1024.  While the mapped range is a
  bit smaller than the orignal scheduler weight range, the dead zones
  on both sides are relatively small and covers wider range than the
  nice value mappings.  This file doesn't make sense in the root
  cgroup and isn't create on root.

* "cpu.cfs_quota_us" and "cpu.cfs_period_us" are replaced by "cpu.max"
  which contains both quota and period.

* "cpu.rt_runtime_us" and "cpu.rt_period_us" are replaced by
  "cpu.rt.max" which contains both runtime and period.

v2: cpu_stats_show() was incorrectly using CONFIG_FAIR_GROUP_SCHED for
    CFS bandwidth stats and also using raw division for u64.  Use
    CONFIG_CFS_BANDWITH and do_div() instead.

    The semantics of "cpu.rt.max" is not fully decided yet.  Dropped
    for now.

Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Li Zefan <lizefan@huawei.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
---
Hello,

Fixed build issues for certain configs and removed cpu.rt.max for now.
The git branch has been updated accordingly.

  git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup.git review-sched-unified-intf

Thanks.

 Documentation/cgroups/unified-hierarchy.txt |  53 +++++++++++
 kernel/sched/core.c                         | 140 ++++++++++++++++++++++++++++
 kernel/sched/cpuacct.c                      |  24 +++++
 kernel/sched/cpuacct.h                      |   5 +
 4 files changed, 222 insertions(+)

diff --git a/Documentation/cgroups/unified-hierarchy.txt b/Documentation/cgroups/unified-hierarchy.txt
index 1ee9caf..09b4a4e 100644
--- a/Documentation/cgroups/unified-hierarchy.txt
+++ b/Documentation/cgroups/unified-hierarchy.txt
@@ -30,6 +30,7 @@ CONTENTS
     5-4-1. blkio
     5-4-2. cpuset
     5-4-3. memory
+    5-4-4. cpu, cpuacct
 6. Planned Changes
   6-1. CAP for resource control
 
@@ -537,6 +538,58 @@ may be specified in any order and not all pairs have to be specified.
   memory.low, memory.high, and memory.max will use the string "max" to
   indicate and set the highest possible value.
 
+5-4-4. cpu, cpuacct
+
+- cpuacct is no longer an independent controller.  It's implicitly
+  enabled by cpu and its information is reported in cpu.stat.
+
+- All time durations, including all stats, are now in microseconds.
+
+- The interface is updated as follows.
+
+  cpu.stat
+
+	Currently reports the following six stats.  All time stats are
+	in microseconds.
+
+	  usage_usec
+	  user_usec
+	  system_usec
+	  nr_periods
+	  nr_throttled
+	  throttled_usec
+
+  cpu.weight
+
+	The weight setting.  The weight is between 1 and 10000 and
+	defaults to 100.
+
+	This file is available only on non-root cgroups.
+
+  cpu.max
+
+	The maximum bandwidth setting.  It's in the following format.
+
+	  $MAX $PERIOD
+
+	which indicates that the group may consume upto $MAX in each
+	$PERIOD duration.  "max" for $MAX indicates no limit.  If only
+	one number is written, $MAX is updated.
+
+	This file is available only on non-root cgroups.
+
+  cpu.rt.max
+
+	The maximum realtime runtime setting.  It's in the following
+	format.
+
+	  $MAX $PERIOD
+
+	which indicates that the group may consume upto $MAX in each
+	$PERIOD duration.  If only one number is written, $MAX is
+	updated.
+
+
 6. Planned Changes
 
 6-1. CAP for resource control
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 6137037..1e72cdd 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -8438,6 +8438,138 @@ static struct cftype cpu_legacy_files[] = {
 	{ }	/* terminate */
 };
 
+static int cpu_stats_show(struct seq_file *sf, void *v)
+{
+	cpuacct_cpu_stats_show(sf);
+
+#ifdef CONFIG_CFS_BANDWIDTH
+	{
+		struct task_group *tg = css_tg(seq_css(sf));
+		struct cfs_bandwidth *cfs_b = &tg->cfs_bandwidth;
+		u64 throttled_usec;
+
+		throttled_usec = cfs_b->throttled_time;
+		do_div(throttled_usec, NSEC_PER_USEC);
+
+		seq_printf(sf, "nr_periods %d\n"
+			   "nr_throttled %d\n"
+			   "throttled_usec %llu\n",
+			   cfs_b->nr_periods, cfs_b->nr_throttled,
+			   throttled_usec);
+	}
+#endif
+	return 0;
+}
+
+#ifdef CONFIG_FAIR_GROUP_SCHED
+static u64 cpu_weight_read_u64(struct cgroup_subsys_state *css,
+			       struct cftype *cft)
+{
+	struct task_group *tg = css_tg(css);
+	u64 weight = scale_load_down(tg->shares);
+
+	return DIV_ROUND_CLOSEST_ULL(weight * CGROUP_WEIGHT_DFL, 1024);
+}
+
+static int cpu_weight_write_u64(struct cgroup_subsys_state *css,
+				struct cftype *cftype, u64 weight)
+{
+	/*
+	 * cgroup weight knobs should use the common MIN, DFL and MAX
+	 * values which are 1, 100 and 10000 respectively.  While it loses
+	 * a bit of range on both ends, it maps pretty well onto the shares
+	 * value used by scheduler and the round-trip conversions preserve
+	 * the original value over the entire range.
+	 */
+	if (weight < CGROUP_WEIGHT_MIN || weight > CGROUP_WEIGHT_MAX)
+		return -ERANGE;
+
+	weight = DIV_ROUND_CLOSEST_ULL(weight * 1024, CGROUP_WEIGHT_DFL);
+
+	return sched_group_set_shares(css_tg(css), scale_load(weight));
+}
+#endif
+
+static void __maybe_unused cpu_period_quota_print(struct seq_file *sf,
+						  long period, long quota)
+{
+	if (quota < 0)
+		seq_puts(sf, "max");
+	else
+		seq_printf(sf, "%ld", quota);
+
+	seq_printf(sf, " %ld\n", period);
+}
+
+/* caller should put the current value in *@periodp before calling */
+static int __maybe_unused cpu_period_quota_parse(char *buf,
+						 u64 *periodp, u64 *quotap)
+{
+	char tok[21];	/* U64_MAX */
+
+	if (!sscanf(buf, "%s %llu", tok, periodp))
+		return -EINVAL;
+
+	*periodp *= NSEC_PER_USEC;
+
+	if (sscanf(tok, "%llu", quotap))
+		*quotap *= NSEC_PER_USEC;
+	else if (!strcmp(tok, "max"))
+		*quotap = RUNTIME_INF;
+	else
+		return -EINVAL;
+
+	return 0;
+}
+
+#ifdef CONFIG_CFS_BANDWIDTH
+static int cpu_max_show(struct seq_file *sf, void *v)
+{
+	struct task_group *tg = css_tg(seq_css(sf));
+
+	cpu_period_quota_print(sf, tg_get_cfs_period(tg), tg_get_cfs_quota(tg));
+	return 0;
+}
+
+static ssize_t cpu_max_write(struct kernfs_open_file *of,
+			     char *buf, size_t nbytes, loff_t off)
+{
+	struct task_group *tg = css_tg(of_css(of));
+	u64 period = tg_get_cfs_period(tg);
+	u64 quota;
+	int ret;
+
+	ret = cpu_period_quota_parse(buf, &period, &quota);
+	if (!ret)
+		ret = tg_set_cfs_bandwidth(tg, period, quota);
+	return ret ?: nbytes;
+}
+#endif
+
+static struct cftype cpu_files[] = {
+	{
+		.name = "stat",
+		.seq_show = cpu_stats_show,
+	},
+#ifdef CONFIG_FAIR_GROUP_SCHED
+	{
+		.name = "weight",
+		.flags = CFTYPE_NOT_ON_ROOT,
+		.read_u64 = cpu_weight_read_u64,
+		.write_u64 = cpu_weight_write_u64,
+	},
+#endif
+#ifdef CONFIG_CFS_BANDWIDTH
+	{
+		.name = "max",
+		.flags = CFTYPE_NOT_ON_ROOT,
+		.seq_show = cpu_max_show,
+		.write = cpu_max_write,
+	},
+#endif
+	{ }	/* terminate */
+};
+
 struct cgroup_subsys cpu_cgrp_subsys = {
 	.css_alloc	= cpu_cgroup_css_alloc,
 	.css_free	= cpu_cgroup_css_free,
@@ -8448,7 +8580,15 @@ struct cgroup_subsys cpu_cgrp_subsys = {
 	.attach		= cpu_cgroup_attach,
 	.exit		= cpu_cgroup_exit,
 	.legacy_cftypes	= cpu_legacy_files,
+	.dfl_cftypes	= cpu_files,
 	.early_init	= 1,
+#ifdef CONFIG_CGROUP_CPUACCT
+	/*
+	 * cpuacct is enabled together with cpu on the unified hierarchy
+	 * and its stats are reported through "cpu.stat".
+	 */
+	.depends_on	= 1 << cpuacct_cgrp_id,
+#endif
 };
 
 #endif	/* CONFIG_CGROUP_SCHED */
diff --git a/kernel/sched/cpuacct.c b/kernel/sched/cpuacct.c
index 42b2dd5..b4d32a6 100644
--- a/kernel/sched/cpuacct.c
+++ b/kernel/sched/cpuacct.c
@@ -224,6 +224,30 @@ static struct cftype files[] = {
 	{ }	/* terminate */
 };
 
+/* used to print cpuacct stats in cpu.stat on the unified hierarchy */
+void cpuacct_cpu_stats_show(struct seq_file *sf)
+{
+	struct cgroup_subsys_state *css;
+	u64 usage, user, sys;
+
+	css = cgroup_get_e_css(seq_css(sf)->cgroup, &cpuacct_cgrp_subsys);
+
+	usage = cpuusage_read(css, seq_cft(sf));
+	cpuacct_stats_read(css_ca(css), &user, &sys);
+
+	user *= TICK_NSEC;
+	sys *= TICK_NSEC;
+	do_div(usage, NSEC_PER_USEC);
+	do_div(user, NSEC_PER_USEC);
+	do_div(sys, NSEC_PER_USEC);
+
+	seq_printf(sf, "usage_usec %llu\n"
+		   "user_usec %llu\n"
+		   "system_usec %llu\n", usage, user, sys);
+
+	css_put(css);
+}
+
 /*
  * charge this task's execution time to its accounting group.
  *
diff --git a/kernel/sched/cpuacct.h b/kernel/sched/cpuacct.h
index ed60562..44eace9 100644
--- a/kernel/sched/cpuacct.h
+++ b/kernel/sched/cpuacct.h
@@ -2,6 +2,7 @@
 
 extern void cpuacct_charge(struct task_struct *tsk, u64 cputime);
 extern void cpuacct_account_field(struct task_struct *p, int index, u64 val);
+extern void cpuacct_cpu_stats_show(struct seq_file *sf);
 
 #else
 
@@ -14,4 +15,8 @@ cpuacct_account_field(struct task_struct *p, int index, u64 val)
 {
 }
 
+static inline void cpuacct_cpu_stats_show(struct seq_file *sf)
+{
+}
+
 #endif
-- 
2.4.3

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1199309 — [PATCH 1/3] cgroup: define controller file conventions

Traditionally, each cgroup controller implemented whatever interface
it wanted leading to interfaces which are widely inconsistent.
Examining the requirements of the controllers readily yield that there
are only a few control schemes shared among all.

Two major controllers already had to implement new interface for the
unified hierarchy due to significant structural changes.  Let's take
the chance to establish common conventions throughout all controllers.

This patch defines CGROUP_WEIGHT_MIN/DFL/MAX to be used on all weight
based control knobs and documents the conventions that controllers
should follow on the unified hierarchy.  Except for io.weight knob,
all existing unified hierarchy knobs are already compliant.  A
follow-up patch will update io.weight.

Signed-off-by: Tejun Heo <tj@kernel.org>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Cc: Li Zefan <lizefan@huawei.com>
---
 Documentation/cgroups/unified-hierarchy.txt | 75 ++++++++++++++++++++++++++---
 include/linux/cgroup.h                      |  9 ++++
 2 files changed, 76 insertions(+), 8 deletions(-)

diff --git a/Documentation/cgroups/unified-hierarchy.txt b/Documentation/cgroups/unified-hierarchy.txt
index 86847a7..fc372b8 100644
--- a/Documentation/cgroups/unified-hierarchy.txt
+++ b/Documentation/cgroups/unified-hierarchy.txt
@@ -23,10 +23,13 @@ CONTENTS
 5. Other Changes
   5-1. [Un]populated Notification
   5-2. Other Core Changes
-  5-3. Per-Controller Changes
-    5-3-1. blkio
-    5-3-2. cpuset
-    5-3-3. memory
+  5-3. Controller file conventions
+    5-3-1. Format
+    5-3-2. Control knobs
+  5-4. Per-Controller Changes
+    5-4-1. blkio
+    5-4-2. cpuset
+    5-4-3. memory
 6. Planned Changes
   6-1. CAP for resource control
 
@@ -372,14 +375,70 @@ supported and the interface files "release_agent" and
 - The "cgroup.clone_children" file is removed.
 
 
-5-3. Per-Controller Changes
+5-3. Controller file conventions
 
-5-3-1. blkio
+5-3-1. Format
+
+In general, all controller files should be in one of the following
+formats whenever possible.
+
+- Values only files
+
+  VAL0 VAL1...\n
+
+- Flat keyed files
+
+  KEY0 VAL0\n
+  KEY1 VAL1\n
+  ...
+
+- Nested keyed files
+
+  KEY0 SUB_KEY0=VAL00 SUB_KEY1=VAL01...
+  KEY1 SUB_KEY0=VAL10 SUB_KEY1=VAL11...
+  ...
+
+For a writeable file, the format for writing should generally match
+reading; however, controllers may allow omitting later fields or
+implement restricted shortcuts for most common use cases.
+
+For both flat and nested keyed files, only the values for a single key
+can be written at a time.  For nested keyed files, the sub key pairs
+may be specified in any order and not all pairs have to be specified.
+
+
+5-3-2. Control knobs
+
+- Settings for a single feature should generally be implemented in a
+  single file.
+
+- In general, the root cgroup should be exempt from resource control
+  and thus shouldn't have resource control knobs.
+
+- If a controller implements ratio based resource distribution, the
+  control knob should be named "weight" and have the range [1, 10000]
+  and 100 should be the default value.  The values are chosen to allow
+  enough and symmetric bias in both directions while keeping it
+  intuitive (the default is 100%).
+
+- If a controller implements an absolute resource limit, the control
+  knob should be named "max".  The special token "max" should be used
+  to represent no limit for both reading and writing.
+
+- If a setting has configurable default value and specific overrides,
+  the default settings should be keyed with "default" and appear as
+  the first entry in the file.  Specific entries can use "default" as
+  its value to indicate inheritance of the default value.
+
+
+5-4. Per-Controller Changes
+
+5-4-1. blkio
 
 - blk-throttle becomes properly hierarchical.
 
 
-5-3-2. cpuset
+5-4-2. cpuset
 
 - Tasks are kept in empty cpusets after hotplug and take on the masks
   of the nearest non-empty ancestor, instead of being moved to it.
@@ -388,7 +447,7 @@ supported and the interface files "release_agent" and
   masks of the nearest non-empty ancestor.
 
 
-5-3-3. memory
+5-4-3. memory
 
 - use_hierarchy is on by default and the cgroup file for the flag is
   not created.
diff --git a/include/linux/cgroup.h b/include/linux/cgroup.h
index a593e29..c6bf9d3 100644
--- a/include/linux/cgroup.h
+++ b/include/linux/cgroup.h
@@ -22,6 +22,15 @@
 
 #ifdef CONFIG_CGROUPS
 
+/*
+ * All weight knobs on the default hierarhcy should use the following min,
+ * default and max values.  The default value is the logarithmic center of
+ * MIN and MAX and allows 100x to be expressed in both directions.
+ */
+#define CGROUP_WEIGHT_MIN		1
+#define CGROUP_WEIGHT_DFL		100
+#define CGROUP_WEIGHT_MAX		10000
+
 /* a css_task_iter should be treated as an opaque object */
 struct css_task_iter {
 	struct cgroup_subsys		*ss;
-- 
2.4.3

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1199534 — Re: [PATCH 1/3] cgroup: define controller file conventions

On Mon, Aug 03, 2015 at 06:41:27PM -0400, Tejun Heo wrote:
> 
> This patch defines CGROUP_WEIGHT_MIN/DFL/MAX to be used on all weight
> based control knobs and documents the conventions that controllers
> should follow on the unified hierarchy.  Except for io.weight knob,
> all existing unified hierarchy knobs are already compliant.  A
> follow-up patch will update io.weight.

> +- If a controller implements ratio based resource distribution, the
> +  control knob should be named "weight" and have the range [1, 10000]
> +  and 100 should be the default value.  The values are chosen to allow
> +  enough and symmetric bias in both directions while keeping it
> +  intuitive (the default is 100%).

Aside from 100% being a sane 'default', what it actually is is a unit.
100% == 1.

So I would suggest naming the thing CGROUP_WEIGHT_UNIT := 100,

> +/*
> + * All weight knobs on the default hierarhcy should use the following min,
> + * default and max values.  The default value is the logarithmic center of
> + * MIN and MAX and allows 100x to be expressed in both directions.
> + */
> +#define CGROUP_WEIGHT_MIN		1
> +#define CGROUP_WEIGHT_DFL		100
> +#define CGROUP_WEIGHT_MAX		10000

That said, I'm not entirely keen on having to change this.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1199984 — Re: [PATCH 1/3] cgroup: define controller file conventions

Hello, Peter.

On Tue, Aug 04, 2015 at 10:48:55AM +0200, Peter Zijlstra wrote:
> > +- If a controller implements ratio based resource distribution, the
> > +  control knob should be named "weight" and have the range [1, 10000]
> > +  and 100 should be the default value.  The values are chosen to allow
> > +  enough and symmetric bias in both directions while keeping it
> > +  intuitive (the default is 100%).
> 
> Aside from 100% being a sane 'default', what it actually is is a unit.
> 100% == 1.
> 
> So I would suggest naming the thing CGROUP_WEIGHT_UNIT := 100,

It's a minor point either way but I think people would generally find
default more familiar.

> > +/*
> > + * All weight knobs on the default hierarhcy should use the following min,
> > + * default and max values.  The default value is the logarithmic center of
> > + * MIN and MAX and allows 100x to be expressed in both directions.
> > + */
> > +#define CGROUP_WEIGHT_MIN		1
> > +#define CGROUP_WEIGHT_DFL		100
> > +#define CGROUP_WEIGHT_MAX		10000
> 
> That said, I'm not entirely keen on having to change this.

Yeah, changing the scale is an icky thing to do but I think the
benefits of unifying the scales across different controllers outweigh
here.

Thanks.

-- 
tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1199535 — Re: [PATCH 1/3] cgroup: define controller file conventions

On Mon, Aug 03, 2015 at 06:41:27PM -0400, Tejun Heo wrote:
> +- If a controller implements an absolute resource limit, the control
> +  knob should be named "max".  The special token "max" should be used
> +  to represent no limit for both reading and writing.

So what do you do with minimal resource guarantees? That's still an
absolute resource limit and 'max' is obviously the wrong name.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1199978 — Re: [PATCH 1/3] cgroup: define controller file conventions

Hello,

On Tue, Aug 04, 2015 at 10:42:57AM +0200, Peter Zijlstra wrote:
> On Mon, Aug 03, 2015 at 06:41:27PM -0400, Tejun Heo wrote:
> > +- If a controller implements an absolute resource limit, the control
> > +  knob should be named "max".  The special token "max" should be used
> > +  to represent no limit for both reading and writing.
> 
> So what do you do with minimal resource guarantees? That's still an
> absolute resource limit and 'max' is obviously the wrong name.

The whole spectrum is min, low, high, max where min, max are absolute
guarantee, upper limit and low, high are best effort ones.  Will
update the doc.

Thanks.

-- 
tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1200246 — [PATCH v2 1/3] cgroup: define controller file conventions

From 6abc8ca19df0078de17dc38340db3002ed489ce7 Mon Sep 17 00:00:00 2001
From: Tejun Heo <tj@kernel.org>
Date: Tue, 4 Aug 2015 15:20:55 -0400

Traditionally, each cgroup controller implemented whatever interface
it wanted leading to interfaces which are widely inconsistent.
Examining the requirements of the controllers readily yield that there
are only a few control schemes shared among all.

Two major controllers already had to implement new interface for the
unified hierarchy due to significant structural changes.  Let's take
the chance to establish common conventions throughout all controllers.

This patch defines CGROUP_WEIGHT_MIN/DFL/MAX to be used on all weight
based control knobs and documents the conventions that controllers
should follow on the unified hierarchy.  Except for io.weight knob,
all existing unified hierarchy knobs are already compliant.  A
follow-up patch will update io.weight.

v2: Added descriptions of min, low and high knobs.

Signed-off-by: Tejun Heo <tj@kernel.org>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Cc: Li Zefan <lizefan@huawei.com>
Cc: Peter Zijlstra <peterz@infradead.org>
---
Hello,

Added low/high descriptions and applied to the following git branch.

 git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup.git for-4.3-unified-base

The branch currently only contains this patch and will stay stable so
that it can be pulled from.  I kept the base weight as DFL for now.
If we decide to change it, I'll apply the change on top.

Thanks.

 Documentation/cgroups/unified-hierarchy.txt | 80 ++++++++++++++++++++++++++---
 include/linux/cgroup.h                      |  9 ++++
 2 files changed, 81 insertions(+), 8 deletions(-)

diff --git a/Documentation/cgroups/unified-hierarchy.txt b/Documentation/cgroups/unified-hierarchy.txt
index 86847a7..1ee9caf 100644
--- a/Documentation/cgroups/unified-hierarchy.txt
+++ b/Documentation/cgroups/unified-hierarchy.txt
@@ -23,10 +23,13 @@ CONTENTS
 5. Other Changes
   5-1. [Un]populated Notification
   5-2. Other Core Changes
-  5-3. Per-Controller Changes
-    5-3-1. blkio
-    5-3-2. cpuset
-    5-3-3. memory
+  5-3. Controller File Conventions
+    5-3-1. Format
+    5-3-2. Control Knobs
+  5-4. Per-Controller Changes
+    5-4-1. blkio
+    5-4-2. cpuset
+    5-4-3. memory
 6. Planned Changes
   6-1. CAP for resource control
 
@@ -372,14 +375,75 @@ supported and the interface files "release_agent" and
 - The "cgroup.clone_children" file is removed.
 
 
-5-3. Per-Controller Changes
+5-3. Controller File Conventions
 
-5-3-1. blkio
+5-3-1. Format
+
+In general, all controller files should be in one of the following
+formats whenever possible.
+
+- Values only files
+
+  VAL0 VAL1...\n
+
+- Flat keyed files
+
+  KEY0 VAL0\n
+  KEY1 VAL1\n
+  ...
+
+- Nested keyed files
+
+  KEY0 SUB_KEY0=VAL00 SUB_KEY1=VAL01...
+  KEY1 SUB_KEY0=VAL10 SUB_KEY1=VAL11...
+  ...
+
+For a writeable file, the format for writing should generally match
+reading; however, controllers may allow omitting later fields or
+implement restricted shortcuts for most common use cases.
+
+For both flat and nested keyed files, only the values for a single key
+can be written at a time.  For nested keyed files, the sub key pairs
+may be specified in any order and not all pairs have to be specified.
+
+
+5-3-2. Control Knobs
+
+- Settings for a single feature should generally be implemented in a
+  single file.
+
+- In general, the root cgroup should be exempt from resource control
+  and thus shouldn't have resource control knobs.
+
+- If a controller implements ratio based resource distribution, the
+  control knob should be named "weight" and have the range [1, 10000]
+  and 100 should be the default value.  The values are chosen to allow
+  enough and symmetric bias in both directions while keeping it
+  intuitive (the default is 100%).
+
+- If a controller implements an absolute resource guarantee and/or
+  limit, the control knobs should be named "min" and "max"
+  respectively.  If a controller implements best effort resource
+  gurantee and/or limit, the control knobs should be named "low" and
+  "high" respectively.
+
+  In the above four control files, the special token "max" should be
+  used to represent upward infinity for both reading and writing.
+
+- If a setting has configurable default value and specific overrides,
+  the default settings should be keyed with "default" and appear as
+  the first entry in the file.  Specific entries can use "default" as
+  its value to indicate inheritance of the default value.
+
+
+5-4. Per-Controller Changes
+
+5-4-1. blkio
 
 - blk-throttle becomes properly hierarchical.
 
 
-5-3-2. cpuset
+5-4-2. cpuset
 
 - Tasks are kept in empty cpusets after hotplug and take on the masks
   of the nearest non-empty ancestor, instead of being moved to it.
@@ -388,7 +452,7 @@ supported and the interface files "release_agent" and
   masks of the nearest non-empty ancestor.
 
 
-5-3-3. memory
+5-4-3. memory
 
 - use_hierarchy is on by default and the cgroup file for the flag is
   not created.
diff --git a/include/linux/cgroup.h b/include/linux/cgroup.h
index a593e29..c6bf9d3 100644
--- a/include/linux/cgroup.h
+++ b/include/linux/cgroup.h
@@ -22,6 +22,15 @@
 
 #ifdef CONFIG_CGROUPS
 
+/*
+ * All weight knobs on the default hierarhcy should use the following min,
+ * default and max values.  The default value is the logarithmic center of
+ * MIN and MAX and allows 100x to be expressed in both directions.
+ */
+#define CGROUP_WEIGHT_MIN		1
+#define CGROUP_WEIGHT_DFL		100
+#define CGROUP_WEIGHT_MAX		10000
+
 /* a css_task_iter should be treated as an opaque object */
 struct css_task_iter {
 	struct cgroup_subsys		*ss;
-- 
2.4.3

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | linux.kernel

csiph-web