Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1662970 > unrolled thread

[PATCHSET for-4.13] cgroup: implement cgroup2 thread mode, v2

Back to article view | Back to linux.kernel

Contents

  [PATCHSET for-4.13] cgroup: implement cgroup2 thread mode, v2 Tejun Heo <tj@kernel.org> - 2017-06-10 16:10 +0200
    [PATCH 06/10] cgroup: implement CSS_TASK_ITER_THREADED Tejun Heo <tj@kernel.org> - 2017-06-10 16:10 +0200
    Re: [PATCHSET for-4.13] cgroup: implement cgroup2 thread mode, v2 Peter Zijlstra <peterz@infradead.org> - 2017-06-12 14:40 +0200
      Re: [PATCHSET for-4.13] cgroup: implement cgroup2 thread mode, v2 Tejun Heo <tj@kernel.org> - 2017-06-12 23:30 +0200
        Re: [PATCHSET for-4.13] cgroup: implement cgroup2 thread mode, v2 Tejun Heo <tj@kernel.org> - 2017-06-15 22:20 +0200
    [PATCH] cgroup: update debug controller to print out thread mode  information Tejun Heo <tj@kernel.org> - 2017-06-15 22:20 +0200

#1662970 — [PATCHSET for-4.13] cgroup: implement cgroup2 thread mode, v2

Hello,

This is v2 of cgroup2 thread mode patchset.  The changes from the last
take[L] are

* Support for mixed thread mode for the root cgroup added.  This
  allows the root cgroup to serve as both a thread root and a parent
  to domain cgroups.  This allows users to use thread mode without any
  nesting while not interfering with domain level cgroup operations.
  Note that this simply makes use of the fact that the system root has
  always been exempt from no-internal-process constraint.  The whole
  resource hierarchy still follows the same basic rules
  w.r.t. resource domains.

* Thread mode enable / disable now piggy backs on the existing control
  mask update mechanism rather than implementing manual css_set update
  mechanism of its own.  This makes the mechanism simpler and more
  flexible.

* Fixes and cleanups, including a fix from Waiman.

It is largely based on the discussions that we had at the plumbers
last year.  Here's the rough outline.

* Thread mode is explicitly enabled on a cgroup by writing "enable"
  into "cgroup.threads" file.  The cgroup shouldn't have any child
  cgroups or enabled controllers.

* Once enabled, arbitrary sub-hierarchy can be created and threads can
  be put anywhere in the subtree by writing TIDs into "cgroup.threads"
  file.  Process granularity and no-internal-process constraint don't
  apply in a threaded subtree.

* To be used in a threaded subtree, controllers should explicitly
  declare thread mode support and should be able to handle internal
  competition in some way.

* The root of a threaded subtree serves as the resource domain for the
  whole subtree.  This is where all the controllers are guaranteed to
  have a common ground and resource consumptions in the threaded
  subtree which aren't tied to a specific thread are charged.
  Non-threaded controllers never see beyond thread root and can assume
  that all controllers will follow the same rules upto that point.

* Root cgroup can enable thread mode anytime and a first level child
  can opt-in to that thread subtree anchored at root by writing "join"
  to "cgroup.threads" files, start its own thread subtree or just be a
  normal cgroup.

This allows threaded controllers to implement thread granular resource
control without getting in the way of system level resource
partitioning.

This patchset contains the following ten patches.  For more details on
the interface and behavior, please refer to 0007.

 0001-cgroup-separate-out-cgroup_has_tasks.patch
 0002-cgroup-reorganize-cgroup.procs-task-write-path.patch
 0003-cgroup-Fix-reference-counting-bug-in-cgroup_procs_wr.patch
 0004-cgroup-add-flags-to-css_task_iter_start-and-implemen.patch
 0005-cgroup-introduce-cgroup-proc_cgrp-and-threaded-css_s.patch
 0006-cgroup-implement-CSS_TASK_ITER_THREADED.patch
 0007-cgroup-implement-cgroup-v2-thread-support.patch
 0008-sched-Misc-preps-for-cgroup-unified-hierarchy-interf.patch
 0009-sched-Implement-interface-for-cgroup-unified-hierarc.patch
 0010-sched-Make-cpu-cpuacct-threaded-controllers.patch

0001-0007 implement cgroup2 thread mode.  0008-0010 enable CPU
controller on cgroup2 and mark them as supporting thread mode.
0008-0010 are included for reference.

The patchset is based on the current master 179145e6312b ("Merge tag
'iommu-fixes-v4.12-rc4' of
git://git.kernel.org/pub/scm/linux/kernel/git/joro/iommu") and also
available in the following git branch.

 git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup.git review-cgroup2-threads-v2

diffstat follows.

 Documentation/cgroup-v2.txt     |   99 +++++
 include/linux/cgroup-defs.h     |   34 +
 include/linux/cgroup.h          |   12 
 kernel/cgroup/cgroup-internal.h |    8 
 kernel/cgroup/cgroup-v1.c       |   64 +++
 kernel/cgroup/cgroup.c          |  727 ++++++++++++++++++++++++++++++++--------
 kernel/cgroup/cpuset.c          |    6 
 kernel/cgroup/freezer.c         |    6 
 kernel/cgroup/pids.c            |    1 
 kernel/events/core.c            |    1 
 kernel/sched/core.c             |  150 ++++++++
 kernel/sched/cpuacct.c          |   53 ++
 kernel/sched/cpuacct.h          |    5 
 mm/memcontrol.c                 |    2 
 net/core/netclassid_cgroup.c    |    2 
 15 files changed, 992 insertions(+), 178 deletions(-)

Thanks.

--
tejun

[L] http://lkml.kernel.org/r/20170202200632.13992-1-tj@kernel.org

[toc] | [next] | [standalone]

#1662971 — [PATCH 06/10] cgroup: implement CSS_TASK_ITER_THREADED

cgroup v2 is in the process of growing thread granularity support.
Once thread mode is enabled, the root cgroup of the subtree serves as
the proc_cgrp to which the processes of the subtree conceptually
belong and domain-level resource consumptions not tied to any specific
task are charged.  In the subtree, threads won't be subject to process
granularity or no-internal-task constraint and can be distributed
arbitrarily across the subtree.

This patch implements a new task iterator flag CSS_TASK_ITER_THREADED,
which, when used on a proc_cgrp, makes the iteration include the tasks
on all the associated threaded css_sets.  "cgroup.procs" read path is
updated to use it so that reading the file on a proc_cgrp lists all
processes.  This will also be used by controller implementations which
need to walk processes or tasks at the resource domain level.

Task iteration is implemented nested in css_set iteration.  If
CSS_TASK_ITER_THREADED is specified, after walking tasks of each
!threaded css_set, all the associated threaded css_sets are visited
before moving onto the next !threaded css_set.

Signed-off-by: Tejun Heo <tj@kernel.org>
---
 include/linux/cgroup.h |  6 ++++
 kernel/cgroup/cgroup.c | 81 +++++++++++++++++++++++++++++++++++++++++---------
 2 files changed, 73 insertions(+), 14 deletions(-)

diff --git a/include/linux/cgroup.h b/include/linux/cgroup.h
index 8755eb5ece18..05da7cc9ea2d 100644
--- a/include/linux/cgroup.h
+++ b/include/linux/cgroup.h
@@ -38,6 +38,8 @@
 
 /* walk only threadgroup leaders */
 #define CSS_TASK_ITER_PROCS		(1U << 0)
+/* walk threaded css_sets as part of their proc_csets */
+#define CSS_TASK_ITER_THREADED		(1U << 1)
 
 /* a css_task_iter should be treated as an opaque object */
 struct css_task_iter {
@@ -47,11 +49,15 @@ struct css_task_iter {
 	struct list_head		*cset_pos;
 	struct list_head		*cset_head;
 
+	struct list_head		*tcset_pos;
+	struct list_head		*tcset_head;
+
 	struct list_head		*task_pos;
 	struct list_head		*tasks_head;
 	struct list_head		*mg_tasks_head;
 
 	struct css_set			*cur_cset;
+	struct css_set			*cur_pcset;
 	struct task_struct		*cur_task;
 	struct list_head		iters_node;	/* css_set->task_iters */
 };
diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c
index 0fa4ffe84933..765c1c27c879 100644
--- a/kernel/cgroup/cgroup.c
+++ b/kernel/cgroup/cgroup.c
@@ -3610,27 +3610,36 @@ bool css_has_online_children(struct cgroup_subsys_state *css)
 	return ret;
 }
 
-/**
- * css_task_iter_advance_css_set - advance a task itererator to the next css_set
- * @it: the iterator to advance
- *
- * Advance @it to the next css_set to walk.
- */
-static void css_task_iter_advance_css_set(struct css_task_iter *it)
+static struct css_set *css_task_iter_next_css_set(struct css_task_iter *it)
 {
-	struct list_head *l = it->cset_pos;
+	bool threaded = it->flags & CSS_TASK_ITER_THREADED;
+	struct list_head *l;
 	struct cgrp_cset_link *link;
 	struct css_set *cset;
 
 	lockdep_assert_held(&css_set_lock);
 
-	/* Advance to the next non-empty css_set */
+	/* find the next threaded cset */
+	if (it->tcset_pos) {
+		l = it->tcset_pos->next;
+
+		if (l != it->tcset_head) {
+			it->tcset_pos = l;
+			return container_of(l, struct css_set,
+					    threaded_csets_node);
+		}
+
+		it->tcset_pos = NULL;
+	}
+
+	/* find the next cset */
+	l = it->cset_pos;
+
 	do {
 		l = l->next;
 		if (l == it->cset_head) {
 			it->cset_pos = NULL;
-			it->task_pos = NULL;
-			return;
+			return NULL;
 		}
 
 		if (it->ss) {
@@ -3640,10 +3649,50 @@ static void css_task_iter_advance_css_set(struct css_task_iter *it)
 			link = list_entry(l, struct cgrp_cset_link, cset_link);
 			cset = link->cset;
 		}
-	} while (!css_set_populated(cset));
+
+		/*
+		 * For threaded iterations, threaded csets are walked
+		 * together with their proc_csets.  Skip here.
+		 */
+	} while (threaded && css_set_threaded(cset));
 
 	it->cset_pos = l;
 
+	/* initialize threaded cset walking */
+	if (threaded) {
+		if (it->cur_pcset)
+			put_css_set_locked(it->cur_pcset);
+		it->cur_pcset = cset;
+		get_css_set(cset);
+
+		it->tcset_head = &cset->threaded_csets;
+		it->tcset_pos = &cset->threaded_csets;
+	}
+
+	return cset;
+}
+
+/**
+ * css_task_iter_advance_css_set - advance a task itererator to the next css_set
+ * @it: the iterator to advance
+ *
+ * Advance @it to the next css_set to walk.
+ */
+static void css_task_iter_advance_css_set(struct css_task_iter *it)
+{
+	struct css_set *cset;
+
+	lockdep_assert_held(&css_set_lock);
+
+	/* Advance to the next non-empty css_set */
+	do {
+		cset = css_task_iter_next_css_set(it);
+		if (!cset) {
+			it->task_pos = NULL;
+			return;
+		}
+	} while (!css_set_populated(cset));
+
 	if (!list_empty(&cset->tasks))
 		it->task_pos = cset->tasks.next;
 	else
@@ -3786,6 +3835,9 @@ void css_task_iter_end(struct css_task_iter *it)
 		spin_unlock_irq(&css_set_lock);
 	}
 
+	if (it->cur_pcset)
+		put_css_set(it->cur_pcset);
+
 	if (it->cur_task)
 		put_task_struct(it->cur_task);
 }
@@ -3811,6 +3863,7 @@ static void *cgroup_procs_start(struct seq_file *s, loff_t *pos)
 	struct kernfs_open_file *of = s->private;
 	struct cgroup *cgrp = seq_css(s)->cgroup;
 	struct css_task_iter *it = of->priv;
+	unsigned iter_flags = CSS_TASK_ITER_PROCS | CSS_TASK_ITER_THREADED;
 
 	/*
 	 * When a seq_file is seeked, it's always traversed sequentially
@@ -3824,10 +3877,10 @@ static void *cgroup_procs_start(struct seq_file *s, loff_t *pos)
 		if (!it)
 			return ERR_PTR(-ENOMEM);
 		of->priv = it;
-		css_task_iter_start(&cgrp->self, CSS_TASK_ITER_PROCS, it);
+		css_task_iter_start(&cgrp->self, iter_flags, it);
 	} else if (!(*pos)++) {
 		css_task_iter_end(it);
-		css_task_iter_start(&cgrp->self, CSS_TASK_ITER_PROCS, it);
+		css_task_iter_start(&cgrp->self, iter_flags, it);
 	}
 
 	return cgroup_procs_next(s, NULL, NULL);
-- 
2.13.0

[toc] | [prev] | [next] | [standalone]

#1663470

Please don't rush this; also, I might not be around much the coming
weeks due to taking some leave 'soon' (kid #3 is imminent).

And I really need more time to look at this (and re-read the old
discussions, because I've forgot most everything again).

On Sat, Jun 10, 2017 at 10:03:41AM -0400, Tejun Heo wrote:

> * Thread mode is explicitly enabled on a cgroup by writing "enable"
>   into "cgroup.threads" file.  The cgroup shouldn't have any child
>   cgroups or enabled controllers.
> 
> * Once enabled, arbitrary sub-hierarchy can be created and threads can
>   be put anywhere in the subtree by writing TIDs into "cgroup.threads"
>   file.  Process granularity and no-internal-process constraint don't
>   apply in a threaded subtree.
> 
> * To be used in a threaded subtree, controllers should explicitly
>   declare thread mode support and should be able to handle internal
>   competition in some way.
> 
> * The root of a threaded subtree serves as the resource domain for the
>   whole subtree.  This is where all the controllers are guaranteed to
>   have a common ground and resource consumptions in the threaded
>   subtree which aren't tied to a specific thread are charged.
>   Non-threaded controllers never see beyond thread root and can assume
>   that all controllers will follow the same rules upto that point.
> 
> * Root cgroup can enable thread mode anytime and a first level child
>   can opt-in to that thread subtree anchored at root by writing "join"
>   to "cgroup.threads" files, start its own thread subtree or just be a
>   normal cgroup.

Yuck... this again is a consequence of tagging the 'wrong' thing. Again,
the primary construct is the resource domain.

If you use that as a tag, you don't need this weird join crap. Because
as soon as you clear the 'resource domain' flag on a group, it instantly
becomes a thread group and 'obviously' connects to the first parent that
is a resource domain.

And, as per the last time, this threaded marker isn't uniquely
identifying things, so it hard prohibits from ever extending the model
to allow resource domains nested in a thread subtree. Now I understand
why you don't implement that now -- you were struggling with the views
API, but that is no excuse to create an API that permanently disables
that feature.

I cannot at this time remember if there was a strong use-case for that
scenario -- like said, I really need to re-read the email threads, but I
might not have enough time to do so now.

Again, please don't rush this.

> This allows threaded controllers to implement thread granular resource
> control without getting in the way of system level resource
> partitioning.
> 
> This patchset contains the following ten patches.  For more details on
> the interface and behavior, please refer to 0007.
> 
>  0001-cgroup-separate-out-cgroup_has_tasks.patch
>  0002-cgroup-reorganize-cgroup.procs-task-write-path.patch
>  0003-cgroup-Fix-reference-counting-bug-in-cgroup_procs_wr.patch
>  0004-cgroup-add-flags-to-css_task_iter_start-and-implemen.patch
>  0005-cgroup-introduce-cgroup-proc_cgrp-and-threaded-css_s.patch
>  0006-cgroup-implement-CSS_TASK_ITER_THREADED.patch
>  0007-cgroup-implement-cgroup-v2-thread-support.patch
>  0008-sched-Misc-preps-for-cgroup-unified-hierarchy-interf.patch
>  0009-sched-Implement-interface-for-cgroup-unified-hierarc.patch
>  0010-sched-Make-cpu-cpuacct-threaded-controllers.patch
> 
> 0001-0007 implement cgroup2 thread mode.  0008-0010 enable CPU
> controller on cgroup2 and mark them as supporting thread mode.
> 0008-0010 are included for reference.

So I really regret the 'shares' interface; we really should have done a
nice thing.

  https://lkml.kernel.org/r/20170410073622.2y6tnpcd2ssuoztz@hirez.programming.kicks-ass.net

So I would like to change to that instead of the weird 100 thing.

As for the RT thing, the runtime/period thing is not a MAX but a MIN
limit (conceptually -- in practise its both).

Also, we need cpuset to be a thread controller.

[toc] | [prev] | [next] | [standalone]

#1664240

Hello, Peter.

On Mon, Jun 12, 2017 at 02:31:50PM +0200, Peter Zijlstra wrote:
> Please don't rush this; also, I might not be around much the coming
> weeks due to taking some leave 'soon' (kid #3 is imminent).

Congrats.  As for this going forward, how can we possibly be slower?

> And I really need more time to look at this (and re-read the old
> discussions, because I've forgot most everything again).

Can we at least unblock the cpu controller part?  We can hash the
details of thread support as long as necessary but I'm not sure it's
reasonable to keep blocking the whole cpu controller at this point.

> > * Root cgroup can enable thread mode anytime and a first level child
> >   can opt-in to that thread subtree anchored at root by writing "join"
> >   to "cgroup.threads" files, start its own thread subtree or just be a
> >   normal cgroup.
> 
> Yuck... this again is a consequence of tagging the 'wrong' thing. Again,
> the primary construct is the resource domain.
> 
> If you use that as a tag, you don't need this weird join crap. Because
> as soon as you clear the 'resource domain' flag on a group, it instantly
> becomes a thread group and 'obviously' connects to the first parent that
> is a resource domain.

It has nothing to do with whether we mark domain or threaded subtrees.
It is solely from whether you wanna express cases where a thread root
is right below another thread root.  Tn's are member cgroup of thread
subtrees where the same number means the same threaded subtree, D's
are of domain cgroups.

The following is straight forward.

	T0
       /  \
      T0   D

The following is too.

	T0
       /  \
      T0   D
            \
	     T1

The question is whether to allow something like the following.

	T0
       /  \
      T0  T1

That's where the "join" thing comes from because we wanna be able to
tell apart whether a cgroup is gonna be a part of the existing thread
subtree or starting its own thread subtree.  There sure are multiple
ways to express that but one way or the other, if you wanna support
topologies like the last one, you have to distinguish the two.

The previous iteration actually was that way, so the only thread mode
operation was setting whether to enable thread or not as before and if
the parent is already thread mode, it'd always join the existing
threaded subtree.  If you like that better, I can post that version
right away.

> And, as per the last time, this threaded marker isn't uniquely
> identifying things, so it hard prohibits from ever extending the model
> to allow resource domains nested in a thread subtree. Now I understand
> why you don't implement that now -- you were struggling with the views
> API, but that is no excuse to create an API that permanently disables
> that feature.

Hmmm?  We can just allow disabling thread mode if we ever get to that.
We can't make arbitrary graphs out of these nodes.  Whatever mode we
put them in, they have to fall in with the overall tree structure, so
I don't think the interface is unnecessarily restricting in that
direction.

> I cannot at this time remember if there was a strong use-case for that
> scenario -- like said, I really need to re-read the email threads, but I
> might not have enough time to do so now.
> 
> Again, please don't rush this.

Well, I don't have a way to do that.

> So I really regret the 'shares' interface; we really should have done a
> nice thing.
> 
>   https://lkml.kernel.org/r/20170410073622.2y6tnpcd2ssuoztz@hirez.programming.kicks-ass.net
> 
> So I would like to change to that instead of the weird 100 thing.

Is it?  Relative weights are pretty fundamental and clearly defined in
expressing work-conserving resource distribution.  Do you have more
details on what you have on mind?

> As for the RT thing, the runtime/period thing is not a MAX but a MIN
> limit (conceptually -- in practise its both).

Yeah, it's a hard allocation.

> Also, we need cpuset to be a thread controller.

Yeah, absolutely.

Thanks.

-- 
tejun

[toc] | [prev] | [next] | [standalone]

#1667231

Hello,

On Mon, Jun 12, 2017 at 05:27:53PM -0400, Tejun Heo wrote:
> The previous iteration actually was that way, so the only thread mode
> operation was setting whether to enable thread or not as before and if
> the parent is already thread mode, it'd always join the existing
> threaded subtree.  If you like that better, I can post that version
> right away.

Just posted v3 of the patch which drops the "join" operation, but do
note that this inherently comes at the cost of losing the ability
express certain topologies.

Thanks.

-- 
tejun

[toc] | [prev] | [next] | [standalone]

#1667234 — [PATCH] cgroup: update debug controller to print out thread mode information

From: Waiman Long <longman@redhat.com>

Update debug controller so that it prints out debug info about thread
mode.

 1) The relationship between proc_cset and threaded_csets are displayed.
 2) The status of being a thread root or threaded cgroup is displayed.

This patch is extracted from Waiman's larger patch.

Patch-originally-by: Waiman Long <longman@redhat.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
---
Hello,

Waiman, this is the thread mode support part of your debug patch.
I'll include this in the thread mode patchset and apply it together.
It seems to work fine but if you spot anything silly, please let me
know.

Thanks.

 kernel/cgroup/debug.c |   57 +++++++++++++++++++++++++++++++++++++-------------
 1 file changed, 43 insertions(+), 14 deletions(-)

--- a/kernel/cgroup/debug.c
+++ b/kernel/cgroup/debug.c
@@ -114,27 +114,53 @@ static int cgroup_css_links_read(struct
 {
 	struct cgroup_subsys_state *css = seq_css(seq);
 	struct cgrp_cset_link *link;
-	int dead_cnt = 0, extra_refs = 0;
+	int dead_cnt = 0, extra_refs = 0, threaded_csets = 0;
 
 	spin_lock_irq(&css_set_lock);
+
+	if (css->cgroup->proc_cgrp)
+		seq_puts(seq, (css->cgroup->proc_cgrp == css->cgroup) ?
+			 "[thread root]\n" : "[threaded]\n");
+
 	list_for_each_entry(link, &css->cgroup->cset_links, cset_link) {
 		struct css_set *cset = link->cset;
 		struct task_struct *task;
 		int count = 0;
 		int refcnt = refcount_read(&cset->refcount);
 
-		seq_printf(seq, " %d", refcnt);
-		if (refcnt - cset->nr_tasks > 0) {
-			int extra = refcnt - cset->nr_tasks;
-
-			seq_printf(seq, " +%d", extra);
-			/*
-			 * Take out the one additional reference in
-			 * init_css_set.
-			 */
-			if (cset == &init_css_set)
-				extra--;
-			extra_refs += extra;
+		/*
+		 * Print out the proc_cset and threaded_cset relationship
+		 * and highlight difference between refcount and task_count.
+		 */
+		seq_printf(seq, "css_set %pK", cset);
+		if (rcu_dereference_protected(cset->proc_cset, 1) != cset) {
+			threaded_csets++;
+			seq_printf(seq, "=>%pK", cset->proc_cset);
+		}
+		if (!list_empty(&cset->threaded_csets)) {
+			struct css_set *tcset;
+			int idx = 0;
+
+			list_for_each_entry(tcset, &cset->threaded_csets,
+					    threaded_csets_node) {
+				seq_puts(seq, idx ? "," : "<=");
+				seq_printf(seq, "%pK", tcset);
+				idx++;
+			}
+		} else {
+			seq_printf(seq, " %d", refcnt);
+			if (refcnt - cset->nr_tasks > 0) {
+				int extra = refcnt - cset->nr_tasks;
+
+				seq_printf(seq, " +%d", extra);
+				/*
+				 * Take out the one additional reference in
+				 * init_css_set.
+				 */
+				if (cset == &init_css_set)
+					extra--;
+				extra_refs += extra;
+			}
 		}
 		seq_puts(seq, "\n");
 
@@ -163,10 +189,12 @@ static int cgroup_css_links_read(struct
 	}
 	spin_unlock_irq(&css_set_lock);
 
-	if (!dead_cnt && !extra_refs)
+	if (!dead_cnt && !extra_refs && !threaded_csets)
 		return 0;
 
 	seq_puts(seq, "\n");
+	if (threaded_csets)
+		seq_printf(seq, "threaded css_sets = %d\n", threaded_csets);
 	if (extra_refs)
 		seq_printf(seq, "extra references = %d\n", extra_refs);
 	if (dead_cnt)
@@ -342,6 +370,7 @@ struct cgroup_subsys debug_cgrp_subsys =
 	.css_alloc	= debug_css_alloc,
 	.css_free	= debug_css_free,
 	.legacy_cftypes	= debug_legacy_files,
+	.threaded	= true,
 };
 
 /*

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web