Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1631074 > unrolled thread

[2/2] sched/fair: Fix O(# total cgroups) in load balance path

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  [2/2] sched/fair: Fix O(# total cgroups) in load balance path Tejun Heo <tj@kernel.org> - 2017-04-26 02:50 +0200
    Re: [2/2] sched/fair: Fix O(# total cgroups) in load balance path Peter Zijlstra <peterz@infradead.org> - 2017-05-01 18:20 +0200
      Re: [2/2] sched/fair: Fix O(# total cgroups) in load balance path Tejun Heo <tj@kernel.org> - 2017-05-01 21:20 +0200

#1631074 — [2/2] sched/fair: Fix O(# total cgroups) in load balance path

Currently, rq->leaf_cfs_rq_list is a traversal ordered list of all
live cfs_rqs which have ever been active on the CPU; unfortunately,
this makes update_blocked_averages() O(# total cgroups) which isn't
scalable at all.

This shows up as a small CPU consumption and scheduling latency
increase in the load balancing path in systems with CPU controller
enabled across most cgroups.  In an edge case where temporary cgroups
were leaking, this caused the kernel to consume good several tens of
percents of CPU cycles running update_blocked_averages(), each run
taking multiple millisecs.

This patch fixes the issue by taking empty and fully decayed cfs_rqs
off the rq->leaf_cfs_rq_list.

Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Paul Turner <pjt@google.com>
Cc: Chris Mason <clm@fb.com>
---
 kernel/sched/fair.c |   19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)

--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -369,8 +369,9 @@ static inline void list_del_leaf_cfs_rq(
 }
 
 /* Iterate thr' all leaf cfs_rq's on a runqueue */
-#define for_each_leaf_cfs_rq(rq, cfs_rq) \
-	list_for_each_entry_rcu(cfs_rq, &rq->leaf_cfs_rq_list, leaf_cfs_rq_list)
+#define for_each_leaf_cfs_rq_safe(rq, cfs_rq, pos)			\
+	list_for_each_entry_safe(cfs_rq, pos, &rq->leaf_cfs_rq_list,	\
+				 leaf_cfs_rq_list)
 
 /* Do the two (enqueued) entities belong to the same group ? */
 static inline struct cfs_rq *
@@ -463,7 +464,7 @@ static inline void list_del_leaf_cfs_rq(
 {
 }
 
-#define for_each_leaf_cfs_rq(rq, cfs_rq) \
+#define for_each_leaf_cfs_rq_safe(rq, cfs_rq, pos)	\
 		for (cfs_rq = &rq->cfs; cfs_rq; cfs_rq = NULL)
 
 static inline struct sched_entity *parent_entity(struct sched_entity *se)
@@ -6983,7 +6984,7 @@ static void attach_tasks(struct lb_env *
 static void update_blocked_averages(int cpu)
 {
 	struct rq *rq = cpu_rq(cpu);
-	struct cfs_rq *cfs_rq;
+	struct cfs_rq *cfs_rq, *pos;
 	struct rq_flags rf;
 
 	rq_lock_irqsave(rq, &rf);
@@ -6993,7 +6994,7 @@ static void update_blocked_averages(int
 	 * Iterates the task_group tree in a bottom up fashion, see
 	 * list_add_leaf_cfs_rq() for details.
 	 */
-	for_each_leaf_cfs_rq(rq, cfs_rq) {
+	for_each_leaf_cfs_rq_safe(rq, cfs_rq, pos) {
 		struct sched_entity *se;
 
 		/* throttled entities do not contribute to load */
@@ -7007,6 +7008,14 @@ static void update_blocked_averages(int
 		se = cfs_rq->tg->se[cpu];
 		if (se && !skip_blocked_update(se))
 			update_load_avg(se, 0);
+
+		/*
+		 * There can be a lot of idle CPU cgroups.  Don't let fully
+		 * decayed cfs_rqs linger on the list.
+		 */
+		if (!cfs_rq->load.weight && !cfs_rq->avg.load_sum &&
+		    !cfs_rq->avg.util_sum && !cfs_rq->runnable_load_sum)
+			list_del_leaf_cfs_rq(cfs_rq);
 	}
 	rq_unlock_irqrestore(rq, &rf);
 }

[toc] | [next] | [standalone]

#1633728

On Tue, Apr 25, 2017 at 05:43:50PM -0700, Tejun Heo wrote:
> @@ -7007,6 +7008,14 @@ static void update_blocked_averages(int
>  		se = cfs_rq->tg->se[cpu];
>  		if (se && !skip_blocked_update(se))
>  			update_load_avg(se, 0);
> +
> +		/*
> +		 * There can be a lot of idle CPU cgroups.  Don't let fully
> +		 * decayed cfs_rqs linger on the list.
> +		 */
> +		if (!cfs_rq->load.weight && !cfs_rq->avg.load_sum &&
> +		    !cfs_rq->avg.util_sum && !cfs_rq->runnable_load_sum)
> +			list_del_leaf_cfs_rq(cfs_rq);
>  	}
>  	rq_unlock_irqrestore(rq, &rf);
>  }

Right this is a 'known' issue and we recently talked about this.

I think you got the condition right, we want to wait for all the stuff
to be decayed out before taking it off the list.

The only 'problem', which Vincent mentioned in that other thread, is that
NOHZ idle doesn't guarantee decay -- then again, you don't want to go
wake a CPU just to decay this crud either. And if we're idle, the list
being long doesn't matter either.

[toc] | [prev] | [next] | [standalone]

#1633818

Hello, Peter.

On Mon, May 01, 2017 at 06:11:58PM +0200, Peter Zijlstra wrote:
> On Tue, Apr 25, 2017 at 05:43:50PM -0700, Tejun Heo wrote:
> > @@ -7007,6 +7008,14 @@ static void update_blocked_averages(int
> >  		se = cfs_rq->tg->se[cpu];
> >  		if (se && !skip_blocked_update(se))
> >  			update_load_avg(se, 0);
> > +
> > +		/*
> > +		 * There can be a lot of idle CPU cgroups.  Don't let fully
> > +		 * decayed cfs_rqs linger on the list.
> > +		 */
> > +		if (!cfs_rq->load.weight && !cfs_rq->avg.load_sum &&
> > +		    !cfs_rq->avg.util_sum && !cfs_rq->runnable_load_sum)
> > +			list_del_leaf_cfs_rq(cfs_rq);
> >  	}
> >  	rq_unlock_irqrestore(rq, &rf);
> >  }
> 
> Right this is a 'known' issue and we recently talked about this.
> 
> I think you got the condition right, we want to wait for all the stuff
> to be decayed out before taking it off the list.
> 
> The only 'problem', which Vincent mentioned in that other thread, is that
> NOHZ idle doesn't guarantee decay -- then again, you don't want to go
> wake a CPU just to decay this crud either. And if we're idle, the list
> being long doesn't matter either.

The list staying long is fine as long as nobody walks it; however, the
list can be *really* long, e.g. hundreds of thousands long, so walking
it repeatedly won't be a good idea even if the system is idle.  As
long as NOHZ decays and trims the list when it ends up walking the
list, and AFAICS it does, it should be fine.

Thanks.

-- 
tejun

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web