Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1639210

Re: [PATCH v2 for-4.12-fixes 2/2] sched/fair: Fix O(# total cgroups) in load balance path

Show all headers | View raw

Hi Tejun,

On 10 May 2017 at 16:44, Tejun Heo <tj@kernel.org> wrote:
> Hello,
>
> On Wed, May 10, 2017 at 08:50:14AM +0200, Vincent Guittot wrote:
>> On 9 May 2017 at 18:18, Tejun Heo <tj@kernel.org> wrote:
>> > Currently, rq->leaf_cfs_rq_list is a traversal ordered list of all
>> > live cfs_rqs which have ever been active on the CPU; unfortunately,
>> > this makes update_blocked_averages() O(# total cgroups) which isn't
>> > scalable at all.
>>
>> Dietmar raised similar optimization in the past. The only question was
>> : what is the impact of  re-adding the cfs_rq in leaf_cfs_rq_list on
>> the wake up path ? Have you done some measurements ?
>
> Didn't do a perf test yet but it's several more branches and a local
> list operation on enqueue, which is already pretty expensive vs. load
> balance being O(total number of cgroups on the system).
>
> Anyways, I'll do some hackbench tests with several levels of layering.
>
>> > @@ -7008,6 +7009,14 @@ static void update_blocked_averages(int
>> >                 se = cfs_rq->tg->se[cpu];
>> >                 if (se && !skip_blocked_update(se))
>> >                         update_load_avg(se, 0);
>> > +
>> > +               /*
>> > +                * There can be a lot of idle CPU cgroups.  Don't let fully
>> > +                * decayed cfs_rqs linger on the list.
>> > +                */
>> > +               if (!cfs_rq->load.weight && !cfs_rq->avg.load_sum &&
>> > +                   !cfs_rq->avg.util_sum && !cfs_rq->runnable_load_sum)
>> > +                       list_del_leaf_cfs_rq(cfs_rq);
>>
>> list_add_leaf_cfs_rq() assumes that we always enqueue cfs_rq bottom-up.
>> By removing  cfs_rq, can't we break this assumption in some cases ?
>
> We queue a cfs_rq on the leaf list when the a se is queued on that
> cfs_rq for the first time, so queueing can happen in any order;

Sorry, what i mean is:
When the group entity of a cfs_rq is enqueued, we are sure that either
the parents is already enqueued or it will be enqueued in the same
sequence. We must be sure that no other branch will be enqueued in the
middle of the sequence and will reset tmp_alone_branch.
This is true with current implementation but I  wondered it can happen
if we del/add the cfs_rq out of order

That said i haven't find a use case that break the sequence


> otherwise, we'd simply be doing list_add_tail().  AFAICS, removing and
> re-adding shouldn't break anything if the code wasn't broken before.
>
> Thanks.
>
> --
> tejun

Back to linux.kernel | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

[PATCH v2 for-4.12-fixes 1/2] sched/fair: Use task_groups instead of  leaf_cfs_rq_list to walk all cfs_rqs Tejun Heo <tj@kernel.org> - 2017-05-09 18:20 +0200
  [PATCH v2 for-4.12-fixes 2/2] sched/fair: Fix O(# total cgroups) in  load balance path Tejun Heo <tj@kernel.org> - 2017-05-09 18:20 +0200
    Re: [PATCH v2 for-4.12-fixes 2/2] sched/fair: Fix O(# total cgroups)  in load balance path Vincent Guittot <vincent.guittot@linaro.org> - 2017-05-10 09:00 +0200
      Re: [PATCH v2 for-4.12-fixes 2/2] sched/fair: Fix O(# total cgroups)  in load balance path Tejun Heo <tj@kernel.org> - 2017-05-10 16:50 +0200
        Re: [PATCH v2 for-4.12-fixes 2/2] sched/fair: Fix O(# total cgroups)  in load balance path Tejun Heo <tj@kernel.org> - 2017-05-10 18:00 +0200
        Re: [PATCH v2 for-4.12-fixes 2/2] sched/fair: Fix O(# total cgroups)  in load balance path Vincent Guittot <vincent.guittot@linaro.org> - 2017-05-11 09:10 +0200
    [PATCH v3 for-4.12-fixes 2/2] sched/fair: Fix O(# total cgroups) in  load balance path Tejun Heo <tj@kernel.org> - 2017-05-10 19:40 +0200

csiph-web