Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes

From	Peter Zijlstra <peterz@infradead.org>
Newsgroups	linux.kernel
Subject	Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes
Date	2016-10-18 11:10 +0200
Message-ID	<styzg-83y-17@gated-at.bofh.it> (permalink)
References	(5 earlier) <stcfn-1yQ-3@gated-at.bofh.it> <steKd-2XK-21@gated-at.bofh.it> <stg9j-40N-9@gated-at.bofh.it> <stgCl-4aL-11@gated-at.bofh.it> <stp2V-1JC-19@gated-at.bofh.it>
Organization	linux.* mail to news gateway

Show all headers | View raw

On Mon, Oct 17, 2016 at 11:52:39PM +0100, Dietmar Eggemann wrote:
> 
> Something looks weird related to the use of for_each_possible_cpu(i) in
> online_fair_sched_group() on my i5-3320M CPU (4 logical cpus).
> 
> In case I print out cpu id and the cpu masks inside the for_each_possible_cpu(i)
> I get:
> 
> [ 5.462368]  cpu=0  cpu_possible_mask=0-7 cpu_online_mask=0-3 cpu_present_mask=0-3 cpu_active_mask=0-3

OK, you have a buggy BIOS :-) It enumerates too many CPU slots. There is
no reason to have 4 empty CPU slots on a machine that cannot do physical
hotplug.

This also explains why it doesn't show on many machines, most machines
will not have this and possible_mask == present_mask == online_mask for
most all cases.

x86 folk, can we detect the lack of physical hotplug capability and FW_WARN
on this and lowering possible_mask to present_mask?

> [ 5.462370]  cpu=1  cpu_possible_mask=0-7 cpu_online_mask=0-3 cpu_present_mask=0-3 cpu_active_mask=0-3
> [ 5.462370]  cpu=2  cpu_possible_mask=0-7 cpu_online_mask=0-3 cpu_present_mask=0-3 cpu_active_mask=0-3
> [ 5.462371]  cpu=3  cpu_possible_mask=0-7 cpu_online_mask=0-3 cpu_present_mask=0-3 cpu_active_mask=0-3
> [ 5.462372] *cpu=4* cpu_possible_mask=0-7 cpu_online_mask=0-3 cpu_present_mask=0-3 cpu_active_mask=0-3
> [ 5.462373] *cpu=5* cpu_possible_mask=0-7 cpu_online_mask=0-3 cpu_present_mask=0-3 cpu_active_mask=0-3
> [ 5.462374] *cpu=6* cpu_possible_mask=0-7 cpu_online_mask=0-3 cpu_present_mask=0-3 cpu_active_mask=0-3
> [ 5.462375] *cpu=7* cpu_possible_mask=0-7 cpu_online_mask=0-3 cpu_present_mask=0-3 cpu_active_mask=0-3
> 
> T430:/sys/fs/cgroup/cpu,cpuacct/system.slice# ls -l | grep '^d' | wc -l
> 80
> 
> /proc/sched_debug:
> 
> cfs_rq[0]:/system.slice
>   ...
>   .tg_load_avg                   : 323584
>   ...
> 
> 80 * 1024 * 4 (not existent cpu4-cpu7) = 327680 (with a little bit of decay,
> this could be this extra load on the systen.slice tg)
> 
> Using for_each_online_cpu(i) instead of for_each_possible_cpu(i) in
> online_fair_sched_group() works on this machine, i.e. the .tg_load_avg
> of system.slice tg is 0 after startup.

Right, so the reason for using present_mask is that it avoids having to
deal with hotplug, also all the per-cpu memory is allocated and present
for !online CPUs anyway, so might as well set it up properly anyway.

(You might want to start booting your laptop with "possible_cpus=4" to
save some memory FWIW.)

But yes, we have a bug here too... /me ponders

So aside from funny BIOSes, this should also show up when creating
cgroups when you have offlined a few CPUs, which is far more common I'd
think.

On IRC you mentioned that adding list_add_leaf_cfs_rq() to
online_fair_sched_group() cures this, this would actually match with
unregister_fair_sched_group() doing list_del_leaf_cfs_rq() and avoid
a few instructions on the enqueue path, so that's all good.

I'm just not immediately seeing how that cures things. The only relevant
user of the leaf_cfs_rq list seems to be update_blocked_averages() which
is called from the balance code (idle_balance() and
rebalance_domains()). But neither should call that for offline (or
!present) CPUs.

Humm..

Thread

Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Vincent Guittot <vincent.guittot@linaro.org> - 2016-10-13 13:10 +0200
  Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Joseph Salisbury <joseph.salisbury@canonical.com> - 2016-10-13 18:00 +0200
    Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Vincent Guittot <vincent.guittot@linaro.org> - 2016-10-13 19:00 +0200
      Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Dietmar Eggemann <dietmar.eggemann@arm.com> - 2016-10-13 21:00 +0200
        Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Vincent Guittot <vincent.guittot@linaro.org> - 2016-10-13 23:40 +0200
          Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Vincent Guittot <vincent.guittot@linaro.org> - 2016-10-14 10:30 +0200
            Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Dietmar Eggemann <dietmar.eggemann@arm.com> - 2016-10-14 15:20 +0200
              Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Vincent Guittot <vincent.guittot@linaro.org> - 2016-10-14 17:20 +0200
                Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Joseph Salisbury <joseph.salisbury@canonical.com> - 2016-10-14 18:10 +0200
                Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Vincent Guittot <vincent.guittot@linaro.org> - 2016-10-17 11:20 +0200
                Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Dietmar Eggemann <dietmar.eggemann@arm.com> - 2016-10-17 14:00 +0200
                Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Peter Zijlstra <peterz@infradead.org> - 2016-10-17 15:30 +0200
                Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Vincent Guittot <vincent.guittot@linaro.org> - 2016-10-17 16:00 +0200
                Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Dietmar Eggemann <dietmar.eggemann@arm.com> - 2016-10-18 01:00 +0200
                Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Vincent Guittot <vincent.guittot@linaro.org> - 2016-10-18 10:50 +0200
                Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Peter Zijlstra <peterz@infradead.org> - 2016-10-18 11:10 +0200
                Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Vincent Guittot <vincent.guittot@linaro.org> - 2016-10-18 11:50 +0200
                Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Peter Zijlstra <peterz@infradead.org> - 2016-10-18 12:40 +0200
                Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Vincent Guittot <vincent.guittot@linaro.org> - 2016-10-18 14:00 +0200
                Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Joonwoo Park <joonwoop@codeaurora.org> - 2016-10-19 00:00 +0200
                Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Vincent Guittot <vincent.guittot@linaro.org> - 2016-10-19 08:50 +0200
                Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Vincent Guittot <vincent.guittot@linaro.org> - 2016-10-19 16:50 +0200
                Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Joseph Salisbury <joseph.salisbury@canonical.com> - 2016-10-19 17:00 +0200
                Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Vincent Guittot <vincent.guittot@linaro.org> - 2016-10-19 17:00 +0200
                Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Dietmar Eggemann <dietmar.eggemann@arm.com> - 2016-10-19 17:40 +0200
                Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Joonwoo Park <joonwoop@codeaurora.org> - 2016-10-19 19:40 +0200
                Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Vincent Guittot <vincent.guittot@linaro.org> - 2016-10-19 20:00 +0200
                Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Vincent Guittot <vincent.guittot@linaro.org> - 2016-10-19 17:50 +0200
                Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Dietmar Eggemann <dietmar.eggemann@arm.com> - 2016-10-19 17:50 +0200
                Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Peter Zijlstra <peterz@infradead.org> - 2016-10-19 18:20 +0200
                Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Morten Rasmussen <morten.rasmussen@arm.com> - 2016-10-19 18:40 +0200
                Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Vincent Guittot <vincent.guittot@linaro.org> - 2016-10-19 19:50 +0200
                Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Morten Rasmussen <morten.rasmussen@arm.com> - 2016-10-20 10:00 +0200
                Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Dietmar Eggemann <dietmar.eggemann@arm.com> - 2016-10-18 13:20 +0200
                Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Peter Zijlstra <peterz@infradead.org> - 2016-10-18 14:10 +0200

csiph-web