Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1491208 > unrolled thread
| Started by | Christian Borntraeger <borntraeger@de.ibm.com> |
|---|---|
| First post | 2016-09-26 12:50 +0200 |
| Last post | 2016-09-26 14:30 +0200 |
| Articles | 9 — 3 participants |
Back to article view | Back to linux.kernel
group scheduler regression since 4.3 (bisect 9d89c257d sched/fair: Rewrite runnable load and utilization average tracking) Christian Borntraeger <borntraeger@de.ibm.com> - 2016-09-26 12:50 +0200
Re: group scheduler regression since 4.3 (bisect 9d89c257d sched/fair: Rewrite runnable load and utilization average tracking) Peter Zijlstra <peterz@infradead.org> - 2016-09-26 13:00 +0200
Re: group scheduler regression since 4.3 (bisect 9d89c257d sched/fair: Rewrite runnable load and utilization average tracking) Christian Borntraeger <borntraeger@de.ibm.com> - 2016-09-26 13:50 +0200
Re: group scheduler regression since 4.3 (bisect 9d89c257d sched/fair: Rewrite runnable load and utilization average tracking) Peter Zijlstra <peterz@infradead.org> - 2016-09-26 14:00 +0200
Re: group scheduler regression since 4.3 (bisect 9d89c257d sched/fair: Rewrite runnable load and utilization average tracking) Christian Borntraeger <borntraeger@de.ibm.com> - 2016-09-26 14:10 +0200
Re: group scheduler regression since 4.3 (bisect 9d89c257d sched/fair: Rewrite runnable load and utilization average tracking) Peter Zijlstra <peterz@infradead.org> - 2016-09-26 14:20 +0200
Re: group scheduler regression since 4.3 (bisect 9d89c257d sched/fair: Rewrite runnable load and utilization average tracking) Christian Borntraeger <borntraeger@de.ibm.com> - 2016-09-26 15:00 +0200
Re: group scheduler regression since 4.3 (bisect 9d89c257d sched/fair: Rewrite runnable load and utilization average tracking) Christian Borntraeger <borntraeger@de.ibm.com> - 2016-09-26 16:20 +0200
Re: group scheduler regression since 4.3 (bisect 9d89c257d sched/fair: Rewrite runnable load and utilization average tracking) Vincent Guittot <vincent.guittot@linaro.org> - 2016-09-26 14:30 +0200
| From | Christian Borntraeger <borntraeger@de.ibm.com> |
|---|---|
| Date | 2016-09-26 12:50 +0200 |
| Subject | group scheduler regression since 4.3 (bisect 9d89c257d sched/fair: Rewrite runnable load and utilization average tracking) |
| Message-ID | <slBDX-2Mp-3@gated-at.bofh.it> |
Folks,
I have seen big scalability degredations sind 4.3 (bisected 9d89c257d
sched/fair: Rewrite runnable load and utilization average tracking)
This has not been fixed by subsequent patches,e.g. the ones that try to
fix this for interactive workload.
The problem is only visible for sleep/wakeup heavy workload which must
be part of the scheduler group (e.g. a sysbench OLTP inside a KVM guest
as libvirt will put KVM guests into cgroup instances).
For example a simple sysbench oltp with mysql inside a KVM guests with
16 CPUs backed by 8 host cpus (16 host threads) scales less (scale up
inside a guest, having multiple instances). This is the numbers of
events per second.
Unmounting /sys/fs/cgroup/cpu,cpuacct (thus forcing libvirt to not
use group scheduling for KVM guests) makes the behaviour much better:
instances group nogroup
1 3406 3002
2 5078 4940
3 6017 6760
4 6471 8216 (+27%)
5 6716 9196
6 6976 9783
7 7127 10170
8 7399 10385 (+40%)
before 9d89c257d ("sched/fair: Rewrite runnable load and utilization
average tracking") there was basically no difference between group
or non-group scheduling. These numbers are with 4.7, older kernels after
9d89c257d show a similar difference.
The bad thing is that there is a lot of idle cpu power in the host
when this happens so the scheduler seems to not realize that this
workload could use more cpus in the host.
I tried some experiments , but I have not found a hack that "fixes" the
degredation, which would give me an indication which part of the code
is broken. So are there any ideas? Is the estimated group load
calculation just not fast enough for sleep/wakeup workload?
Christian
[toc] | [next] | [standalone]
| From | Peter Zijlstra <peterz@infradead.org> |
|---|---|
| Date | 2016-09-26 13:00 +0200 |
| Subject | Re: group scheduler regression since 4.3 (bisect 9d89c257d sched/fair: Rewrite runnable load and utilization average tracking) |
| Message-ID | <slBND-2PA-1@gated-at.bofh.it> |
| In reply to | #1491208 |
On Mon, Sep 26, 2016 at 12:42:22PM +0200, Christian Borntraeger wrote:
> Folks,
>
> I have seen big scalability degredations sind 4.3 (bisected 9d89c257d
> sched/fair: Rewrite runnable load and utilization average tracking)
> This has not been fixed by subsequent patches,e.g. the ones that try to
> fix this for interactive workload.
>
> The problem is only visible for sleep/wakeup heavy workload which must
> be part of the scheduler group (e.g. a sysbench OLTP inside a KVM guest
> as libvirt will put KVM guests into cgroup instances).
>
> For example a simple sysbench oltp with mysql inside a KVM guests with
> 16 CPUs backed by 8 host cpus (16 host threads) scales less (scale up
> inside a guest, having multiple instances). This is the numbers of
> events per second.
> Unmounting /sys/fs/cgroup/cpu,cpuacct (thus forcing libvirt to not
> use group scheduling for KVM guests) makes the behaviour much better:
>
>
> instances group nogroup
> 1 3406 3002
> 2 5078 4940
> 3 6017 6760
> 4 6471 8216 (+27%)
> 5 6716 9196
> 6 6976 9783
> 7 7127 10170
> 8 7399 10385 (+40%)
>
> before 9d89c257d ("sched/fair: Rewrite runnable load and utilization
> average tracking") there was basically no difference between group
> or non-group scheduling. These numbers are with 4.7, older kernels after
> 9d89c257d show a similar difference.
>
> The bad thing is that there is a lot of idle cpu power in the host
> when this happens so the scheduler seems to not realize that this
> workload could use more cpus in the host.
>
> I tried some experiments , but I have not found a hack that "fixes" the
> degredation, which would give me an indication which part of the code
> is broken. So are there any ideas? Is the estimated group load
> calculation just not fast enough for sleep/wakeup workload?
One of the differences in the old and new thing is being addressed by
these patches:
https://lkml.kernel.org/r/1473666472-13749-1-git-send-email-vincent.guittot@linaro.org
Could you see if those patches make a difference? If not, we'll have to
go poke elsewhere ofcourse ;-)
[toc] | [prev] | [next] | [standalone]
| From | Christian Borntraeger <borntraeger@de.ibm.com> |
|---|---|
| Date | 2016-09-26 13:50 +0200 |
| Subject | Re: group scheduler regression since 4.3 (bisect 9d89c257d sched/fair: Rewrite runnable load and utilization average tracking) |
| Message-ID | <slCA2-3kx-25@gated-at.bofh.it> |
| In reply to | #1491210 |
On 09/26/2016 12:56 PM, Peter Zijlstra wrote:
> On Mon, Sep 26, 2016 at 12:42:22PM +0200, Christian Borntraeger wrote:
>> Folks,
>>
>> I have seen big scalability degredations sind 4.3 (bisected 9d89c257d
>> sched/fair: Rewrite runnable load and utilization average tracking)
>> This has not been fixed by subsequent patches,e.g. the ones that try to
>> fix this for interactive workload.
>>
>> The problem is only visible for sleep/wakeup heavy workload which must
>> be part of the scheduler group (e.g. a sysbench OLTP inside a KVM guest
>> as libvirt will put KVM guests into cgroup instances).
>>
>> For example a simple sysbench oltp with mysql inside a KVM guests with
>> 16 CPUs backed by 8 host cpus (16 host threads) scales less (scale up
>> inside a guest, having multiple instances). This is the numbers of
>> events per second.
>> Unmounting /sys/fs/cgroup/cpu,cpuacct (thus forcing libvirt to not
>> use group scheduling for KVM guests) makes the behaviour much better:
>>
>>
>> instances group nogroup
>> 1 3406 3002
>> 2 5078 4940
>> 3 6017 6760
>> 4 6471 8216 (+27%)
>> 5 6716 9196
>> 6 6976 9783
>> 7 7127 10170
>> 8 7399 10385 (+40%)
>>
>> before 9d89c257d ("sched/fair: Rewrite runnable load and utilization
>> average tracking") there was basically no difference between group
>> or non-group scheduling. These numbers are with 4.7, older kernels after
>> 9d89c257d show a similar difference.
>>
>> The bad thing is that there is a lot of idle cpu power in the host
>> when this happens so the scheduler seems to not realize that this
>> workload could use more cpus in the host.
>>
>> I tried some experiments , but I have not found a hack that "fixes" the
>> degredation, which would give me an indication which part of the code
>> is broken. So are there any ideas? Is the estimated group load
>> calculation just not fast enough for sleep/wakeup workload?
>
> One of the differences in the old and new thing is being addressed by
> these patches:
>
> https://lkml.kernel.org/r/1473666472-13749-1-git-send-email-vincent.guittot@linaro.org
>
> Could you see if those patches make a difference? If not, we'll have to
> go poke elsewhere ofcourse ;-)
Those patches do not apply cleanly on v4.7, linux/master or next/master.
Is there a good branch to test these patches?
[toc] | [prev] | [next] | [standalone]
| From | Peter Zijlstra <peterz@infradead.org> |
|---|---|
| Date | 2016-09-26 14:00 +0200 |
| Subject | Re: group scheduler regression since 4.3 (bisect 9d89c257d sched/fair: Rewrite runnable load and utilization average tracking) |
| Message-ID | <slCJI-3nN-29@gated-at.bofh.it> |
| In reply to | #1491235 |
On Mon, Sep 26, 2016 at 01:42:05PM +0200, Christian Borntraeger wrote: > On 09/26/2016 12:56 PM, Peter Zijlstra wrote: > > One of the differences in the old and new thing is being addressed by > > these patches: > > > > https://lkml.kernel.org/r/1473666472-13749-1-git-send-email-vincent.guittot@linaro.org > > > > Could you see if those patches make a difference? If not, we'll have to > > go poke elsewhere ofcourse ;-) > > Those patches do not apply cleanly on v4.7, linux/master or next/master. > Is there a good branch to test these patches? They seemed to apply for me on tip/sched/core, I pushed out a branch for you that has them on. git://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git sched/propagate I didn't boot the result though; but they applied without issue.
[toc] | [prev] | [next] | [standalone]
| From | Christian Borntraeger <borntraeger@de.ibm.com> |
|---|---|
| Date | 2016-09-26 14:10 +0200 |
| Subject | Re: group scheduler regression since 4.3 (bisect 9d89c257d sched/fair: Rewrite runnable load and utilization average tracking) |
| Message-ID | <slCTn-3FS-5@gated-at.bofh.it> |
| In reply to | #1491240 |
On 09/26/2016 01:53 PM, Peter Zijlstra wrote: > On Mon, Sep 26, 2016 at 01:42:05PM +0200, Christian Borntraeger wrote: >> On 09/26/2016 12:56 PM, Peter Zijlstra wrote: > >>> One of the differences in the old and new thing is being addressed by >>> these patches: >>> >>> https://lkml.kernel.org/r/1473666472-13749-1-git-send-email-vincent.guittot@linaro.org >>> >>> Could you see if those patches make a difference? If not, we'll have to >>> go poke elsewhere ofcourse ;-) >> >> Those patches do not apply cleanly on v4.7, linux/master or next/master. >> Is there a good branch to test these patches? > > They seemed to apply for me on tip/sched/core, I pushed out a branch for > you that has them on. > > git://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git sched/propagate > > I didn't boot the result though; but they applied without issue. They applied ok on next from 9/13. Things go even worse. With this host configuration: CPU NODE BOOK SOCKET CORE L1d:L1i:L2d:L2i ONLINE CONFIGURED ADDRESS 0 0 0 0 0 0:0:0:0 yes yes 0 1 0 0 0 0 1:1:1:1 yes yes 1 2 0 0 0 1 2:2:2:2 yes yes 2 3 0 0 0 1 3:3:3:3 yes yes 3 4 0 0 1 2 4:4:4:4 yes yes 4 5 0 0 1 2 5:5:5:5 yes yes 5 6 0 0 1 3 6:6:6:6 yes yes 6 7 0 0 1 3 7:7:7:7 yes yes 7 8 0 0 1 4 8:8:8:8 yes yes 8 9 0 0 1 4 9:9:9:9 yes yes 9 10 0 0 1 5 10:10:10:10 yes yes 10 11 0 0 1 5 11:11:11:11 yes yes 11 12 0 0 1 6 12:12:12:12 yes yes 12 13 0 0 1 6 13:13:13:13 yes yes 13 14 0 0 1 7 14:14:14:14 yes yes 14 15 0 0 1 7 15:15:15:15 yes yes 15 the guest was running either on 0-3 or on 4-15, but never used the full system. With group scheduling disabled everything was good again. So looks like that this bug has also some dependency on on the host topology. Christian
[toc] | [prev] | [next] | [standalone]
| From | Peter Zijlstra <peterz@infradead.org> |
|---|---|
| Date | 2016-09-26 14:20 +0200 |
| Subject | Re: group scheduler regression since 4.3 (bisect 9d89c257d sched/fair: Rewrite runnable load and utilization average tracking) |
| Message-ID | <slD33-3J5-23@gated-at.bofh.it> |
| In reply to | #1491246 |
On Mon, Sep 26, 2016 at 02:01:43PM +0200, Christian Borntraeger wrote: > They applied ok on next from 9/13. Things go even worse. > With this host configuration: > > CPU NODE BOOK SOCKET CORE L1d:L1i:L2d:L2i ONLINE CONFIGURED ADDRESS > 0 0 0 0 0 0:0:0:0 yes yes 0 > 1 0 0 0 0 1:1:1:1 yes yes 1 > 2 0 0 0 1 2:2:2:2 yes yes 2 > 3 0 0 0 1 3:3:3:3 yes yes 3 > 4 0 0 1 2 4:4:4:4 yes yes 4 > 5 0 0 1 2 5:5:5:5 yes yes 5 > 6 0 0 1 3 6:6:6:6 yes yes 6 > 7 0 0 1 3 7:7:7:7 yes yes 7 > 8 0 0 1 4 8:8:8:8 yes yes 8 > 9 0 0 1 4 9:9:9:9 yes yes 9 > 10 0 0 1 5 10:10:10:10 yes yes 10 > 11 0 0 1 5 11:11:11:11 yes yes 11 > 12 0 0 1 6 12:12:12:12 yes yes 12 > 13 0 0 1 6 13:13:13:13 yes yes 13 > 14 0 0 1 7 14:14:14:14 yes yes 14 > 15 0 0 1 7 15:15:15:15 yes yes 15 > > the guest was running either on 0-3 or on 4-15, but never > used the full system. With group scheduling disabled everything was good > again. So looks like that this bug has also some dependency on on the > host topology. OK, so CPU affinities that unevenly straddle topology boundaries like that are hard (and is generally not recommended), but its not immediately obvious why it would be so much worse with cgroups enabled.
[toc] | [prev] | [next] | [standalone]
| From | Christian Borntraeger <borntraeger@de.ibm.com> |
|---|---|
| Date | 2016-09-26 15:00 +0200 |
| Subject | Re: group scheduler regression since 4.3 (bisect 9d89c257d sched/fair: Rewrite runnable load and utilization average tracking) |
| Message-ID | <slDFL-3W3-1@gated-at.bofh.it> |
| In reply to | #1491249 |
On 09/26/2016 02:10 PM, Peter Zijlstra wrote: > On Mon, Sep 26, 2016 at 02:01:43PM +0200, Christian Borntraeger wrote: >> They applied ok on next from 9/13. Things go even worse. >> With this host configuration: >> >> CPU NODE BOOK SOCKET CORE L1d:L1i:L2d:L2i ONLINE CONFIGURED ADDRESS >> 0 0 0 0 0 0:0:0:0 yes yes 0 >> 1 0 0 0 0 1:1:1:1 yes yes 1 >> 2 0 0 0 1 2:2:2:2 yes yes 2 >> 3 0 0 0 1 3:3:3:3 yes yes 3 >> 4 0 0 1 2 4:4:4:4 yes yes 4 >> 5 0 0 1 2 5:5:5:5 yes yes 5 >> 6 0 0 1 3 6:6:6:6 yes yes 6 >> 7 0 0 1 3 7:7:7:7 yes yes 7 >> 8 0 0 1 4 8:8:8:8 yes yes 8 >> 9 0 0 1 4 9:9:9:9 yes yes 9 >> 10 0 0 1 5 10:10:10:10 yes yes 10 >> 11 0 0 1 5 11:11:11:11 yes yes 11 >> 12 0 0 1 6 12:12:12:12 yes yes 12 >> 13 0 0 1 6 13:13:13:13 yes yes 13 >> 14 0 0 1 7 14:14:14:14 yes yes 14 >> 15 0 0 1 7 15:15:15:15 yes yes 15 >> >> the guest was running either on 0-3 or on 4-15, but never >> used the full system. With group scheduling disabled everything was good >> again. So looks like that this bug has also some dependency on on the >> host topology. > > OK, so CPU affinities that unevenly straddle topology boundaries like > that are hard (and is generally not recommended), but its not > immediately obvious why it would be so much worse with cgroups enabled. Well thats what I get from LPAR... With CPUs 0-3 disabled things are better, but there is still 10% difference between group/nogroup. Will test Vincents v4 soon. In any case, would a 5 second sequence of /proc/sched_debug for the good/bad case with all 16 host CPUs (or the reduced 12 cpu set) be useful? Christian
[toc] | [prev] | [next] | [standalone]
| From | Christian Borntraeger <borntraeger@de.ibm.com> |
|---|---|
| Date | 2016-09-26 16:20 +0200 |
| Subject | Re: group scheduler regression since 4.3 (bisect 9d89c257d sched/fair: Rewrite runnable load and utilization average tracking) |
| Message-ID | <slEVb-4Tt-1@gated-at.bofh.it> |
| In reply to | #1491249 |
On 09/26/2016 02:10 PM, Peter Zijlstra wrote: > On Mon, Sep 26, 2016 at 02:01:43PM +0200, Christian Borntraeger wrote: >> They applied ok on next from 9/13. Things go even worse. >> With this host configuration: >> >> CPU NODE BOOK SOCKET CORE L1d:L1i:L2d:L2i ONLINE CONFIGURED ADDRESS >> 0 0 0 0 0 0:0:0:0 yes yes 0 >> 1 0 0 0 0 1:1:1:1 yes yes 1 >> 2 0 0 0 1 2:2:2:2 yes yes 2 >> 3 0 0 0 1 3:3:3:3 yes yes 3 >> 4 0 0 1 2 4:4:4:4 yes yes 4 >> 5 0 0 1 2 5:5:5:5 yes yes 5 >> 6 0 0 1 3 6:6:6:6 yes yes 6 >> 7 0 0 1 3 7:7:7:7 yes yes 7 >> 8 0 0 1 4 8:8:8:8 yes yes 8 >> 9 0 0 1 4 9:9:9:9 yes yes 9 >> 10 0 0 1 5 10:10:10:10 yes yes 10 >> 11 0 0 1 5 11:11:11:11 yes yes 11 >> 12 0 0 1 6 12:12:12:12 yes yes 12 >> 13 0 0 1 6 13:13:13:13 yes yes 13 >> 14 0 0 1 7 14:14:14:14 yes yes 14 >> 15 0 0 1 7 15:15:15:15 yes yes 15 >> >> the guest was running either on 0-3 or on 4-15, but never >> used the full system. With group scheduling disabled everything was good >> again. So looks like that this bug has also some dependency on on the >> host topology. > > OK, so CPU affinities that unevenly straddle topology boundaries like > that are hard (and is generally not recommended), but its not Ok so I created with cpu hotplug a symmetrical CPU topology: CPU NODE BOOK SOCKET CORE L1d:L1i:L2d:L2i ONLINE CONFIGURED ADDRESS 0 0 0 0 0 0:0:0:0 yes yes 0 1 0 0 0 0 1:1:1:1 yes yes 1 2 0 0 0 1 2:2:2:2 yes yes 2 3 0 0 0 1 3:3:3:3 yes yes 3 4 0 0 0 2 4:4:4:4 yes yes 4 5 0 0 0 2 5:5:5:5 yes yes 5 6 0 0 0 3 6:6:6:6 yes yes 6 7 0 0 0 3 7:7:7:7 yes yes 7 8 0 0 0 - ::: no yes 8 9 0 0 0 - ::: no yes 9 10 0 0 0 - ::: no yes 10 11 0 0 0 - ::: no yes 11 12 0 0 1 4 8:8:8:8 yes yes 12 13 0 0 1 4 9:9:9:9 yes yes 13 14 0 0 1 5 10:10:10:10 yes yes 14 15 0 0 1 5 11:11:11:11 yes yes 15 16 0 0 1 6 12:12:12:12 yes yes 16 17 0 0 1 6 13:13:13:13 yes yes 17 18 0 0 1 7 14:14:14:14 yes yes 18 19 0 0 1 7 15:15:15:15 yes yes 19 Same effect: Only half of the CPUs are used, but the number of guest CPUs == number of host cpus. Turns out that this is totally unrelated to this patch set, so it must be something else.
[toc] | [prev] | [next] | [standalone]
| From | Vincent Guittot <vincent.guittot@linaro.org> |
|---|---|
| Date | 2016-09-26 14:30 +0200 |
| Subject | Re: group scheduler regression since 4.3 (bisect 9d89c257d sched/fair: Rewrite runnable load and utilization average tracking) |
| Message-ID | <slDcJ-3Ml-13@gated-at.bofh.it> |
| In reply to | #1491210 |
On 26 September 2016 at 12:56, Peter Zijlstra <peterz@infradead.org> wrote:
> On Mon, Sep 26, 2016 at 12:42:22PM +0200, Christian Borntraeger wrote:
>> Folks,
>>
>> I have seen big scalability degredations sind 4.3 (bisected 9d89c257d
>> sched/fair: Rewrite runnable load and utilization average tracking)
>> This has not been fixed by subsequent patches,e.g. the ones that try to
>> fix this for interactive workload.
>>
>> The problem is only visible for sleep/wakeup heavy workload which must
>> be part of the scheduler group (e.g. a sysbench OLTP inside a KVM guest
>> as libvirt will put KVM guests into cgroup instances).
>>
>> For example a simple sysbench oltp with mysql inside a KVM guests with
>> 16 CPUs backed by 8 host cpus (16 host threads) scales less (scale up
>> inside a guest, having multiple instances). This is the numbers of
>> events per second.
>> Unmounting /sys/fs/cgroup/cpu,cpuacct (thus forcing libvirt to not
>> use group scheduling for KVM guests) makes the behaviour much better:
>>
>>
>> instances group nogroup
>> 1 3406 3002
>> 2 5078 4940
>> 3 6017 6760
>> 4 6471 8216 (+27%)
>> 5 6716 9196
>> 6 6976 9783
>> 7 7127 10170
>> 8 7399 10385 (+40%)
>>
>> before 9d89c257d ("sched/fair: Rewrite runnable load and utilization
>> average tracking") there was basically no difference between group
>> or non-group scheduling. These numbers are with 4.7, older kernels after
>> 9d89c257d show a similar difference.
>>
>> The bad thing is that there is a lot of idle cpu power in the host
>> when this happens so the scheduler seems to not realize that this
>> workload could use more cpus in the host.
>>
>> I tried some experiments , but I have not found a hack that "fixes" the
>> degredation, which would give me an indication which part of the code
>> is broken. So are there any ideas? Is the estimated group load
>> calculation just not fast enough for sleep/wakeup workload?
>
> One of the differences in the old and new thing is being addressed by
> these patches:
>
> https://lkml.kernel.org/r/1473666472-13749-1-git-send-email-vincent.guittot@linaro.org
I have just sent a new version which fix one issue with
runnable_load_avg that was raised by Dietmar.
The patchset is also available here:
https://git.linaro.org/people/vincent.guittot/kernel.git sched/pelt
>
> Could you see if those patches make a difference? If not, we'll have to
> go poke elsewhere ofcourse ;-)
[toc] | [prev] | [standalone]
Back to top | Article view | linux.kernel
csiph-web