Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1650020 > unrolled thread
| Started by | Tim Chen <tim.c.chen@linux.intel.com> |
|---|---|
| First post | 2017-05-25 01:50 +0200 |
| Last post | 2017-05-27 03:30 +0200 |
| Articles | 3 — 2 participants |
Back to article view | Back to linux.kernel
This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by
below is the oldest one visible, not the original post.
Re: [PATCH v2 for-4.12-fixes 1/2] sched/fair: Use task_groups instead of leaf_cfs_rq_list to walk all cfs_rqs Tim Chen <tim.c.chen@linux.intel.com> - 2017-05-25 01:50 +0200
Re: [PATCH v2 for-4.12-fixes 1/2] sched/fair: Use task_groups instead of leaf_cfs_rq_list to walk all cfs_rqs Tejun Heo <tj@kernel.org> - 2017-05-25 16:40 +0200
Re: [PATCH v2 for-4.12-fixes 1/2] sched/fair: Use task_groups instead of leaf_cfs_rq_list to walk all cfs_rqs Tim Chen <tim.c.chen@linux.intel.com> - 2017-05-27 03:30 +0200
| From | Tim Chen <tim.c.chen@linux.intel.com> |
|---|---|
| Date | 2017-05-25 01:50 +0200 |
| Subject | Re: [PATCH v2 for-4.12-fixes 1/2] sched/fair: Use task_groups instead of leaf_cfs_rq_list to walk all cfs_rqs |
| Message-ID | <tKOcp-pv-1@gated-at.bofh.it> |
On 05/09/2017 09:17 AM, Tejun Heo wrote: > Currently, rq->leaf_cfs_rq_list is a traversal ordered list of all > live cfs_rqs which have ever been active on the CPU; unfortunately, > this makes update_blocked_averages() O(total number of CPU cgroups) > which isn't scalable at all. > > The next patch will make rq->leaf_cfs_rq_list only contain the cfs_rqs > which are currently active. In preparation, this patch converts users > which need to traverse all cfs_rqs to use task_groups list instead. > > task_groups list is protected by its own lock and allows RCU protected > traversal and the order of operations guarantees that all online > cfs_rqs will be visited, but holding rq->lock won't protect against > iterating an already unregistered cfs_rq. However, the operations of > the two users that get converted - update_runtime_enabled() and > unthrottle_offline_cfs_rqs() - should be safe to perform on already > dead cfs_rqs, so adding rcu read protection around them should be > enough. > > Note that print_cfs_stats() is not converted. The next patch will > change its behavior to print out only active cfs_rqs, which is > intended as there's not much point in printing out idle cfs_rqs. > > v2: Dropped strong synchronization around removal and left > print_cfs_stats() unchanged as suggested by Peterz. > > Tejun, We did some preliminary testing of this patchset for a well known database benchmark on a 4 socket Skylake server system. It provides a 3.7% throughput boost which is significant for this benchmark. Thanks. Tim
[toc] | [next] | [standalone]
| From | Tejun Heo <tj@kernel.org> |
|---|---|
| Date | 2017-05-25 16:40 +0200 |
| Subject | Re: [PATCH v2 for-4.12-fixes 1/2] sched/fair: Use task_groups instead of leaf_cfs_rq_list to walk all cfs_rqs |
| Message-ID | <tL25I-WE-17@gated-at.bofh.it> |
| In reply to | #1650020 |
On Wed, May 24, 2017 at 04:40:34PM -0700, Tim Chen wrote: > We did some preliminary testing of this patchset for a well > known database benchmark on a 4 socket Skylake server system. > It provides a 3.7% throughput boost which is significant for > this benchmark. That's great to hear. Yeah, the walk can be noticeably expensive even with moderate number of cgroups. Thanks for sharing the result. -- tejun
[toc] | [prev] | [next] | [standalone]
| From | Tim Chen <tim.c.chen@linux.intel.com> |
|---|---|
| Date | 2017-05-27 03:30 +0200 |
| Message-ID | <tLyIn-4Vw-175@gated-at.bofh.it> |
| In reply to | #1650566 |
On 05/25/2017 07:39 AM, Tejun Heo wrote:
> On Wed, May 24, 2017 at 04:40:34PM -0700, Tim Chen wrote:
>> We did some preliminary testing of this patchset for a well
>> known database benchmark on a 4 socket Skylake server system.
>> It provides a 3.7% throughput boost which is significant for
>> this benchmark.
>
> That's great to hear. Yeah, the walk can be noticeably expensive even
> with moderate number of cgroups. Thanks for sharing the result.
>
Yes, the walk in update_blocked_averages has bad scaling property as it
iterates over *all* cfs_rq's leaf tasks, making it very expensive. It
consumes 11.7% of our cpu cycles for this benchmark when CGROUP
is on. Your patchset skips unused cgroup and reduce the overhead to
10.4%. CPU cycles profile is attached below for your reference.
The scheduler's frequent update of cgroup's laod averages, and
having to iterate all the leaf tasks for each load balance causes
update_blocked_averages to be one of the most expensive functions in the
system, making CGROUP costly. Without CGROUP, schedule only cost 3.3%
of cpu cycles vs 16.4% with CGROUP turned on. Your patchset does reduce
it to 14.9%.
This benchmark has thousands of running tasks, so it puts a good
deal of stress to the scheduler.
Tim
CPU cycles profile:
4.11 Before your patchset with CGROUP:
---------------------------------------
16.42% 0.03% 280 [kernel.vmlinux] [k] schedule
|
--16.39%--schedule
|
--16.31%--__sched_text_start
|
|--12.85%--pick_next_task_fair
| |
| --11.71%--update_blocked_averages
| |
| --5.00%--update_load_avg
|
|--2.04%--finish_task_switch
| |
| |--0.85%--ret_from_intr
| | |
| | --0.85%--do_IRQ
| |
| --0.75%--apic_timer_interrupt
| |
| --0.75%--smp_apic_timer_interrupt
| |
| --0.55%--irq_exit
| |
| --0.55%--__do_softirq
|
--0.51%--deactivate_task
4.11 After your patchset with CGROUP:
-------------------------------------
14.90% 0.04% 337 [kernel.vmlinux] [k] schedule
|
--14.86%--schedule
|
--14.78%--__sched_text_start
|
|--11.51%--pick_next_task_fair
| |
| --10.37%--update_blocked_averages
| |
| --4.55%--update_load_avg
|
|--1.79%--finish_task_switch
| |
| |--0.77%--ret_from_intr
| | |
| | --0.77%--do_IRQ
| |
| --0.65%--apic_timer_interrupt
| |
| --0.65%--smp_apic_timer_interrupt
|
--0.53%--deactivate_task
4.11 with No CGROUP:
--------------------
3.33% 0.04% 336 [kernel.vmlinux] [k] schedule
|
--3.29%--schedule
|
--3.19%--__sched_text_start
|
--1.45%--pick_next_task_fair
|
--1.15%--load_balance
|
--0.87%--find_busiest_group
|
--0.82%--update_sd_lb_stats
[toc] | [prev] | [standalone]
Back to top | Article view | linux.kernel
csiph-web