Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1498082 > unrolled thread
| Started by | Wanpeng Li <kernellwp@gmail.com> |
|---|---|
| First post | 2016-10-10 06:20 +0200 |
| Last post | 2016-10-10 14:30 +0200 |
| Articles | 3 — 2 participants |
Back to article view | Back to linux.kernel
[PATCH] sched/core: Fix kick offline cpu to do nohz idle load balance Wanpeng Li <kernellwp@gmail.com> - 2016-10-10 06:20 +0200
Re: [PATCH] sched/core: Fix kick offline cpu to do nohz idle load balance Wanpeng Li <kernellwp@gmail.com> - 2016-10-10 11:00 +0200
Re: [PATCH] sched/core: Fix kick offline cpu to do nohz idle load balance Peter Zijlstra <peterz@infradead.org> - 2016-10-10 14:30 +0200
| From | Wanpeng Li <kernellwp@gmail.com> |
|---|---|
| Date | 2016-10-10 06:20 +0200 |
| Subject | [PATCH] sched/core: Fix kick offline cpu to do nohz idle load balance |
| Message-ID | <sqAed-4Wd-1@gated-at.bofh.it> |
From: Wanpeng Li <wanpeng.li@hotmail.com>
WARNING: CPU: 0 PID: 3404 at arch/x86/kernel/smp.c:125 native_smp_send_reschedule+0x3f/0x50
CPU: 0 PID: 3404 Comm: qemu-system-x86 Not tainted 4.8.0+ #21
Call Trace:
__warn+0xd1/0xf0
warn_slowpath_null+0x1d/0x20
native_smp_send_reschedule+0x3f/0x50
trigger_load_balance+0x29c/0x4a0
? trigger_load_balance+0x72/0x4a0
scheduler_tick+0x9f/0xd0
? tick_sched_do_timer+0x50/0x50
update_process_times+0x47/0x60
tick_sched_handle.isra.24+0x25/0x60
tick_sched_timer+0x3d/0x70
__hrtimer_run_queues+0xf4/0x510
hrtimer_interrupt+0xb7/0x1d0
local_apic_timer_interrupt+0x35/0x60
smp_apic_timer_interrupt+0x3d/0x50
apic_timer_interrupt+0x96/0xa0
If there is a need to kick the idle load balancer, an ILB will be selected
to perform nohz idle load balance, however, if the selected ILB is in the
process of offline, smp_sched_reschedule() which generates a sched IPI will
splat as above.
CPU0 CPU1
find_new_ilb()
set_rq_offline()
smp_sched_reschedule() Oops
nohz_balance_exit_idle()
This patch fix it by exiting nohz idle balance before set cpu offline.
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Wanpeng Li <wanpeng.li@hotmail.com>
---
kernel/sched/core.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 94732d1..7c83f99 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -7412,6 +7412,7 @@ int sched_cpu_dying(unsigned int cpu)
/* Handle pending wakeups and then migrate everything off */
sched_ttwu_pending();
+ nohz_balance_exit_idle(cpu);
raw_spin_lock_irqsave(&rq->lock, flags);
if (rq->rd) {
BUG_ON(!cpumask_test_cpu(cpu, rq->rd->span));
@@ -7422,7 +7423,6 @@ int sched_cpu_dying(unsigned int cpu)
raw_spin_unlock_irqrestore(&rq->lock, flags);
calc_load_migrate(rq);
update_max_interval();
- nohz_balance_exit_idle(cpu);
hrtick_clear(rq);
return 0;
}
--
1.9.1
[toc] | [next] | [standalone]
| From | Wanpeng Li <kernellwp@gmail.com> |
|---|---|
| Date | 2016-10-10 11:00 +0200 |
| Message-ID | <sqEBb-7uz-1@gated-at.bofh.it> |
| In reply to | #1498082 |
2016-10-10 12:10 GMT+08:00 Wanpeng Li <kernellwp@gmail.com>:
> From: Wanpeng Li <wanpeng.li@hotmail.com>
>
> WARNING: CPU: 0 PID: 3404 at arch/x86/kernel/smp.c:125 native_smp_send_reschedule+0x3f/0x50
> CPU: 0 PID: 3404 Comm: qemu-system-x86 Not tainted 4.8.0+ #21
> Call Trace:
> __warn+0xd1/0xf0
> warn_slowpath_null+0x1d/0x20
> native_smp_send_reschedule+0x3f/0x50
> trigger_load_balance+0x29c/0x4a0
> ? trigger_load_balance+0x72/0x4a0
> scheduler_tick+0x9f/0xd0
> ? tick_sched_do_timer+0x50/0x50
> update_process_times+0x47/0x60
> tick_sched_handle.isra.24+0x25/0x60
> tick_sched_timer+0x3d/0x70
> __hrtimer_run_queues+0xf4/0x510
> hrtimer_interrupt+0xb7/0x1d0
> local_apic_timer_interrupt+0x35/0x60
> smp_apic_timer_interrupt+0x3d/0x50
> apic_timer_interrupt+0x96/0xa0
>
> If there is a need to kick the idle load balancer, an ILB will be selected
> to perform nohz idle load balance, however, if the selected ILB is in the
> process of offline, smp_sched_reschedule() which generates a sched IPI will
> splat as above.
>
> CPU0 CPU1
>
> find_new_ilb()
> set_rq_offline()
> smp_sched_reschedule() Oops
> nohz_balance_exit_idle()
>
> This patch fix it by exiting nohz idle balance before set cpu offline.
CPU 0 CPU1
find_new_ilb()
nohz_balance_exit_idle()
set_rq_offline()
smp_sched_reschedule()
It seems that the patch still can't avoid this race, so any proposal
is a great appreciated. :)
Regards,
Wanpeng Li
[toc] | [prev] | [next] | [standalone]
| From | Peter Zijlstra <peterz@infradead.org> |
|---|---|
| Date | 2016-10-10 14:30 +0200 |
| Subject | Re: [PATCH] sched/core: Fix kick offline cpu to do nohz idle load balance |
| Message-ID | <sqHSp-1dv-13@gated-at.bofh.it> |
| In reply to | #1498178 |
On Mon, Oct 10, 2016 at 04:34:48PM +0800, Wanpeng Li wrote: > > If there is a need to kick the idle load balancer, an ILB will be selected > > to perform nohz idle load balance, however, if the selected ILB is in the > > process of offline, smp_sched_reschedule() which generates a sched IPI will > > splat as above. > > > > CPU0 CPU1 > > > > find_new_ilb() > > set_rq_offline() > > smp_sched_reschedule() Oops > > nohz_balance_exit_idle() > > > > This patch fix it by exiting nohz idle balance before set cpu offline. > > CPU 0 CPU1 > > find_new_ilb() > nohz_balance_exit_idle() > set_rq_offline() > smp_sched_reschedule() > > It seems that the patch still can't avoid this race, so any proposal > is a great appreciated. :) Not sure how this can happen, scheduler_tick() -> trigger_load_balance() -> nohz_balancer_kick() is called with IRQs disabled, this too implies a RCU-sched read side section. And hotplug explicitly includes a rcu_sync_sched(). It would be find_new_ilb() is 'broken' in that it considers !active CPUs. That's not immediately obvious.
[toc] | [prev] | [standalone]
Back to top | Article view | linux.kernel
csiph-web