Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1668239 > unrolled thread

[PATCH RFC] sched: Allow migrating kthreads into online but inactive CPUs

Back to article view | Back to linux.kernel

Contents

  [PATCH RFC] sched: Allow migrating kthreads into online but inactive  CPUs Tejun Heo <tj@kernel.org> - 2017-06-17 14:20 +0200
    Re: simple repro case Tejun Heo <tj@kernel.org> - 2017-06-17 14:20 +0200

#1668239 — [PATCH RFC] sched: Allow migrating kthreads into online but inactive CPUs

Per-cpu workqueues have been tripping CPU affinity sanity checks while
a CPU is being offlined.  A per-cpu kworker ends up running on a CPU
which isn't its target CPU while the CPU is online but inactive.

While the scheduler allows kthreads to wake up on an online but
inactive CPU, it doesn't allow a running kthread to be migrated to
such a CPU, which leads to an odd situation where setting affinity on
a sleeping and running kthread leads to different results.

Each mem-reclaim workqueue has one rescuer which guarantees forward
progress and the rescuer needs to bind itself to the CPU which needs
help in making forward progress; however, due to the above issue,
while set_cpus_allowed_ptr() succeeds, the rescuer doesn't end up on
the correct CPU if the CPU is in the process of going offline,
tripping the sanity check and executing the work item on the wrong
CPU.

This patch updates __migrate_task() so that kthreads can be migrated
into an inactive but online CPU.

Signed-off-by: Tejun Heo <tj@kernel.org>
Reported-by: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Reported-by: Steven Rostedt <rostedt@goodmis.org>
---
 kernel/sched/core.c |    9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 803c3bc274c4..1500217ce4b4 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -980,8 +980,13 @@ struct migration_arg {
 static struct rq *__migrate_task(struct rq *rq, struct rq_flags *rf,
 				 struct task_struct *p, int dest_cpu)
 {
-	if (unlikely(!cpu_active(dest_cpu)))
-		return rq;
+	if (p->flags & PF_KTHREAD) {
+		if (unlikely(!cpu_online(dest_cpu)))
+			return rq;
+	} else {
+		if (unlikely(!cpu_active(dest_cpu)))
+			return rq;
+	}
 
 	/* Affinity changed (again). */
 	if (!cpumask_test_cpu(dest_cpu, &p->cpus_allowed))

[toc] | [next] | [standalone]

#1668240 — Re: simple repro case

Here's a simple rerpo.  The test code runs whenever a CPU goes
off/online.  The test kthread is created on a different CPU and
migrated to the target CPU while running.  Without the previous patch
applied, the kthread ends up running on the wrong CPU.

Thanks.

diff --git a/kernel/workqueue.c b/kernel/workqueue.c
index c74bf39ef764..faed30edbb21 100644
--- a/kernel/workqueue.c
+++ b/kernel/workqueue.c
@@ -4648,12 +4648,56 @@ int workqueue_prepare_cpu(unsigned int cpu)
 	return 0;
 }
 
+#include <linux/delay.h>
+
+static int test_last_cpu = -1;
+
+static int test_kthread_migration_threadfn(void *data)
+{
+	while (!kthread_should_stop()) {
+		test_last_cpu = raw_smp_processor_id();
+		cond_resched();
+	}
+	return 0;
+}
+
+static void test_kthread_migration(int inactive_cpu)
+{
+	int start_cpu = cpumask_any(cpu_online_mask);
+	struct task_struct *task;
+
+	printk("TEST: cpu %d inactive, starting on %d and migrating (active/online=%*pbl/%*pbl)\n",
+	       inactive_cpu, start_cpu, cpumask_pr_args(cpu_active_mask),
+	       cpumask_pr_args(cpu_online_mask));
+
+	task = kthread_create(test_kthread_migration_threadfn, NULL, "test");
+	if (IS_ERR(task)) {
+		printk("TEST: kthread_create failed with %ld\n", PTR_ERR(task));
+		return;
+	}
+
+	kthread_bind(task, start_cpu);
+	wake_up_process(task);
+	msleep(100);
+	printk("TEST: test_last_cpu=%d cpus_allowed=%*pbl\n",
+	       test_last_cpu, cpumask_pr_args(&task->cpus_allowed));
+	printk("TEST: migrating to inactve cpu %d\n", inactive_cpu);
+	set_cpus_allowed_ptr(task, cpumask_of(inactive_cpu));
+	msleep(100);
+	printk("TEST: test_last_cpu=%d cpus_allowed=%*pbl\n",
+	       test_last_cpu, cpumask_pr_args(&task->cpus_allowed));
+	kthread_stop(task);
+	return;
+}
+
 int workqueue_online_cpu(unsigned int cpu)
 {
 	struct worker_pool *pool;
 	struct workqueue_struct *wq;
 	int pi;
 
+	test_kthread_migration(cpu);
+
 	mutex_lock(&wq_pool_mutex);
 
 	for_each_pool(pool, pi) {
@@ -4680,6 +4724,8 @@ int workqueue_offline_cpu(unsigned int cpu)
 	struct work_struct unbind_work;
 	struct workqueue_struct *wq;
 
+	test_kthread_migration(cpu);
+
 	/* unbinding per-cpu workers should happen on the local CPU */
 	INIT_WORK_ONSTACK(&unbind_work, wq_unbind_fn);
 	queue_work_on(cpu, system_highpri_wq, &unbind_work);

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web