Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1166351 > unrolled thread

[PATCH] sched,numa: document and fix numa_preferred_nid setting

Back to article view | Back to linux.kernel

Contents

  [PATCH] sched,numa: document and fix numa_preferred_nid setting Rik van Riel <riel@redhat.com> - 2015-06-16 22:00 +0200
    Re: [PATCH] sched,numa: document and fix numa_preferred_nid setting Rik van Riel <riel@redhat.com> - 2015-06-23 00:30 +0200

#1166351 — [PATCH] sched,numa: document and fix numa_preferred_nid setting

There are two places where the numa balancing code sets a task's
numa_preferred_nid.

The primary location is task_numa_placement(), where the kernel
examines the NUMA fault statistics to determine the location where
most of the memory that the task (or numa_group) accesses is.

The second location is only used for large workloads, where a
numa_group has enough tasks that the tasks are spread out over
several NUMA nodes, and multiple nodes are in the numa group's
active_nodes mask.

In order to allow those workloads to settle down, we pretend
that any node inside the numa_group's active_nodes mask is the
task's new preferred node. This dissuades task_numa_fault()
from continuously retrying to migrate the task to the group's
preferred node, and allows a multi-node workload to settle down.
This in turn improves locality of private faults inside a numa
group.

Reported-by: Shrikar Dronamraju <srikar@linux.vnet.ibm.com>
Signed-off-by: Rik van Riel <riel@redhat.com>
---
 kernel/sched/fair.c | 23 +++++++++++++++--------
 1 file changed, 15 insertions(+), 8 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index c2980e8733bc..54bb57f09e75 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -1485,7 +1485,12 @@ static int task_numa_migrate(struct task_struct *p)
 				groupweight = group_weight(p, env.src_nid, dist);
 			}
 
-			/* Only consider nodes where both task and groups benefit */
+			/*
+			 * Only consider nodes where placement is better for
+			 * either the group (help large workloads converge),
+			 * or the task (placement of tasks within a numa group,
+			 * and single threaded processes).
+			 */
 			taskimp = task_weight(p, nid, dist) - taskweight;
 			groupimp = group_weight(p, nid, dist) - groupweight;
 			if (taskimp < 0 && groupimp < 0)
@@ -1499,12 +1504,14 @@ static int task_numa_migrate(struct task_struct *p)
 	}
 
 	/*
-	 * If the task is part of a workload that spans multiple NUMA nodes,
-	 * and is migrating into one of the workload's active nodes, remember
-	 * this node as the task's preferred numa node, so the workload can
-	 * settle down.
-	 * A task that migrated to a second choice node will be better off
-	 * trying for a better one later. Do not set the preferred node here.
+	 * The primary place for setting a task's numa_preferred_nid is in
+	 * task_numa_placement(). If a task is moved to a sub-optimal node,
+	 * leave numa_preferred_nid alone, so task_numa_fault() will retry
+	 * migrating the task to where it really belongs.
+	 * The exception is a task that belongs to a large numa_group, which
+	 * spans multiple NUMA nodes. If that task migrates into one of the
+	 * workload's active nodes, remember that node as the task's
+	 * numa_preferred_nid, so the workload can settle down.
 	 */
 	if (p->numa_group) {
 		if (env.best_cpu == -1)
@@ -1513,7 +1520,7 @@ static int task_numa_migrate(struct task_struct *p)
 			nid = env.dst_nid;
 
 		if (node_isset(nid, p->numa_group->active_nodes))
-			sched_setnuma(p, env.dst_nid);
+			sched_setnuma(p, nid);
 	}
 
 	/* No better CPU than the current one was found. */

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1170229

On 06/22/2015 12:13 PM, Srikar Dronamraju wrote:
>> +	 * migrating the task to where it really belongs.
>> +	 * The exception is a task that belongs to a large numa_group, which
>> +	 * spans multiple NUMA nodes. If that task migrates into one of the
>> +	 * workload's active nodes, remember that node as the task's
>> +	 * numa_preferred_nid, so the workload can settle down.
>>  	 */
>>  	if (p->numa_group) {
>>  		if (env.best_cpu == -1)
>> @@ -1513,7 +1520,7 @@ static int task_numa_migrate(struct task_struct *p)
>>  			nid = env.dst_nid;
>>  
>>  		if (node_isset(nid, p->numa_group->active_nodes))
>> -			sched_setnuma(p, env.dst_nid);
>> +			sched_setnuma(p, nid);
>>  	}
>>  
>>  	/* No better CPU than the current one was found. */
>>
> 
> When I refer to the Modified Rik's patch, I mean to remove the
> node_isset() check before the sched_setnuma. With that change, we kind
> of reduce the numa02 and 1JVMper System regression while getting as good
> numbers as Rik's patch with 2JVM and 4JVM per System.
> 
> The idea behind removing the node_isset check is:
> node_isset is mostly used to track mem movement to nodes where cpus are
> running and not vice versa.  This is as per comment in
> update_numa_active_node_mask. There could be a sitation where task memory
> is all in a node and the node has capacity to accomodate but no tasks
> associated with the task have run enuf on that node. In such a case, we
> shouldnt be ruling out migrating the task to the node.

That is a good point.

However, if overriding the preferred_nid that task_numa_placement
identified is a good idea in task_numa_migrate, would it also be
a good idea for tasks that are NOT part of a numa group?

What are the consequences of never setting preferred_nid from
task_numa_migrate?   (we would try to migrate the task to a
better node more frequently)

What are the consequences of always setting preferred_nid from
task_numa_migrate?   (we would only try migrating the task once,
and it could get stuck in a sub-optimal location)

The patch seems to work, but I do not understand why, and would
like to know your ideas on why you think the patch works.

I am really not looking forward to the idea of maintaining code
that nobody understands...

-- 
All rights reversed
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web