Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1233723 > unrolled thread

[PATCH RFC 0/5] powerpc:numa Add serial nid support

Back to article view | Back to linux.kernel

Contents

  [PATCH RFC  0/5] powerpc:numa Add serial nid support Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com> - 2015-09-27 20:30 +0200
    [PATCH RFC  5/5] powerpc:numa Use chipid to nid mapping to get serial numa node ids Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com> - 2015-09-27 20:30 +0200
    [PATCH RFC  4/5] powerpc:numa Add helper functions to maintain chipid to nid mapping Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com> - 2015-09-27 20:30 +0200
      Re: [PATCH RFC  4/5] powerpc:numa Add helper functions to maintain  chipid to nid mapping Nishanth Aravamudan <nacc@linux.vnet.ibm.com> - 2015-09-28 19:40 +0200
        Re: [PATCH RFC  4/5] powerpc:numa Add helper functions to maintain  chipid to nid mapping Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com> - 2015-09-29 21:10 +0200
    Re: [PATCH RFC 0/5] powerpc:numa Add serial nid support Denis Kirjanov <kda@linux-powerpc.org> - 2015-09-28 13:00 +0200
      Re: [PATCH RFC 0/5] powerpc:numa Add serial nid support Nishanth Aravamudan <nacc@linux.vnet.ibm.com> - 2015-09-28 19:10 +0200
        Re: [PATCH RFC 0/5] powerpc:numa Add serial nid support Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com> - 2015-09-29 20:20 +0200
          Re: [PATCH RFC 0/5] powerpc:numa Add serial nid support Denis Kirjanov <kda@linux-powerpc.org> - 2015-09-29 21:50 +0200
            Re: [PATCH RFC 0/5] powerpc:numa Add serial nid support Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com> - 2015-09-30 08:20 +0200
    Re: [PATCH RFC  0/5] powerpc:numa Add serial nid support Nishanth Aravamudan <nacc@linux.vnet.ibm.com> - 2015-09-28 19:40 +0200
      Re: [PATCH RFC  0/5] powerpc:numa Add serial nid support Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com> - 2015-09-29 21:20 +0200
    Re: [PATCH RFC  0/5] powerpc:numa Add serial nid support Michael Ellerman <mpe@ellerman.id.au> - 2015-10-06 12:30 +0200
      Re: [PATCH RFC  0/5] powerpc:numa Add serial nid support Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com> - 2015-10-06 13:20 +0200

#1233723 — [PATCH RFC 0/5] powerpc:numa Add serial nid support

Problem description:
Powerpc has sparse node numbering, i.e. on a 4 node system nodes are
numbered (possibly) as 0,1,16,17. At a lower level, we map the chipid
got from device tree is naturally mapped (directly) to nid.

Potential side effect of that is:

1) There are several places in kernel that assumes serial node numbering.
and memory allocations assume that all the nodes from 0-(highest nid)
exist inturn ending up allocating memory for the nodes that does not exist.

2) For virtualization use cases (such as qemu, libvirt, openstack), mapping
sparse nid of the host system to contiguous nids of guest (numa affinity,
placement) could be a challenge.

Possible Solutions:
1) Handling the memory allocations is kernel case by case: Though in some
cases it is easy to achieve, some cases may be intrusive/not trivial. 
at the end it does not handle side effect (2) above.

2) Map the sparse chipid got from device tree to a serial nid at kernel
level (The idea proposed in this series).
Pro: It is more natural to handle at kernel level than at lower (OPAL) layer.
con: The chipid is in device tree no longer the same as nid in kernel

3) Let the lower layer (OPAL) give the serial node ids after parsing the
chipid and the associativity etc [ either as a separate item in device tree
or by compacting the chipid numbers ]
Pros: kernel, device tree are on same page and less change in kernel
Con: is it the functionality expected in lower layer

As mentioned above, current patch series tries to map chipid from lower layer
to a contiguos nid at kernel level keeping the node distance calculation and
so on intact.

Result:
Before the patch: numactl -H

available: 4 nodes (0-1,16-17)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
node 0 size: 31665 MB
node 0 free: 29836 MB
node 1 cpus: 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
node 1 size: 32722 MB
node 1 free: 32019 MB
node 16 cpus: 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95
node 16 size: 32571 MB
node 16 free: 31222 MB
node 17 cpus: 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127
node 17 size: 0 MB
node 17 free: 0 MB
node distances:
node   0   1  16  17 
  0:  10  20  40  40 
  1:  20  10  40  40 
 16:  40  40  10  20 
 17:  40  40  20  10 

After the patch: numactl -H

available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
node 0 size: 31665 MB
node 0 free: 30657 MB
node 1 cpus: 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
node 1 size: 32722 MB
node 1 free: 32566 MB
node 2 cpus: 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95
node 2 size: 32571 MB
node 2 free: 32401 MB
node 3 cpus: 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127
node 3 size: 0 MB
node 3 free: 0 MB
node distances:
node   0   1   2   3 
  0:  10  20  40  40 
  1:  20  10  40  40 
  2:  40  40  10  20 
  3:  40  40  20  10 

(note that numa distances are intact). Apart from this, The following tests
are done with the patched kernel (both baremetal and KVM guest with multiple
nodes) to ensure there is no breakage.

1) offlining and onlining of memory in /sys/devices/system/node/nodeX path

2) offlining and onlining of cpus in /sys/devices/system/cpu/ path

3) Numactl tests from
ftp://oss.sgi.com/www/projects/libnuma/download/numactl-2.0.10.tar.gz

(infact there were more breakage before the patch because of sparse nid
and memoryless node cases of powerpc)

4) Thousands of docker containers were spawned.

Please let me know your comments.

 patch 1-3: cleanup patches
 patch 4: Adds helper function to map nid and chipid
 patch 5: Uses the mapping to get serial nid 

Raghavendra K T (5):
  powerpc:numa Add numa_cpu_lookup function to update lookup table
  powerpc:numa Rename functions referring to nid as chipid
  powerpc:numa create 1:1 mappaing between chipid and nid
  powerpc:numa Add helper functions to maintain chipid to nid mapping
  powerpc:numa Use chipid to nid mapping to get serial numa node ids

 arch/powerpc/include/asm/mmzone.h |   2 +-
 arch/powerpc/kernel/smp.c         |  10 ++--
 arch/powerpc/mm/numa.c            | 121 +++++++++++++++++++++++++++++++-------
 3 files changed, 105 insertions(+), 28 deletions(-)

-- 
1.7.11.7

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1233724 — [PATCH RFC 5/5] powerpc:numa Use chipid to nid mapping to get serial numa node ids

Also properly initialize numa distance table for serial nids.

Problem: Powerpc supports sparse nid numbering which could affect
1) memory footprint 2) virtualization use cases

Current solution:
The patch maps sprase chipid got fromn device tree to serail
nids.

Result before:
node   0   1  16  17
  0:  10  20  40  40
  1:  20  10  40  40
 16:  40  40  10  20
 17:  40  40  20  10

After:
node   0   1   2   3
  0:  10  20  40  40
  1:  20  10  40  40
  2:  40  40  10  20
  3:  40  40  20  10

Testing: Scenarios tested on  baremetal and KVM guest with 4 nodes
1) offlining and onlining memory and cpus
2) Running the tests from numactl source.
3) Creating 1000s of docker containers stressing the system

Signed-off-by: Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com>
---
 arch/powerpc/mm/numa.c | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

diff --git a/arch/powerpc/mm/numa.c b/arch/powerpc/mm/numa.c
index f015cad..873ac8c 100644
--- a/arch/powerpc/mm/numa.c
+++ b/arch/powerpc/mm/numa.c
@@ -304,7 +304,8 @@ static int associativity_to_chipid(const __be32 *associativity)
 		/*
 		 * Skip the length field and send start of associativity array
 		 */
-		initialize_distance_lookup_table(chipid, associativity + 1);
+		initialize_distance_lookup_table(chipid_to_nid(chipid),
+						 associativity + 1);
 	}
 
 out:
@@ -314,9 +315,10 @@ out:
  /* Return the nid from associativity */
 static int associativity_to_nid(const __be32 *associativity)
 {
-	int nid;
+	int chipid, nid;
 
-	nid = associativity_to_chipid(associativity);
+	chipid = associativity_to_chipid(associativity);
+	nid = map_chipid_to_nid(chipid);
 	return nid;
 }
 
@@ -340,9 +342,10 @@ static int of_node_to_chipid_single(struct device_node *device)
  */
 static int of_node_to_nid_single(struct device_node *device)
 {
-	int nid;
+	int chipid, nid;
 
-	nid = of_node_to_chipid_single(device);
+	chipid = of_node_to_chipid_single(device);
+	nid = map_chipid_to_nid(chipid);
 	return nid;
 }
 
-- 
1.7.11.7

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1233725 — [PATCH RFC 4/5] powerpc:numa Add helper functions to maintain chipid to nid mapping

Create arrays that maps serial nids and sparse chipids.

Note: My original idea had only two arrays of chipid to nid map. Final
code is inspired by driver/acpi/numa.c that maps a proximity node with
a logical node by Takayoshi Kochi <t-kochi@bq.jp.nec.com>, and thus
uses an additional chipid_map nodemask. The mask helps in first unused
nid easily by knowing first unset bit in the mask.

No change in functionality.

Signed-off-by: Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com>
---
 arch/powerpc/mm/numa.c | 48 +++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 47 insertions(+), 1 deletion(-)

diff --git a/arch/powerpc/mm/numa.c b/arch/powerpc/mm/numa.c
index dd2073b..f015cad 100644
--- a/arch/powerpc/mm/numa.c
+++ b/arch/powerpc/mm/numa.c
@@ -63,6 +63,11 @@ static int form1_affinity;
 static int distance_ref_points_depth;
 static const __be32 *distance_ref_points;
 static int distance_lookup_table[MAX_NUMNODES][MAX_DISTANCE_REF_POINTS];
+static nodemask_t chipid_map = NODE_MASK_NONE;
+static int chipid_to_nid_map[MAX_NUMNODES]
+				= { [0 ... MAX_NUMNODES - 1] = NUMA_NO_NODE };
+static int nid_to_chipid_map[MAX_NUMNODES]
+				= { [0 ... MAX_NUMNODES - 1] = NUMA_NO_NODE };
 
 /*
  * Allocate node_to_cpumask_map based on number of available nodes
@@ -133,6 +138,48 @@ static int __init fake_numa_create_new_node(unsigned long end_pfn,
 	return 0;
 }
 
+int chipid_to_nid(int chipid)
+{
+	if (chipid < 0)
+		return NUMA_NO_NODE;
+	return chipid_to_nid_map[chipid];
+}
+
+int nid_to_chipid(int nid)
+{
+	if (nid < 0)
+		return NUMA_NO_NODE;
+	return nid_to_chipid_map[nid];
+}
+
+static void __map_chipid_to_nid(int chipid, int nid)
+{
+	if (chipid_to_nid_map[chipid] == NUMA_NO_NODE
+	     || nid < chipid_to_nid_map[chipid])
+		chipid_to_nid_map[chipid] = nid;
+	if (nid_to_chipid_map[nid] == NUMA_NO_NODE
+	    || chipid < nid_to_chipid_map[nid])
+		nid_to_chipid_map[nid] = chipid;
+}
+
+int map_chipid_to_nid(int chipid)
+{
+	int nid;
+
+	if (chipid < 0 || chipid >= MAX_NUMNODES)
+		return NUMA_NO_NODE;
+
+	nid = chipid_to_nid_map[chipid];
+	if (nid == NUMA_NO_NODE) {
+		if (nodes_weight(chipid_map) >= MAX_NUMNODES)
+			return NUMA_NO_NODE;
+		nid = first_unset_node(chipid_map);
+		__map_chipid_to_nid(chipid, nid);
+		node_set(nid, chipid_map);
+	}
+	return nid;
+}
+
 int numa_cpu_lookup(int cpu)
 {
 	return numa_cpu_lookup_table[cpu];
@@ -264,7 +311,6 @@ out:
 	return chipid;
 }
 
-
  /* Return the nid from associativity */
 static int associativity_to_nid(const __be32 *associativity)
 {
-- 
1.7.11.7

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1234320 — Re: [PATCH RFC 4/5] powerpc:numa Add helper functions to maintain chipid to nid mapping

On 27.09.2015 [23:59:12 +0530], Raghavendra K T wrote:
> Create arrays that maps serial nids and sparse chipids.
> 
> Note: My original idea had only two arrays of chipid to nid map. Final
> code is inspired by driver/acpi/numa.c that maps a proximity node with
> a logical node by Takayoshi Kochi <t-kochi@bq.jp.nec.com>, and thus
> uses an additional chipid_map nodemask. The mask helps in first unused
> nid easily by knowing first unset bit in the mask.
> 
> No change in functionality.
> 
> Signed-off-by: Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com>
> ---
>  arch/powerpc/mm/numa.c | 48 +++++++++++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 47 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/powerpc/mm/numa.c b/arch/powerpc/mm/numa.c
> index dd2073b..f015cad 100644
> --- a/arch/powerpc/mm/numa.c
> +++ b/arch/powerpc/mm/numa.c
> @@ -63,6 +63,11 @@ static int form1_affinity;
>  static int distance_ref_points_depth;
>  static const __be32 *distance_ref_points;
>  static int distance_lookup_table[MAX_NUMNODES][MAX_DISTANCE_REF_POINTS];
> +static nodemask_t chipid_map = NODE_MASK_NONE;
> +static int chipid_to_nid_map[MAX_NUMNODES]
> +				= { [0 ... MAX_NUMNODES - 1] = NUMA_NO_NODE };

Hrm, conceptually there are *more* chips than nodes, right? So what
guarantees we won't see > MAX_NUMNODES chips?

> +static int nid_to_chipid_map[MAX_NUMNODES]
> +				= { [0 ... MAX_NUMNODES - 1] = NUMA_NO_NODE };
> 
>  /*
>   * Allocate node_to_cpumask_map based on number of available nodes
> @@ -133,6 +138,48 @@ static int __init fake_numa_create_new_node(unsigned long end_pfn,
>  	return 0;
>  }
> 
> +int chipid_to_nid(int chipid)
> +{
> +	if (chipid < 0)
> +		return NUMA_NO_NODE;

Do you really want to support these cases? Or should they be
bugs/warnings indicating that you got an unexpected input? Or at least
WARN_ON_ONCE?

> +	return chipid_to_nid_map[chipid];
> +}
> +
> +int nid_to_chipid(int nid)
> +{
> +	if (nid < 0)
> +		return NUMA_NO_NODE;
> +	return nid_to_chipid_map[nid];
> +}
> +
> +static void __map_chipid_to_nid(int chipid, int nid)
> +{
> +	if (chipid_to_nid_map[chipid] == NUMA_NO_NODE
> +	     || nid < chipid_to_nid_map[chipid])
> +		chipid_to_nid_map[chipid] = nid;
> +	if (nid_to_chipid_map[nid] == NUMA_NO_NODE
> +	    || chipid < nid_to_chipid_map[nid])
> +		nid_to_chipid_map[nid] = chipid;
> +}

chip <-> node mapping is a static (physical) concept, right? Should we
emit some debugging if for some reason we get a runtime call to remap
an already mapped chip to a new node?

> +
> +int map_chipid_to_nid(int chipid)
> +{
> +	int nid;
> +
> +	if (chipid < 0 || chipid >= MAX_NUMNODES)
> +		return NUMA_NO_NODE;
> +
> +	nid = chipid_to_nid_map[chipid];
> +	if (nid == NUMA_NO_NODE) {
> +		if (nodes_weight(chipid_map) >= MAX_NUMNODES)
> +			return NUMA_NO_NODE;

If you create a KVM guest with a bogus topology, doesn't this just start
losing NUMA information for very high-noded guests?

> +		nid = first_unset_node(chipid_map);
> +		__map_chipid_to_nid(chipid, nid);
> +		node_set(nid, chipid_map);
> +	}
> +	return nid;
> +}
> +
>  int numa_cpu_lookup(int cpu)
>  {
>  	return numa_cpu_lookup_table[cpu];
> @@ -264,7 +311,6 @@ out:
>  	return chipid;
>  }
> 
> -

stray change?

>   /* Return the nid from associativity */
>  static int associativity_to_nid(const __be32 *associativity)
>  {
> -- 
> 1.7.11.7
> 

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1235494 — Re: [PATCH RFC 4/5] powerpc:numa Add helper functions to maintain chipid to nid mapping

On 09/28/2015 11:02 PM, Nishanth Aravamudan wrote:
> On 27.09.2015 [23:59:12 +0530], Raghavendra K T wrote:
>> Create arrays that maps serial nids and sparse chipids.
>>
>> Note: My original idea had only two arrays of chipid to nid map. Final
>> code is inspired by driver/acpi/numa.c that maps a proximity node with
>> a logical node by Takayoshi Kochi <t-kochi@bq.jp.nec.com>, and thus
>> uses an additional chipid_map nodemask. The mask helps in first unused
>> nid easily by knowing first unset bit in the mask.
>>
>> No change in functionality.
>>
>> Signed-off-by: Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com>
>> ---
>>   arch/powerpc/mm/numa.c | 48 +++++++++++++++++++++++++++++++++++++++++++++++-
>>   1 file changed, 47 insertions(+), 1 deletion(-)
>>
>> diff --git a/arch/powerpc/mm/numa.c b/arch/powerpc/mm/numa.c
>> index dd2073b..f015cad 100644
>> --- a/arch/powerpc/mm/numa.c
>> +++ b/arch/powerpc/mm/numa.c
>> @@ -63,6 +63,11 @@ static int form1_affinity;
>>   static int distance_ref_points_depth;
>>   static const __be32 *distance_ref_points;
>>   static int distance_lookup_table[MAX_NUMNODES][MAX_DISTANCE_REF_POINTS];
>> +static nodemask_t chipid_map = NODE_MASK_NONE;
>> +static int chipid_to_nid_map[MAX_NUMNODES]
>> +				= { [0 ... MAX_NUMNODES - 1] = NUMA_NO_NODE };
>
> Hrm, conceptually there are *more* chips than nodes, right? So what
> guarantees we won't see > MAX_NUMNODES chips?

You are correct that nid <= chipids.
and #nids = #chipids when all possible slots are populated. Considering
we assume that maximum chip slots are no more than MAX_NUMNODES,


how about having

#define MAX_CHIPNODES MAX_NUMNODES
and
chipid_to_nid_map[MAX_CHIPNODES] = { [0 ... MAX_CHIPNODES - 1] = ..

>
>> +static int nid_to_chipid_map[MAX_NUMNODES]
>> +				= { [0 ... MAX_NUMNODES - 1] = NUMA_NO_NODE };
>>
>>   /*
>>    * Allocate node_to_cpumask_map based on number of available nodes
>> @@ -133,6 +138,48 @@ static int __init fake_numa_create_new_node(unsigned long end_pfn,
>>   	return 0;
>>   }
>>
>> +int chipid_to_nid(int chipid)
>> +{
>> +	if (chipid < 0)
>> +		return NUMA_NO_NODE;
>
> Do you really want to support these cases? Or should they be
> bugs/warnings indicating that you got an unexpected input? Or at least
> WARN_ON_ONCE?
>

Right. Querying for nid of an invalid chipid should be atleast
WARN_ON_ONCE(). But 'll check once if there is any valid scenario
before the change.

>> +	return chipid_to_nid_map[chipid];
>> +}
>> +
>> +int nid_to_chipid(int nid)
>> +{
>> +	if (nid < 0)
>> +		return NUMA_NO_NODE;
>> +	return nid_to_chipid_map[nid];
>> +}
>> +
>> +static void __map_chipid_to_nid(int chipid, int nid)
>> +{
>> +	if (chipid_to_nid_map[chipid] == NUMA_NO_NODE
>> +	     || nid < chipid_to_nid_map[chipid])
>> +		chipid_to_nid_map[chipid] = nid;
>> +	if (nid_to_chipid_map[nid] == NUMA_NO_NODE
>> +	    || chipid < nid_to_chipid_map[nid])
>> +		nid_to_chipid_map[nid] = chipid;
>> +}
>
> chip <-> node mapping is a static (physical) concept, right? Should we
> emit some debugging if for some reason we get a runtime call to remap
> an already mapped chip to a new node?
>

Good point. Already mapped chipid to a different nid is unexpected
whereas mapping chipid to same nid is expected.(because mapping comes
from cpus belonging to same node).

WARN_ON() should suffice here?


>> +
>> +int map_chipid_to_nid(int chipid)
>> +{
>> +	int nid;
>> +
>> +	if (chipid < 0 || chipid >= MAX_NUMNODES)
>> +		return NUMA_NO_NODE;
>> +
>> +	nid = chipid_to_nid_map[chipid];
>> +	if (nid == NUMA_NO_NODE) {
>> +		if (nodes_weight(chipid_map) >= MAX_NUMNODES)
>> +			return NUMA_NO_NODE;
>
> If you create a KVM guest with a bogus topology, doesn't this just start
> losing NUMA information for very high-noded guests?
>

'll try to see if it is possible to hit this case, ideally we should
not allow more than MAX_NUMNODES for chipids and we should abort early.

>> +		nid = first_unset_node(chipid_map);
>> +		__map_chipid_to_nid(chipid, nid);
>> +		node_set(nid, chipid_map);
>> +	}
>> +	return nid;
>> +}
>> +
>>   int numa_cpu_lookup(int cpu)
>>   {
>>   	return numa_cpu_lookup_table[cpu];
>> @@ -264,7 +311,6 @@ out:
>>   	return chipid;
>>   }
>>
>> -
>
> stray change?
>

yep, will correct that.


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1234047 — Re: [PATCH RFC 0/5] powerpc:numa Add serial nid support

On 9/27/15, Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com> wrote:
> Problem description:
> Powerpc has sparse node numbering, i.e. on a 4 node system nodes are
> numbered (possibly) as 0,1,16,17. At a lower level, we map the chipid
> got from device tree is naturally mapped (directly) to nid.

Interesting thing to play with, I'll try to test it on my POWER7 box,
but it doesn't have the OPAL layer :(

>
> Potential side effect of that is:
>
> 1) There are several places in kernel that assumes serial node numbering.
> and memory allocations assume that all the nodes from 0-(highest nid)
> exist inturn ending up allocating memory for the nodes that does not exist.
>
> 2) For virtualization use cases (such as qemu, libvirt, openstack), mapping
> sparse nid of the host system to contiguous nids of guest (numa affinity,
> placement) could be a challenge.
>
> Possible Solutions:
> 1) Handling the memory allocations is kernel case by case: Though in some
> cases it is easy to achieve, some cases may be intrusive/not trivial.
> at the end it does not handle side effect (2) above.
>
> 2) Map the sparse chipid got from device tree to a serial nid at kernel
> level (The idea proposed in this series).
> Pro: It is more natural to handle at kernel level than at lower (OPAL)
> layer.
> con: The chipid is in device tree no longer the same as nid in kernel
>
> 3) Let the lower layer (OPAL) give the serial node ids after parsing the
> chipid and the associativity etc [ either as a separate item in device tree
> or by compacting the chipid numbers ]
> Pros: kernel, device tree are on same page and less change in kernel
> Con: is it the functionality expected in lower layer
>
> As mentioned above, current patch series tries to map chipid from lower
> layer
> to a contiguos nid at kernel level keeping the node distance calculation and
> so on intact.
>
> Result:
> Before the patch: numactl -H
>
> available: 4 nodes (0-1,16-17)
> node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
> 24 25 26 27 28 29 30 31
> node 0 size: 31665 MB
> node 0 free: 29836 MB
> node 1 cpus: 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52
> 53 54 55 56 57 58 59 60 61 62 63
> node 1 size: 32722 MB
> node 1 free: 32019 MB
> node 16 cpus: 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84
> 85 86 87 88 89 90 91 92 93 94 95
> node 16 size: 32571 MB
> node 16 free: 31222 MB
> node 17 cpus: 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111
> 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127
> node 17 size: 0 MB
> node 17 free: 0 MB
> node distances:
> node   0   1  16  17
>   0:  10  20  40  40
>   1:  20  10  40  40
>  16:  40  40  10  20
>  17:  40  40  20  10
>
> After the patch: numactl -H
>
> available: 4 nodes (0-3)
> node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
> 24 25 26 27 28 29 30 31
> node 0 size: 31665 MB
> node 0 free: 30657 MB
> node 1 cpus: 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52
> 53 54 55 56 57 58 59 60 61 62 63
> node 1 size: 32722 MB
> node 1 free: 32566 MB
> node 2 cpus: 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84
> 85 86 87 88 89 90 91 92 93 94 95
> node 2 size: 32571 MB
> node 2 free: 32401 MB
> node 3 cpus: 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112
> 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127
> node 3 size: 0 MB
> node 3 free: 0 MB
> node distances:
> node   0   1   2   3
>   0:  10  20  40  40
>   1:  20  10  40  40
>   2:  40  40  10  20
>   3:  40  40  20  10
>
> (note that numa distances are intact). Apart from this, The following tests
> are done with the patched kernel (both baremetal and KVM guest with multiple
> nodes) to ensure there is no breakage.
>
> 1) offlining and onlining of memory in /sys/devices/system/node/nodeX path
>
> 2) offlining and onlining of cpus in /sys/devices/system/cpu/ path
>
> 3) Numactl tests from
> ftp://oss.sgi.com/www/projects/libnuma/download/numactl-2.0.10.tar.gz
>
> (infact there were more breakage before the patch because of sparse nid
> and memoryless node cases of powerpc)
>
> 4) Thousands of docker containers were spawned.
>
> Please let me know your comments.
>
>  patch 1-3: cleanup patches
>  patch 4: Adds helper function to map nid and chipid
>  patch 5: Uses the mapping to get serial nid
>
> Raghavendra K T (5):
>   powerpc:numa Add numa_cpu_lookup function to update lookup table
>   powerpc:numa Rename functions referring to nid as chipid
>   powerpc:numa create 1:1 mappaing between chipid and nid
>   powerpc:numa Add helper functions to maintain chipid to nid mapping
>   powerpc:numa Use chipid to nid mapping to get serial numa node ids
>
>  arch/powerpc/include/asm/mmzone.h |   2 +-
>  arch/powerpc/kernel/smp.c         |  10 ++--
>  arch/powerpc/mm/numa.c            | 121
> +++++++++++++++++++++++++++++++-------
>  3 files changed, 105 insertions(+), 28 deletions(-)
>
> --
> 1.7.11.7
>
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/linuxppc-dev
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1234304 — Re: [PATCH RFC 0/5] powerpc:numa Add serial nid support

On 28.09.2015 [13:44:42 +0300], Denis Kirjanov wrote:
> On 9/27/15, Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com> wrote:
> > Problem description:
> > Powerpc has sparse node numbering, i.e. on a 4 node system nodes are
> > numbered (possibly) as 0,1,16,17. At a lower level, we map the chipid
> > got from device tree is naturally mapped (directly) to nid.
> 
> Interesting thing to play with, I'll try to test it on my POWER7 box,
> but it doesn't have the OPAL layer :(

Note that it's also interesting to try it under PowerVM, with odd NUMA
topologies and report any issues found :)


-Nish

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1235460 — Re: [PATCH RFC 0/5] powerpc:numa Add serial nid support

On 09/28/2015 10:34 PM, Nishanth Aravamudan wrote:
> On 28.09.2015 [13:44:42 +0300], Denis Kirjanov wrote:
>> On 9/27/15, Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com> wrote:
>>> Problem description:
>>> Powerpc has sparse node numbering, i.e. on a 4 node system nodes are
>>> numbered (possibly) as 0,1,16,17. At a lower level, we map the chipid
>>> got from device tree is naturally mapped (directly) to nid.
>>
>> Interesting thing to play with, I'll try to test it on my POWER7 box,
>> but it doesn't have the OPAL layer :(

Hi Denis,
Thanks for your interest. I have pushed the patches to

https://github.com/ktraghavendra/linux/tree/serialnuma_v1 if it makes
  patches easy to grab.

>
> Note that it's also interesting to try it under PowerVM, with odd NUMA
> topologies and report any issues found :)
>

Thanks Nish, I 'll also grab a powerVM and test.




--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1235511 — Re: [PATCH RFC 0/5] powerpc:numa Add serial nid support

On 9/29/15, Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com> wrote:
> On 09/28/2015 10:34 PM, Nishanth Aravamudan wrote:
>> On 28.09.2015 [13:44:42 +0300], Denis Kirjanov wrote:
>>> On 9/27/15, Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com> wrote:
>>>> Problem description:
>>>> Powerpc has sparse node numbering, i.e. on a 4 node system nodes are
>>>> numbered (possibly) as 0,1,16,17. At a lower level, we map the chipid
>>>> got from device tree is naturally mapped (directly) to nid.
>>>
>>> Interesting thing to play with, I'll try to test it on my POWER7 box,
>>> but it doesn't have the OPAL layer :(
>
> Hi Denis,
> Thanks for your interest. I have pushed the patches to
>
> https://github.com/ktraghavendra/linux/tree/serialnuma_v1 if it makes
>   patches easy to grab.

Thanks!
One sad thing is that I can't test the actual node id mapping now
since currently I have an access to machine with only one memory node
:/ Can we fake it through qemu?

>
>>
>> Note that it's also interesting to try it under PowerVM, with odd NUMA
>> topologies and report any issues found :)
>>
>
> Thanks Nish, I 'll also grab a powerVM and test.
>
>
>
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1235789 — Re: [PATCH RFC 0/5] powerpc:numa Add serial nid support

On 09/30/2015 01:16 AM, Denis Kirjanov wrote:
> On 9/29/15, Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com> wrote:
>> On 09/28/2015 10:34 PM, Nishanth Aravamudan wrote:
>>> On 28.09.2015 [13:44:42 +0300], Denis Kirjanov wrote:
>>>> On 9/27/15, Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com> wrote:
>>>>> Problem description:
>>>>> Powerpc has sparse node numbering, i.e. on a 4 node system nodes are
>>>>> numbered (possibly) as 0,1,16,17. At a lower level, we map the chipid
>>>>> got from device tree is naturally mapped (directly) to nid.
>>>>
>>>> Interesting thing to play with, I'll try to test it on my POWER7 box,
>>>> but it doesn't have the OPAL layer :(
>>
>> Hi Denis,
>> Thanks for your interest. I have pushed the patches to
>>
>> https://github.com/ktraghavendra/linux/tree/serialnuma_v1 if it makes
>>    patches easy to grab.
>
> Thanks!
> One sad thing is that I can't test the actual node id mapping now
> since currently I have an access to machine with only one memory node
> :/ Can we fake it through qemu?
>

faking the sparse numa ids is possible with Nish's patch for qemu:

https://lists.gnu.org/archive/html/qemu-devel/2014-06/msg05826.html

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1234318

On 27.09.2015 [23:59:08 +0530], Raghavendra K T wrote:
> Problem description:
> Powerpc has sparse node numbering, i.e. on a 4 node system nodes are
> numbered (possibly) as 0,1,16,17. At a lower level, we map the chipid
> got from device tree is naturally mapped (directly) to nid.

chipid is a OPAL concept, I believe, and not documented in PAPR... How
does this work under PowerVM?

> Potential side effect of that is:
> 
> 1) There are several places in kernel that assumes serial node numbering.
> and memory allocations assume that all the nodes from 0-(highest nid)
> exist inturn ending up allocating memory for the nodes that does not exist.
> 
> 2) For virtualization use cases (such as qemu, libvirt, openstack), mapping
> sparse nid of the host system to contiguous nids of guest (numa affinity,
> placement) could be a challenge.
> 
> Possible Solutions:
> 1) Handling the memory allocations is kernel case by case: Though in some
> cases it is easy to achieve, some cases may be intrusive/not trivial. 
> at the end it does not handle side effect (2) above.
> 
> 2) Map the sparse chipid got from device tree to a serial nid at kernel
> level (The idea proposed in this series).
> Pro: It is more natural to handle at kernel level than at lower (OPAL) layer.
> con: The chipid is in device tree no longer the same as nid in kernel

Is there any debugging/logging? Looks like not -- so how does a sysadmin
map from firmware-provided values to the Linux values? That's going to
make debugging of large systems (PowerVM or otherwise) less than
pleasant, it seems? Possibly you could put something in sysfs?

-Nish

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1235496

On 09/28/2015 11:04 PM, Nishanth Aravamudan wrote:
> On 27.09.2015 [23:59:08 +0530], Raghavendra K T wrote:
[...]
>>
>> 2) Map the sparse chipid got from device tree to a serial nid at kernel
>> level (The idea proposed in this series).
>> Pro: It is more natural to handle at kernel level than at lower (OPAL) layer.
>> con: The chipid is in device tree no longer the same as nid in kernel
>
> Is there any debugging/logging? Looks like not -- so how does a sysadmin
> map from firmware-provided values to the Linux values? That's going to
> make debugging of large systems (PowerVM or otherwise) less than
> pleasant, it seems? Possibly you could put something in sysfs?

I see 2 things could be done here:

1) while doing dump_numa_cpu_topology() we can dump nid_to_chipid()
as additional information.

2) sysfs->
Does /sys/devices/system/node/nodeX/*chipid* looks good. May be we
should add only for powerpc or otherwise we need to have chipid = nid
populated for other archs. [ I think this change may be done slowly ]





--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1240344

On Sun, 2015-09-27 at 23:59 +0530, Raghavendra K T wrote:
> Problem description:
> Powerpc has sparse node numbering, i.e. on a 4 node system nodes are
> numbered (possibly) as 0,1,16,17. At a lower level, we map the chipid
> got from device tree is naturally mapped (directly) to nid.
> 
> Potential side effect of that is:
> 
> 1) There are several places in kernel that assumes serial node numbering.
> and memory allocations assume that all the nodes from 0-(highest nid)
> exist inturn ending up allocating memory for the nodes that does not exist.

Is it several? Or lots?

If it's several, ie. more than two but not lots, then we should probably just
fix those places. Or is that /really/ hard for some reason?

Do we ever get whole nodes hotplugged in under PowerVM? I don't think so, but I
don't remember for sure.

> 2) For virtualization use cases (such as qemu, libvirt, openstack), mapping
> sparse nid of the host system to contiguous nids of guest (numa affinity,
> placement) could be a challenge.

Can you elaborate? That's a bit vague.

> Possible Solutions:
> 1) Handling the memory allocations is kernel case by case: Though in some
> cases it is easy to achieve, some cases may be intrusive/not trivial. 
> at the end it does not handle side effect (2) above.
> 
> 2) Map the sparse chipid got from device tree to a serial nid at kernel
> level (The idea proposed in this series).
> Pro: It is more natural to handle at kernel level than at lower (OPAL) layer.
> con: The chipid is in device tree no longer the same as nid in kernel
> 
> 3) Let the lower layer (OPAL) give the serial node ids after parsing the
> chipid and the associativity etc [ either as a separate item in device tree
> or by compacting the chipid numbers ]
> Pros: kernel, device tree are on same page and less change in kernel
> Con: is it the functionality expected in lower layer

...

> 3) Numactl tests from
> ftp://oss.sgi.com/www/projects/libnuma/download/numactl-2.0.10.tar.gz
> 
> (infact there were more breakage before the patch because of sparse nid
> and memoryless node cases of powerpc)

This is probably the best argument for your series. ie. userspace is dumb and
fixing every broken app that assumes linear node numbering is not feasible.


So on the whole I think the concept is good. This series though is a bit
confusing because of all the renaming etc. etc. Nish made lots of good comments
so I'll wait for a v2 based on those.

cheers



--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1240373

On 10/06/2015 03:55 PM, Michael Ellerman wrote:
> On Sun, 2015-09-27 at 23:59 +0530, Raghavendra K T wrote:
>> Problem description:
>> Powerpc has sparse node numbering, i.e. on a 4 node system nodes are
>> numbered (possibly) as 0,1,16,17. At a lower level, we map the chipid
>> got from device tree is naturally mapped (directly) to nid.
>>
>> Potential side effect of that is:
>>
>> 1) There are several places in kernel that assumes serial node numbering.
>> and memory allocations assume that all the nodes from 0-(highest nid)
>> exist inturn ending up allocating memory for the nodes that does not exist.
>
> Is it several? Or lots?
>
> If it's several, ie. more than two but not lots, then we should probably just
> fix those places. Or is that /really/ hard for some reason?
>

It is several and I did attempt to fix them. But the rest of the places
(like memcg, work queue, scheduler and so on) are tricky to fix because
the memory allocations are glued with other things.
and similar fix may be expected in future too..


> Do we ever get whole nodes hotplugged in under PowerVM? I don't think so, but I
> don't remember for sure.
>

Even on powervm we do have discontiguous  numa nodes. [Adding more to 
it, we could even end up creating a dummy node 0 just to make kernel
happy]
for e.g.,
available: 2 nodes (0,7)
node 0 cpus:
node 0 size: 0 MB
node 0 free: 0 MB
node 7 cpus: 0 1 2 3 4 5 6 7
node 7 size: 10240 MB
node 7 free: 8174 MB
node distances:
node   0   7
   0:  10  40
   7:  40  10

note that node zero neither has any cpu nor memory.

>> 2) For virtualization use cases (such as qemu, libvirt, openstack), mapping
>> sparse nid of the host system to contiguous nids of guest (numa affinity,
>> placement) could be a challenge.
>
> Can you elaborate? That's a bit vague.

one e.g., i can think of: (though libvirt/openstack people will know 
more about it) suppose one wishes to have half of the vcpus bind to one
physical node and rest of the vcpus to second numa node, we cant say
whether second node is 1,8, or 16. and same libvirtxml on a two node
system may not be valid for another two numa node system.
[ i believe it may cause some migration problem too ].

>
>> Possible Solutions:
>> 1) Handling the memory allocations is kernel case by case: Though in some
>> cases it is easy to achieve, some cases may be intrusive/not trivial.
>> at the end it does not handle side effect (2) above.
>>
>> 2) Map the sparse chipid got from device tree to a serial nid at kernel
>> level (The idea proposed in this series).
>> Pro: It is more natural to handle at kernel level than at lower (OPAL) layer.
>> con: The chipid is in device tree no longer the same as nid in kernel
>>
>> 3) Let the lower layer (OPAL) give the serial node ids after parsing the
>> chipid and the associativity etc [ either as a separate item in device tree
>> or by compacting the chipid numbers ]
>> Pros: kernel, device tree are on same page and less change in kernel
>> Con: is it the functionality expected in lower layer
>
> ...
>
>> 3) Numactl tests from
>> ftp://oss.sgi.com/www/projects/libnuma/download/numactl-2.0.10.tar.gz
>>
>> (infact there were more breakage before the patch because of sparse nid
>> and memoryless node cases of powerpc)
>
> This is probably the best argument for your series. ie. userspace is dumb and
> fixing every broken app that assumes linear node numbering is not feasible.
>
>
> So on the whole I think the concept is good. This series though is a bit
> confusing because of all the renaming etc. etc. Nish made lots of good comments
> so I'll wait for a v2 based on those.
>

Yes, will be sending V2 soon extending my patch to fix powervm case too.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web