Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1648184 > unrolled thread

[Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc

Back to article view | Back to linux.kernel

Contents

  [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc Michael Bringmann <mwb@linux.vnet.ibm.com> - 2017-05-23 17:20 +0200
    Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc Reza Arbab <arbab@linux.vnet.ibm.com> - 2017-05-23 18:00 +0200
      Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc Michael Bringmann <mwb@linux.vnet.ibm.com> - 2017-05-23 22:10 +0200
        Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc Reza Arbab <arbab@linux.vnet.ibm.com> - 2017-05-23 23:50 +0200
          Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc Michael Bringmann <mwb@linux.vnet.ibm.com> - 2017-05-24 00:50 +0200
          Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc Michael Bringmann <mwb@linux.vnet.ibm.com> - 2017-05-24 00:50 +0200
            Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc Michael Ellerman <mpe@ellerman.id.au> - 2017-05-24 13:30 +0200
            Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc Reza Arbab <arbab@linux.vnet.ibm.com> - 2017-05-24 16:40 +0200

#1648184 — [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc

Removing or adding memory via the PowerPC hotplug interface shows
anomalies in the association between memory and nodes.  The code
was updated to initialize more possible nodes to make them available
to subsequent DLPAR hotplug-memory operations, even if they are not
needed at boot time.

Signed-off-by: Michael Bringmann <mwb@linux.vnet.ibm.com>
---
 arch/powerpc/mm/numa.c |   44 ++++++++++++++++++++++++++++++++------------
 1 file changed, 32 insertions(+), 12 deletions(-)

diff --git a/arch/powerpc/mm/numa.c b/arch/powerpc/mm/numa.c
index 15c2dd5..3d58c1f 100644
--- a/arch/powerpc/mm/numa.c
+++ b/arch/powerpc/mm/numa.c
@@ -870,7 +870,7 @@ void __init dump_numa_cpu_topology(void)
 }
 
 /* Initialize NODE_DATA for a node on the local memory */
-static void __init setup_node_data(int nid, u64 start_pfn, u64 end_pfn)
+static void setup_node_data(int nid, u64 start_pfn, u64 end_pfn)
 {
 	u64 spanned_pages = end_pfn - start_pfn;
 	const size_t nd_size = roundup(sizeof(pg_data_t), SMP_CACHE_BYTES);
@@ -878,23 +878,41 @@ static void __init setup_node_data(int nid, u64 start_pfn, u64 end_pfn)
 	void *nd;
 	int tnid;
 
-	nd_pa = memblock_alloc_try_nid(nd_size, SMP_CACHE_BYTES, nid);
-	nd = __va(nd_pa);
+	if (!node_data[nid]) {
+		nd_pa = memblock_alloc_try_nid(nd_size, SMP_CACHE_BYTES, nid);
+		nd = __va(nd_pa);
 
-	/* report and initialize */
-	pr_info("  NODE_DATA [mem %#010Lx-%#010Lx]\n",
-		nd_pa, nd_pa + nd_size - 1);
-	tnid = early_pfn_to_nid(nd_pa >> PAGE_SHIFT);
-	if (tnid != nid)
-		pr_info("    NODE_DATA(%d) on node %d\n", nid, tnid);
+		node_data[nid] = nd;
+		memset(NODE_DATA(nid), 0, sizeof(pg_data_t));
+		NODE_DATA(nid)->node_id = nid;
+
+		/* report and initialize */
+		pr_info("  NODE_DATA [mem %#010Lx-%#010Lx]\n",
+			nd_pa, nd_pa + nd_size - 1);
+		tnid = early_pfn_to_nid(nd_pa >> PAGE_SHIFT);
+		if (tnid != nid)
+			pr_info("    NODE_DATA(%d) on node %d\n", nid, tnid);
+	} else {
+		nd_pa = (u64) node_data[nid];
+		nd = __va(nd_pa);
+	}
 
-	node_data[nid] = nd;
-	memset(NODE_DATA(nid), 0, sizeof(pg_data_t));
-	NODE_DATA(nid)->node_id = nid;
 	NODE_DATA(nid)->node_start_pfn = start_pfn;
 	NODE_DATA(nid)->node_spanned_pages = spanned_pages;
 }
 
+static void setup_nodes(void)
+{
+	int i, l = 32 /* MAX_NUMNODES */;
+
+	for (i = 0; i < l; i++) {
+		if (!node_possible(i)) {
+			setup_node_data(i, 0, 0);
+			node_set(i, node_possible_map);
+		}
+	}
+}
+
 void __init initmem_init(void)
 {
 	int nid, cpu;
@@ -914,6 +932,8 @@ void __init initmem_init(void)
 	 */
 	nodes_and(node_possible_map, node_possible_map, node_online_map);
 
+	setup_nodes();
+
 	for_each_online_node(nid) {
 		unsigned long start_pfn, end_pfn;
 

[toc] | [next] | [standalone]

#1648223

On Tue, May 23, 2017 at 10:15:44AM -0500, Michael Bringmann wrote:
>+static void setup_nodes(void)
>+{
>+	int i, l = 32 /* MAX_NUMNODES */;
>+
>+	for (i = 0; i < l; i++) {
>+		if (!node_possible(i)) {
>+			setup_node_data(i, 0, 0);
>+			node_set(i, node_possible_map);
>+		}
>+	}
>+}

This seems to be a workaround for 3af229f2071f ("powerpc/numa: Reset 
node_possible_map to only node_online_map").

Balbir, you have a patchset which reverts it. Do you think that will be 
getting merged?

http://lkml.kernel.org/r/1479253501-26261-1-git-send-email-bsingharora@gmail.com
(see patch 3/3)

-- 
Reza Arbab

[toc] | [prev] | [next] | [standalone]

#1648387

On 05/23/2017 10:52 AM, Reza Arbab wrote:
> On Tue, May 23, 2017 at 10:15:44AM -0500, Michael Bringmann wrote:
>> +static void setup_nodes(void)
>> +{
>> +    int i, l = 32 /* MAX_NUMNODES */;
>> +
>> +    for (i = 0; i < l; i++) {
>> +        if (!node_possible(i)) {
>> +            setup_node_data(i, 0, 0);
>> +            node_set(i, node_possible_map);
>> +        }
>> +    }
>> +}
> 
> This seems to be a workaround for 3af229f2071f ("powerpc/numa: Reset node_possible_map to only node_online_map").

They may be related, but that commit is not a replacement.  The above patch ensures that
there are enough of the nodes initialized at startup to allow for memory hot-add into a
node that was not used at boot.  (See 'setup_node_data' function in 'numa.c'.)  That and
recording that the node was initialized.

I didn't see where any part of commit 3af229f2071f would touch the 'node_possible_map'
which is needed by 'numa.c' and 'workqueue.c'.  The nodemask created and updated by
'mem_cgroup_may_update_nodemask()' does not appear to be the same mask.

> 
> Balbir, you have a patchset which reverts it. Do you think that will be getting merged?
> 
> http://lkml.kernel.org/r/1479253501-26261-1-git-send-email-bsingharora@gmail.com
> (see patch 3/3)
> 

-- 
Michael W. Bringmann
Linux Technology Center
IBM Corporation
Tie-Line  363-5196
External: (512) 286-5196
Cell:       (512) 466-0650
mwb@linux.vnet.ibm.com

[toc] | [prev] | [next] | [standalone]

#1648847

On Tue, May 23, 2017 at 03:05:08PM -0500, Michael Bringmann wrote:
>On 05/23/2017 10:52 AM, Reza Arbab wrote:
>> On Tue, May 23, 2017 at 10:15:44AM -0500, Michael Bringmann wrote:
>>> +static void setup_nodes(void)
>>> +{
>>> +    int i, l = 32 /* MAX_NUMNODES */;
>>> +
>>> +    for (i = 0; i < l; i++) {
>>> +        if (!node_possible(i)) {
>>> +            setup_node_data(i, 0, 0);
>>> +            node_set(i, node_possible_map);
>>> +        }
>>> +    }
>>> +}
>>
>> This seems to be a workaround for 3af229f2071f ("powerpc/numa: Reset node_possible_map to only node_online_map").
>
>They may be related, but that commit is not a replacement.  The above patch ensures that
>there are enough of the nodes initialized at startup to allow for memory hot-add into a
>node that was not used at boot.  (See 'setup_node_data' function in 'numa.c'.)  That and
>recording that the node was initialized.

Is it really necessary to preinitialize these empty nodes using 
setup_node_data()? When you do memory hotadd into a node that was not 
used at boot, the node data already gets set up by

add_memory
  add_memory_resource
    hotadd_new_pgdat
      arch_alloc_nodedata <-- allocs the pg_data_t
      ...
      free_area_init_node <-- sets NODE_DATA(nid)->node_id, etc.

Removing setup_node_data() from that loop leaves only the call to 
node_set(). If 3af229f2071f (which reduces node_possible_map) was 
reverted, you wouldn't need to do that either.

>I didn't see where any part of commit 3af229f2071f would touch the 'node_possible_map'
>which is needed by 'numa.c' and 'workqueue.c'.  The nodemask created and updated by
>'mem_cgroup_may_update_nodemask()' does not appear to be the same mask.

Are you sure you're looking at 3af229f2071f? It only adds one line of 
code; the reduction of node_possible_map.

-- 
Reza Arbab

[toc] | [prev] | [next] | [standalone]

#1648988

On 05/23/2017 04:49 PM, Reza Arbab wrote:
> On Tue, May 23, 2017 at 03:05:08PM -0500, Michael Bringmann wrote:
>> On 05/23/2017 10:52 AM, Reza Arbab wrote:
>>> On Tue, May 23, 2017 at 10:15:44AM -0500, Michael Bringmann wrote:
>>>> +static void setup_nodes(void)
>>>> +{
>>>> +    int i, l = 32 /* MAX_NUMNODES */;
>>>> +
>>>> +    for (i = 0; i < l; i++) {
>>>> +        if (!node_possible(i)) {
>>>> +            setup_node_data(i, 0, 0);
>>>> +            node_set(i, node_possible_map);
>>>> +        }
>>>> +    }
>>>> +}
>>>
>>> This seems to be a workaround for 3af229f2071f ("powerpc/numa: Reset node_possible_map to only node_online_map").
>>
>> They may be related, but that commit is not a replacement.  The above patch ensures that
>> there are enough of the nodes initialized at startup to allow for memory hot-add into a
>> node that was not used at boot.  (See 'setup_node_data' function in 'numa.c'.)  That and
>> recording that the node was initialized.
> 
> Is it really necessary to preinitialize these empty nodes using setup_node_data()? When you do memory hotadd into a node that was not used at boot, the node data already gets set up by
> 
> add_memory
>  add_memory_resource
>    hotadd_new_pgdat
>      arch_alloc_nodedata <-- allocs the pg_data_t
>      ...
>      free_area_init_node <-- sets NODE_DATA(nid)->node_id, etc.

I see that code now, but for some reason it did not work when I hot-added
memory.

> 
> Removing setup_node_data() from that loop leaves only the call to node_set(). If 3af229f2071f (which reduces node_possible_map) was reverted, you wouldn't need to do that either.
> 
>> I didn't see where any part of commit 3af229f2071f would touch the 'node_possible_map'
>> which is needed by 'numa.c' and 'workqueue.c'.  The nodemask created and updated by
>> 'mem_cgroup_may_update_nodemask()' does not appear to be the same mask.
> 
> Are you sure you're looking at 3af229f2071f? It only adds one line of code; the reduction of node_possible_map.
> 

The 3rd file in the patch set removes,

-	nodes_and(node_possible_map, node_possible_map, node_online_map);

I need to add bits to 'node_possible_map' -- bits which may not be used
for the memory at boot, but which would be used when memory is hot-added
later.  I haven't found anything outside of the boot code that adds bits
to the 'possible' mask.

-- 
Michael W. Bringmann
Linux Technology Center
IBM Corporation
Tie-Line  363-5196
External: (512) 286-5196
Cell:       (512) 466-0650
mwb@linux.vnet.ibm.com

[toc] | [prev] | [next] | [standalone]

#1648992

On 05/23/2017 04:49 PM, Reza Arbab wrote:
> On Tue, May 23, 2017 at 03:05:08PM -0500, Michael Bringmann wrote:
>> On 05/23/2017 10:52 AM, Reza Arbab wrote:
>>> On Tue, May 23, 2017 at 10:15:44AM -0500, Michael Bringmann wrote:
>>>> +static void setup_nodes(void)
>>>> +{
>>>> +    int i, l = 32 /* MAX_NUMNODES */;
>>>> +
>>>> +    for (i = 0; i < l; i++) {
>>>> +        if (!node_possible(i)) {
>>>> +            setup_node_data(i, 0, 0);
>>>> +            node_set(i, node_possible_map);
>>>> +        }
>>>> +    }
>>>> +}
>>>
>>> This seems to be a workaround for 3af229f2071f ("powerpc/numa: Reset node_possible_map to only node_online_map").
>>
>> They may be related, but that commit is not a replacement.  The above patch ensures that
>> there are enough of the nodes initialized at startup to allow for memory hot-add into a
>> node that was not used at boot.  (See 'setup_node_data' function in 'numa.c'.)  That and
>> recording that the node was initialized.
> 
> Is it really necessary to preinitialize these empty nodes using setup_node_data()? When you do memory hotadd into a node that was not used at boot, the node data already gets set up by
> 
> add_memory
>  add_memory_resource
>    hotadd_new_pgdat
>      arch_alloc_nodedata <-- allocs the pg_data_t
>      ...
>      free_area_init_node <-- sets NODE_DATA(nid)->node_id, etc.
> 
> Removing setup_node_data() from that loop leaves only the call to node_set(). If 3af229f2071f (which reduces node_possible_map) was reverted, you wouldn't need to do that either.

With or without 3af229f2071f, we would still need to add something, somewhere to add new
bits to the 'node_possible_map'.  That is not being done.

> 
>> I didn't see where any part of commit 3af229f2071f would touch the 'node_possible_map'
>> which is needed by 'numa.c' and 'workqueue.c'.  The nodemask created and updated by
>> 'mem_cgroup_may_update_nodemask()' does not appear to be the same mask.
> 
> Are you sure you're looking at 3af229f2071f? It only adds one line of code; the reduction of node_possible_map.
> 

-- 
Michael W. Bringmann
Linux Technology Center
IBM Corporation
Tie-Line  363-5196
External: (512) 286-5196
Cell:       (512) 466-0650
mwb@linux.vnet.ibm.com

[toc] | [prev] | [next] | [standalone]

#1649487

Michael Bringmann <mwb@linux.vnet.ibm.com> writes:

> On 05/23/2017 04:49 PM, Reza Arbab wrote:
>> On Tue, May 23, 2017 at 03:05:08PM -0500, Michael Bringmann wrote:
>>> On 05/23/2017 10:52 AM, Reza Arbab wrote:
>>>> On Tue, May 23, 2017 at 10:15:44AM -0500, Michael Bringmann wrote:
>>>>> +static void setup_nodes(void)
>>>>> +{
>>>>> +    int i, l = 32 /* MAX_NUMNODES */;
>>>>> +
>>>>> +    for (i = 0; i < l; i++) {
>>>>> +        if (!node_possible(i)) {
>>>>> +            setup_node_data(i, 0, 0);
>>>>> +            node_set(i, node_possible_map);
>>>>> +        }
>>>>> +    }
>>>>> +}
>>>>
>>>> This seems to be a workaround for 3af229f2071f ("powerpc/numa: Reset node_possible_map to only node_online_map").
>>>
>>> They may be related, but that commit is not a replacement.  The above patch ensures that
>>> there are enough of the nodes initialized at startup to allow for memory hot-add into a
>>> node that was not used at boot.  (See 'setup_node_data' function in 'numa.c'.)  That and
>>> recording that the node was initialized.
>> 
>> Is it really necessary to preinitialize these empty nodes using setup_node_data()? When you do memory hotadd into a node that was not used at boot, the node data already gets set up by
>> 
>> add_memory
>>  add_memory_resource
>>    hotadd_new_pgdat
>>      arch_alloc_nodedata <-- allocs the pg_data_t
>>      ...
>>      free_area_init_node <-- sets NODE_DATA(nid)->node_id, etc.
>> 
>> Removing setup_node_data() from that loop leaves only the call to node_set(). If 3af229f2071f (which reduces node_possible_map) was reverted, you wouldn't need to do that either.
>
> With or without 3af229f2071f, we would still need to add something, somewhere to add new
> bits to the 'node_possible_map'.  That is not being done.

You mustn't add bits to the possible map after boot.

That's its purpose, to tell you what nodes could ever *possibly* exist.

cheers

[toc] | [prev] | [next] | [standalone]

#1649667

On Tue, May 23, 2017 at 05:44:23PM -0500, Michael Bringmann wrote:
>On 05/23/2017 04:49 PM, Reza Arbab wrote:
>> On Tue, May 23, 2017 at 03:05:08PM -0500, Michael Bringmann wrote:
>>> On 05/23/2017 10:52 AM, Reza Arbab wrote:
>>>> On Tue, May 23, 2017 at 10:15:44AM -0500, Michael Bringmann wrote:
>>>>> +static void setup_nodes(void)
>>>>> +{
>>>>> +    int i, l = 32 /* MAX_NUMNODES */;
>>>>> +
>>>>> +    for (i = 0; i < l; i++) {
>>>>> +        if (!node_possible(i)) {
>>>>> +            setup_node_data(i, 0, 0);
>>>>> +            node_set(i, node_possible_map);
>>>>> +        }
>>>>> +    }
>>>>> +}
>>>>
>>>> This seems to be a workaround for 3af229f2071f ("powerpc/numa: Reset node_possible_map to only node_online_map").
>>>
>>> They may be related, but that commit is not a replacement.  The above patch ensures that
>>> there are enough of the nodes initialized at startup to allow for memory hot-add into a
>>> node that was not used at boot.  (See 'setup_node_data' function in 'numa.c'.)  That and
>>> recording that the node was initialized.
>>
>> Is it really necessary to preinitialize these empty nodes using setup_node_data()? When you do memory hotadd into a node that was not used at boot, the node data already gets set up by
>>
>> add_memory
>>  add_memory_resource
>>    hotadd_new_pgdat
>>      arch_alloc_nodedata <-- allocs the pg_data_t
>>      ...
>>      free_area_init_node <-- sets NODE_DATA(nid)->node_id, etc.
>>
>> Removing setup_node_data() from that loop leaves only the call to node_set(). If 3af229f2071f (which reduces node_possible_map) was reverted, you wouldn't need to do that either.
>
>With or without 3af229f2071f, we would still need to add something, somewhere to add new
>bits to the 'node_possible_map'.  That is not being done.

Without 3af229f2071f, those bits would already BE set in 
node_possible_map. You wouldn't have to do anything.

-- 
Reza Arbab

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web