Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1208325 > unrolled thread

[Patch V3 0/9] Enable memoryless node support for x86

Started byJiang Liu <jiang.liu@linux.intel.com>
First post2015-08-17 05:20 +0200
Last post2015-08-19 10:10 +0200
Articles 11 on this page of 31 — 9 participants

Back to article view | Back to linux.kernel

Contents

  [Patch V3 0/9] Enable memoryless node support for x86 Jiang Liu <jiang.liu@linux.intel.com> - 2015-08-17 05:20 +0200
    [Patch V3 6/9] i40evf: Use numa_mem_id() to better support memoryless node Jiang Liu <jiang.liu@linux.intel.com> - 2015-08-17 05:20 +0200
      RE: [Intel-wired-lan] [Patch V3 6/9] i40evf: Use numa_mem_id() to  better support memoryless node "Patil, Kiran" <kiran.patil@intel.com> - 2015-08-17 21:10 +0200
        Re: [Intel-wired-lan] [Patch V3 6/9] i40evf: Use numa_mem_id() to  better support memoryless node Jeff Kirsher <jeffrey.t.kirsher@intel.com> - 2015-08-18 23:40 +0200
    [Patch V3 9/9] mm, x86: Enable memoryless node support to better support CPU/memory hotplug Jiang Liu <jiang.liu@linux.intel.com> - 2015-08-17 05:20 +0200
      Re: [Patch V3 9/9] mm, x86: Enable memoryless node support to better  support CPU/memory hotplug Tang Chen <tangchen@cn.fujitsu.com> - 2015-08-18 08:20 +0200
        Re: [Patch V3 9/9] mm, x86: Enable memoryless node support to better  support CPU/memory hotplug Tang Chen <tangchen@cn.fujitsu.com> - 2015-08-18 13:40 +0200
      Re: [Patch V3 9/9] mm, x86: Enable memoryless node support to better  support CPU/memory hotplug Ingo Molnar <mingo@kernel.org> - 2015-08-18 09:40 +0200
    [Patch V3 1/9] x86, NUMA, ACPI: Online node earlier when doing CPU hot-addition Jiang Liu <jiang.liu@linux.intel.com> - 2015-08-17 05:20 +0200
    [Patch V3 2/9] kernel/profile.c: Replace cpu_to_mem() with cpu_to_node() Jiang Liu <jiang.liu@linux.intel.com> - 2015-08-17 05:20 +0200
      Re: [Patch V3 2/9] kernel/profile.c: Replace cpu_to_mem() with  cpu_to_node() David Rientjes <rientjes@google.com> - 2015-08-18 02:40 +0200
        Re: [Patch V3 2/9] kernel/profile.c: Replace cpu_to_mem() with  cpu_to_node() Jiang Liu <jiang.liu@linux.intel.com> - 2015-08-19 09:20 +0200
          Re: [Patch V3 2/9] kernel/profile.c: Replace cpu_to_mem() with  cpu_to_node() David Rientjes <rientjes@google.com> - 2015-08-20 02:10 +0200
    [Patch V3 4/9] openvswitch: Replace cpu_to_node() with cpu_to_mem() to support memoryless node Jiang Liu <jiang.liu@linux.intel.com> - 2015-08-17 05:20 +0200
      Re: [Patch V3 4/9] openvswitch: Replace cpu_to_node() with  cpu_to_mem() to support memoryless node Pravin Shelar <pshelar@nicira.com> - 2015-08-18 02:20 +0200
    [Patch V3 7/9] x86, numa: Kill useless code to improve code readability Jiang Liu <jiang.liu@linux.intel.com> - 2015-08-17 05:20 +0200
    [Patch V3 3/9] sgi-xp: Replace cpu_to_node() with cpu_to_mem() to support memoryless node Jiang Liu <jiang.liu@linux.intel.com> - 2015-08-17 05:20 +0200
      Re: [Patch V3 3/9] sgi-xp: Replace cpu_to_node() with cpu_to_mem()  to support memoryless node David Rientjes <rientjes@google.com> - 2015-08-18 02:30 +0200
        Re: [Patch V3 3/9] sgi-xp: Replace cpu_to_node() with cpu_to_mem() to  support memoryless node Jiang Liu <jiang.liu@linux.intel.com> - 2015-08-19 10:30 +0200
          Re: [Patch V3 3/9] sgi-xp: Replace cpu_to_node() with cpu_to_mem()  to support memoryless node David Rientjes <rientjes@google.com> - 2015-08-20 02:10 +0200
            Re: [Patch V3 3/9] sgi-xp: Replace cpu_to_node() with cpu_to_mem() to  support memoryless node Jiang Liu <jiang.liu@linux.intel.com> - 2015-08-20 08:40 +0200
      Re: [Patch V3 3/9] sgi-xp: Replace cpu_to_node() with cpu_to_mem() to  support memoryless node Robin Holt <robinmholt@gmail.com> - 2015-08-19 14:00 +0200
        Re: [Patch V3 3/9] sgi-xp: Replace cpu_to_node() with cpu_to_mem() to  support memoryless node Jiang Liu <jiang.liu@linux.intel.com> - 2015-08-19 14:50 +0200
    [Patch V3 5/9] i40e: Use numa_mem_id() to better support memoryless node Jiang Liu <jiang.liu@linux.intel.com> - 2015-08-17 05:20 +0200
      Re: [Patch V3 5/9] i40e: Use numa_mem_id() to better support memoryless  node David Rientjes <rientjes@google.com> - 2015-08-18 02:40 +0200
      RE: [Intel-wired-lan] [Patch V3 5/9] i40e: Use numa_mem_id() to  better support memoryless node "Patil, Kiran" <kiran.patil@intel.com> - 2015-08-20 00:40 +0200
        RE: [Intel-wired-lan] [Patch V3 5/9] i40e: Use numa_mem_id() to  better support memoryless node David Rientjes <rientjes@google.com> - 2015-08-20 02:20 +0200
    [Patch V3 8/9] mm: Update _mem_id_[] for every possible CPU when memory configuration changes Jiang Liu <jiang.liu@linux.intel.com> - 2015-08-17 05:20 +0200
    Re: [Patch V3 0/9] Enable memoryless node support for x86 Andrew Morton <akpm@linux-foundation.org> - 2015-08-17 23:40 +0200
    Re: [Patch V3 0/9] Enable memoryless node support for x86 Tang Chen <tangchen@cn.fujitsu.com> - 2015-08-18 12:10 +0200
      Re: [Patch V3 0/9] Enable memoryless node support for x86 Jiang Liu <jiang.liu@linux.intel.com> - 2015-08-19 10:10 +0200

Page 2 of 2 — ← Prev page 1 [2]

#1210184 — Re: [Patch V3 3/9] sgi-xp: Replace cpu_to_node() with cpu_to_mem() to support memoryless node

FromJiang Liu <jiang.liu@linux.intel.com>
Date2015-08-20 08:40 +0200
SubjectRe: [Patch V3 3/9] sgi-xp: Replace cpu_to_node() with cpu_to_mem() to support memoryless node
Message-ID<pZrG2-5if-23@gated-at.bofh.it>
In reply to#1210090
On 2015/8/20 8:02, David Rientjes wrote:
> On Wed, 19 Aug 2015, Jiang Liu wrote:
> 
>>> Why not simply fix build_zonelists_node() so that the __GFP_THISNODE 
>>> zonelists are set up to reference the zones of cpu_to_mem() for memoryless 
>>> nodes?
>>>
>>> It seems much better than checking and maintaining every __GFP_THISNODE 
>>> user to determine if they are using a memoryless node or not.  I don't 
>>> feel that this solution is maintainable in the longterm.
>> Hi David,
>> 	There are some usage cases, such as memory migration,
>> expect the page allocator rejecting memory allocation requests
>> if there is no memory on local node. So we have:
>> 1) alloc_pages_node(cpu_to_node(), __GFP_THISNODE) to only allocate
>> memory from local node.
>> 2) alloc_pages_node(cpu_to_mem(), __GFP_THISNODE) to allocate memory
>> from local node or from nearest node if local node is memoryless.
>>
> 
> Right, so do you think it would be better to make the default zonelists be 
> setup so that cpu_to_node()->zonelists == cpu_to_mem()->zonelists and then 
> individual callers that want to fail for memoryless nodes check 
> populated_zone() themselves?
Hi David,
	Great idea:) I think that means we are going to kill the
concept of memoryless node, and we only need to specially handle
a few callers who really care about whether there is memory on
local node.
	Then I need some time to audit all usages of __GFP_THISNODE
and update you whether it's doable.
Thanks!
Gerry

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1209797 — Re: [Patch V3 3/9] sgi-xp: Replace cpu_to_node() with cpu_to_mem() to support memoryless node

FromRobin Holt <robinmholt@gmail.com>
Date2015-08-19 14:00 +0200
SubjectRe: [Patch V3 3/9] sgi-xp: Replace cpu_to_node() with cpu_to_mem() to support memoryless node
Message-ID<pZaca-50h-9@gated-at.bofh.it>
In reply to#1208332
On Sun, Aug 16, 2015 at 10:19 PM, Jiang Liu <jiang.liu@linux.intel.com> wrote:
> Function xpc_create_gru_mq_uv() allocates memory with __GFP_THISNODE
> flag set, which may cause permanent memory allocation failure on
> memoryless node. So replace cpu_to_node() with cpu_to_mem() to better
> support memoryless node. For node with memory, cpu_to_mem() is the same
> as cpu_to_node().
>
> Signed-off-by: Jiang Liu <jiang.liu@linux.intel.com>
> ---
>  drivers/misc/sgi-xp/xpc_uv.c |    2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/misc/sgi-xp/xpc_uv.c b/drivers/misc/sgi-xp/xpc_uv.c
> index 95c894482fdd..9210981c0d5b 100644
> --- a/drivers/misc/sgi-xp/xpc_uv.c
> +++ b/drivers/misc/sgi-xp/xpc_uv.c
> @@ -238,7 +238,7 @@ xpc_create_gru_mq_uv(unsigned int mq_size, int cpu, char *irq_name,
>
>         mq->mmr_blade = uv_cpu_to_blade_id(cpu);
>
> -       nid = cpu_to_node(cpu);
> +       nid = cpu_to_mem(cpu);

I would recommend rejecting this.  First, SGI's UV system does not and
can not support memory-less nodes.  Additionally the hardware _REALLY_
wants the memory to be local to the CPU.  We will register this memory
region with the node firmware.  That will set the hardware up to watch
this memory block and raise an IRQ targeting the registered CPU when
anything is written into the memory block.  This is all part of how
cross-partition communications expects to work.

Additionally, the interrupt handler will read the memory region, so
having node-local memory is extremely helpful.

Thanks,
Robin
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1209820 — Re: [Patch V3 3/9] sgi-xp: Replace cpu_to_node() with cpu_to_mem() to support memoryless node

FromJiang Liu <jiang.liu@linux.intel.com>
Date2015-08-19 14:50 +0200
SubjectRe: [Patch V3 3/9] sgi-xp: Replace cpu_to_node() with cpu_to_mem() to support memoryless node
Message-ID<pZaYy-6b0-9@gated-at.bofh.it>
In reply to#1209797
On 2015/8/19 19:52, Robin Holt wrote:
> On Sun, Aug 16, 2015 at 10:19 PM, Jiang Liu <jiang.liu@linux.intel.com> wrote:
>> Function xpc_create_gru_mq_uv() allocates memory with __GFP_THISNODE
>> flag set, which may cause permanent memory allocation failure on
>> memoryless node. So replace cpu_to_node() with cpu_to_mem() to better
>> support memoryless node. For node with memory, cpu_to_mem() is the same
>> as cpu_to_node().
>>
>> Signed-off-by: Jiang Liu <jiang.liu@linux.intel.com>
>> ---
>>  drivers/misc/sgi-xp/xpc_uv.c |    2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/drivers/misc/sgi-xp/xpc_uv.c b/drivers/misc/sgi-xp/xpc_uv.c
>> index 95c894482fdd..9210981c0d5b 100644
>> --- a/drivers/misc/sgi-xp/xpc_uv.c
>> +++ b/drivers/misc/sgi-xp/xpc_uv.c
>> @@ -238,7 +238,7 @@ xpc_create_gru_mq_uv(unsigned int mq_size, int cpu, char *irq_name,
>>
>>         mq->mmr_blade = uv_cpu_to_blade_id(cpu);
>>
>> -       nid = cpu_to_node(cpu);
>> +       nid = cpu_to_mem(cpu);
> 
> I would recommend rejecting this.  First, SGI's UV system does not and
> can not support memory-less nodes.  Additionally the hardware _REALLY_
> wants the memory to be local to the CPU.  We will register this memory
> region with the node firmware.  That will set the hardware up to watch
> this memory block and raise an IRQ targeting the registered CPU when
> anything is written into the memory block.  This is all part of how
> cross-partition communications expects to work.
> 
> Additionally, the interrupt handler will read the memory region, so
> having node-local memory is extremely helpful.
Hi Robin,
	Thanks for review, I will drop this patch in next version.
Actually, if SGI UV systems don't support memoryless node, cpu_to_mem()
is the same as cpu_to_node().
Thanks!
Gerry
> 
> Thanks,
> Robin
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1208333 — [Patch V3 5/9] i40e: Use numa_mem_id() to better support memoryless node

FromJiang Liu <jiang.liu@linux.intel.com>
Date2015-08-17 05:20 +0200
Subject[Patch V3 5/9] i40e: Use numa_mem_id() to better support memoryless node
Message-ID<pYj7Q-3i1-21@gated-at.bofh.it>
In reply to#1208325
Function i40e_clean_rx_irq() tries to reuse memory pages allocated
from the nearest node. To better support memoryless node, use
numa_mem_id() instead of numa_node_id() to get the nearest node with
memory.

This change should only affect performance.

Signed-off-by: Jiang Liu <jiang.liu@linux.intel.com>
---
 drivers/net/ethernet/intel/i40e/i40e_txrx.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/intel/i40e/i40e_txrx.c b/drivers/net/ethernet/intel/i40e/i40e_txrx.c
index 9a4f2bc70cd2..a8f618cb8eb0 100644
--- a/drivers/net/ethernet/intel/i40e/i40e_txrx.c
+++ b/drivers/net/ethernet/intel/i40e/i40e_txrx.c
@@ -1516,7 +1516,7 @@ static int i40e_clean_rx_irq_ps(struct i40e_ring *rx_ring, int budget)
 	unsigned int total_rx_bytes = 0, total_rx_packets = 0;
 	u16 rx_packet_len, rx_header_len, rx_sph, rx_hbo;
 	u16 cleaned_count = I40E_DESC_UNUSED(rx_ring);
-	const int current_node = numa_node_id();
+	const int current_node = numa_mem_id();
 	struct i40e_vsi *vsi = rx_ring->vsi;
 	u16 i = rx_ring->next_to_clean;
 	union i40e_rx_desc *rx_desc;
-- 
1.7.10.4

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1208981 — Re: [Patch V3 5/9] i40e: Use numa_mem_id() to better support memoryless node

FromDavid Rientjes <rientjes@google.com>
Date2015-08-18 02:40 +0200
SubjectRe: [Patch V3 5/9] i40e: Use numa_mem_id() to better support memoryless node
Message-ID<pYD6x-6P7-9@gated-at.bofh.it>
In reply to#1208333
On Mon, 17 Aug 2015, Jiang Liu wrote:

> Function i40e_clean_rx_irq() tries to reuse memory pages allocated

s/i40e_clean_rx_irq/i40e_clean_rx_irq_ps/

> from the nearest node. To better support memoryless node, use
> numa_mem_id() instead of numa_node_id() to get the nearest node with
> memory.
> 

Out of curiosity, what prevents the cpu to be preempted and current_node 
to no longer match numa_mem_id()?

> This change should only affect performance.
> 
> Signed-off-by: Jiang Liu <jiang.liu@linux.intel.com>
> ---
>  drivers/net/ethernet/intel/i40e/i40e_txrx.c |    2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/drivers/net/ethernet/intel/i40e/i40e_txrx.c b/drivers/net/ethernet/intel/i40e/i40e_txrx.c
> index 9a4f2bc70cd2..a8f618cb8eb0 100644
> --- a/drivers/net/ethernet/intel/i40e/i40e_txrx.c
> +++ b/drivers/net/ethernet/intel/i40e/i40e_txrx.c
> @@ -1516,7 +1516,7 @@ static int i40e_clean_rx_irq_ps(struct i40e_ring *rx_ring, int budget)
>  	unsigned int total_rx_bytes = 0, total_rx_packets = 0;
>  	u16 rx_packet_len, rx_header_len, rx_sph, rx_hbo;
>  	u16 cleaned_count = I40E_DESC_UNUSED(rx_ring);
> -	const int current_node = numa_node_id();
> +	const int current_node = numa_mem_id();
>  	struct i40e_vsi *vsi = rx_ring->vsi;
>  	u16 i = rx_ring->next_to_clean;
>  	union i40e_rx_desc *rx_desc;
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1210058 — RE: [Intel-wired-lan] [Patch V3 5/9] i40e: Use numa_mem_id() to better support memoryless node

From"Patil, Kiran" <kiran.patil@intel.com>
Date2015-08-20 00:40 +0200
SubjectRE: [Intel-wired-lan] [Patch V3 5/9] i40e: Use numa_mem_id() to better support memoryless node
Message-ID<pZkbv-2Gm-5@gated-at.bofh.it>
In reply to#1208333
Acked-by: Kiran Patil <kiran.patil@intel.com>

-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces@lists.osuosl.org] On Behalf Of Jiang Liu
Sent: Sunday, August 16, 2015 8:19 PM
To: Andrew Morton; Mel Gorman; David Rientjes; Mike Galbraith; Peter Zijlstra; Wysocki, Rafael J; Tang Chen; Tejun Heo; Kirsher, Jeffrey T; Brandeburg, Jesse; Nelson, Shannon; Wyborny, Carolyn; Skidmore, Donald C; Vick, Matthew; Ronciak, John; Williams, Mitch A
Cc: Luck, Tony; netdev@vger.kernel.org; x86@kernel.org; linux-hotplug@vger.kernel.org; linux-kernel@vger.kernel.org; linux-mm@kvack.org; intel-wired-lan@lists.osuosl.org; Jiang Liu
Subject: [Intel-wired-lan] [Patch V3 5/9] i40e: Use numa_mem_id() to better support memoryless node

Function i40e_clean_rx_irq() tries to reuse memory pages allocated from the nearest node. To better support memoryless node, use
numa_mem_id() instead of numa_node_id() to get the nearest node with memory.

This change should only affect performance.

Signed-off-by: Jiang Liu <jiang.liu@linux.intel.com>
---
 drivers/net/ethernet/intel/i40e/i40e_txrx.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/intel/i40e/i40e_txrx.c b/drivers/net/ethernet/intel/i40e/i40e_txrx.c
index 9a4f2bc70cd2..a8f618cb8eb0 100644
--- a/drivers/net/ethernet/intel/i40e/i40e_txrx.c
+++ b/drivers/net/ethernet/intel/i40e/i40e_txrx.c
@@ -1516,7 +1516,7 @@ static int i40e_clean_rx_irq_ps(struct i40e_ring *rx_ring, int budget)
 	unsigned int total_rx_bytes = 0, total_rx_packets = 0;
 	u16 rx_packet_len, rx_header_len, rx_sph, rx_hbo;
 	u16 cleaned_count = I40E_DESC_UNUSED(rx_ring);
-	const int current_node = numa_node_id();
+	const int current_node = numa_mem_id();
 	struct i40e_vsi *vsi = rx_ring->vsi;
 	u16 i = rx_ring->next_to_clean;
 	union i40e_rx_desc *rx_desc;
--
1.7.10.4

_______________________________________________
Intel-wired-lan mailing list
Intel-wired-lan@lists.osuosl.org
http://lists.osuosl.org/mailman/listinfo/intel-wired-lan
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1210094 — RE: [Intel-wired-lan] [Patch V3 5/9] i40e: Use numa_mem_id() to better support memoryless node

FromDavid Rientjes <rientjes@google.com>
Date2015-08-20 02:20 +0200
SubjectRE: [Intel-wired-lan] [Patch V3 5/9] i40e: Use numa_mem_id() to better support memoryless node
Message-ID<pZlKi-503-11@gated-at.bofh.it>
In reply to#1210058
On Wed, 19 Aug 2015, Patil, Kiran wrote:

> Acked-by: Kiran Patil <kiran.patil@intel.com>

Where's the call to preempt_disable() to prevent kernels with preemption 
from making numa_node_id() invalid during this iteration?
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1208334 — [Patch V3 8/9] mm: Update _mem_id_[] for every possible CPU when memory configuration changes

FromJiang Liu <jiang.liu@linux.intel.com>
Date2015-08-17 05:20 +0200
Subject[Patch V3 8/9] mm: Update _mem_id_[] for every possible CPU when memory configuration changes
Message-ID<pYj7Q-3i1-23@gated-at.bofh.it>
In reply to#1208325
Current kernel only updates _mem_id_[cpu] for onlined CPUs when memory
configuration changes. So kernel may allocate memory from remote node
for a CPU if the CPU is still in absent or offline state even if the
node associated with the CPU has already been onlined. This patch tries
to improve performance by updating _mem_id_[cpu] for each possible CPU
when memory configuration changes, thus kernel could always allocate
from local node once the node is onlined.

We check node_online(cpu_to_node(cpu)) because:
1) local_memory_node(nid) needs to access NODE_DATA(nid)
2) try_offline_node(nid) just zeroes out NODE_DATA(nid) instead of free it

Signed-off-by: Jiang Liu <jiang.liu@linux.intel.com>
---
 mm/page_alloc.c |   10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index beda41710802..bcfd66e66820 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -4334,13 +4334,13 @@ static int __build_all_zonelists(void *data)
 		/*
 		 * We now know the "local memory node" for each node--
 		 * i.e., the node of the first zone in the generic zonelist.
-		 * Set up numa_mem percpu variable for on-line cpus.  During
-		 * boot, only the boot cpu should be on-line;  we'll init the
-		 * secondary cpus' numa_mem as they come on-line.  During
-		 * node/memory hotplug, we'll fixup all on-line cpus.
+		 * Set up numa_mem percpu variable for all possible cpus
+		 * if associated node has been onlined.
 		 */
-		if (cpu_online(cpu))
+		if (node_online(cpu_to_node(cpu)))
 			set_cpu_numa_mem(cpu, local_memory_node(cpu_to_node(cpu)));
+		else
+			set_cpu_numa_mem(cpu, NUMA_NO_NODE);
 #endif
 	}
 
-- 
1.7.10.4

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1208902

FromAndrew Morton <akpm@linux-foundation.org>
Date2015-08-17 23:40 +0200
Message-ID<pYAil-2Fp-1@gated-at.bofh.it>
In reply to#1208325
On Mon, 17 Aug 2015 11:18:57 +0800 Jiang Liu <jiang.liu@linux.intel.com> wrote:

> This is the third version to enable memoryless node support on x86
> platforms.

I'll grab this for inclusion in linux-next after the 4.2 release.

It's basically an x86 patch so if someone else was planning on looking
after it, please tell me off.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1209158

FromTang Chen <tangchen@cn.fujitsu.com>
Date2015-08-18 12:10 +0200
Message-ID<pYM0a-310-7@gated-at.bofh.it>
In reply to#1208325
On 08/17/2015 11:18 AM, Jiang Liu wrote:
> This is the third version to enable memoryless node support on x86
> platforms. The previous version (https://lkml.org/lkml/2014/7/11/75)
> blindly replaces numa_node_id()/cpu_to_node() with numa_mem_id()/
> cpu_to_mem(). That's not the right solution as pointed out by Tejun
> and Peter due to:
> 1) We shouldn't shift the burden to normal slab users.
> 2) Details of memoryless node should be hidden in arch and mm code
>     as much as possible.
>
> After digging into more code and documentation, we found the rules to
> deal with memoryless node should be:
> 1) Arch code should online corresponding NUMA node before onlining any
>     CPU or memory, otherwise it may cause invalid memory access when
>     accessing NODE_DATA(nid).
> 2) For normal memory allocations without __GFP_THISNODE setting in the
>     gfp_flags, we should prefer numa_node_id()/cpu_to_node() instead of
>     numa_mem_id()/cpu_to_mem() because the latter loses hardware topology
>     information as pointed out by Tejun:
> 	   A - B - X - C - D
> 	Where X is the memless node.  numa_mem_id() on X would return
> 	either B or C, right?  If B or C can't satisfy the allocation,
> 	the allocator would fallback to A from B and D for C, both of
> 	which aren't optimal. It should first fall back to C or B
> 	respectively, which the allocator can't do anymoe because the
> 	information is lost when the caller side performs numa_mem_id().

Hi Liu,

BTW, how is this A - B - X - C - D problem solved ?
I don't quite follow this.

I cannot tell the difference between numa_node_id()/cpu_to_node() and
numa_mem_id()/cpu_to_mem() on this point. Even with hardware topology
info, how could it avoid this problem ?

Isn't it still possible falling back to A from B and D for C ?

Thanks.

> 3) For memory allocation with __GFP_THISNODE setting in gfp_flags,
>     numa_node_id()/cpu_to_node() should be used if caller only wants to
>     allocate from local memory, otherwise numa_mem_id()/cpu_to_mem()
>     should be used if caller wants to allocate from the nearest node
>     with memory.
> 4) numa_mem_id()/cpu_to_mem() should be used if caller wants to check
>     whether a page is allocated from the nearest node.
>
> Based on above rules, this patch set
> 1) Patch 1 is a bugfix to resolve a crash caused by socket hot-addition
> 2) Patch 2 replaces numa_mem_id() with numa_node_id() when __GFP_THISNODE
>     isn't set in gfp_flags.
> 3) Patch 3-6 replaces numa_node_id()/cpu_to_node() with numa_mem_id()/
>     cpu_to_mem() if caller wants to allocate from local node only.
> 4) Patch 7-9 enables support of memoryless node on x86.
>
> With this patch set applied, on a system with two sockets enabled at boot,
> one with memory and the other without memory, we got following numa
> topology after boot:
> root@bkd04sdp:~# numactl --hardware
> available: 2 nodes (0-1)
> node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44
> node 0 size: 15940 MB
> node 0 free: 15397 MB
> node 1 cpus: 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59
> node 1 size: 0 MB
> node 1 free: 0 MB
> node distances:
> node   0   1
>    0:  10  21
>    1:  21  10
>
> After hot-adding the third socket without memory, we got:
> root@bkd04sdp:~# numactl --hardware
> available: 3 nodes (0-2)
> node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44
> node 0 size: 15940 MB
> node 0 free: 15142 MB
> node 1 cpus: 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59
> node 1 size: 0 MB
> node 1 free: 0 MB
> node 2 cpus:
> node 2 size: 0 MB
> node 2 free: 0 MB
> node distances:
> node   0   1   2
>    0:  10  21  21
>    1:  21  10  21
>    2:  21  21  10
>
> Jiang Liu (9):
>    x86, NUMA, ACPI: Online node earlier when doing CPU hot-addition
>    kernel/profile.c: Replace cpu_to_mem() with cpu_to_node()
>    sgi-xp: Replace cpu_to_node() with cpu_to_mem() to support memoryless
>      node
>    openvswitch: Replace cpu_to_node() with cpu_to_mem() to support
>      memoryless node
>    i40e: Use numa_mem_id() to better support memoryless node
>    i40evf: Use numa_mem_id() to better support memoryless node
>    x86, numa: Kill useless code to improve code readability
>    mm: Update _mem_id_[] for every possible CPU when memory
>      configuration changes
>    mm, x86: Enable memoryless node support to better support CPU/memory
>      hotplug
>
>   arch/x86/Kconfig                              |    3 ++
>   arch/x86/kernel/acpi/boot.c                   |    9 +++-
>   arch/x86/kernel/smpboot.c                     |    2 +
>   arch/x86/mm/numa.c                            |   59 +++++++++++++++----------
>   drivers/misc/sgi-xp/xpc_uv.c                  |    2 +-
>   drivers/net/ethernet/intel/i40e/i40e_txrx.c   |    2 +-
>   drivers/net/ethernet/intel/i40evf/i40e_txrx.c |    2 +-
>   kernel/profile.c                              |    2 +-
>   mm/page_alloc.c                               |   10 ++---
>   net/openvswitch/flow.c                        |    2 +-
>   10 files changed, 59 insertions(+), 34 deletions(-)
>

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1209692

FromJiang Liu <jiang.liu@linux.intel.com>
Date2015-08-19 10:10 +0200
Message-ID<pZ6Bz-8uo-1@gated-at.bofh.it>
In reply to#1209158
On 2015/8/18 18:02, Tang Chen wrote:
> 
> On 08/17/2015 11:18 AM, Jiang Liu wrote:
>> This is the third version to enable memoryless node support on x86
>> platforms. The previous version (https://lkml.org/lkml/2014/7/11/75)
>> blindly replaces numa_node_id()/cpu_to_node() with numa_mem_id()/
>> cpu_to_mem(). That's not the right solution as pointed out by Tejun
>> and Peter due to:
>> 1) We shouldn't shift the burden to normal slab users.
>> 2) Details of memoryless node should be hidden in arch and mm code
>>     as much as possible.
>>
>> After digging into more code and documentation, we found the rules to
>> deal with memoryless node should be:
>> 1) Arch code should online corresponding NUMA node before onlining any
>>     CPU or memory, otherwise it may cause invalid memory access when
>>     accessing NODE_DATA(nid).
>> 2) For normal memory allocations without __GFP_THISNODE setting in the
>>     gfp_flags, we should prefer numa_node_id()/cpu_to_node() instead of
>>     numa_mem_id()/cpu_to_mem() because the latter loses hardware topology
>>     information as pointed out by Tejun:
>>        A - B - X - C - D
>>     Where X is the memless node.  numa_mem_id() on X would return
>>     either B or C, right?  If B or C can't satisfy the allocation,
>>     the allocator would fallback to A from B and D for C, both of
>>     which aren't optimal. It should first fall back to C or B
>>     respectively, which the allocator can't do anymoe because the
>>     information is lost when the caller side performs numa_mem_id().
> 
> Hi Liu,
> 
> BTW, how is this A - B - X - C - D problem solved ?
> I don't quite follow this.
> 
> I cannot tell the difference between numa_node_id()/cpu_to_node() and
> numa_mem_id()/cpu_to_mem() on this point. Even with hardware topology
> info, how could it avoid this problem ?
> 
> Isn't it still possible falling back to A from B and D for C ?
Hi Chen,
For the imagined topology, A<->B<->X<->C<->D, where A, B, C, D has
memory and X is memoryless.
Possible fallback lists are:
B: [ B, A, C, D]
X: [ B, C, A, D]
C: [ C, D, B, A]

cpu_to_mem(X) will either return B or C. Let's assume it returns B.
Then we will use "B: [ B, A, C, D]" to allocate memory for X, which
is not the optimal fallback list for X. And cpu_to_node(X) returns
X, and "X: [ B, C, A, D]" is the optimal fallback list for X.
Thanks!
Gerry
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | linux.kernel

csiph-web