Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1243481 > unrolled thread

Re: [PATCH 3/3] x86/apic: Use smaller array for __apicid_to_node[] mapping

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH 3/3] x86/apic: Use smaller array for __apicid_to_node[]  mapping Jiang Liu <jiang.liu@linux.intel.com> - 2015-10-09 17:40 +0200
    Re: [PATCH 3/3] x86/apic: Use smaller array for __apicid_to_node[]  mapping Thomas Gleixner <tglx@linutronix.de> - 2015-10-12 12:30 +0200
      Re: [PATCH 3/3] x86/apic: Use smaller array for __apicid_to_node[]  mapping Jiang Liu <jiang.liu@linux.intel.com> - 2015-10-13 11:40 +0200
        Re: [PATCH 3/3] x86/apic: Use smaller array for __apicid_to_node[]  mapping Thomas Gleixner <tglx@linutronix.de> - 2015-10-13 15:00 +0200
    Re: [PATCH 3/3] x86/apic: Use smaller array for __apicid_to_node[]  mapping Daniel J Blueman <daniel@numascale.com> - 2015-10-12 12:40 +0200

#1243481 — Re: [PATCH 3/3] x86/apic: Use smaller array for __apicid_to_node[] mapping

On 2015/10/3 3:12, Denys Vlasenko wrote:
> From: Daniel J Blueman <daniel@numascale.com>
> 
> The Intel x2APIC spec states the upper 16-bits of APIC ID is the
> cluster ID [1, p2-12], intended for future distributed systems. Beyond
> the legacy 8-bit APIC ID, Numascale NumaConnect uses 4-bits for the
> position of a server on each axis of a multi-dimension torus; SGI
> NUMAlink also structures the APIC ID space.
> 
> Instead, define an array based on NR_CPUs to achieve a 1:1 mapping and
> perform linear search; this addresses the binary bloat and the present
> artificial APIC ID limits. With CONFIG_NR_CPUS=256:
> 
> $ size vmlinux vmlinux-patched
>   text      data     bss      dec     hex filename
> 18232877 1849656 2281472 22364005 1553f65 vmlinux
> 18233034 1786168 2281472 22300674 1544802 vmlinux-patched
> 
> That is, ~64 kbytes less data.
> 
> Works peachy on a 256-core system with a 20-bit APIC ID space, and on a
> 48-core legacy 8-bit APIC ID system. If we care, I can make
> numa_cpu_node O(1) lookup for typical cases.
> 
> Signed-off-by: Daniel J Blueman <daniel@numascale.com>
> CC: Ingo Molnar <mingo@kernel.org>
> CC: Daniel J Blueman <daniel@numascale.com>
> CC: Jiang Liu <jiang.liu@linux.intel.com>
> CC: Thomas Gleixner <tglx@linutronix.de>
> CC: Len Brown <len.brown@intel.com>
> CC: x86@kernel.org
> CC: linux-kernel@vger.kernel.org
> 
> [1]
> http://www.intel.com/content/dam/doc/specification-update/64-architecture-x2apic-specification.pdf
> ---
> 
> I added forgotten change in arch/x86/mm/numa_emulation.c (Denys)
> 
>  arch/x86/include/asm/numa.h  | 13 +++++++------
>  arch/x86/kernel/cpu/amd.c    |  8 ++++----
>  arch/x86/mm/numa.c           | 31 +++++++++++++++++++++++--------
>  arch/x86/mm/numa_emulation.c |  6 +++---
>  4 files changed, 37 insertions(+), 21 deletions(-)
> 
> diff --git a/arch/x86/include/asm/numa.h b/arch/x86/include/asm/numa.h
> index c2ecfd0..33becb8 100644
> --- a/arch/x86/include/asm/numa.h
> +++ b/arch/x86/include/asm/numa.h
> @@ -17,6 +17,11 @@
>   */
>  #define NODE_MIN_SIZE (4*1024*1024)
>  
> +struct apicid_to_node {
> +	int apicid;
> +	s16 node;
> +};
> +
>  extern int numa_off;
>  
>  /*
> @@ -27,17 +32,13 @@ extern int numa_off;
>   * should be accessed by the accessors - set_apicid_to_node() and
>   * numa_cpu_node().
>   */
> -extern s16 __apicid_to_node[MAX_LOCAL_APICID];
> +extern struct apicid_to_node __apicid_to_node[NR_CPUS];
Hi Denys and Daniel,
	I still have some concerns about limiting the array to NR_CPUS.
__apicid_to_node are populated according to the order that CPUs are
listed in ACPI SRAT table. And CPU IDs are allocated according to the
order that CPUs are listed in ACPI MADT(APIC) order. So it may cause
trouble if:
1) system has more than NR_CPUS CPUs
2) CPUs are listed in different order in SRAT and MADT tables.

<snit>
> @@ -607,9 +625,6 @@ static int __init numa_init(int (*init_func)(void))
>  	int i;
>  	int ret;
>  
> -	for (i = 0; i < MAX_LOCAL_APICID; i++)
> -		set_apicid_to_node(i, NUMA_NO_NODE);
> -
	Why remove above code? numa_init() may be called multiple times
so it needs to reset __apicid_to_node array on the second and following
calls. So we need another way to reset __apicid_to_node array instead
of simply deleting above code.
Thanks,
Gerry

>  	nodes_clear(numa_nodes_parsed);
>  	nodes_clear(node_possible_map);
>  	nodes_clear(node_online_map);
> diff --git a/arch/x86/mm/numa_emulation.c b/arch/x86/mm/numa_emulation.c
> index a8f90ce..1a0e112 100644
> --- a/arch/x86/mm/numa_emulation.c
> +++ b/arch/x86/mm/numa_emulation.c
> @@ -399,12 +399,12 @@ void __init numa_emulation(struct numa_meminfo *numa_meminfo, int numa_dist_cnt)
>  	 * back to zero just in case.
>  	 */
>  	for (i = 0; i < ARRAY_SIZE(__apicid_to_node); i++) {
> -		if (__apicid_to_node[i] == NUMA_NO_NODE)
> +		if (__apicid_to_node[i].node == NUMA_NO_NODE)
>  			continue;
>  		for (j = 0; j < ARRAY_SIZE(emu_nid_to_phys); j++)
> -			if (__apicid_to_node[i] == emu_nid_to_phys[j])
> +			if (__apicid_to_node[i].node == emu_nid_to_phys[j])
>  				break;
> -		__apicid_to_node[i] = j < ARRAY_SIZE(emu_nid_to_phys) ? j : 0;
> +		__apicid_to_node[i].node = j < ARRAY_SIZE(emu_nid_to_phys) ? j : 0;
>  	}
>  
>  	/* make sure all emulated nodes are mapped to a physical node */
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1244572

On Mon, 12 Oct 2015, Daniel J Blueman wrote:
> Another approach which may be suitable without changing SRAT parsing to be
> after the memory allocator is up, is to exploit the associativity of the
> bottom APIC ID bits.

What's the problem with moving (SRAT/ACPI/whatever) APIC parsing after
the memory allocator is up and available?

Thanks,

	tglx
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1245509

On 2015/10/12 18:25, Thomas Gleixner wrote:
> On Mon, 12 Oct 2015, Daniel J Blueman wrote:
>> Another approach which may be suitable without changing SRAT parsing to be
>> after the memory allocator is up, is to exploit the associativity of the
>> bottom APIC ID bits.
> 
> What's the problem with moving (SRAT/ACPI/whatever) APIC parsing after
> the memory allocator is up and available?
Hi Thomas,
	The work flow is as below at boot:
1) figure out memory NUMA topology info by walking ACPI table or probing
   AMD northbirdge.
2) initialize memory allocation based on memory NUMA topology.

And to make code simple, it also scan CPU NUMA topology in step 1, so
we could avoid walking ACPI tables twice. On the other hand, there are
several subsystems having code pattern as follow before booting APs.
up:
for_each_possible_cpu(cpu)
	alloc_page_node(size, cpu_to_node(cpu))
So it's a little hard to find a suitable hook point to delay CPU NUMA
topology scanning after memory allocator is ready.
Thanks,
Gerry

> 
> Thanks,
> 
> 	tglx
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1245685

On Tue, 13 Oct 2015, Jiang Liu wrote:
> On 2015/10/12 18:25, Thomas Gleixner wrote:
> > On Mon, 12 Oct 2015, Daniel J Blueman wrote:
> >> Another approach which may be suitable without changing SRAT parsing to be
> >> after the memory allocator is up, is to exploit the associativity of the
> >> bottom APIC ID bits.
> > 
> > What's the problem with moving (SRAT/ACPI/whatever) APIC parsing after
> > the memory allocator is up and available?
> Hi Thomas,
> 	The work flow is as below at boot:
> 1) figure out memory NUMA topology info by walking ACPI table or probing
>    AMD northbirdge.
> 2) initialize memory allocation based on memory NUMA topology.
> 
> And to make code simple, it also scan CPU NUMA topology in step 1, so
> we could avoid walking ACPI tables twice. On the other hand, there are
> several subsystems having code pattern as follow before booting APs.
> up:
> for_each_possible_cpu(cpu)
> 	alloc_page_node(size, cpu_to_node(cpu))
> So it's a little hard to find a suitable hook point to delay CPU NUMA
> topology scanning after memory allocator is ready.

Not really. The memory allocator is available very early and long
before smp_prepare_cpus(). So we should try hard to move it after that
point, even if that means that we need to walk the tables twice.

Alternatively, use a bootmem allocation and convert it to a radix tree
when the allocator is up.

Thanks,

	tglx


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1244580

On Fri, Oct 9, 2015 at 11:35 PM, Jiang Liu <jiang.liu@linux.intel.com> 
wrote:
> On 2015/10/3 3:12, Denys Vlasenko wrote:
>>  From: Daniel J Blueman <daniel@numascale.com>
>> 
>>  The Intel x2APIC spec states the upper 16-bits of APIC ID is the
>>  cluster ID [1, p2-12], intended for future distributed systems. 
>> Beyond
>>  the legacy 8-bit APIC ID, Numascale NumaConnect uses 4-bits for the
>>  position of a server on each axis of a multi-dimension torus; SGI
>>  NUMAlink also structures the APIC ID space.
>> 
>>  Instead, define an array based on NR_CPUs to achieve a 1:1 mapping 
>> and
>>  perform linear search; this addresses the binary bloat and the 
>> present
>>  artificial APIC ID limits. With CONFIG_NR_CPUS=256:
>> 
>>  $ size vmlinux vmlinux-patched
>>    text      data     bss      dec     hex filename
>>  18232877 1849656 2281472 22364005 1553f65 vmlinux
>>  18233034 1786168 2281472 22300674 1544802 vmlinux-patched
>> 
>>  That is, ~64 kbytes less data.
>> 
>>  Works peachy on a 256-core system with a 20-bit APIC ID space, and 
>> on a
>>  48-core legacy 8-bit APIC ID system. If we care, I can make
>>  numa_cpu_node O(1) lookup for typical cases.
>> 
>>  Signed-off-by: Daniel J Blueman <daniel@numascale.com>
>>  CC: Ingo Molnar <mingo@kernel.org>
>>  CC: Daniel J Blueman <daniel@numascale.com>
>>  CC: Jiang Liu <jiang.liu@linux.intel.com>
>>  CC: Thomas Gleixner <tglx@linutronix.de>
>>  CC: Len Brown <len.brown@intel.com>
>>  CC: x86@kernel.org
>>  CC: linux-kernel@vger.kernel.org
>> 
>>  [1]
>>  
>> http://www.intel.com/content/dam/doc/specification-update/64-architecture-x2apic-specification.pdf
>>  ---
>> 
>>  I added forgotten change in arch/x86/mm/numa_emulation.c (Denys)
>> 
>>   arch/x86/include/asm/numa.h  | 13 +++++++------
>>   arch/x86/kernel/cpu/amd.c    |  8 ++++----
>>   arch/x86/mm/numa.c           | 31 +++++++++++++++++++++++--------
>>   arch/x86/mm/numa_emulation.c |  6 +++---
>>   4 files changed, 37 insertions(+), 21 deletions(-)
>> 
>>  diff --git a/arch/x86/include/asm/numa.h 
>> b/arch/x86/include/asm/numa.h
>>  index c2ecfd0..33becb8 100644
>>  --- a/arch/x86/include/asm/numa.h
>>  +++ b/arch/x86/include/asm/numa.h
>>  @@ -17,6 +17,11 @@
>>    */
>>   #define NODE_MIN_SIZE (4*1024*1024)
>> 
>>  +struct apicid_to_node {
>>  +	int apicid;
>>  +	s16 node;
>>  +};
>>  +
>>   extern int numa_off;
>> 
>>   /*
>>  @@ -27,17 +32,13 @@ extern int numa_off;
>>    * should be accessed by the accessors - set_apicid_to_node() and
>>    * numa_cpu_node().
>>    */
>>  -extern s16 __apicid_to_node[MAX_LOCAL_APICID];
>>  +extern struct apicid_to_node __apicid_to_node[NR_CPUS];
> Hi Denys and Daniel,
> 	I still have some concerns about limiting the array to NR_CPUS.
> __apicid_to_node are populated according to the order that CPUs are
> listed in ACPI SRAT table. And CPU IDs are allocated according to the
> order that CPUs are listed in ACPI MADT(APIC) order. So it may cause
> trouble if:
> 1) system has more than NR_CPUS CPUs
> 2) CPUs are listed in different order in SRAT and MADT tables.

Another approach which may be suitable without changing SRAT parsing to 
be after the memory allocator is up, is to exploit the associativity of 
the bottom APIC ID bits.

We'd have a searchable static array based on NUMA_SHIFT and use the 
bit-shift encoded in the MSRs. That said, this may run into the issue 
Jiang cited albeit with CONFIG_NUMA_SHIFT. Perhaps the constraints or 
risk of restructuring SRAT parsing aren't worth the payoff?

Finally, the only alternative is as the current mapping is initialised 
in numa_init, we can drop the static initialisation and move the 64KB 
to the BSS to avoid bloating the binary image, but this may not achieve 
the initial goal of runtime footprint reduction.

Daniel

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web