Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1658204 > unrolled thread

[PATCH v2] arch/sparc: support NR_CPUS = 4096

Back to article view | Back to linux.kernel

Contents

  [PATCH v2] arch/sparc: support NR_CPUS = 4096 Jane Chu <jane.chu@oracle.com> - 2017-06-06 01:00 +0200
    Re: [PATCH v2] arch/sparc: support NR_CPUS = 4096 David Miller <davem@davemloft.net> - 2017-06-06 03:00 +0200
      Re: [PATCH v2] arch/sparc: support NR_CPUS = 4096 jane.chu@oracle.com - 2017-06-06 05:10 +0200
        Re: [PATCH v2] arch/sparc: support NR_CPUS = 4096 David Miller <davem@davemloft.net> - 2017-06-06 05:30 +0200
          Re: [PATCH v2] arch/sparc: support NR_CPUS = 4096 jane.chu@oracle.com - 2017-06-06 19:10 +0200

#1658204 — [PATCH v2] arch/sparc: support NR_CPUS = 4096

Linux SPARC64 limits NR_CPUS to 4064 because init_cpu_send_mondo_info()
only allocates a single page for NR_CPUS mondo entries. Thus we cannot
use all 4096 CPUs on some SPARC platforms.

To fix, allocate (2^order) pages where order is set according to the size
of cpu_list for possible cpus. Since cpu_list_pa and cpu_mondo_block_pa
are not used in asm code, there are no imm13 offsets from the base PA
that will break because they can only reach one page.

Orabug: 25505750

Signed-off-by: Jane Chu <jane.chu@oracle.com>

Reviewed-by: Bob Picco <bob.picco@oracle.com>
Reviewed-by: Atish Patra <atish.patra@oracle.com>
---
 arch/sparc/Kconfig         |    4 ++--
 arch/sparc/kernel/irq_64.c |   16 ++++++++++++----
 2 files changed, 14 insertions(+), 6 deletions(-)

diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig
index 58243b0..4399be7 100644
--- a/arch/sparc/Kconfig
+++ b/arch/sparc/Kconfig
@@ -192,9 +192,9 @@ config NR_CPUS
 	int "Maximum number of CPUs"
 	depends on SMP
 	range 2 32 if SPARC32
-	range 2 1024 if SPARC64
+	range 2 4096 if SPARC64
 	default 32 if SPARC32
-	default 64 if SPARC64
+	default 4096 if SPARC64
 
 source kernel/Kconfig.hz
 
diff --git a/arch/sparc/kernel/irq_64.c b/arch/sparc/kernel/irq_64.c
index 4d0248a..cc8f6c3 100644
--- a/arch/sparc/kernel/irq_64.c
+++ b/arch/sparc/kernel/irq_64.c
@@ -1034,17 +1034,25 @@ static void __init init_cpu_send_mondo_info(struct trap_per_cpu *tb)
 {
 #ifdef CONFIG_SMP
 	unsigned long page;
+	void *mondo;
 
-	BUILD_BUG_ON((NR_CPUS * sizeof(u16)) > (PAGE_SIZE - 64));
+	BUILD_BUG_ON((NR_CPUS * sizeof(u16)) > PAGE_SIZE);
+
+	/* Make sure mondo block is 64byte aligned */
+	mondo = kzalloc(64, GFP_KERNEL);
+	if (!mondo) {
+		prom_printf("SUN4V: Error, cannot allocate mondo block.\n");
+		prom_halt();
+	}
+	tb->cpu_mondo_block_pa = __pa(mondo);
 
 	page = get_zeroed_page(GFP_KERNEL);
 	if (!page) {
-		prom_printf("SUN4V: Error, cannot allocate cpu mondo page.\n");
+		prom_printf("SUN4V: Error, cannot allocate cpu list page.\n");
 		prom_halt();
 	}
 
-	tb->cpu_mondo_block_pa = __pa(page);
-	tb->cpu_list_pa = __pa(page + 64);
+	tb->cpu_list_pa = __pa(page);
 #endif
 }
 
-- 
1.7.1

[toc] | [next] | [standalone]

#1658278

From: Jane Chu <jane.chu@oracle.com>
Date: Mon,  5 Jun 2017 16:48:31 -0600

> Linux SPARC64 limits NR_CPUS to 4064 because init_cpu_send_mondo_info()
> only allocates a single page for NR_CPUS mondo entries. Thus we cannot
> use all 4096 CPUs on some SPARC platforms.
> 
> To fix, allocate (2^order) pages where order is set according to the size
> of cpu_list for possible cpus. Since cpu_list_pa and cpu_mondo_block_pa
> are not used in asm code, there are no imm13 offsets from the base PA
> that will break because they can only reach one page.
> 
> Orabug: 25505750
> 
> Signed-off-by: Jane Chu <jane.chu@oracle.com>
> 
> Reviewed-by: Bob Picco <bob.picco@oracle.com>
> Reviewed-by: Atish Patra <atish.patra@oracle.com>
> ---
>  arch/sparc/Kconfig         |    4 ++--
>  arch/sparc/kernel/irq_64.c |   16 ++++++++++++----
>  2 files changed, 14 insertions(+), 6 deletions(-)
> 
> diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig
> index 58243b0..4399be7 100644
> --- a/arch/sparc/Kconfig
> +++ b/arch/sparc/Kconfig
> @@ -192,9 +192,9 @@ config NR_CPUS
>  	int "Maximum number of CPUs"
>  	depends on SMP
>  	range 2 32 if SPARC32
> -	range 2 1024 if SPARC64
> +	range 2 4096 if SPARC64
>  	default 32 if SPARC32
> -	default 64 if SPARC64
> +	default 4096 if SPARC64
>  
>  source kernel/Kconfig.hz
>  
> diff --git a/arch/sparc/kernel/irq_64.c b/arch/sparc/kernel/irq_64.c
> index 4d0248a..cc8f6c3 100644
> --- a/arch/sparc/kernel/irq_64.c
> +++ b/arch/sparc/kernel/irq_64.c
> @@ -1034,17 +1034,25 @@ static void __init init_cpu_send_mondo_info(struct trap_per_cpu *tb)
>  {
>  #ifdef CONFIG_SMP
>  	unsigned long page;
> +	void *mondo;
>  
> -	BUILD_BUG_ON((NR_CPUS * sizeof(u16)) > (PAGE_SIZE - 64));
> +	BUILD_BUG_ON((NR_CPUS * sizeof(u16)) > PAGE_SIZE);
> +
> +	/* Make sure mondo block is 64byte aligned */
> +	mondo = kzalloc(64, GFP_KERNEL);
> +	if (!mondo) {
> +		prom_printf("SUN4V: Error, cannot allocate mondo block.\n");
> +		prom_halt();
> +	}
> +	tb->cpu_mondo_block_pa = __pa(mondo);

Hmmm, you said that this has to be 64 byte aligned right?  We might have
to do something in order to insure that, as kmalloc() only guarantees
ARCH_KMALLOC_MINALIGN which I think is 8 on sparc.

I suppose this would work:

	mondo = kzalloc(64 + 63, GFP_KERNEL);

and then 64-byte align that pointer.

[toc] | [prev] | [next] | [standalone]

#1658327

On 06/05/2017 05:57 PM, David Miller wrote:

> From: Jane Chu <jane.chu@oracle.com>
> Date: Mon,  5 Jun 2017 16:48:31 -0600
>
>> Linux SPARC64 limits NR_CPUS to 4064 because init_cpu_send_mondo_info()
>> only allocates a single page for NR_CPUS mondo entries. Thus we cannot
>> use all 4096 CPUs on some SPARC platforms.
>>
>> To fix, allocate (2^order) pages where order is set according to the size
>> of cpu_list for possible cpus. Since cpu_list_pa and cpu_mondo_block_pa
>> are not used in asm code, there are no imm13 offsets from the base PA
>> that will break because they can only reach one page.
>>
>> Orabug: 25505750
>>
>> Signed-off-by: Jane Chu <jane.chu@oracle.com>
>>
>> Reviewed-by: Bob Picco <bob.picco@oracle.com>
>> Reviewed-by: Atish Patra <atish.patra@oracle.com>
>> ---
>>   arch/sparc/Kconfig         |    4 ++--
>>   arch/sparc/kernel/irq_64.c |   16 ++++++++++++----
>>   2 files changed, 14 insertions(+), 6 deletions(-)
>>
>> diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig
>> index 58243b0..4399be7 100644
>> --- a/arch/sparc/Kconfig
>> +++ b/arch/sparc/Kconfig
>> @@ -192,9 +192,9 @@ config NR_CPUS
>>   	int "Maximum number of CPUs"
>>   	depends on SMP
>>   	range 2 32 if SPARC32
>> -	range 2 1024 if SPARC64
>> +	range 2 4096 if SPARC64
>>   	default 32 if SPARC32
>> -	default 64 if SPARC64
>> +	default 4096 if SPARC64
>>   
>>   source kernel/Kconfig.hz
>>   
>> diff --git a/arch/sparc/kernel/irq_64.c b/arch/sparc/kernel/irq_64.c
>> index 4d0248a..cc8f6c3 100644
>> --- a/arch/sparc/kernel/irq_64.c
>> +++ b/arch/sparc/kernel/irq_64.c
>> @@ -1034,17 +1034,25 @@ static void __init init_cpu_send_mondo_info(struct trap_per_cpu *tb)
>>   {
>>   #ifdef CONFIG_SMP
>>   	unsigned long page;
>> +	void *mondo;
>>   
>> -	BUILD_BUG_ON((NR_CPUS * sizeof(u16)) > (PAGE_SIZE - 64));
>> +	BUILD_BUG_ON((NR_CPUS * sizeof(u16)) > PAGE_SIZE);
>> +
>> +	/* Make sure mondo block is 64byte aligned */
>> +	mondo = kzalloc(64, GFP_KERNEL);
>> +	if (!mondo) {
>> +		prom_printf("SUN4V: Error, cannot allocate mondo block.\n");
>> +		prom_halt();
>> +	}
>> +	tb->cpu_mondo_block_pa = __pa(mondo);
> Hmmm, you said that this has to be 64 byte aligned right?  We might have
> to do something in order to insure that, as kmalloc() only guarantees
> ARCH_KMALLOC_MINALIGN which I think is 8 on sparc.
Yes.
>
> I suppose this would work:
>
> 	mondo = kzalloc(64 + 63, GFP_KERNEL);
>
> and then 64-byte align that pointer.

On sun4v sparc,  it looks like kzalloc(64, GFP_KERNEL)  ends up allocating
from kmalloc_caches[6] - a 64-byte kmem-cache allocated by
kmem_cache_init() with SLAB_HWCACHE_ALIGN flag set, so it's in 
l3-cache-line-size
alignment, i.e. 64byte.  I printed out the 'mondo' pa and verified that.

thanks!
-jane

[toc] | [prev] | [next] | [standalone]

#1658331

From: jane.chu@oracle.com
Date: Mon, 5 Jun 2017 20:03:28 -0700

> On sun4v sparc, it looks like kzalloc(64, GFP_KERNEL) ends up
> allocating from kmalloc_caches[6] - a 64-byte kmem-cache allocated
> by kmem_cache_init() with SLAB_HWCACHE_ALIGN flag set, so it's in
> l3-cache-line-size alignment, i.e. 64byte.  I printed out the
> 'mondo' pa and verified that.

Please check this, because that is an implementation detail.  We have
three SLAB allocators.

For example, if the user enables SLOB, I bet you don't get that
alignment.

The only thing we are universally guaranteed is ARCH_KMALLOC_MINALIGN.

Please implement this properly, rather than relying upon things like
the above.

Thank you.

[toc] | [prev] | [next] | [standalone]

#1658951

On 06/05/2017 08:27 PM, David Miller wrote:

> From: jane.chu@oracle.com
> Date: Mon, 5 Jun 2017 20:03:28 -0700
>
>> On sun4v sparc, it looks like kzalloc(64, GFP_KERNEL) ends up
>> allocating from kmalloc_caches[6] - a 64-byte kmem-cache allocated
>> by kmem_cache_init() with SLAB_HWCACHE_ALIGN flag set, so it's in
>> l3-cache-line-size alignment, i.e. 64byte.  I printed out the
>> 'mondo' pa and verified that.
> Please check this, because that is an implementation detail.  We have
> three SLAB allocators.
>
> For example, if the user enables SLOB, I bet you don't get that
> alignment.
>
> The only thing we are universally guaranteed is ARCH_KMALLOC_MINALIGN.
>
> Please implement this properly, rather than relying upon things like
> the above.

Good point, agreed.  I will update the patch.

thanks!
-jane

>
> Thank you.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web