Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1233064 > unrolled thread

[PATCH RFC] x86: Reduce MAX_LOCAL_APIC and MAX_IO_APICS

Started byDenys Vlasenko <dvlasenk@redhat.com>
First post2015-09-25 22:40 +0200
Last post2015-09-30 19:20 +0200
Articles 6 — 3 participants

Back to article view | Back to linux.kernel

Contents

  [PATCH RFC] x86: Reduce MAX_LOCAL_APIC and MAX_IO_APICS Denys Vlasenko <dvlasenk@redhat.com> - 2015-09-25 22:40 +0200
    Re: [PATCH RFC] x86: Reduce MAX_LOCAL_APIC and MAX_IO_APICS Thomas Gleixner <tglx@linutronix.de> - 2015-09-30 17:20 +0200
      Re: [PATCH RFC] x86: Reduce MAX_LOCAL_APIC and MAX_IO_APICS Denys Vlasenko <dvlasenk@redhat.com> - 2015-09-30 18:00 +0200
        Re: [PATCH RFC] x86: Reduce MAX_LOCAL_APIC and MAX_IO_APICS Jiang Liu <jiang.liu@linux.intel.com> - 2015-09-30 19:10 +0200
        Re: [PATCH RFC] x86: Reduce MAX_LOCAL_APIC and MAX_IO_APICS Thomas Gleixner <tglx@linutronix.de> - 2015-09-30 19:50 +0200
    Re: [PATCH RFC] x86: Reduce MAX_LOCAL_APIC and MAX_IO_APICS Jiang Liu <jiang.liu@linux.intel.com> - 2015-09-30 19:20 +0200

#1233064 — [PATCH RFC] x86: Reduce MAX_LOCAL_APIC and MAX_IO_APICS

FromDenys Vlasenko <dvlasenk@redhat.com>
Date2015-09-25 22:40 +0200
Subject[PATCH RFC] x86: Reduce MAX_LOCAL_APIC and MAX_IO_APICS
Message-ID<qcHWG-3PA-5@gated-at.bofh.it>
Before this change MAX_LOCAL_APIC had the fixed value of 32*1024.
Such a big value causes several data arrays to be quite oversized:

phys_cpu_present_map is 4 kbytes (one bit per apic id),
__apicid_to_node[] is 64 kbytes,
apic_version[] is 128 kbytes.

On "usual" systems, APIC ids simply go from zero
to maximum logical CPU number, mirroring CPU ids.

On broken and unusual multi-socket systems
APIC ids can be non-contiguous.

This patch changes MAX_LOCAL_APIC definition as follows:

 = It is guaranteed to be at least 16.
 = If NR_CPUS > 16, then it's equal to NR_CPUS.
 = A new CONFIG_MAX_LAPIC_ID can be used to increase it
   (but not decrease).

MAX_IO_APICS was 128. This is a bit large too, making,
for example, ioapics[] array 9216 bytes big.

After this patch, MAX_IO_APICS is at least 8, at most 128.
If NR_CPUS is in this range, then MAX_IO_APICS = NR_CPUS.

apic_version[] array is changed from int to u8 -
APIC version values as of year 2015 are no larger than 0x1f
on all known CPUs.

A bit of code added to ensure that the statement
	apic_version[apicid] = version;
in generic_processor_info() is safe wrt bad values in both
'apicid' and 'version' variables.

This change reduces NR_CPUS=64 kernel's data size by 204661 bytes:

    text     data      bss       dec     hex filename
91353669 13825744 19021824 124201237 7672915 vmlinux.before
91353680 13760336 18882560 123996576 76409a0 vmlinux

Signed-off-by: Denys Vlasenko <dvlasenk@redhat.com>
CC: Ingo Molnar <mingo@kernel.org>
CC: Jiang Liu <jiang.liu@linux.intel.com>
CC: Thomas Gleixner <tglx@linutronix.de>
CC: Len Brown <len.brown@intel.com>
CC: x86@kernel.org
CC: linux-kernel@vger.kernel.org
---
 arch/x86/Kconfig               | 11 +++++++++++
 arch/x86/include/asm/apicdef.h | 23 +++++++++++++++++------
 arch/x86/include/asm/mpspec.h  |  2 +-
 arch/x86/kernel/apic/apic.c    | 19 ++++++++++++++++++-
 4 files changed, 47 insertions(+), 8 deletions(-)

diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 328c835..9e7c4c1 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -872,6 +872,17 @@ config NR_CPUS
 	  This is purely to save memory - each supported CPU adds
 	  approximately eight kilobytes to the kernel image.
 
+config MAX_LAPIC_ID
+	int "Maximum APIC ID"
+	range 8 32768
+	default "8"
+	---help---
+	  Use this option to set maximum allowed Local APIC ID higher than
+	  maximum number of CPUs. This may be necessary for machines
+	  with large number of processor sockets and non-contiguous
+	  LAPIC numbering.
+	  This setting will be automatically rounded up, if necessary.
+
 config SCHED_SMT
 	bool "SMT (Hyperthreading) scheduler support"
 	depends on SMP
diff --git a/arch/x86/include/asm/apicdef.h b/arch/x86/include/asm/apicdef.h
index c46bb99..64e2476 100644
--- a/arch/x86/include/asm/apicdef.h
+++ b/arch/x86/include/asm/apicdef.h
@@ -147,15 +147,26 @@
 #define XAPIC_ENABLE	(1UL << 11)
 #define X2APIC_ENABLE	(1UL << 10)
 
-#ifdef CONFIG_X86_32
-# define MAX_IO_APICS 64
-# define MAX_LOCAL_APIC 256
-#else
-# define MAX_IO_APICS 128
-# define MAX_LOCAL_APIC 32768
+/*
+ * Allow non-contiguous APIC IDs for small machines:
+ * APIC ids 0..15 are valid in any config.
+ * Typical SMP machines have contiguous APIC IDs: 0..NR_CPUS-1.
+ * CONFIG_MAX_LAPIC_ID can override.
+ */
+#define MAX_LOCAL_APIC (NR_CPUS < 16 ? 16 : NR_CPUS)
+#if MAX_LOCAL_APIC < CONFIG_MAX_LAPIC_ID
+# undef  MAX_LOCAL_APIC
+# define MAX_LOCAL_APIC CONFIG_MAX_LAPIC_ID
 #endif
 
 /*
+ * Minimum is 8.
+ * For largish NR_CPUS, we expect to have no more IOAPICs than CPUs.
+ * No matter how large NR_CPUS is, max is 128.
+ */
+#define MAX_IO_APICS (NR_CPUS < 8 ? 8 : NR_CPUS < 128 ? NR_CPUS : 128)
+
+/*
  * All x86-64 systems are xAPIC compatible.
  * In the following, "apicid" is a physical APIC ID.
  */
diff --git a/arch/x86/include/asm/mpspec.h b/arch/x86/include/asm/mpspec.h
index b07233b..8d0c2e6 100644
--- a/arch/x86/include/asm/mpspec.h
+++ b/arch/x86/include/asm/mpspec.h
@@ -6,7 +6,7 @@
 #include <asm/x86_init.h>
 #include <asm/apicdef.h>
 
-extern int apic_version[];
+extern u8 apic_version[];
 extern int pic_mode;
 
 #ifdef CONFIG_X86_32
diff --git a/arch/x86/kernel/apic/apic.c b/arch/x86/kernel/apic/apic.c
index 24e94ce..f49a956 100644
--- a/arch/x86/kernel/apic/apic.c
+++ b/arch/x86/kernel/apic/apic.c
@@ -1798,7 +1798,7 @@ void __init register_lapic_address(unsigned long address)
 	}
 }
 
-int apic_version[MAX_LOCAL_APIC];
+u8 apic_version[MAX_LOCAL_APIC];
 
 /*
  * Local APIC interrupts
@@ -2054,6 +2054,23 @@ int generic_processor_info(int apicid, int version)
 		return -EINVAL;
 	}
 
+	if ((unsigned)apicid >= ARRAY_SIZE(apic_version)) {
+		int thiscpu = max + disabled_cpus;
+		pr_warning("APIC: APIC id 0x%x is too large."
+			   " Processor %d ignored.\n",
+			   apicid, thiscpu);
+		disabled_cpus++;
+		return -EINVAL;
+	}
+	if ((unsigned)version > 255) {
+		int thiscpu = max + disabled_cpus;
+		pr_warning("APIC: APIC version 0x%x is too large."
+			   " Processor %d ignored.\n",
+			   version, thiscpu);
+		disabled_cpus++;
+		return -EINVAL;
+	}
+
 	num_processors++;
 	if (apicid == boot_cpu_physical_apicid) {
 		/*
-- 
1.8.1.4

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1236433

FromThomas Gleixner <tglx@linutronix.de>
Date2015-09-30 17:20 +0200
Message-ID<qerkJ-kW-13@gated-at.bofh.it>
In reply to#1233064
On Fri, 25 Sep 2015, Denys Vlasenko wrote:
>  
> +config MAX_LAPIC_ID
> +	int "Maximum APIC ID"
> +	range 8 32768
> +	default "8"
> +	---help---
> +	  Use this option to set maximum allowed Local APIC ID higher than
> +	  maximum number of CPUs. This may be necessary for machines
> +	  with large number of processor sockets and non-contiguous
> +	  LAPIC numbering.
> +	  This setting will be automatically rounded up, if necessary.

This is wrong. If you would limit the APIC IDs then you really break
stuff. You can only limit the number of APICs.

ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x02] enabled)

And that's not a really large machine..

Thanks,

	tglx
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1236471

FromDenys Vlasenko <dvlasenk@redhat.com>
Date2015-09-30 18:00 +0200
Message-ID<qerXt-14n-41@gated-at.bofh.it>
In reply to#1236433
On 09/30/2015 05:11 PM, Thomas Gleixner wrote:
> On Fri, 25 Sep 2015, Denys Vlasenko wrote:
>>  
>> +config MAX_LAPIC_ID
>> +	int "Maximum APIC ID"
>> +	range 8 32768
>> +	default "8"
>> +	---help---
>> +	  Use this option to set maximum allowed Local APIC ID higher than
>> +	  maximum number of CPUs. This may be necessary for machines
>> +	  with large number of processor sockets and non-contiguous
>> +	  LAPIC numbering.
>> +	  This setting will be automatically rounded up, if necessary.
> 
> This is wrong. If you would limit the APIC IDs then you really break
> stuff. You can only limit the number of APICs.

This CONFIG setting allows to _increase_ max APIC ID.

Check out this part of the patch:

+/*
+ * Allow non-contiguous APIC IDs for small machines:
+ * APIC ids 0..15 are valid in any config.
+ * Typical SMP machines have contiguous APIC IDs: 0..NR_CPUS-1.
+ * CONFIG_MAX_LAPIC_ID can override.
+ */
+#define MAX_LOCAL_APIC (NR_CPUS < 16 ? 16 : NR_CPUS)
+#if MAX_LOCAL_APIC < CONFIG_MAX_LAPIC_ID
+# undef  MAX_LOCAL_APIC
+# define MAX_LOCAL_APIC CONFIG_MAX_LAPIC_ID
 #endif


For example, if you'd build with NR_CPUS=128
(for example, Fedora kernels do that),
max accepted APIC id will be NR_CPUS-1 = 127
even if CONFIX_MAX_LAPIC_ID is 8.

If Fedora would want to support APIC ids up to
255, it will need to set CONFIG_MAX_LAPIC_ID=256.

Otherwise, if it's happy with "only" supporting up to 128,
it does not need to change CONFIG_MAX_LAPIC_ID from default.

With current kernels, max APIC id for any kernel is 32768,
which is in most cases way bigger than necessary.


Perhaps I need to update the text.
Something like:

- This setting will be automatically rounded up, if necessary
+ This setting will be increased to NR_CPUS, if necessary


> ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
> ACPI: LAPIC (acpi_id[0x01] lapic_id[0x02] enabled)

Does it mean that on a 2-CPU machine, CPU #1 has APIC_ID=2?

My patch will work fine for this machine,
with any CONFIG_MAX_LAPIC_ID.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1236549

FromJiang Liu <jiang.liu@linux.intel.com>
Date2015-09-30 19:10 +0200
Message-ID<qet3c-2QO-29@gated-at.bofh.it>
In reply to#1236471
On 2015/9/30 23:49, Denys Vlasenko wrote:
> On 09/30/2015 05:11 PM, Thomas Gleixner wrote:
>> On Fri, 25 Sep 2015, Denys Vlasenko wrote:
>>>  
>>> +config MAX_LAPIC_ID
>>> +	int "Maximum APIC ID"
>>> +	range 8 32768
>>> +	default "8"
>>> +	---help---
>>> +	  Use this option to set maximum allowed Local APIC ID higher than
>>> +	  maximum number of CPUs. This may be necessary for machines
>>> +	  with large number of processor sockets and non-contiguous
>>> +	  LAPIC numbering.
>>> +	  This setting will be automatically rounded up, if necessary.
>>
>> This is wrong. If you would limit the APIC IDs then you really break
>> stuff. You can only limit the number of APICs.
> 
> This CONFIG setting allows to _increase_ max APIC ID.
> 
> Check out this part of the patch:
> 
> +/*
> + * Allow non-contiguous APIC IDs for small machines:
> + * APIC ids 0..15 are valid in any config.
> + * Typical SMP machines have contiguous APIC IDs: 0..NR_CPUS-1.
> + * CONFIG_MAX_LAPIC_ID can override.
> + */
> +#define MAX_LOCAL_APIC (NR_CPUS < 16 ? 16 : NR_CPUS)
> +#if MAX_LOCAL_APIC < CONFIG_MAX_LAPIC_ID
> +# undef  MAX_LOCAL_APIC
> +# define MAX_LOCAL_APIC CONFIG_MAX_LAPIC_ID
>  #endif
> 
> 
> For example, if you'd build with NR_CPUS=128
> (for example, Fedora kernels do that),
> max accepted APIC id will be NR_CPUS-1 = 127
> even if CONFIX_MAX_LAPIC_ID is 8.
> 
> If Fedora would want to support APIC ids up to
> 255, it will need to set CONFIG_MAX_LAPIC_ID=256.
> 
> Otherwise, if it's happy with "only" supporting up to 128,
> it does not need to change CONFIG_MAX_LAPIC_ID from default.
> 
> With current kernels, max APIC id for any kernel is 32768,
> which is in most cases way bigger than necessary.
> 
> 
> Perhaps I need to update the text.
> Something like:
> 
> - This setting will be automatically rounded up, if necessary
> + This setting will be increased to NR_CPUS, if necessary
> 
> 
>> ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
>> ACPI: LAPIC (acpi_id[0x01] lapic_id[0x02] enabled)
> 
> Does it mean that on a 2-CPU machine, CPU #1 has APIC_ID=2?
Yes, APIC IDs are assigned by BIOS and may not be continuous.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1236586

FromThomas Gleixner <tglx@linutronix.de>
Date2015-09-30 19:50 +0200
Message-ID<qetFU-3Ao-1@gated-at.bofh.it>
In reply to#1236471
On Wed, 30 Sep 2015, Denys Vlasenko wrote:
> On 09/30/2015 05:11 PM, Thomas Gleixner wrote:
> > On Fri, 25 Sep 2015, Denys Vlasenko wrote:
> >>  
> >> +config MAX_LAPIC_ID
> >> +	int "Maximum APIC ID"
> >> +	range 8 32768
> >> +	default "8"
> >> +	---help---
> >> +	  Use this option to set maximum allowed Local APIC ID higher than
> >> +	  maximum number of CPUs. This may be necessary for machines
> >> +	  with large number of processor sockets and non-contiguous
> >> +	  LAPIC numbering.
> >> +	  This setting will be automatically rounded up, if necessary.
> > 
> > This is wrong. If you would limit the APIC IDs then you really break
> > stuff. You can only limit the number of APICs.
> 
> This CONFIG setting allows to _increase_ max APIC ID.

NO. This is crap. I don't want to tweak a gazillion of knobs just to
build a kernel with CONFIG_NR_CPUS=8. Really not.

If you really want to make that space saving, then make it a runtime
allocation.
 
Thanks,

	tglx
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1236553

FromJiang Liu <jiang.liu@linux.intel.com>
Date2015-09-30 19:20 +0200
Message-ID<qetcR-327-1@gated-at.bofh.it>
In reply to#1233064
On 2015/9/26 3:48, Denys Vlasenko wrote:
> Before this change MAX_LOCAL_APIC had the fixed value of 32*1024.
> Such a big value causes several data arrays to be quite oversized:
> 
> phys_cpu_present_map is 4 kbytes (one bit per apic id),
> __apicid_to_node[] is 64 kbytes,
> apic_version[] is 128 kbytes.
> 
> On "usual" systems, APIC ids simply go from zero
> to maximum logical CPU number, mirroring CPU ids.
> 
> On broken and unusual multi-socket systems
> APIC ids can be non-contiguous.
> 
> This patch changes MAX_LOCAL_APIC definition as follows:
> 
>  = It is guaranteed to be at least 16.
>  = If NR_CPUS > 16, then it's equal to NR_CPUS.
>  = A new CONFIG_MAX_LAPIC_ID can be used to increase it
>    (but not decrease).
> 
> MAX_IO_APICS was 128. This is a bit large too, making,
> for example, ioapics[] array 9216 bytes big.
> 
> After this patch, MAX_IO_APICS is at least 8, at most 128.
> If NR_CPUS is in this range, then MAX_IO_APICS = NR_CPUS.
> 
> apic_version[] array is changed from int to u8 -
> APIC version values as of year 2015 are no larger than 0x1f
> on all known CPUs.
> 
> A bit of code added to ensure that the statement
> 	apic_version[apicid] = version;
> in generic_processor_info() is safe wrt bad values in both
> 'apicid' and 'version' variables.
> 
> This change reduces NR_CPUS=64 kernel's data size by 204661 bytes:
> 
>     text     data      bss       dec     hex filename
> 91353669 13825744 19021824 124201237 7672915 vmlinux.before
> 91353680 13760336 18882560 123996576 76409a0 vmlinux
> 
> Signed-off-by: Denys Vlasenko <dvlasenk@redhat.com>
> CC: Ingo Molnar <mingo@kernel.org>
> CC: Jiang Liu <jiang.liu@linux.intel.com>
> CC: Thomas Gleixner <tglx@linutronix.de>
> CC: Len Brown <len.brown@intel.com>
> CC: x86@kernel.org
> CC: linux-kernel@vger.kernel.org
> ---
>  arch/x86/Kconfig               | 11 +++++++++++
>  arch/x86/include/asm/apicdef.h | 23 +++++++++++++++++------
>  arch/x86/include/asm/mpspec.h  |  2 +-
>  arch/x86/kernel/apic/apic.c    | 19 ++++++++++++++++++-
>  4 files changed, 47 insertions(+), 8 deletions(-)
> 
> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
> index 328c835..9e7c4c1 100644
> --- a/arch/x86/Kconfig
> +++ b/arch/x86/Kconfig
> @@ -872,6 +872,17 @@ config NR_CPUS
>  	  This is purely to save memory - each supported CPU adds
>  	  approximately eight kilobytes to the kernel image.
>  
> +config MAX_LAPIC_ID
> +	int "Maximum APIC ID"
> +	range 8 32768
> +	default "8"
> +	---help---
> +	  Use this option to set maximum allowed Local APIC ID higher than
> +	  maximum number of CPUs. This may be necessary for machines
> +	  with large number of processor sockets and non-contiguous
> +	  LAPIC numbering.
> +	  This setting will be automatically rounded up, if necessary.
> +
>  config SCHED_SMT
>  	bool "SMT (Hyperthreading) scheduler support"
>  	depends on SMP
> diff --git a/arch/x86/include/asm/apicdef.h b/arch/x86/include/asm/apicdef.h
> index c46bb99..64e2476 100644
> --- a/arch/x86/include/asm/apicdef.h
> +++ b/arch/x86/include/asm/apicdef.h
> @@ -147,15 +147,26 @@
>  #define XAPIC_ENABLE	(1UL << 11)
>  #define X2APIC_ENABLE	(1UL << 10)
>  
> -#ifdef CONFIG_X86_32
> -# define MAX_IO_APICS 64
> -# define MAX_LOCAL_APIC 256
> -#else
> -# define MAX_IO_APICS 128
> -# define MAX_LOCAL_APIC 32768
> +/*
> + * Allow non-contiguous APIC IDs for small machines:
> + * APIC ids 0..15 are valid in any config.
> + * Typical SMP machines have contiguous APIC IDs: 0..NR_CPUS-1.
> + * CONFIG_MAX_LAPIC_ID can override.
> + */
> +#define MAX_LOCAL_APIC (NR_CPUS < 16 ? 16 : NR_CPUS)
> +#if MAX_LOCAL_APIC < CONFIG_MAX_LAPIC_ID
> +# undef  MAX_LOCAL_APIC
> +# define MAX_LOCAL_APIC CONFIG_MAX_LAPIC_ID
>  #endif
>  
>  /*
> + * Minimum is 8.
> + * For largish NR_CPUS, we expect to have no more IOAPICs than CPUs.
> + * No matter how large NR_CPUS is, max is 128.
> + */
> +#define MAX_IO_APICS (NR_CPUS < 8 ? 8 : NR_CPUS < 128 ? NR_CPUS : 128)
This is a little risky. For example, a typical eight-socket Intel
platform will have nine IOAPICs. IO devices may get inaccessible
if some IOAPICs are ignored due to MAX_IO_APICS limitation. It's
a surprising if IO devices get lost if user runs a kernel built with
low NR_CPUS.
Thanks!
Gerry
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web