Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1470296 > unrolled thread

[PATCH 0/2] arm64: Use static keys for CPU features

Started byCatalin Marinas <catalin.marinas@arm.com>
First post2016-08-25 19:30 +0200
Last post2016-09-02 18:10 +0200
Articles 5 — 2 participants

Back to article view | Back to linux.kernel

Contents

  [PATCH 0/2] arm64: Use static keys for CPU features Catalin Marinas <catalin.marinas@arm.com> - 2016-08-25 19:30 +0200
    [PATCH 2/2] arm64: Use static keys for CPU features Catalin Marinas <catalin.marinas@arm.com> - 2016-08-25 19:30 +0200
      Re: [PATCH 2/2] arm64: Use static keys for CPU features Suzuki K Poulose <Suzuki.Poulose@arm.com> - 2016-08-26 11:30 +0200
        Re: [PATCH 2/2] arm64: Use static keys for CPU features Catalin Marinas <catalin.marinas@arm.com> - 2016-09-02 18:00 +0200
          Re: [PATCH 2/2] arm64: Use static keys for CPU features Suzuki K Poulose <Suzuki.Poulose@arm.com> - 2016-09-02 18:10 +0200

#1470296 — [PATCH 0/2] arm64: Use static keys for CPU features

FromCatalin Marinas <catalin.marinas@arm.com>
Date2016-08-25 19:30 +0200
Subject[PATCH 0/2] arm64: Use static keys for CPU features
Message-ID<sa6Dv-3XY-27@gated-at.bofh.it>
This series is aimed to optimise the arm64 cpus_have_cap() functionality
(checking for the presence of certain CPU capabilities/features) to
avoid a bitmap test and use a jump label instead, patched at boot time.

While this series may not provide a clear performance improvement with
the current kernel, it will be more beneficial with new features like
TTBR0 PAN are which are used on hot paths (get_user/put_user, thread
switching).

Catalin Marinas (2):
  jump_labels: Allow array initialisers
  arm64: Use static keys for CPU features

 Documentation/static-keys.txt       |  8 ++++++++
 arch/arm64/include/asm/cpufeature.h | 14 +++++++++++---
 arch/arm64/kernel/cpufeature.c      |  3 +++
 arch/arm64/kernel/cpuinfo.c         |  6 ++++++
 include/linux/jump_label.h          | 12 ++++++++++++
 5 files changed, 40 insertions(+), 3 deletions(-)

[toc] | [next] | [standalone]

#1470297 — [PATCH 2/2] arm64: Use static keys for CPU features

FromCatalin Marinas <catalin.marinas@arm.com>
Date2016-08-25 19:30 +0200
Subject[PATCH 2/2] arm64: Use static keys for CPU features
Message-ID<sa6Dv-3XY-25@gated-at.bofh.it>
In reply to#1470296
This patch adds static keys transparently for all the cpu_hwcaps
features by implementing an array of default-false static keys and
enabling them when detected. The cpus_have_cap() check uses the static
keys if the feature being checked is a constant, otherwise the compiler
generates the bitmap test.

Because of the early call to static_branch_enable() via
check_local_cpu_errata() -> update_cpu_capabilities(), the jump labels
are initialised in cpuinfo_store_boot_cpu().

Cc: Will Deacon <will.deacon@arm.com>
Cc: Suzuki K. Poulose <Suzuki.Poulose@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
---
 arch/arm64/include/asm/cpufeature.h | 14 +++++++++++---
 arch/arm64/kernel/cpufeature.c      |  3 +++
 arch/arm64/kernel/cpuinfo.c         |  6 ++++++
 3 files changed, 20 insertions(+), 3 deletions(-)

diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h
index 7099f26e3702..c9dfb1e4c435 100644
--- a/arch/arm64/include/asm/cpufeature.h
+++ b/arch/arm64/include/asm/cpufeature.h
@@ -9,6 +9,8 @@
 #ifndef __ASM_CPUFEATURE_H
 #define __ASM_CPUFEATURE_H
 
+#include <linux/jump_label.h>
+
 #include <asm/hwcap.h>
 #include <asm/sysreg.h>
 
@@ -109,6 +111,7 @@ struct arm64_cpu_capabilities {
 };
 
 extern DECLARE_BITMAP(cpu_hwcaps, ARM64_NCAPS);
+extern struct static_key_false cpu_hwcap_keys[ARM64_NCAPS];
 
 bool this_cpu_has_cap(unsigned int cap);
 
@@ -121,16 +124,21 @@ static inline bool cpus_have_cap(unsigned int num)
 {
 	if (num >= ARM64_NCAPS)
 		return false;
-	return test_bit(num, cpu_hwcaps);
+	if (__builtin_constant_p(num))
+		return static_branch_unlikely(&cpu_hwcap_keys[num]);
+	else
+		return test_bit(num, cpu_hwcaps);
 }
 
 static inline void cpus_set_cap(unsigned int num)
 {
-	if (num >= ARM64_NCAPS)
+	if (num >= ARM64_NCAPS) {
 		pr_warn("Attempt to set an illegal CPU capability (%d >= %d)\n",
 			num, ARM64_NCAPS);
-	else
+	} else {
 		__set_bit(num, cpu_hwcaps);
+		static_branch_enable(&cpu_hwcap_keys[num]);
+	}
 }
 
 static inline int __attribute_const__
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index 62272eac1352..919b2d0d68ae 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -46,6 +46,9 @@ unsigned int compat_elf_hwcap2 __read_mostly;
 
 DECLARE_BITMAP(cpu_hwcaps, ARM64_NCAPS);
 
+DEFINE_STATIC_KEY_ARRAY_FALSE(cpu_hwcap_keys, ARM64_NCAPS);
+EXPORT_SYMBOL(cpu_hwcap_keys);
+
 #define __ARM64_FTR_BITS(SIGNED, STRICT, TYPE, SHIFT, WIDTH, SAFE_VAL) \
 	{						\
 		.sign = SIGNED,				\
diff --git a/arch/arm64/kernel/cpuinfo.c b/arch/arm64/kernel/cpuinfo.c
index ed1b84fe6925..6a141e399daf 100644
--- a/arch/arm64/kernel/cpuinfo.c
+++ b/arch/arm64/kernel/cpuinfo.c
@@ -377,6 +377,12 @@ void cpuinfo_store_cpu(void)
 void __init cpuinfo_store_boot_cpu(void)
 {
 	struct cpuinfo_arm64 *info = &per_cpu(cpu_data, 0);
+
+	/*
+	 * Initialise the static keys early as they may be enabled by
+	 * check_local_cpu_errata() -> update_cpu_capabilities().
+	 */
+	jump_label_init();
 	__cpuinfo_store_cpu(info);
 
 	boot_cpu_data = *info;

[toc] | [prev] | [next] | [standalone]

#1470614 — Re: [PATCH 2/2] arm64: Use static keys for CPU features

FromSuzuki K Poulose <Suzuki.Poulose@arm.com>
Date2016-08-26 11:30 +0200
SubjectRe: [PATCH 2/2] arm64: Use static keys for CPU features
Message-ID<salCy-57G-29@gated-at.bofh.it>
In reply to#1470297
On 25/08/16 18:26, Catalin Marinas wrote:
> This patch adds static keys transparently for all the cpu_hwcaps
> features by implementing an array of default-false static keys and
> enabling them when detected. The cpus_have_cap() check uses the static
> keys if the feature being checked is a constant, otherwise the compiler
> generates the bitmap test.
>
> Because of the early call to static_branch_enable() via
> check_local_cpu_errata() -> update_cpu_capabilities(), the jump labels
> are initialised in cpuinfo_store_boot_cpu().
>
> Cc: Will Deacon <will.deacon@arm.com>
> Cc: Suzuki K. Poulose <Suzuki.Poulose@arm.com>
> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
> ---


>  static inline int __attribute_const__
> diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
> index 62272eac1352..919b2d0d68ae 100644
> --- a/arch/arm64/kernel/cpufeature.c
> +++ b/arch/arm64/kernel/cpufeature.c
> @@ -46,6 +46,9 @@ unsigned int compat_elf_hwcap2 __read_mostly;
>
>  DECLARE_BITMAP(cpu_hwcaps, ARM64_NCAPS);
>
> +DEFINE_STATIC_KEY_ARRAY_FALSE(cpu_hwcap_keys, ARM64_NCAPS);
> +EXPORT_SYMBOL(cpu_hwcap_keys);
> +
>  #define __ARM64_FTR_BITS(SIGNED, STRICT, TYPE, SHIFT, WIDTH, SAFE_VAL) \
>  	{						\
>  		.sign = SIGNED,				\
> diff --git a/arch/arm64/kernel/cpuinfo.c b/arch/arm64/kernel/cpuinfo.c
> index ed1b84fe6925..6a141e399daf 100644
> --- a/arch/arm64/kernel/cpuinfo.c
> +++ b/arch/arm64/kernel/cpuinfo.c
> @@ -377,6 +377,12 @@ void cpuinfo_store_cpu(void)
>  void __init cpuinfo_store_boot_cpu(void)
>  {
>  	struct cpuinfo_arm64 *info = &per_cpu(cpu_data, 0);
> +
> +	/*
> +	 * Initialise the static keys early as they may be enabled by
> +	 * check_local_cpu_errata() -> update_cpu_capabilities().
> +	 */
> +	jump_label_init();
>  	__cpuinfo_store_cpu(info);

Catalin,

Just a heads up. I have a patch [1] which moves the "check_local_cpu_errata()"
around to smp_prepare_boot_cpu(). This patch should still work fine with that
case. Only that may be we could move the jump_lable_init() to smp_prepare_boot_cpu(),
before we call "update_cpu_errata_work_arounds()" for Boot CPU.

Either way, this will be useful for some of the other feature checks.

Thanks
Suzuki

[1] https://lkml.kernel.org/r/1471525832-21209-4-git-send-email-suzuki.poulose@arm.com

[toc] | [prev] | [next] | [standalone]

#1475294 — Re: [PATCH 2/2] arm64: Use static keys for CPU features

FromCatalin Marinas <catalin.marinas@arm.com>
Date2016-09-02 18:00 +0200
SubjectRe: [PATCH 2/2] arm64: Use static keys for CPU features
Message-ID<scZ2P-1kr-57@gated-at.bofh.it>
In reply to#1470614
On Fri, Aug 26, 2016 at 10:22:13AM +0100, Suzuki K. Poulose wrote:
> On 25/08/16 18:26, Catalin Marinas wrote:
> > static inline int __attribute_const__
> >diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
> >index 62272eac1352..919b2d0d68ae 100644
> >--- a/arch/arm64/kernel/cpufeature.c
> >+++ b/arch/arm64/kernel/cpufeature.c
> >@@ -46,6 +46,9 @@ unsigned int compat_elf_hwcap2 __read_mostly;
> >
> > DECLARE_BITMAP(cpu_hwcaps, ARM64_NCAPS);
> >
> >+DEFINE_STATIC_KEY_ARRAY_FALSE(cpu_hwcap_keys, ARM64_NCAPS);
> >+EXPORT_SYMBOL(cpu_hwcap_keys);
> >+
> > #define __ARM64_FTR_BITS(SIGNED, STRICT, TYPE, SHIFT, WIDTH, SAFE_VAL) \
> > 	{						\
> > 		.sign = SIGNED,				\
> >diff --git a/arch/arm64/kernel/cpuinfo.c b/arch/arm64/kernel/cpuinfo.c
> >index ed1b84fe6925..6a141e399daf 100644
> >--- a/arch/arm64/kernel/cpuinfo.c
> >+++ b/arch/arm64/kernel/cpuinfo.c
> >@@ -377,6 +377,12 @@ void cpuinfo_store_cpu(void)
> > void __init cpuinfo_store_boot_cpu(void)
> > {
> > 	struct cpuinfo_arm64 *info = &per_cpu(cpu_data, 0);
> >+
> >+	/*
> >+	 * Initialise the static keys early as they may be enabled by
> >+	 * check_local_cpu_errata() -> update_cpu_capabilities().
> >+	 */
> >+	jump_label_init();
> > 	__cpuinfo_store_cpu(info);
> 
> Just a heads up. I have a patch [1] which moves the "check_local_cpu_errata()"
> around to smp_prepare_boot_cpu(). This patch should still work fine with that
> case. Only that may be we could move the jump_lable_init() to smp_prepare_boot_cpu(),
> before we call "update_cpu_errata_work_arounds()" for Boot CPU.

IIUC, we wouldn't call update_cpu_errata_work_arounds() until the CPU
feature infrastructure is initialised via cpuinfo_store_boot_cpu(). So
I don't think moving the jump_label_init() call above is necessary.

> [1] https://lkml.kernel.org/r/1471525832-21209-4-git-send-email-suzuki.poulose@arm.com

-- 
Catalin

[toc] | [prev] | [next] | [standalone]

#1475314 — Re: [PATCH 2/2] arm64: Use static keys for CPU features

FromSuzuki K Poulose <Suzuki.Poulose@arm.com>
Date2016-09-02 18:10 +0200
SubjectRe: [PATCH 2/2] arm64: Use static keys for CPU features
Message-ID<scZct-1CU-5@gated-at.bofh.it>
In reply to#1475294
On 02/09/16 16:52, Catalin Marinas wrote:
> On Fri, Aug 26, 2016 at 10:22:13AM +0100, Suzuki K. Poulose wrote:
>> On 25/08/16 18:26, Catalin Marinas wrote:

>>
>> Just a heads up. I have a patch [1] which moves the "check_local_cpu_errata()"
>> around to smp_prepare_boot_cpu(). This patch should still work fine with that
>> case. Only that may be we could move the jump_lable_init() to smp_prepare_boot_cpu(),
>> before we call "update_cpu_errata_work_arounds()" for Boot CPU.
>
> IIUC, we wouldn't call update_cpu_errata_work_arounds() until the CPU
> feature infrastructure is initialised via cpuinfo_store_boot_cpu(). So
> I don't think moving the jump_label_init() call above is necessary.

Right, as I said, your patch should work fine even with that change. Its just that,
jump_label_init() (a generic kernel setup) can be called from a better visible
place (smp_prepare_boot_cpu()) than from a less interesting place with the patch
below.

Cheers
Suzuki


>
>> [1] https://lkml.kernel.org/r/1471525832-21209-4-git-send-email-suzuki.poulose@arm.com
>

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web