Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1325771 > unrolled thread

[PATCH v3 00/23] arm64: Virtualization Host Extension support

Back to article view | Back to linux.kernel

Contents

  [PATCH v3 00/23] arm64: Virtualization Host Extension support Marc Zyngier <marc.zyngier@arm.com> - 2016-02-03 19:10 +0100
    [PATCH v3 14/23] arm64: KVM: VHE: Make __fpsimd_enabled VHE aware Marc Zyngier <marc.zyngier@arm.com> - 2016-02-03 19:10 +0100
    [PATCH v3 22/23] arm64: VHE: Add support for running Linux in EL2 mode Marc Zyngier <marc.zyngier@arm.com> - 2016-02-03 19:10 +0100
      Re: [PATCH v3 22/23] arm64: VHE: Add support for running Linux in  EL2 mode Christoffer Dall <christoffer.dall@linaro.org> - 2016-02-04 20:30 +0100
      Re: [PATCH v3 22/23] arm64: VHE: Add support for running Linux in  EL2 mode Catalin Marinas <catalin.marinas@arm.com> - 2016-02-08 17:00 +0100
    [PATCH v3 19/23] arm64: KVM: Move most of the fault decoding to C Marc Zyngier <marc.zyngier@arm.com> - 2016-02-03 19:10 +0100
      Re: [PATCH v3 19/23] arm64: KVM: Move most of the fault decoding to C Christoffer Dall <christoffer.dall@linaro.org> - 2016-02-04 20:30 +0100
      Re: [PATCH v3 19/23] arm64: KVM: Move most of the fault decoding to C Catalin Marinas <catalin.marinas@arm.com> - 2016-02-08 15:50 +0100
    [PATCH v3 13/23] arm64: KVM: VHE: Enable minimal sysreg save/restore Marc Zyngier <marc.zyngier@arm.com> - 2016-02-03 19:10 +0100
    [PATCH v3 16/23] arm64: KVM: VHE: Use unified sysreg accessors for timer Marc Zyngier <marc.zyngier@arm.com> - 2016-02-03 19:10 +0100
    [PATCH v3 01/23] arm/arm64: KVM: Add hook for C-based stage2 init Marc Zyngier <marc.zyngier@arm.com> - 2016-02-03 19:20 +0100
    Re: [PATCH v3 00/23] arm64: Virtualization Host Extension support Christoffer Dall <christoffer.dall@linaro.org> - 2016-02-04 20:30 +0100
      Re: [PATCH v3 00/23] arm64: Virtualization Host Extension support Marc Zyngier <marc.zyngier@arm.com> - 2016-02-05 10:00 +0100

#1325771 — [PATCH v3 00/23] arm64: Virtualization Host Extension support

ARMv8.1 comes with the "Virtualization Host Extension" (VHE for
short), which enables simpler support of Type-2 hypervisors.

This extension allows the kernel to directly run at EL2, and
significantly reduces the number of system registers shared between
host and guest, reducing the overhead of virtualization.

In order to have the same kernel binary running on all versions of the
architecture, this series makes heavy use of runtime code patching.

The first 22 patches massage the KVM code to deal with VHE and enable
Linux to run at EL2. The last patch catches an ugly case when VHE
capable CPUs are paired with some of their less capable siblings. This
should never happen, but hey...

I have deliberately left out some of the more "advanced"
optimizations, as they are likely to distract the reviewer from the
core infrastructure, which is what I care about at the moment.

Note: GDB is currently busted on VHE systems, as it checks for version
      6 on the debug architecture, while VHE is version 7. The
      binutils people are on the case.

This has been tested on the FVP_Base_SLV-V8-A model, and based on
v4.5-rc2. I've put a branch out on:

git://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git kvm-arm64/vhe

* From v2:
  - Added support for perf to count kernel events in EL2
  - Added support for EL2 breakpoints
  - Moved the VTCR_EL2 setup from assembly to C
  - Made the fault handling easier to understand (hopefuly)
  - Plenty of smaller fixups

* From v1:
  - Full rewrite now that the World Switch is written in C code.
  - Dropped the "early IRQ handling" for the moment.

Marc Zyngier (23):
  arm/arm64: KVM: Add hook for C-based stage2 init
  arm64: KVM: Switch to C-based stage2 init
  arm/arm64: Add new is_kernel_in_hyp_mode predicate
  arm64: Allow the arch timer to use the HYP timer
  arm64: Add ARM64_HAS_VIRT_HOST_EXTN feature
  arm64: KVM: Skip HYP setup when already running in HYP
  arm64: KVM: VHE: Patch out use of HVC
  arm64: KVM: VHE: Patch out kern_hyp_va
  arm64: KVM: VHE: Introduce unified system register accessors
  arm64: KVM: VHE: Differenciate host/guest sysreg save/restore
  arm64: KVM: VHE: Split save/restore of registers shared between guest
    and host
  arm64: KVM: VHE: Use unified system register accessors
  arm64: KVM: VHE: Enable minimal sysreg save/restore
  arm64: KVM: VHE: Make __fpsimd_enabled VHE aware
  arm64: KVM: VHE: Implement VHE activate/deactivate_traps
  arm64: KVM: VHE: Use unified sysreg accessors for timer
  arm64: KVM: VHE: Add fpsimd enabling on guest access
  arm64: KVM: VHE: Add alternative panic handling
  arm64: KVM: Move most of the fault decoding to C
  arm64: perf: Count EL2 events if the kernel is running in HYP
  arm64: hw_breakpoint: Allow EL2 breakpoints if running in HYP
  arm64: VHE: Add support for running Linux in EL2 mode
  arm64: Panic when VHE and non VHE CPUs coexist

 arch/arm/include/asm/kvm_host.h        |   4 +
 arch/arm/include/asm/virt.h            |   5 +
 arch/arm/kvm/arm.c                     | 174 +++++++++++++++++++----------
 arch/arm/kvm/mmu.c                     |   7 ++
 arch/arm64/Kconfig                     |  13 +++
 arch/arm64/include/asm/cpufeature.h    |   3 +-
 arch/arm64/include/asm/hw_breakpoint.h |  49 ++++++---
 arch/arm64/include/asm/kvm_arm.h       |   6 +-
 arch/arm64/include/asm/kvm_asm.h       |   2 +
 arch/arm64/include/asm/kvm_emulate.h   |   3 +
 arch/arm64/include/asm/kvm_host.h      |   6 +
 arch/arm64/include/asm/kvm_mmu.h       |  12 +-
 arch/arm64/include/asm/virt.h          |  27 +++++
 arch/arm64/kernel/asm-offsets.c        |   3 -
 arch/arm64/kernel/cpufeature.c         |  11 ++
 arch/arm64/kernel/head.S               |  48 +++++++-
 arch/arm64/kernel/perf_event.c         |  14 ++-
 arch/arm64/kernel/smp.c                |   3 +
 arch/arm64/kvm/hyp-init.S              |  18 ---
 arch/arm64/kvm/hyp.S                   |   7 ++
 arch/arm64/kvm/hyp/Makefile            |   1 +
 arch/arm64/kvm/hyp/entry.S             |   6 +
 arch/arm64/kvm/hyp/hyp-entry.S         | 109 ++++++------------
 arch/arm64/kvm/hyp/hyp.h               | 108 ++++++++++++++++--
 arch/arm64/kvm/hyp/s2-setup.c          |  44 ++++++++
 arch/arm64/kvm/hyp/switch.c            | 196 ++++++++++++++++++++++++++++++---
 arch/arm64/kvm/hyp/sysreg-sr.c         | 147 ++++++++++++++++---------
 arch/arm64/kvm/hyp/timer-sr.c          |  10 +-
 drivers/clocksource/arm_arch_timer.c   |  96 +++++++++-------
 29 files changed, 837 insertions(+), 295 deletions(-)
 create mode 100644 arch/arm64/kvm/hyp/s2-setup.c

-- 
2.1.4

[toc] | [next] | [standalone]

#1325772 — [PATCH v3 14/23] arm64: KVM: VHE: Make __fpsimd_enabled VHE aware

As non-VHE and VHE have different ways to express the trapping of
FPSIMD registers to EL2, make __fpsimd_enabled a patchable predicate
and provide a VHE implementation.

Reviewed-by: Christoffer Dall <christoffer.dall@linaro.org>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm64/include/asm/kvm_arm.h |  3 +++
 arch/arm64/kvm/hyp/hyp.h         |  5 +----
 arch/arm64/kvm/hyp/switch.c      | 19 +++++++++++++++++++
 3 files changed, 23 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h
index 738a95f..498335e 100644
--- a/arch/arm64/include/asm/kvm_arm.h
+++ b/arch/arm64/include/asm/kvm_arm.h
@@ -217,4 +217,7 @@
 	ECN(SOFTSTP_CUR), ECN(WATCHPT_LOW), ECN(WATCHPT_CUR), \
 	ECN(BKPT32), ECN(VECTOR32), ECN(BRK64)
 
+#define CPACR_EL1_FPEN		(3 << 20)
+#define CPACR_EL1_TTA		(1 << 28)
+
 #endif /* __ARM64_KVM_ARM_H__ */
diff --git a/arch/arm64/kvm/hyp/hyp.h b/arch/arm64/kvm/hyp/hyp.h
index 5dfa883..44eaff7 100644
--- a/arch/arm64/kvm/hyp/hyp.h
+++ b/arch/arm64/kvm/hyp/hyp.h
@@ -171,10 +171,7 @@ void __debug_cond_restore_host_state(struct kvm_vcpu *vcpu);
 
 void __fpsimd_save_state(struct user_fpsimd_state *fp_regs);
 void __fpsimd_restore_state(struct user_fpsimd_state *fp_regs);
-static inline bool __fpsimd_enabled(void)
-{
-	return !(read_sysreg(cptr_el2) & CPTR_EL2_TFP);
-}
+bool __fpsimd_enabled(void);
 
 u64 __guest_enter(struct kvm_vcpu *vcpu, struct kvm_cpu_context *host_ctxt);
 void __noreturn __hyp_do_panic(unsigned long, ...);
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index 9071dee..0db161e 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -17,6 +17,25 @@
 
 #include "hyp.h"
 
+static bool __hyp_text __fpsimd_enabled_nvhe(void)
+{
+	return !(read_sysreg(cptr_el2) & CPTR_EL2_TFP);
+}
+
+static bool __hyp_text __fpsimd_enabled_vhe(void)
+{
+	return !!(read_sysreg(cpacr_el1) & CPACR_EL1_FPEN);
+}
+
+static hyp_alternate_select(__fpsimd_is_enabled,
+			    __fpsimd_enabled_nvhe, __fpsimd_enabled_vhe,
+			    ARM64_HAS_VIRT_HOST_EXTN);
+
+bool __hyp_text __fpsimd_enabled(void)
+{
+	return __fpsimd_is_enabled()();
+}
+
 static void __hyp_text __activate_traps(struct kvm_vcpu *vcpu)
 {
 	u64 val;
-- 
2.1.4

[toc] | [prev] | [next] | [standalone]

#1325773 — [PATCH v3 22/23] arm64: VHE: Add support for running Linux in EL2 mode

With ARMv8.1 VHE, the architecture is able to (almost) transparently
run the kernel at EL2, despite being written for EL1.

This patch takes care of the "almost" part, mostly preventing the kernel
from dropping from EL2 to EL1, and setting up the HYP configuration.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm64/Kconfig       | 13 +++++++++++++
 arch/arm64/kernel/head.S | 28 +++++++++++++++++++++++++++-
 2 files changed, 40 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 8cc6228..cf118d9 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -750,6 +750,19 @@ config ARM64_LSE_ATOMICS
 	  not support these instructions and requires the kernel to be
 	  built with binutils >= 2.25.
 
+config ARM64_VHE
+	bool "Enable support for Virtualization Host Extensions (VHE)"
+	default y
+	help
+	  Virtualization Host Extensions (VHE) allow the kernel to run
+	  directly at EL2 (instead of EL1) on processors that support
+	  it. This leads to better performance for KVM, as they reduce
+	  the cost of the world switch.
+
+	  Selecting this option allows the VHE feature to be detected
+	  at runtime, and does not affect processors that do not
+	  implement this feature.
+
 endmenu
 
 endmenu
diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index 917d981..6f2f377 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -30,6 +30,7 @@
 #include <asm/cache.h>
 #include <asm/cputype.h>
 #include <asm/kernel-pgtable.h>
+#include <asm/kvm_arm.h>
 #include <asm/memory.h>
 #include <asm/pgtable-hwdef.h>
 #include <asm/pgtable.h>
@@ -464,9 +465,27 @@ CPU_LE(	bic	x0, x0, #(3 << 24)	)	// Clear the EE and E0E bits for EL1
 	isb
 	ret
 
+2:
+#ifdef CONFIG_ARM64_VHE
+	/*
+	 * Check for VHE being present. For the rest of the EL2 setup,
+	 * x2 being non-zero indicates that we do have VHE, and that the
+	 * kernel is intended to run at EL2.
+	 */
+	mrs	x2, id_aa64mmfr1_el1
+	ubfx	x2, x2, #8, #4
+#else
+	mov	x2, xzr
+#endif
+
 	/* Hyp configuration. */
-2:	mov	x0, #(1 << 31)			// 64-bit EL1
+	mov	x0, #HCR_RW			// 64-bit EL1
+	cbz	x2, set_hcr
+	orr	x0, x0, #HCR_TGE		// Enable Host Extensions
+	orr	x0, x0, #HCR_E2H
+set_hcr:
 	msr	hcr_el2, x0
+	isb
 
 	/* Generic timers. */
 	mrs	x0, cnthctl_el2
@@ -526,6 +545,13 @@ CPU_LE(	movk	x0, #0x30d0, lsl #16	)	// Clear EE and E0E on LE systems
 	/* Stage-2 translation */
 	msr	vttbr_el2, xzr
 
+	cbz	x2, install_el2_stub
+
+	mov	w20, #BOOT_CPU_MODE_EL2		// This CPU booted in EL2
+	isb
+	ret
+
+install_el2_stub:
 	/* Hypervisor stub */
 	adrp	x0, __hyp_stub_vectors
 	add	x0, x0, #:lo12:__hyp_stub_vectors
-- 
2.1.4

[toc] | [prev] | [next] | [standalone]

#1327125 — Re: [PATCH v3 22/23] arm64: VHE: Add support for running Linux in EL2 mode

On Wed, Feb 03, 2016 at 06:00:15PM +0000, Marc Zyngier wrote:
> With ARMv8.1 VHE, the architecture is able to (almost) transparently
> run the kernel at EL2, despite being written for EL1.
> 
> This patch takes care of the "almost" part, mostly preventing the kernel
> from dropping from EL2 to EL1, and setting up the HYP configuration.
> 
> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>

Reviewed-by: Christoffer Dall <christoffer.dall@linaro.org>

[toc] | [prev] | [next] | [standalone]

#1329235 — Re: [PATCH v3 22/23] arm64: VHE: Add support for running Linux in EL2 mode

On Wed, Feb 03, 2016 at 06:00:15PM +0000, Marc Zyngier wrote:
> With ARMv8.1 VHE, the architecture is able to (almost) transparently
> run the kernel at EL2, despite being written for EL1.
> 
> This patch takes care of the "almost" part, mostly preventing the kernel
> from dropping from EL2 to EL1, and setting up the HYP configuration.
> 
> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>

Acked-by: Catalin Marinas <catalin.marinas@arm.com>

[toc] | [prev] | [next] | [standalone]

#1325776 — [PATCH v3 19/23] arm64: KVM: Move most of the fault decoding to C

The fault decoding process (including computing the IPA in the case
of a permission fault) would be much better done in C code, as we
have a reasonable infrastructure to deal with the VHE/non-VHE
differences.

Let's move the whole thing to C, including the workaround for
erratum 834220, and just patch the odd ESR_EL2 access remaining
in hyp-entry.S.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm64/kernel/asm-offsets.c |  3 --
 arch/arm64/kvm/hyp/hyp-entry.S  | 69 +++------------------------------
 arch/arm64/kvm/hyp/switch.c     | 85 +++++++++++++++++++++++++++++++++++++++++
 3 files changed, 90 insertions(+), 67 deletions(-)

diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
index fffa4ac6..b0ab4e9 100644
--- a/arch/arm64/kernel/asm-offsets.c
+++ b/arch/arm64/kernel/asm-offsets.c
@@ -110,9 +110,6 @@ int main(void)
   DEFINE(CPU_USER_PT_REGS,	offsetof(struct kvm_regs, regs));
   DEFINE(CPU_FP_REGS,		offsetof(struct kvm_regs, fp_regs));
   DEFINE(VCPU_FPEXC32_EL2,	offsetof(struct kvm_vcpu, arch.ctxt.sys_regs[FPEXC32_EL2]));
-  DEFINE(VCPU_ESR_EL2,		offsetof(struct kvm_vcpu, arch.fault.esr_el2));
-  DEFINE(VCPU_FAR_EL2,		offsetof(struct kvm_vcpu, arch.fault.far_el2));
-  DEFINE(VCPU_HPFAR_EL2,	offsetof(struct kvm_vcpu, arch.fault.hpfar_el2));
   DEFINE(VCPU_HOST_CONTEXT,	offsetof(struct kvm_vcpu, arch.host_cpu_context));
 #endif
 #ifdef CONFIG_CPU_PM
diff --git a/arch/arm64/kvm/hyp/hyp-entry.S b/arch/arm64/kvm/hyp/hyp-entry.S
index 1bdeee7..3488894 100644
--- a/arch/arm64/kvm/hyp/hyp-entry.S
+++ b/arch/arm64/kvm/hyp/hyp-entry.S
@@ -19,7 +19,6 @@
 
 #include <asm/alternative.h>
 #include <asm/assembler.h>
-#include <asm/asm-offsets.h>
 #include <asm/cpufeature.h>
 #include <asm/kvm_arm.h>
 #include <asm/kvm_asm.h>
@@ -69,7 +68,11 @@ ENDPROC(__vhe_hyp_call)
 el1_sync:				// Guest trapped into EL2
 	save_x0_to_x3
 
+alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
 	mrs	x1, esr_el2
+alternative_else
+	mrs	x1, esr_el1
+alternative_endif
 	lsr	x2, x1, #ESR_ELx_EC_SHIFT
 
 	cmp	x2, #ESR_ELx_EC_HVC64
@@ -105,72 +108,10 @@ el1_trap:
 	cmp	x2, #ESR_ELx_EC_FP_ASIMD
 	b.eq	__fpsimd_guest_restore
 
-	cmp	x2, #ESR_ELx_EC_DABT_LOW
-	mov	x0, #ESR_ELx_EC_IABT_LOW
-	ccmp	x2, x0, #4, ne
-	b.ne	1f		// Not an abort we care about
-
-	/* This is an abort. Check for permission fault */
-alternative_if_not ARM64_WORKAROUND_834220
-	and	x2, x1, #ESR_ELx_FSC_TYPE
-	cmp	x2, #FSC_PERM
-	b.ne	1f		// Not a permission fault
-alternative_else
-	nop			// Use the permission fault path to
-	nop			// check for a valid S1 translation,
-	nop			// regardless of the ESR value.
-alternative_endif
-
-	/*
-	 * Check for Stage-1 page table walk, which is guaranteed
-	 * to give a valid HPFAR_EL2.
-	 */
-	tbnz	x1, #7, 1f	// S1PTW is set
-
-	/* Preserve PAR_EL1 */
-	mrs	x3, par_el1
-	stp	x3, xzr, [sp, #-16]!
-
-	/*
-	 * Permission fault, HPFAR_EL2 is invalid.
-	 * Resolve the IPA the hard way using the guest VA.
-	 * Stage-1 translation already validated the memory access rights.
-	 * As such, we can use the EL1 translation regime, and don't have
-	 * to distinguish between EL0 and EL1 access.
-	 */
-	mrs	x2, far_el2
-	at	s1e1r, x2
-	isb
-
-	/* Read result */
-	mrs	x3, par_el1
-	ldp	x0, xzr, [sp], #16	// Restore PAR_EL1 from the stack
-	msr	par_el1, x0
-	tbnz	x3, #0, 3f		// Bail out if we failed the translation
-	ubfx	x3, x3, #12, #36	// Extract IPA
-	lsl	x3, x3, #4		// and present it like HPFAR
-	b	2f
-
-1:	mrs	x3, hpfar_el2
-	mrs	x2, far_el2
-
-2:	mrs	x0, tpidr_el2
-	str	w1, [x0, #VCPU_ESR_EL2]
-	str	x2, [x0, #VCPU_FAR_EL2]
-	str	x3, [x0, #VCPU_HPFAR_EL2]
-
+	mrs	x0, tpidr_el2
 	mov	x1, #ARM_EXCEPTION_TRAP
 	b	__guest_exit
 
-	/*
-	 * Translation failed. Just return to the guest and
-	 * let it fault again. Another CPU is probably playing
-	 * behind our back.
-	 */
-3:	restore_x0_to_x3
-
-	eret
-
 el1_irq:
 	save_x0_to_x3
 	mrs	x0, tpidr_el2
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index e90683a..a192357 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -15,6 +15,7 @@
  * along with this program.  If not, see <http://www.gnu.org/licenses/>.
  */
 
+#include <linux/types.h>
 #include <asm/kvm_asm.h>
 
 #include "hyp.h"
@@ -145,6 +146,86 @@ static void __hyp_text __vgic_restore_state(struct kvm_vcpu *vcpu)
 	__vgic_call_restore_state()(vcpu);
 }
 
+static bool __hyp_text __true_value(void)
+{
+	return true;
+}
+
+static bool __hyp_text __false_value(void)
+{
+	return false;
+}
+
+static hyp_alternate_select(__check_arm_834220,
+			    __false_value, __true_value,
+			    ARM64_WORKAROUND_834220);
+
+static bool __hyp_text __translate_far_to_hpfar(u64 far, u64 *hpfar)
+{
+	u64 par, tmp;
+
+	/*
+	 * Resolve the IPA the hard way using the guest VA.
+	 *
+	 * Stage-1 translation already validated the memory access
+	 * rights. As such, we can use the EL1 translation regime, and
+	 * don't have to distinguish between EL0 and EL1 access.
+	 *
+	 * We do need to save/restore PAR_EL1 though, as we haven't
+	 * saved the guest context yet, and we may return early...
+	 */
+	par = read_sysreg(par_el1);
+	asm volatile("at s1e1r, %0" : : "r" (far));
+	isb();
+
+	tmp = read_sysreg(par_el1);
+	write_sysreg(par, par_el1);
+
+	if (unlikely(tmp & 1))
+		return false; /* Translation failed, back to guest */
+
+	/* Convert PAR to HPFAR format */
+	*hpfar = ((tmp >> 12) & ((1UL << 36) - 1)) << 4;
+	return true;
+}
+
+static bool __hyp_text __populate_fault_info(struct kvm_vcpu *vcpu)
+{
+	u64 esr = read_sysreg_el2(esr);
+	u8 ec = esr >> ESR_ELx_EC_SHIFT;
+	u64 hpfar, far;
+
+	vcpu->arch.fault.esr_el2 = esr;
+
+	if (ec != ESR_ELx_EC_DABT_LOW && ec != ESR_ELx_EC_IABT_LOW)
+		return true;
+
+	far = read_sysreg_el2(far);
+
+	/*
+	 * The HPFAR can be invalid if the stage 2 fault did not
+	 * happen during a stage 1 page table walk (the ESR_EL2.S1PTW
+	 * bit is clear) and one of the two following cases are true:
+	 *   1. The fault was due to a permission fault
+	 *   2. The processor carries errata 834220
+	 *
+	 * Therefore, for all non S1PTW faults where we either have a
+	 * permission fault or the errata workaround is enabled, we
+	 * resolve the IPA using the AT instruction.
+	 */
+	if (!(esr & ESR_ELx_S1PTW) &&
+	    (__check_arm_834220()() || (esr & ESR_ELx_FSC_TYPE) == FSC_PERM)) {
+		if (!__translate_far_to_hpfar(far, &hpfar))
+			return false;
+	} else {
+		hpfar = read_sysreg(hpfar_el2);
+	}
+
+	vcpu->arch.fault.far_el2 = far;
+	vcpu->arch.fault.hpfar_el2 = hpfar;
+	return true;
+}
+
 static int __hyp_text __guest_run(struct kvm_vcpu *vcpu)
 {
 	struct kvm_cpu_context *host_ctxt;
@@ -176,9 +257,13 @@ static int __hyp_text __guest_run(struct kvm_vcpu *vcpu)
 	__debug_restore_state(vcpu, kern_hyp_va(vcpu->arch.debug_ptr), guest_ctxt);
 
 	/* Jump in the fire! */
+again:
 	exit_code = __guest_enter(vcpu, host_ctxt);
 	/* And we're baaack! */
 
+	if (exit_code == ARM_EXCEPTION_TRAP && !__populate_fault_info(vcpu))
+		goto again;
+
 	fp_enabled = __fpsimd_enabled();
 
 	__sysreg_save_guest_state(guest_ctxt);
-- 
2.1.4

[toc] | [prev] | [next] | [standalone]

#1327117 — Re: [PATCH v3 19/23] arm64: KVM: Move most of the fault decoding to C

On Wed, Feb 03, 2016 at 06:00:12PM +0000, Marc Zyngier wrote:
> The fault decoding process (including computing the IPA in the case
> of a permission fault) would be much better done in C code, as we
> have a reasonable infrastructure to deal with the VHE/non-VHE
> differences.
> 
> Let's move the whole thing to C, including the workaround for
> erratum 834220, and just patch the odd ESR_EL2 access remaining
> in hyp-entry.S.
> 
> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
> ---
>  arch/arm64/kernel/asm-offsets.c |  3 --
>  arch/arm64/kvm/hyp/hyp-entry.S  | 69 +++------------------------------
>  arch/arm64/kvm/hyp/switch.c     | 85 +++++++++++++++++++++++++++++++++++++++++
>  3 files changed, 90 insertions(+), 67 deletions(-)
> 
> diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
> index fffa4ac6..b0ab4e9 100644
> --- a/arch/arm64/kernel/asm-offsets.c
> +++ b/arch/arm64/kernel/asm-offsets.c
> @@ -110,9 +110,6 @@ int main(void)
>    DEFINE(CPU_USER_PT_REGS,	offsetof(struct kvm_regs, regs));
>    DEFINE(CPU_FP_REGS,		offsetof(struct kvm_regs, fp_regs));
>    DEFINE(VCPU_FPEXC32_EL2,	offsetof(struct kvm_vcpu, arch.ctxt.sys_regs[FPEXC32_EL2]));
> -  DEFINE(VCPU_ESR_EL2,		offsetof(struct kvm_vcpu, arch.fault.esr_el2));
> -  DEFINE(VCPU_FAR_EL2,		offsetof(struct kvm_vcpu, arch.fault.far_el2));
> -  DEFINE(VCPU_HPFAR_EL2,	offsetof(struct kvm_vcpu, arch.fault.hpfar_el2));
>    DEFINE(VCPU_HOST_CONTEXT,	offsetof(struct kvm_vcpu, arch.host_cpu_context));
>  #endif
>  #ifdef CONFIG_CPU_PM
> diff --git a/arch/arm64/kvm/hyp/hyp-entry.S b/arch/arm64/kvm/hyp/hyp-entry.S
> index 1bdeee7..3488894 100644
> --- a/arch/arm64/kvm/hyp/hyp-entry.S
> +++ b/arch/arm64/kvm/hyp/hyp-entry.S
> @@ -19,7 +19,6 @@
>  
>  #include <asm/alternative.h>
>  #include <asm/assembler.h>
> -#include <asm/asm-offsets.h>
>  #include <asm/cpufeature.h>
>  #include <asm/kvm_arm.h>
>  #include <asm/kvm_asm.h>
> @@ -69,7 +68,11 @@ ENDPROC(__vhe_hyp_call)
>  el1_sync:				// Guest trapped into EL2
>  	save_x0_to_x3
>  
> +alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
>  	mrs	x1, esr_el2
> +alternative_else
> +	mrs	x1, esr_el1
> +alternative_endif
>  	lsr	x2, x1, #ESR_ELx_EC_SHIFT
>  
>  	cmp	x2, #ESR_ELx_EC_HVC64
> @@ -105,72 +108,10 @@ el1_trap:
>  	cmp	x2, #ESR_ELx_EC_FP_ASIMD
>  	b.eq	__fpsimd_guest_restore
>  
> -	cmp	x2, #ESR_ELx_EC_DABT_LOW
> -	mov	x0, #ESR_ELx_EC_IABT_LOW
> -	ccmp	x2, x0, #4, ne
> -	b.ne	1f		// Not an abort we care about
> -
> -	/* This is an abort. Check for permission fault */
> -alternative_if_not ARM64_WORKAROUND_834220
> -	and	x2, x1, #ESR_ELx_FSC_TYPE
> -	cmp	x2, #FSC_PERM
> -	b.ne	1f		// Not a permission fault
> -alternative_else
> -	nop			// Use the permission fault path to
> -	nop			// check for a valid S1 translation,
> -	nop			// regardless of the ESR value.
> -alternative_endif
> -
> -	/*
> -	 * Check for Stage-1 page table walk, which is guaranteed
> -	 * to give a valid HPFAR_EL2.
> -	 */
> -	tbnz	x1, #7, 1f	// S1PTW is set
> -
> -	/* Preserve PAR_EL1 */
> -	mrs	x3, par_el1
> -	stp	x3, xzr, [sp, #-16]!
> -
> -	/*
> -	 * Permission fault, HPFAR_EL2 is invalid.
> -	 * Resolve the IPA the hard way using the guest VA.
> -	 * Stage-1 translation already validated the memory access rights.
> -	 * As such, we can use the EL1 translation regime, and don't have
> -	 * to distinguish between EL0 and EL1 access.
> -	 */
> -	mrs	x2, far_el2
> -	at	s1e1r, x2
> -	isb
> -
> -	/* Read result */
> -	mrs	x3, par_el1
> -	ldp	x0, xzr, [sp], #16	// Restore PAR_EL1 from the stack
> -	msr	par_el1, x0
> -	tbnz	x3, #0, 3f		// Bail out if we failed the translation
> -	ubfx	x3, x3, #12, #36	// Extract IPA
> -	lsl	x3, x3, #4		// and present it like HPFAR
> -	b	2f
> -
> -1:	mrs	x3, hpfar_el2
> -	mrs	x2, far_el2
> -
> -2:	mrs	x0, tpidr_el2
> -	str	w1, [x0, #VCPU_ESR_EL2]
> -	str	x2, [x0, #VCPU_FAR_EL2]
> -	str	x3, [x0, #VCPU_HPFAR_EL2]
> -
> +	mrs	x0, tpidr_el2
>  	mov	x1, #ARM_EXCEPTION_TRAP
>  	b	__guest_exit
>  
> -	/*
> -	 * Translation failed. Just return to the guest and
> -	 * let it fault again. Another CPU is probably playing
> -	 * behind our back.
> -	 */
> -3:	restore_x0_to_x3
> -
> -	eret
> -
>  el1_irq:
>  	save_x0_to_x3
>  	mrs	x0, tpidr_el2
> diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
> index e90683a..a192357 100644
> --- a/arch/arm64/kvm/hyp/switch.c
> +++ b/arch/arm64/kvm/hyp/switch.c
> @@ -15,6 +15,7 @@
>   * along with this program.  If not, see <http://www.gnu.org/licenses/>.
>   */
>  
> +#include <linux/types.h>
>  #include <asm/kvm_asm.h>
>  
>  #include "hyp.h"
> @@ -145,6 +146,86 @@ static void __hyp_text __vgic_restore_state(struct kvm_vcpu *vcpu)
>  	__vgic_call_restore_state()(vcpu);
>  }
>  
> +static bool __hyp_text __true_value(void)
> +{
> +	return true;
> +}
> +
> +static bool __hyp_text __false_value(void)
> +{
> +	return false;
> +}
> +
> +static hyp_alternate_select(__check_arm_834220,
> +			    __false_value, __true_value,
> +			    ARM64_WORKAROUND_834220);
> +
> +static bool __hyp_text __translate_far_to_hpfar(u64 far, u64 *hpfar)
> +{
> +	u64 par, tmp;
> +
> +	/*
> +	 * Resolve the IPA the hard way using the guest VA.
> +	 *
> +	 * Stage-1 translation already validated the memory access
> +	 * rights. As such, we can use the EL1 translation regime, and
> +	 * don't have to distinguish between EL0 and EL1 access.
> +	 *
> +	 * We do need to save/restore PAR_EL1 though, as we haven't
> +	 * saved the guest context yet, and we may return early...
> +	 */
> +	par = read_sysreg(par_el1);
> +	asm volatile("at s1e1r, %0" : : "r" (far));
> +	isb();
> +
> +	tmp = read_sysreg(par_el1);
> +	write_sysreg(par, par_el1);
> +
> +	if (unlikely(tmp & 1))
> +		return false; /* Translation failed, back to guest */
> +
> +	/* Convert PAR to HPFAR format */
> +	*hpfar = ((tmp >> 12) & ((1UL << 36) - 1)) << 4;
> +	return true;
> +}
> +
> +static bool __hyp_text __populate_fault_info(struct kvm_vcpu *vcpu)
> +{
> +	u64 esr = read_sysreg_el2(esr);
> +	u8 ec = esr >> ESR_ELx_EC_SHIFT;
> +	u64 hpfar, far;
> +
> +	vcpu->arch.fault.esr_el2 = esr;
> +
> +	if (ec != ESR_ELx_EC_DABT_LOW && ec != ESR_ELx_EC_IABT_LOW)
> +		return true;
> +
> +	far = read_sysreg_el2(far);
> +
> +	/*
> +	 * The HPFAR can be invalid if the stage 2 fault did not
> +	 * happen during a stage 1 page table walk (the ESR_EL2.S1PTW
> +	 * bit is clear) and one of the two following cases are true:
> +	 *   1. The fault was due to a permission fault
> +	 *   2. The processor carries errata 834220
> +	 *
> +	 * Therefore, for all non S1PTW faults where we either have a
> +	 * permission fault or the errata workaround is enabled, we
> +	 * resolve the IPA using the AT instruction.
> +	 */
> +	if (!(esr & ESR_ELx_S1PTW) &&
> +	    (__check_arm_834220()() || (esr & ESR_ELx_FSC_TYPE) == FSC_PERM)) {
> +		if (!__translate_far_to_hpfar(far, &hpfar))
> +			return false;
> +	} else {
> +		hpfar = read_sysreg(hpfar_el2);
> +	}
> +
> +	vcpu->arch.fault.far_el2 = far;
> +	vcpu->arch.fault.hpfar_el2 = hpfar;
> +	return true;
> +}
> +
>  static int __hyp_text __guest_run(struct kvm_vcpu *vcpu)
>  {
>  	struct kvm_cpu_context *host_ctxt;
> @@ -176,9 +257,13 @@ static int __hyp_text __guest_run(struct kvm_vcpu *vcpu)
>  	__debug_restore_state(vcpu, kern_hyp_va(vcpu->arch.debug_ptr), guest_ctxt);
>  
>  	/* Jump in the fire! */
> +again:
>  	exit_code = __guest_enter(vcpu, host_ctxt);
>  	/* And we're baaack! */
>  
> +	if (exit_code == ARM_EXCEPTION_TRAP && !__populate_fault_info(vcpu))
> +		goto again;
> +
>  	fp_enabled = __fpsimd_enabled();
>  
>  	__sysreg_save_guest_state(guest_ctxt);
> -- 
> 2.1.4
> 

Thanks for the rewrite, I find this code really nice now, especially
comparing to the confusing job-security-creating stuff we had in
assembly before.

-Christoffer

[toc] | [prev] | [next] | [standalone]

#1329133 — Re: [PATCH v3 19/23] arm64: KVM: Move most of the fault decoding to C

On Wed, Feb 03, 2016 at 06:00:12PM +0000, Marc Zyngier wrote:
> The fault decoding process (including computing the IPA in the case
> of a permission fault) would be much better done in C code, as we
> have a reasonable infrastructure to deal with the VHE/non-VHE
> differences.
> 
> Let's move the whole thing to C, including the workaround for
> erratum 834220, and just patch the odd ESR_EL2 access remaining
> in hyp-entry.S.
> 
> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>

Acked-by: Catalin Marinas <catalin.marinas@arm.com>

[toc] | [prev] | [next] | [standalone]

#1325779 — [PATCH v3 13/23] arm64: KVM: VHE: Enable minimal sysreg save/restore

We're now in a position where we can introduce VHE's minimal
save/restore, which is limited to the handful of shared sysregs.

Add the required alternative function calls that result in a
"do nothing" call on VHE, and the normal save/restore for non-VHE.

Reviewed-by: Christoffer Dall <christoffer.dall@linaro.org>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm64/kvm/hyp/sysreg-sr.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c
index 7d7d757..74b5f81 100644
--- a/arch/arm64/kvm/hyp/sysreg-sr.c
+++ b/arch/arm64/kvm/hyp/sysreg-sr.c
@@ -23,6 +23,9 @@
 
 #include "hyp.h"
 
+/* Yes, this does nothing, on purpose */
+static void __hyp_text __sysreg_do_nothing(struct kvm_cpu_context *ctxt) { }
+
 /*
  * Non-VHE: Both host and guest must save everything.
  *
@@ -67,9 +70,13 @@ static void __hyp_text __sysreg_save_state(struct kvm_cpu_context *ctxt)
 	ctxt->gp_regs.spsr[KVM_SPSR_EL1]= read_sysreg_el1(spsr);
 }
 
+static hyp_alternate_select(__sysreg_call_save_host_state,
+			    __sysreg_save_state, __sysreg_do_nothing,
+			    ARM64_HAS_VIRT_HOST_EXTN);
+
 void __hyp_text __sysreg_save_host_state(struct kvm_cpu_context *ctxt)
 {
-	__sysreg_save_state(ctxt);
+	__sysreg_call_save_host_state()(ctxt);
 	__sysreg_save_common_state(ctxt);
 }
 
@@ -116,9 +123,13 @@ static void __hyp_text __sysreg_restore_state(struct kvm_cpu_context *ctxt)
 	write_sysreg_el1(ctxt->gp_regs.spsr[KVM_SPSR_EL1],spsr);
 }
 
+static hyp_alternate_select(__sysreg_call_restore_host_state,
+			    __sysreg_restore_state, __sysreg_do_nothing,
+			    ARM64_HAS_VIRT_HOST_EXTN);
+
 void __hyp_text __sysreg_restore_host_state(struct kvm_cpu_context *ctxt)
 {
-	__sysreg_restore_state(ctxt);
+	__sysreg_call_restore_host_state()(ctxt);
 	__sysreg_restore_common_state(ctxt);
 }
 
-- 
2.1.4

[toc] | [prev] | [next] | [standalone]

#1325780 — [PATCH v3 16/23] arm64: KVM: VHE: Use unified sysreg accessors for timer

Switch the timer code to the unified sysreg accessors.

Reviewed-by: Christoffer Dall <christoffer.dall@linaro.org>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm64/kvm/hyp/timer-sr.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/arch/arm64/kvm/hyp/timer-sr.c b/arch/arm64/kvm/hyp/timer-sr.c
index 1051e5d..f276d9e 100644
--- a/arch/arm64/kvm/hyp/timer-sr.c
+++ b/arch/arm64/kvm/hyp/timer-sr.c
@@ -31,12 +31,12 @@ void __hyp_text __timer_save_state(struct kvm_vcpu *vcpu)
 	u64 val;
 
 	if (kvm->arch.timer.enabled) {
-		timer->cntv_ctl = read_sysreg(cntv_ctl_el0);
-		timer->cntv_cval = read_sysreg(cntv_cval_el0);
+		timer->cntv_ctl = read_sysreg_el0(cntv_ctl);
+		timer->cntv_cval = read_sysreg_el0(cntv_cval);
 	}
 
 	/* Disable the virtual timer */
-	write_sysreg(0, cntv_ctl_el0);
+	write_sysreg_el0(0, cntv_ctl);
 
 	/* Allow physical timer/counter access for the host */
 	val = read_sysreg(cnthctl_el2);
@@ -64,8 +64,8 @@ void __hyp_text __timer_restore_state(struct kvm_vcpu *vcpu)
 
 	if (kvm->arch.timer.enabled) {
 		write_sysreg(kvm->arch.timer.cntvoff, cntvoff_el2);
-		write_sysreg(timer->cntv_cval, cntv_cval_el0);
+		write_sysreg_el0(timer->cntv_cval, cntv_cval);
 		isb();
-		write_sysreg(timer->cntv_ctl, cntv_ctl_el0);
+		write_sysreg_el0(timer->cntv_ctl, cntv_ctl);
 	}
 }
-- 
2.1.4

[toc] | [prev] | [next] | [standalone]

#1325801 — [PATCH v3 01/23] arm/arm64: KVM: Add hook for C-based stage2 init

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm/include/asm/kvm_host.h   | 4 ++++
 arch/arm/kvm/arm.c                | 1 +
 arch/arm64/include/asm/kvm_host.h | 4 ++++
 3 files changed, 9 insertions(+)

diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h
index f9f2779..f1e86f1 100644
--- a/arch/arm/include/asm/kvm_host.h
+++ b/arch/arm/include/asm/kvm_host.h
@@ -220,6 +220,10 @@ static inline void __cpu_init_hyp_mode(phys_addr_t boot_pgd_ptr,
 	kvm_call_hyp((void*)hyp_stack_ptr, vector_ptr, pgd_ptr);
 }
 
+static inline void __cpu_init_stage2(void)
+{
+}
+
 static inline int kvm_arch_dev_ioctl_check_extension(long ext)
 {
 	return 0;
diff --git a/arch/arm/kvm/arm.c b/arch/arm/kvm/arm.c
index dda1959..6b76e01 100644
--- a/arch/arm/kvm/arm.c
+++ b/arch/arm/kvm/arm.c
@@ -985,6 +985,7 @@ static void cpu_init_hyp_mode(void *dummy)
 	vector_ptr = (unsigned long)__kvm_hyp_vector;
 
 	__cpu_init_hyp_mode(boot_pgd_ptr, pgd_ptr, hyp_stack_ptr, vector_ptr);
+	__cpu_init_stage2();
 
 	kvm_arm_init_debug();
 }
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 689d4c9..fe86cf9 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -332,6 +332,10 @@ static inline void __cpu_init_hyp_mode(phys_addr_t boot_pgd_ptr,
 		     hyp_stack_ptr, vector_ptr);
 }
 
+static inline void __cpu_init_stage2(void)
+{
+}
+
 static inline void kvm_arch_hardware_disable(void) {}
 static inline void kvm_arch_hardware_unsetup(void) {}
 static inline void kvm_arch_sync_events(struct kvm *kvm) {}
-- 
2.1.4

[toc] | [prev] | [next] | [standalone]

#1327128

On Wed, Feb 03, 2016 at 05:59:53PM +0000, Marc Zyngier wrote:
> ARMv8.1 comes with the "Virtualization Host Extension" (VHE for
> short), which enables simpler support of Type-2 hypervisors.
> 
> This extension allows the kernel to directly run at EL2, and
> significantly reduces the number of system registers shared between
> host and guest, reducing the overhead of virtualization.
> 
> In order to have the same kernel binary running on all versions of the
> architecture, this series makes heavy use of runtime code patching.
> 
> The first 22 patches massage the KVM code to deal with VHE and enable
> Linux to run at EL2. The last patch catches an ugly case when VHE
> capable CPUs are paired with some of their less capable siblings. This
> should never happen, but hey...
> 
> I have deliberately left out some of the more "advanced"
> optimizations, as they are likely to distract the reviewer from the
> core infrastructure, which is what I care about at the moment.
> 
> Note: GDB is currently busted on VHE systems, as it checks for version
>       6 on the debug architecture, while VHE is version 7. The
>       binutils people are on the case.
> 
> This has been tested on the FVP_Base_SLV-V8-A model, and based on
> v4.5-rc2. I've put a branch out on:
> 
> git://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git kvm-arm64/vhe
> 

You can have my reviewed-by on all patches that I didn't already review
or ack explicitly.

Only exception is the debug stuff where I didn't manage to page in the
context, so hopefully familiar with that code can have a look.

Then this is ready to be queued.

Thanks,
-Christoffer

[toc] | [prev] | [next] | [standalone]

#1327540

On 04/02/16 19:26, Christoffer Dall wrote:
> On Wed, Feb 03, 2016 at 05:59:53PM +0000, Marc Zyngier wrote:
>> ARMv8.1 comes with the "Virtualization Host Extension" (VHE for
>> short), which enables simpler support of Type-2 hypervisors.
>>
>> This extension allows the kernel to directly run at EL2, and
>> significantly reduces the number of system registers shared between
>> host and guest, reducing the overhead of virtualization.
>>
>> In order to have the same kernel binary running on all versions of the
>> architecture, this series makes heavy use of runtime code patching.
>>
>> The first 22 patches massage the KVM code to deal with VHE and enable
>> Linux to run at EL2. The last patch catches an ugly case when VHE
>> capable CPUs are paired with some of their less capable siblings. This
>> should never happen, but hey...
>>
>> I have deliberately left out some of the more "advanced"
>> optimizations, as they are likely to distract the reviewer from the
>> core infrastructure, which is what I care about at the moment.
>>
>> Note: GDB is currently busted on VHE systems, as it checks for version
>>       6 on the debug architecture, while VHE is version 7. The
>>       binutils people are on the case.
>>
>> This has been tested on the FVP_Base_SLV-V8-A model, and based on
>> v4.5-rc2. I've put a branch out on:
>>
>> git://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git kvm-arm64/vhe
>>
> 
> You can have my reviewed-by on all patches that I didn't already review
> or ack explicitly.

Thanks for the review, much appreciated.

> Only exception is the debug stuff where I didn't manage to page in the
> context, so hopefully familiar with that code can have a look.

I think this will have to be Mr Deacon.

Thanks again,

	M.
-- 
Jazz is not dead. It just smells funny...

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web