Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1573169 > unrolled thread

[RFC v4 00/10] Provide the EL1 physical timer to the VM

Back to article view | Back to linux.kernel

Contents

  [RFC v4 00/10] Provide the EL1 physical timer to the VM Jintack Lim <jintack@cs.columbia.edu> - 2017-02-03 16:30 +0100
    [RFC v4 09/10] KVM: arm64: Add the EL1 physical timer access handler Jintack Lim <jintack@cs.columbia.edu> - 2017-02-03 16:30 +0100
    [RFC v4 08/10] KVM: arm/arm64: Set up a background timer for the physical timer emulation Jintack Lim <jintack@cs.columbia.edu> - 2017-02-03 16:30 +0100
    [RFC v4 10/10] KVM: arm/arm64: Emulate the EL1 phys timer registers Jintack Lim <jintack@cs.columbia.edu> - 2017-02-03 16:30 +0100
    Re: [RFC v4 00/10] Provide the EL1 physical timer to the VM Marc Zyngier <marc.zyngier@arm.com> - 2017-02-03 17:20 +0100
      Re: [RFC v4 00/10] Provide the EL1 physical timer to the VM Jintack Lim <jintack@cs.columbia.edu> - 2017-02-03 18:00 +0100
        Re: [RFC v4 00/10] Provide the EL1 physical timer to the VM Marc Zyngier <marc.zyngier@arm.com> - 2017-02-03 18:00 +0100

#1573169 — [RFC v4 00/10] Provide the EL1 physical timer to the VM

The ARM architecture defines the EL1 physical timer and the virtual timer,
and it is reasonable for an OS to expect to be able to access both.
However, the current KVM implementation does not provide the EL1 physical
timer to VMs but terminates VMs on access to the timer.

This patch series enables VMs to use the EL1 physical timer through
trap-and-emulate only on arm64. The KVM host emulates each EL1 physical
timer register access and sets up the background timer accordingly.  When
the background timer expires, the KVM host injects EL1 physical timer
interrupts to the VM.  Alternatively, it's also possible to allow VMs to
access the EL1 physical timer without trapping.  However, this requires
somehow using the EL2 physical timer for the Linux host while running the
VM instead of the EL1 physical timer.  Right now I just implemented
trap-and-emulate because this was straightforward to do, and I leave it to
future work to determine if transferring the EL1 physical timer state to
the EL2 timer provides any performance benefit.

This feature will be useful for any OS that wishes to access the EL1
physical timer. Nested virtualization is one of those use cases. A nested
hypervisor running inside a VM would think it has full access to the
hardware and naturally tries to use the EL1 physical timer as Linux would
do. Other nested hypervisors may try to use the EL2 physical timer as Xen
would do, but supporting the EL2 physical timer to the VM is out of scope
of this patch series. This patch series will make it easy to add the EL2
timer support in the future, though.

Note that Linux VMs booting in EL1 will be unaffected by this patch series
and will continue to use only the virtual timer and this patch series will
therefore not introduce any performance degredation as a result of
trap-and-emulate.

v3 => v4:
 - Fix a bug that prevents a VM from booting on 32-bit architecture
 - Clarify that the emulated physical timer is only supported on arm64
   in the cover letter

v2 => v3:
 - Rebase on kvmarm/queue
 - Take kvm->lock to synchronize cntvoff across all vtimers
 - Remove unnecessary function parameters
 - Add comments

v1 => v2:
 - Rebase on kvm-arm-for-4.10-rc4
 - To make it simple, schedule the background timer for the EL1 physical timer
   emulation on every entry to the VM and cancel it on exit.
 - Change timer_context structure to have cntvoff and restore enable field back
   to arch_timer_cpu structure
Jintack Lim (10):
  KVM: arm/arm64: Abstract virtual timer context into separate structure
  KVM: arm/arm64: Move cntvoff to each timer context
  KVM: arm/arm64: Decouple kvm timer functions from virtual timer
  KVM: arm/arm64: Add the EL1 physical timer context
  KVM: arm/arm64: Initialize the emulated EL1 physical timer
  KVM: arm/arm64: Update the physical timer interrupt level
  KVM: arm/arm64: Set a background timer to the earliest timer
    expiration
  KVM: arm/arm64: Set up a background timer for the physical timer
    emulation
  KVM: arm64: Add the EL1 physical timer access handler
  KVM: arm/arm64: Emulate the EL1 phys timer registers

 arch/arm/include/asm/kvm_host.h   |   3 -
 arch/arm/kvm/arm.c                |   4 +-
 arch/arm/kvm/reset.c              |   9 +-
 arch/arm64/include/asm/kvm_host.h |   3 -
 arch/arm64/kvm/reset.c            |   9 +-
 arch/arm64/kvm/sys_regs.c         |  65 +++++++++++++
 include/kvm/arm_arch_timer.h      |  39 ++++----
 virt/kvm/arm/arch_timer.c         | 200 ++++++++++++++++++++++++++------------
 virt/kvm/arm/hyp/timer-sr.c       |  13 +--
 9 files changed, 249 insertions(+), 96 deletions(-)

-- 
1.9.1

[toc] | [next] | [standalone]

#1573171 — [RFC v4 09/10] KVM: arm64: Add the EL1 physical timer access handler

KVM traps on the EL1 phys timer accesses from VMs, but it doesn't handle
those traps. This results in terminating VMs. Instead, set a handler for
the EL1 phys timer access, and inject an undefined exception as an
intermediate step.

Signed-off-by: Jintack Lim <jintack@cs.columbia.edu>
Reviewed-by: Christoffer Dall <christoffer.dall@linaro.org>
---
 arch/arm64/kvm/sys_regs.c | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)

diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
index caa47ce..1cd3464 100644
--- a/arch/arm64/kvm/sys_regs.c
+++ b/arch/arm64/kvm/sys_regs.c
@@ -820,6 +820,30 @@ static bool access_pmuserenr(struct kvm_vcpu *vcpu, struct sys_reg_params *p,
 	  CRm((0b1100 | (((n) >> 3) & 0x3))), Op2(((n) & 0x7)),		\
 	  access_pmu_evtyper, reset_unknown, (PMEVTYPER0_EL0 + n), }
 
+static bool access_cntp_tval(struct kvm_vcpu *vcpu,
+		struct sys_reg_params *p,
+		const struct sys_reg_desc *r)
+{
+	kvm_inject_undefined(vcpu);
+	return true;
+}
+
+static bool access_cntp_ctl(struct kvm_vcpu *vcpu,
+		struct sys_reg_params *p,
+		const struct sys_reg_desc *r)
+{
+	kvm_inject_undefined(vcpu);
+	return true;
+}
+
+static bool access_cntp_cval(struct kvm_vcpu *vcpu,
+		struct sys_reg_params *p,
+		const struct sys_reg_desc *r)
+{
+	kvm_inject_undefined(vcpu);
+	return true;
+}
+
 /*
  * Architected system registers.
  * Important: Must be sorted ascending by Op0, Op1, CRn, CRm, Op2
@@ -1029,6 +1053,16 @@ static bool access_pmuserenr(struct kvm_vcpu *vcpu, struct sys_reg_params *p,
 	{ Op0(0b11), Op1(0b011), CRn(0b1101), CRm(0b0000), Op2(0b011),
 	  NULL, reset_unknown, TPIDRRO_EL0 },
 
+	/* CNTP_TVAL_EL0 */
+	{ Op0(0b11), Op1(0b011), CRn(0b1110), CRm(0b0010), Op2(0b000),
+	  access_cntp_tval },
+	/* CNTP_CTL_EL0 */
+	{ Op0(0b11), Op1(0b011), CRn(0b1110), CRm(0b0010), Op2(0b001),
+	  access_cntp_ctl },
+	/* CNTP_CVAL_EL0 */
+	{ Op0(0b11), Op1(0b011), CRn(0b1110), CRm(0b0010), Op2(0b010),
+	  access_cntp_cval },
+
 	/* PMEVCNTRn_EL0 */
 	PMU_PMEVCNTR_EL0(0),
 	PMU_PMEVCNTR_EL0(1),
-- 
1.9.1

[toc] | [prev] | [next] | [standalone]

#1573173 — [RFC v4 08/10] KVM: arm/arm64: Set up a background timer for the physical timer emulation

Set a background timer for the EL1 physical timer emulation while VMs
are running, so that VMs get the physical timer interrupts in a timely
manner.

Schedule the background timer on entry to the VM and cancel it on exit.
This would not have any performance impact to the guest OSes that
currently use the virtual timer since the physical timer is always not
enabled.

Signed-off-by: Jintack Lim <jintack@cs.columbia.edu>
Reviewed-by: Christoffer Dall <christoffer.dall@linaro.org>
---
 virt/kvm/arm/arch_timer.c | 25 ++++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)

diff --git a/virt/kvm/arm/arch_timer.c b/virt/kvm/arm/arch_timer.c
index 0ea7452..33257b5 100644
--- a/virt/kvm/arm/arch_timer.c
+++ b/virt/kvm/arm/arch_timer.c
@@ -229,6 +229,22 @@ static int kvm_timer_update_state(struct kvm_vcpu *vcpu)
 	return 0;
 }
 
+/* Schedule the background timer for the emulated timer. */
+static void kvm_timer_emulate(struct kvm_vcpu *vcpu,
+			      struct arch_timer_context *timer_ctx)
+{
+	struct arch_timer_cpu *timer = &vcpu->arch.timer_cpu;
+
+	if (kvm_timer_should_fire(timer_ctx))
+		return;
+
+	if (!kvm_timer_irq_can_fire(timer_ctx))
+		return;
+
+	/*  The timer has not yet expired, schedule a background timer */
+	timer_arm(timer, kvm_timer_compute_delta(timer_ctx));
+}
+
 /*
  * Schedule the background timer before calling kvm_vcpu_block, so that this
  * thread is removed from its waitqueue and made runnable when there's a timer
@@ -286,6 +302,9 @@ void kvm_timer_flush_hwstate(struct kvm_vcpu *vcpu)
 	if (kvm_timer_update_state(vcpu))
 		return;
 
+	/* Set the background timer for the physical timer emulation. */
+	kvm_timer_emulate(vcpu, vcpu_ptimer(vcpu));
+
 	/*
 	* If we enter the guest with the virtual input level to the VGIC
 	* asserted, then we have already told the VGIC what we need to, and
@@ -348,7 +367,11 @@ void kvm_timer_sync_hwstate(struct kvm_vcpu *vcpu)
 {
 	struct arch_timer_cpu *timer = &vcpu->arch.timer_cpu;
 
-	BUG_ON(timer_is_armed(timer));
+	/*
+	 * This is to cancel the background timer for the physical timer
+	 * emulation if it is set.
+	 */
+	timer_disarm(timer);
 
 	/*
 	 * The guest could have modified the timer registers or the timer
-- 
1.9.1

[toc] | [prev] | [next] | [standalone]

#1573175 — [RFC v4 10/10] KVM: arm/arm64: Emulate the EL1 phys timer registers

Emulate read and write operations to CNTP_TVAL, CNTP_CVAL and CNTP_CTL.
Now VMs are able to use the EL1 physical timer.

Signed-off-by: Jintack Lim <jintack@cs.columbia.edu>
Reviewed-by: Christoffer Dall <christoffer.dall@linaro.org>
---
 arch/arm64/kvm/sys_regs.c    | 37 ++++++++++++++++++++++++++++++++++---
 include/kvm/arm_arch_timer.h |  2 ++
 virt/kvm/arm/arch_timer.c    |  2 +-
 3 files changed, 37 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
index 1cd3464..0e26f8c 100644
--- a/arch/arm64/kvm/sys_regs.c
+++ b/arch/arm64/kvm/sys_regs.c
@@ -824,7 +824,14 @@ static bool access_cntp_tval(struct kvm_vcpu *vcpu,
 		struct sys_reg_params *p,
 		const struct sys_reg_desc *r)
 {
-	kvm_inject_undefined(vcpu);
+	struct arch_timer_context *ptimer = vcpu_ptimer(vcpu);
+	u64 now = kvm_phys_timer_read();
+
+	if (p->is_write)
+		ptimer->cnt_cval = p->regval + now;
+	else
+		p->regval = ptimer->cnt_cval - now;
+
 	return true;
 }
 
@@ -832,7 +839,25 @@ static bool access_cntp_ctl(struct kvm_vcpu *vcpu,
 		struct sys_reg_params *p,
 		const struct sys_reg_desc *r)
 {
-	kvm_inject_undefined(vcpu);
+	struct arch_timer_context *ptimer = vcpu_ptimer(vcpu);
+
+	if (p->is_write) {
+		/* ISTATUS bit is read-only */
+		ptimer->cnt_ctl = p->regval & ~ARCH_TIMER_CTRL_IT_STAT;
+	} else {
+		u64 now = kvm_phys_timer_read();
+
+		p->regval = ptimer->cnt_ctl;
+		/*
+		 * Set ISTATUS bit if it's expired.
+		 * Note that according to ARMv8 ARM Issue A.k, ISTATUS bit is
+		 * UNKNOWN when ENABLE bit is 0, so we chose to set ISTATUS bit
+		 * regardless of ENABLE bit for our implementation convenience.
+		 */
+		if (ptimer->cnt_cval <= now)
+			p->regval |= ARCH_TIMER_CTRL_IT_STAT;
+	}
+
 	return true;
 }
 
@@ -840,7 +865,13 @@ static bool access_cntp_cval(struct kvm_vcpu *vcpu,
 		struct sys_reg_params *p,
 		const struct sys_reg_desc *r)
 {
-	kvm_inject_undefined(vcpu);
+	struct arch_timer_context *ptimer = vcpu_ptimer(vcpu);
+
+	if (p->is_write)
+		ptimer->cnt_cval = p->regval;
+	else
+		p->regval = ptimer->cnt_cval;
+
 	return true;
 }
 
diff --git a/include/kvm/arm_arch_timer.h b/include/kvm/arm_arch_timer.h
index f1d2fba0..fe797d6 100644
--- a/include/kvm/arm_arch_timer.h
+++ b/include/kvm/arm_arch_timer.h
@@ -72,6 +72,8 @@ int kvm_timer_vcpu_reset(struct kvm_vcpu *vcpu,
 void kvm_timer_schedule(struct kvm_vcpu *vcpu);
 void kvm_timer_unschedule(struct kvm_vcpu *vcpu);
 
+u64 kvm_phys_timer_read(void);
+
 void kvm_timer_vcpu_put(struct kvm_vcpu *vcpu);
 
 void kvm_timer_init_vhe(void);
diff --git a/virt/kvm/arm/arch_timer.c b/virt/kvm/arm/arch_timer.c
index 33257b5..35d7100 100644
--- a/virt/kvm/arm/arch_timer.c
+++ b/virt/kvm/arm/arch_timer.c
@@ -40,7 +40,7 @@ void kvm_timer_vcpu_put(struct kvm_vcpu *vcpu)
 	vcpu_vtimer(vcpu)->active_cleared_last = false;
 }
 
-static u64 kvm_phys_timer_read(void)
+u64 kvm_phys_timer_read(void)
 {
 	return timecounter->cc->read(timecounter->cc);
 }
-- 
1.9.1

[toc] | [prev] | [next] | [standalone]

#1573234

On 03/02/17 15:19, Jintack Lim wrote:
> The ARM architecture defines the EL1 physical timer and the virtual timer,
> and it is reasonable for an OS to expect to be able to access both.
> However, the current KVM implementation does not provide the EL1 physical
> timer to VMs but terminates VMs on access to the timer.
> 
> This patch series enables VMs to use the EL1 physical timer through
> trap-and-emulate only on arm64. The KVM host emulates each EL1 physical
> timer register access and sets up the background timer accordingly.  When
> the background timer expires, the KVM host injects EL1 physical timer
> interrupts to the VM.  Alternatively, it's also possible to allow VMs to
> access the EL1 physical timer without trapping.  However, this requires
> somehow using the EL2 physical timer for the Linux host while running the
> VM instead of the EL1 physical timer.  Right now I just implemented
> trap-and-emulate because this was straightforward to do, and I leave it to
> future work to determine if transferring the EL1 physical timer state to
> the EL2 timer provides any performance benefit.
> 
> This feature will be useful for any OS that wishes to access the EL1
> physical timer. Nested virtualization is one of those use cases. A nested
> hypervisor running inside a VM would think it has full access to the
> hardware and naturally tries to use the EL1 physical timer as Linux would
> do. Other nested hypervisors may try to use the EL2 physical timer as Xen
> would do, but supporting the EL2 physical timer to the VM is out of scope
> of this patch series. This patch series will make it easy to add the EL2
> timer support in the future, though.
> 
> Note that Linux VMs booting in EL1 will be unaffected by this patch series
> and will continue to use only the virtual timer and this patch series will
> therefore not introduce any performance degredation as a result of
> trap-and-emulate.
> 
> v3 => v4:
>  - Fix a bug that prevents a VM from booting on 32-bit architecture
>  - Clarify that the emulated physical timer is only supported on arm64
>    in the cover letter

Hi Jintack,

I've now applied this to queue, and will push it out later today.

Out of curiosity, is there any reason why this is arm64 only? As far as
I can tell, we're only missing the cp15 handling (both for arm and in
the 32bit handling in arm64).

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

[toc] | [prev] | [next] | [standalone]

#1573274

On Fri, Feb 3, 2017 at 11:14 AM, Marc Zyngier <marc.zyngier@arm.com> wrote:
> On 03/02/17 15:19, Jintack Lim wrote:
>> The ARM architecture defines the EL1 physical timer and the virtual timer,
>> and it is reasonable for an OS to expect to be able to access both.
>> However, the current KVM implementation does not provide the EL1 physical
>> timer to VMs but terminates VMs on access to the timer.
>>
>> This patch series enables VMs to use the EL1 physical timer through
>> trap-and-emulate only on arm64. The KVM host emulates each EL1 physical
>> timer register access and sets up the background timer accordingly.  When
>> the background timer expires, the KVM host injects EL1 physical timer
>> interrupts to the VM.  Alternatively, it's also possible to allow VMs to
>> access the EL1 physical timer without trapping.  However, this requires
>> somehow using the EL2 physical timer for the Linux host while running the
>> VM instead of the EL1 physical timer.  Right now I just implemented
>> trap-and-emulate because this was straightforward to do, and I leave it to
>> future work to determine if transferring the EL1 physical timer state to
>> the EL2 timer provides any performance benefit.
>>
>> This feature will be useful for any OS that wishes to access the EL1
>> physical timer. Nested virtualization is one of those use cases. A nested
>> hypervisor running inside a VM would think it has full access to the
>> hardware and naturally tries to use the EL1 physical timer as Linux would
>> do. Other nested hypervisors may try to use the EL2 physical timer as Xen
>> would do, but supporting the EL2 physical timer to the VM is out of scope
>> of this patch series. This patch series will make it easy to add the EL2
>> timer support in the future, though.
>>
>> Note that Linux VMs booting in EL1 will be unaffected by this patch series
>> and will continue to use only the virtual timer and this patch series will
>> therefore not introduce any performance degredation as a result of
>> trap-and-emulate.
>>
>> v3 => v4:
>>  - Fix a bug that prevents a VM from booting on 32-bit architecture
>>  - Clarify that the emulated physical timer is only supported on arm64
>>    in the cover letter
>
> Hi Jintack,
>
> I've now applied this to queue, and will push it out later today.

Thanks, Marc.

>
> Out of curiosity, is there any reason why this is arm64 only?

It was simply because I didn't have a convenient 32bit architecture
develop environment at hand and didn't spend time to set it up myself
:(
(As specified in the nesting RFC patch series cover letter, the
nesting patches are compiled, but not tested on 32-bit architecture
yet.)
I guess it's time to set it up.

> As far as
> I can tell, we're only missing the cp15 handling (both for arm and in
> the 32bit handling in arm64).

I think so, too. I can't promise when, but I'll try to add those once
I set the develop environment.

>
> Thanks,
>
>         M.
> --
> Jazz is not dead. It just smells funny...
>

[toc] | [prev] | [next] | [standalone]

#1573276

On 03/02/17 16:51, Jintack Lim wrote:
> On Fri, Feb 3, 2017 at 11:14 AM, Marc Zyngier <marc.zyngier@arm.com> wrote:
>> On 03/02/17 15:19, Jintack Lim wrote:
>>> The ARM architecture defines the EL1 physical timer and the virtual timer,
>>> and it is reasonable for an OS to expect to be able to access both.
>>> However, the current KVM implementation does not provide the EL1 physical
>>> timer to VMs but terminates VMs on access to the timer.
>>>
>>> This patch series enables VMs to use the EL1 physical timer through
>>> trap-and-emulate only on arm64. The KVM host emulates each EL1 physical
>>> timer register access and sets up the background timer accordingly.  When
>>> the background timer expires, the KVM host injects EL1 physical timer
>>> interrupts to the VM.  Alternatively, it's also possible to allow VMs to
>>> access the EL1 physical timer without trapping.  However, this requires
>>> somehow using the EL2 physical timer for the Linux host while running the
>>> VM instead of the EL1 physical timer.  Right now I just implemented
>>> trap-and-emulate because this was straightforward to do, and I leave it to
>>> future work to determine if transferring the EL1 physical timer state to
>>> the EL2 timer provides any performance benefit.
>>>
>>> This feature will be useful for any OS that wishes to access the EL1
>>> physical timer. Nested virtualization is one of those use cases. A nested
>>> hypervisor running inside a VM would think it has full access to the
>>> hardware and naturally tries to use the EL1 physical timer as Linux would
>>> do. Other nested hypervisors may try to use the EL2 physical timer as Xen
>>> would do, but supporting the EL2 physical timer to the VM is out of scope
>>> of this patch series. This patch series will make it easy to add the EL2
>>> timer support in the future, though.
>>>
>>> Note that Linux VMs booting in EL1 will be unaffected by this patch series
>>> and will continue to use only the virtual timer and this patch series will
>>> therefore not introduce any performance degredation as a result of
>>> trap-and-emulate.
>>>
>>> v3 => v4:
>>>  - Fix a bug that prevents a VM from booting on 32-bit architecture
>>>  - Clarify that the emulated physical timer is only supported on arm64
>>>    in the cover letter
>>
>> Hi Jintack,
>>
>> I've now applied this to queue, and will push it out later today.
> 
> Thanks, Marc.
> 
>>
>> Out of curiosity, is there any reason why this is arm64 only?
> 
> It was simply because I didn't have a convenient 32bit architecture
> develop environment at hand and didn't spend time to set it up myself
> :(
> (As specified in the nesting RFC patch series cover letter, the
> nesting patches are compiled, but not tested on 32-bit architecture
> yet.)
> I guess it's time to set it up.
> 
>> As far as
>> I can tell, we're only missing the cp15 handling (both for arm and in
>> the 32bit handling in arm64).
> 
> I think so, too. I can't promise when, but I'll try to add those once
> I set the develop environment.

That's fine, we can add these later (and maybe I'll just do it, since it
is pretty trivial).

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web