Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1672249 > unrolled thread

[PATCH v4 0/4] KVM: async_pf: Fix async_pf exception injection

Started byWanpeng Li <kernellwp@gmail.com>
First post2017-06-22 04:10 +0200
Last post2017-06-28 13:50 +0200
Articles 5 — 2 participants

Back to article view | Back to linux.kernel

Contents

  [PATCH v4 0/4] KVM: async_pf: Fix async_pf exception injection Wanpeng Li <kernellwp@gmail.com> - 2017-06-22 04:10 +0200
    [PATCH v4 3/4] KVM: async_pf: Force a nested vmexit if the injected #PF is async_pf Wanpeng Li <kernellwp@gmail.com> - 2017-06-22 04:10 +0200
      Re: [PATCH v4 3/4] KVM: async_pf: Force a nested vmexit if the  injected #PF is async_pf Paolo Bonzini <pbonzini@redhat.com> - 2017-06-27 15:50 +0200
        Re: [PATCH v4 3/4] KVM: async_pf: Force a nested vmexit if the  injected #PF is async_pf Wanpeng Li <kernellwp@gmail.com> - 2017-06-28 00:40 +0200
          Re: [PATCH v4 3/4] KVM: async_pf: Force a nested vmexit if the  injected #PF is async_pf Paolo Bonzini <pbonzini@redhat.com> - 2017-06-28 13:50 +0200

#1672249 — [PATCH v4 0/4] KVM: async_pf: Fix async_pf exception injection

FromWanpeng Li <kernellwp@gmail.com>
Date2017-06-22 04:10 +0200
Subject[PATCH v4 0/4] KVM: async_pf: Fix async_pf exception injection
Message-ID<tUZJf-6uJ-3@gated-at.bofh.it>
 INFO: task gnome-terminal-:1734 blocked for more than 120 seconds.
       Not tainted 4.12.0-rc4+ #8
 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
 gnome-terminal- D    0  1734   1015 0x00000000
 Call Trace:
  __schedule+0x3cd/0xb30
  schedule+0x40/0x90
  kvm_async_pf_task_wait+0x1cc/0x270
  ? __vfs_read+0x37/0x150
  ? prepare_to_swait+0x22/0x70
  do_async_page_fault+0x77/0xb0
  ? do_async_page_fault+0x77/0xb0
  async_page_fault+0x28/0x30

This is triggered by running both win7 and win2016 on L1 KVM simultaneously, 
and then gives stress to memory on L1, I can observed this hang on L1 when 
at least ~70% swap area is occupied on L0.

This is due to async pf was injected to L2 which should be injected to L1, 
L2 guest starts receiving pagefault w/ bogus %cr2(apf token from the host 
actually), and L1 guest starts accumulating tasks stuck in D state in 
kvm_async_pf_task_wait() since missing PAGE_READY async_pfs.

This patchset fixes it according to Radim's proposal "force a nested VM exit 
from nested_vmx_check_exception if the injected #PF is async_pf and handle 
the #PF VM exit in L1". https://www.spinics.net/lists/kvm/msg142498.html

Note: The patchset almost not touch SVM since I don't have AMD CPU to verify
the modification.

v3 -> v4:
 * reuse pad field in kvm_vcpu_events for async_page_fault
 * update kvm_vcpu_events API documentations
 * change async_page_fault type in vcpu->arch.exception from bool to u8

v2 -> v3:
 * add the flag to the userspace interface(KVM_GET/PUT_VCPU_EVENTS)

v1 -> v2:
 * remove nested_vmx_check_exception nr parameter
 * construct a simple special vm-exit information field for async pf
 * introduce nested_apf_token to vcpu->arch.apf to avoid change the CR2 
   visible in L2 guest 
 * avoid pass the apf directed towards it (L1) into L2 if there is L3 
   at the moment

Wanpeng Li (4):
  KVM: x86: Simple kvm_x86_ops->queue_exception parameter
  KVM: async_pf: Add L1 guest async_pf #PF vmexit handler
  KVM: async_pf: Force a nested vmexit if the injected #PF is async_pf
  KVM: async_pf: Let host know whether the guest support delivery async_pf as #PF vmexit

 Documentation/virtual/kvm/api.txt     |  8 +++--
 Documentation/virtual/kvm/msr.txt     |  5 +--
 arch/x86/include/asm/kvm_emulate.h    |  1 +
 arch/x86/include/asm/kvm_host.h       |  7 ++--
 arch/x86/include/uapi/asm/kvm.h       |  3 +-
 arch/x86/include/uapi/asm/kvm_para.h  |  1 +
 arch/x86/kernel/kvm.c                 |  1 +
 arch/x86/kvm/mmu.c                    |  2 +-
 arch/x86/kvm/svm.c                    |  8 +++--
 arch/x86/kvm/vmx.c                    | 61 +++++++++++++++++++++++++++--------
 arch/x86/kvm/x86.c                    | 28 ++++++++++------
 tools/arch/x86/include/uapi/asm/kvm.h |  3 +-
 12 files changed, 92 insertions(+), 36 deletions(-)

-- 
2.7.4

[toc] | [next] | [standalone]

#1672252 — [PATCH v4 3/4] KVM: async_pf: Force a nested vmexit if the injected #PF is async_pf

FromWanpeng Li <kernellwp@gmail.com>
Date2017-06-22 04:10 +0200
Subject[PATCH v4 3/4] KVM: async_pf: Force a nested vmexit if the injected #PF is async_pf
Message-ID<tUZJf-6uJ-15@gated-at.bofh.it>
In reply to#1672249
From: Wanpeng Li <wanpeng.li@hotmail.com>

Add an async_page_fault field to vcpu->arch.exception to identify an async 
page fault, and constructs the expected vm-exit information fields. Force 
a nested VM exit from nested_vmx_check_exception() if the injected #PF 
is async page fault. Extending the userspace interface KVM_GET_VCPU_EVENTS
and KVM_SET_VCPU_EVENTS for live migration.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Radim Krčmář <rkrcmar@redhat.com>
Signed-off-by: Wanpeng Li <wanpeng.li@hotmail.com>
---
 Documentation/virtual/kvm/api.txt     |  8 ++++++--
 arch/x86/include/asm/kvm_emulate.h    |  1 +
 arch/x86/include/asm/kvm_host.h       |  2 ++
 arch/x86/include/uapi/asm/kvm.h       |  3 ++-
 arch/x86/kvm/vmx.c                    | 17 ++++++++++++++---
 arch/x86/kvm/x86.c                    | 18 ++++++++++++++----
 tools/arch/x86/include/uapi/asm/kvm.h |  3 ++-
 7 files changed, 41 insertions(+), 11 deletions(-)

diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
index 4029943..db32882 100644
--- a/Documentation/virtual/kvm/api.txt
+++ b/Documentation/virtual/kvm/api.txt
@@ -832,7 +832,7 @@ struct kvm_vcpu_events {
 		__u8 injected;
 		__u8 nr;
 		__u8 has_error_code;
-		__u8 pad;
+		__u8 async_page_fault;
 		__u32 error_code;
 	} exception;
 	struct {
@@ -857,7 +857,7 @@ struct kvm_vcpu_events {
 	} smi;
 };
 
-Only two fields are defined in the flags field:
+Only three fields are defined in the flags field:
 
 - KVM_VCPUEVENT_VALID_SHADOW may be set in the flags field to signal that
   interrupt.shadow contains a valid state.
@@ -865,6 +865,9 @@ Only two fields are defined in the flags field:
 - KVM_VCPUEVENT_VALID_SMM may be set in the flags field to signal that
   smi contains a valid state.
 
+- KVM_VCPUEVENT_VALID_ASYNC_PF may be set in the flags field to signal that
+  the exception is an async page fault.
+
 4.32 KVM_SET_VCPU_EVENTS
 
 Capability: KVM_CAP_VCPU_EVENTS
@@ -887,6 +890,7 @@ suppress overwriting the current in-kernel state. The bits are:
 KVM_VCPUEVENT_VALID_NMI_PENDING - transfer nmi.pending to the kernel
 KVM_VCPUEVENT_VALID_SIPI_VECTOR - transfer sipi_vector
 KVM_VCPUEVENT_VALID_SMM         - transfer the smi sub-struct.
+KVM_VCPUEVENT_VALID_ASYNC_PF    - transfer async page fault
 
 If KVM_CAP_INTR_SHADOW is available, KVM_VCPUEVENT_VALID_SHADOW can be set in
 the flags field to signal that interrupt.shadow contains a valid state and
diff --git a/arch/x86/include/asm/kvm_emulate.h b/arch/x86/include/asm/kvm_emulate.h
index 0559626..6d54c8f 100644
--- a/arch/x86/include/asm/kvm_emulate.h
+++ b/arch/x86/include/asm/kvm_emulate.h
@@ -23,6 +23,7 @@ struct x86_exception {
 	u16 error_code;
 	bool nested_page_fault;
 	u64 address; /* cr2 or nested page fault gpa */
+	u8 async_page_fault;
 };
 
 /*
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 1f01bfb..2e84cc4 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -545,6 +545,7 @@ struct kvm_vcpu_arch {
 		bool reinject;
 		u8 nr;
 		u32 error_code;
+		u8 async_page_fault;
 	} exception;
 
 	struct kvm_queued_interrupt {
@@ -645,6 +646,7 @@ struct kvm_vcpu_arch {
 		u64 msr_val;
 		u32 id;
 		bool send_user_only;
+		unsigned long nested_apf_token;
 	} apf;
 
 	/* OSVW MSRs (AMD only) */
diff --git a/arch/x86/include/uapi/asm/kvm.h b/arch/x86/include/uapi/asm/kvm.h
index c2824d0..27b368b 100644
--- a/arch/x86/include/uapi/asm/kvm.h
+++ b/arch/x86/include/uapi/asm/kvm.h
@@ -287,6 +287,7 @@ struct kvm_reinject_control {
 #define KVM_VCPUEVENT_VALID_SIPI_VECTOR	0x00000002
 #define KVM_VCPUEVENT_VALID_SHADOW	0x00000004
 #define KVM_VCPUEVENT_VALID_SMM		0x00000008
+#define KVM_VCPUEVENT_VALID_ASYNC_PF 0x00000010
 
 /* Interrupt shadow states */
 #define KVM_X86_SHADOW_INT_MOV_SS	0x01
@@ -298,7 +299,7 @@ struct kvm_vcpu_events {
 		__u8 injected;
 		__u8 nr;
 		__u8 has_error_code;
-		__u8 pad;
+		__u8 async_page_fault;
 		__u32 error_code;
 	} exception;
 	struct {
diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c
index f533cc1..e7b9844 100644
--- a/arch/x86/kvm/vmx.c
+++ b/arch/x86/kvm/vmx.c
@@ -2419,13 +2419,24 @@ static void skip_emulated_instruction(struct kvm_vcpu *vcpu)
  * KVM wants to inject page-faults which it got to the guest. This function
  * checks whether in a nested guest, we need to inject them to L1 or L2.
  */
-static int nested_vmx_check_exception(struct kvm_vcpu *vcpu, unsigned nr)
+static int nested_vmx_check_exception(struct kvm_vcpu *vcpu)
 {
 	struct vmcs12 *vmcs12 = get_vmcs12(vcpu);
+	unsigned int nr = vcpu->arch.exception.nr;
 
-	if (!(vmcs12->exception_bitmap & (1u << nr)))
+	if (!((vmcs12->exception_bitmap & (1u << nr)) ||
+		(nr == PF_VECTOR && vcpu->arch.exception.async_page_fault)))
 		return 0;
 
+	if (vcpu->arch.exception.async_page_fault) {
+		vmcs_write32(VM_EXIT_INTR_ERROR_CODE, vcpu->arch.exception.error_code);
+		nested_vmx_vmexit(vcpu, EXIT_REASON_EXCEPTION_NMI,
+			PF_VECTOR | INTR_TYPE_HARD_EXCEPTION |
+			INTR_INFO_DELIVER_CODE_MASK | INTR_INFO_VALID_MASK,
+			vcpu->arch.apf.nested_apf_token);
+		return 1;
+	}
+
 	nested_vmx_vmexit(vcpu, EXIT_REASON_EXCEPTION_NMI,
 			  vmcs_read32(VM_EXIT_INTR_INFO),
 			  vmcs_readl(EXIT_QUALIFICATION));
@@ -2442,7 +2453,7 @@ static void vmx_queue_exception(struct kvm_vcpu *vcpu)
 	u32 intr_info = nr | INTR_INFO_VALID_MASK;
 
 	if (!reinject && is_guest_mode(vcpu) &&
-	    nested_vmx_check_exception(vcpu, nr))
+	    nested_vmx_check_exception(vcpu))
 		return;
 
 	if (has_error_code) {
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 1b28a31..9a2410c 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -452,7 +452,11 @@ EXPORT_SYMBOL_GPL(kvm_complete_insn_gp);
 void kvm_inject_page_fault(struct kvm_vcpu *vcpu, struct x86_exception *fault)
 {
 	++vcpu->stat.pf_guest;
-	vcpu->arch.cr2 = fault->address;
+	vcpu->arch.exception.async_page_fault = fault->async_page_fault;
+	if (is_guest_mode(vcpu) && vcpu->arch.exception.async_page_fault)
+		vcpu->arch.apf.nested_apf_token = fault->address;
+	else
+		vcpu->arch.cr2 = fault->address;
 	kvm_queue_exception_e(vcpu, PF_VECTOR, fault->error_code);
 }
 EXPORT_SYMBOL_GPL(kvm_inject_page_fault);
@@ -3072,7 +3076,7 @@ static void kvm_vcpu_ioctl_x86_get_vcpu_events(struct kvm_vcpu *vcpu,
 		!kvm_exception_is_soft(vcpu->arch.exception.nr);
 	events->exception.nr = vcpu->arch.exception.nr;
 	events->exception.has_error_code = vcpu->arch.exception.has_error_code;
-	events->exception.pad = 0;
+	events->exception.async_page_fault = vcpu->arch.exception.async_page_fault;
 	events->exception.error_code = vcpu->arch.exception.error_code;
 
 	events->interrupt.injected =
@@ -3096,7 +3100,8 @@ static void kvm_vcpu_ioctl_x86_get_vcpu_events(struct kvm_vcpu *vcpu,
 
 	events->flags = (KVM_VCPUEVENT_VALID_NMI_PENDING
 			 | KVM_VCPUEVENT_VALID_SHADOW
-			 | KVM_VCPUEVENT_VALID_SMM);
+			 | KVM_VCPUEVENT_VALID_SMM
+			 | KVM_VCPUEVENT_VALID_ASYNC_PF);
 	memset(&events->reserved, 0, sizeof(events->reserved));
 }
 
@@ -3108,7 +3113,8 @@ static int kvm_vcpu_ioctl_x86_set_vcpu_events(struct kvm_vcpu *vcpu,
 	if (events->flags & ~(KVM_VCPUEVENT_VALID_NMI_PENDING
 			      | KVM_VCPUEVENT_VALID_SIPI_VECTOR
 			      | KVM_VCPUEVENT_VALID_SHADOW
-			      | KVM_VCPUEVENT_VALID_SMM))
+			      | KVM_VCPUEVENT_VALID_SMM
+			      | KVM_VCPUEVENT_VALID_ASYNC_PF))
 		return -EINVAL;
 
 	if (events->exception.injected &&
@@ -3126,6 +3132,8 @@ static int kvm_vcpu_ioctl_x86_set_vcpu_events(struct kvm_vcpu *vcpu,
 	vcpu->arch.exception.pending = events->exception.injected;
 	vcpu->arch.exception.nr = events->exception.nr;
 	vcpu->arch.exception.has_error_code = events->exception.has_error_code;
+	if (events->flags & KVM_VCPUEVENT_VALID_ASYNC_PF)
+		vcpu->arch.exception.async_page_fault = events->exception.async_page_fault;
 	vcpu->arch.exception.error_code = events->exception.error_code;
 
 	vcpu->arch.interrupt.pending = events->interrupt.injected;
@@ -8571,6 +8579,7 @@ void kvm_arch_async_page_not_present(struct kvm_vcpu *vcpu,
 		fault.error_code = 0;
 		fault.nested_page_fault = false;
 		fault.address = work->arch.token;
+		fault.async_page_fault = true;
 		kvm_inject_page_fault(vcpu, &fault);
 	}
 }
@@ -8593,6 +8602,7 @@ void kvm_arch_async_page_present(struct kvm_vcpu *vcpu,
 		fault.error_code = 0;
 		fault.nested_page_fault = false;
 		fault.address = work->arch.token;
+		fault.async_page_fault = true;
 		kvm_inject_page_fault(vcpu, &fault);
 	}
 	vcpu->arch.apf.halted = false;
diff --git a/tools/arch/x86/include/uapi/asm/kvm.h b/tools/arch/x86/include/uapi/asm/kvm.h
index c2824d0..27b368b 100644
--- a/tools/arch/x86/include/uapi/asm/kvm.h
+++ b/tools/arch/x86/include/uapi/asm/kvm.h
@@ -287,6 +287,7 @@ struct kvm_reinject_control {
 #define KVM_VCPUEVENT_VALID_SIPI_VECTOR	0x00000002
 #define KVM_VCPUEVENT_VALID_SHADOW	0x00000004
 #define KVM_VCPUEVENT_VALID_SMM		0x00000008
+#define KVM_VCPUEVENT_VALID_ASYNC_PF 0x00000010
 
 /* Interrupt shadow states */
 #define KVM_X86_SHADOW_INT_MOV_SS	0x01
@@ -298,7 +299,7 @@ struct kvm_vcpu_events {
 		__u8 injected;
 		__u8 nr;
 		__u8 has_error_code;
-		__u8 pad;
+		__u8 async_page_fault;
 		__u32 error_code;
 	} exception;
 	struct {
-- 
2.7.4

[toc] | [prev] | [next] | [standalone]

#1675711 — Re: [PATCH v4 3/4] KVM: async_pf: Force a nested vmexit if the injected #PF is async_pf

FromPaolo Bonzini <pbonzini@redhat.com>
Date2017-06-27 15:50 +0200
SubjectRe: [PATCH v4 3/4] KVM: async_pf: Force a nested vmexit if the injected #PF is async_pf
Message-ID<tWZ2q-1ch-5@gated-at.bofh.it>
In reply to#1672252

On 22/06/2017 04:06, Wanpeng Li wrote:
> From: Wanpeng Li <wanpeng.li@hotmail.com>
> 
> Add an async_page_fault field to vcpu->arch.exception to identify an async 
> page fault, and constructs the expected vm-exit information fields. Force 
> a nested VM exit from nested_vmx_check_exception() if the injected #PF 
> is async page fault. Extending the userspace interface KVM_GET_VCPU_EVENTS
> and KVM_SET_VCPU_EVENTS for live migration.

I am not sure what would happen if new kernel (that can produce
async_page_fault=1) runs on top of old userspace (that can consume it).

I think it would be safer to make the new field "nested_apf", and only
set it if in guest_mode, like

	vcpu->arch.exception.nested_apf =
		is_guest_mode(vcpu) && fault->async_page_fault;
	if (vcpu->arch.exception.nested_apf)
		vcpu->arch.apf.nested_apf_token = fault->address;
	else
		vcpu->arch.cr2 = fault->address;

What do you think?

Paolo

[toc] | [prev] | [next] | [standalone]

#1676224 — Re: [PATCH v4 3/4] KVM: async_pf: Force a nested vmexit if the injected #PF is async_pf

FromWanpeng Li <kernellwp@gmail.com>
Date2017-06-28 00:40 +0200
SubjectRe: [PATCH v4 3/4] KVM: async_pf: Force a nested vmexit if the injected #PF is async_pf
Message-ID<tX7jk-6UY-7@gated-at.bofh.it>
In reply to#1675711
2017-06-27 21:40 GMT+08:00 Paolo Bonzini <pbonzini@redhat.com>:
>
>
> On 22/06/2017 04:06, Wanpeng Li wrote:
>> From: Wanpeng Li <wanpeng.li@hotmail.com>
>>
>> Add an async_page_fault field to vcpu->arch.exception to identify an async
>> page fault, and constructs the expected vm-exit information fields. Force
>> a nested VM exit from nested_vmx_check_exception() if the injected #PF
>> is async page fault. Extending the userspace interface KVM_GET_VCPU_EVENTS
>> and KVM_SET_VCPU_EVENTS for live migration.
>
> I am not sure what would happen if new kernel (that can produce
> async_page_fault=1) runs on top of old userspace (that can consume it).
>
> I think it would be safer to make the new field "nested_apf", and only
> set it if in guest_mode, like
>
>         vcpu->arch.exception.nested_apf =
>                 is_guest_mode(vcpu) && fault->async_page_fault;
>         if (vcpu->arch.exception.nested_apf)
>                 vcpu->arch.apf.nested_apf_token = fault->address;
>         else
>                 vcpu->arch.cr2 = fault->address;

I have already added the same logic in kvm_inject_page_fault in patch
3/4, in addition, there is a guarantee it is in guest mode when we set
svm->vmcb->control.xxxx in nested_svm_check_exception, how about just
as what we do in nested_vmx_check_exception?

+ if (svm->vcpu.arch.exception.async_page_fault)
+     svm->vmcb->control.exit_info_2 = svm->vcpu.arch.apf.nested_apf_token;
+ else
+     svm->vmcb->control.exit_info_2 = svm->vcpu.arch.cr2;

Regards,
Wanpeng Li

[toc] | [prev] | [next] | [standalone]

#1676576 — Re: [PATCH v4 3/4] KVM: async_pf: Force a nested vmexit if the injected #PF is async_pf

FromPaolo Bonzini <pbonzini@redhat.com>
Date2017-06-28 13:50 +0200
SubjectRe: [PATCH v4 3/4] KVM: async_pf: Force a nested vmexit if the injected #PF is async_pf
Message-ID<tXjDQ-6i4-15@gated-at.bofh.it>
In reply to#1676224

On 28/06/2017 00:33, Wanpeng Li wrote:
> I have already added the same logic in kvm_inject_page_fault in patch
> 3/4, in addition, there is a guarantee it is in guest mode when we set
> svm->vmcb->control.xxxx in nested_svm_check_exception, how about just
> as what we do in nested_vmx_check_exception?
> 
> + if (svm->vcpu.arch.exception.async_page_fault)
> +     svm->vmcb->control.exit_info_2 = svm->vcpu.arch.apf.nested_apf_token;
> + else
> +     svm->vmcb->control.exit_info_2 = svm->vcpu.arch.cr2;

Yes, see my reply to [PATCH v5 3/4].

What do you think of moving "is_guest_mode(vcpu) &&" from this "if":

> 
>  void kvm_inject_page_fault(struct kvm_vcpu *vcpu, struct x86_exception *fault)
>  {
>  	++vcpu->stat.pf_guest;
> -	vcpu->arch.cr2 = fault->address;
> +	vcpu->arch.exception.async_page_fault = fault->async_page_fault;
> +	if (is_guest_mode(vcpu) && vcpu->arch.exception.async_page_fault)
> +		vcpu->arch.apf.nested_apf_token = fault->address;


to the previous assignment of vcpu->arch.exception.async_page_fault (so
that it can be renamed to nested_apf)?

Thanks,

Paolo

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web