Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1576009 > unrolled thread

Re: [PATCH 6/6] kvm: x86: do not use KVM_REQ_EVENT for APICv interrupt injection

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH 6/6] kvm: x86: do not use KVM_REQ_EVENT for APICv  interrupt injection Radim Krčmář <rkrcmar@redhat.com> - 2017-02-07 21:10 +0100
    Re: [PATCH 6/6] kvm: x86: do not use KVM_REQ_EVENT for APICv  interrupt injection Paolo Bonzini <pbonzini@redhat.com> - 2017-02-08 18:30 +0100
      Re: [PATCH 6/6] kvm: x86: do not use KVM_REQ_EVENT for APICv  interrupt injection Radim Krčmář <rkrcmar@redhat.com> - 2017-02-09 16:20 +0100

#1576009 — Re: [PATCH 6/6] kvm: x86: do not use KVM_REQ_EVENT for APICv interrupt injection

2016-12-19 17:17+0100, Paolo Bonzini:
> Since bf9f6ac8d749 ("KVM: Update Posted-Interrupts Descriptor when vCPU
> is blocked", 2015-09-18) the posted interrupt descriptor is checked
> unconditionally for PIR.ON.  Therefore we don't need KVM_REQ_EVENT to
> trigger the scan and, if NMIs or SMIs are not involved, we can avoid
> the complicated event injection path.
> 
> Calling kvm_vcpu_kick if PIR.ON=1 is also useless, though it has been
> there since APICv was introduced.
> 
> However, without the KVM_REQ_EVENT safety net KVM needs to be much
> more careful about races between vmx_deliver_posted_interrupt and
> vcpu_enter_guest.  First, the IPI for posted interrupts may be issued
> between setting vcpu->mode = IN_GUEST_MODE and disabling interrupts.
> If that happens, kvm_trigger_posted_interrupt returns true, but
> smp_kvm_posted_intr_ipi doesn't do anything about it.  The guest is
> entered with PIR.ON, but the posted interrupt IPI has not been sent
> and the interrupt is only delivered to the guest on the next vmentry
> (if any).  To fix this, disable interrupts before setting vcpu->mode.
> This ensures that the IPI is delayed until the guest enters non-root mode;
> it is then trapped by the processor causing the interrupt to be injected.
> 
> Second, the IPI may be issued between
> 
>                         kvm_x86_ops->hwapic_irr_update(vcpu,
>                                 kvm_lapic_find_highest_irr(vcpu));
> 
> and vcpu->mode = IN_GUEST_MODE.  In this case, kvm_vcpu_kick is called
> but it (correctly) doesn't do anything because it sees vcpu->mode ==
> OUTSIDE_GUEST_MODE.  Again, the guest is entered with PIR.ON but no
> posted interrupt IPI is pending; this time, the fix for this is to move
> the RVI update after IN_GUEST_MODE.
> 
> Both issues were previously masked by the liberal usage of KVM_REQ_EVENT.
> In both race scenarios KVM_REQ_EVENT would cancel guest entry, resulting
> in another vmentry which would inject the interrupt.
> 
> This saves about 300 cycles on the self_ipi_* tests of vmexit.flat.

Please mention that this also fixes an existing problem with posted
interrupts from devices.  If we didn't check PIR.ON after disabling host
interrupts, we might delay delivery to the next VM exit or posted
interrupt.  (It was recently posted.)

> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> @@ -6767,20 +6754,39 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
>  	kvm_x86_ops->prepare_guest_switch(vcpu);
>  	if (vcpu->fpu_active)
>  		kvm_load_guest_fpu(vcpu);
> +
> +	/*
> +	 * Disabling IRQs before setting IN_GUEST_MODE.  Posted interrupt
> +	 * IPI are then delayed after guest entry, which ensures that they
> +	 * result in virtual interrupt delivery.
> +	 */
> +	local_irq_disable();
>  	vcpu->mode = IN_GUEST_MODE;
>  
>  	srcu_read_unlock(&vcpu->kvm->srcu, vcpu->srcu_idx);
>  
>  	/*
> -	 * We should set ->mode before check ->requests,
> -	 * Please see the comment in kvm_make_all_cpus_request.
> -	 * This also orders the write to mode from any reads
> -	 * to the page tables done while the VCPU is running.
> -	 * Please see the comment in kvm_flush_remote_tlbs.
> +	 * 1) We should set ->mode before checking ->requests.  Please see
> +	 * the comment in kvm_make_all_cpus_request.
> +	 *
> +	 * 2) For APICv, we should set ->mode before checking PIR.ON.  This
> +	 * pairs with the memory barrier implicit in pi_test_and_set_on
> +	 * (see vmx_deliver_posted_interrupt).
> +	 *
> +	 * 3) This also orders the write to mode from any reads to the page
> +	 * tables done while the VCPU is running.  Please see the comment
> +	 * in kvm_flush_remote_tlbs.
>  	 */
>  	smp_mb__after_srcu_read_unlock();
> -	local_irq_disable();
> +	if (kvm_lapic_enabled(vcpu)) {
> +		/*
> +		 * This handles the case where a posted interrupt was
> +		 * notified with kvm_vcpu_kick.
> +		 */
> +		if (kvm_x86_ops->sync_pir_to_irr)
> +			kvm_x86_ops->sync_pir_to_irr(vcpu);

Hm, this is not working well when nesting while L1 has assigned devices:
if the posted interrupt arrives just before local_irq_disable(), then
we'll just enter L2 instead of doing a nested VM exit (in case we have
interrupt exiting).

And after reading the code a bit, I think we allow posted interrupts in
L2 while L1 has assigned devices that use posted interrupts, and that it
doesn't work.

Am I missing something?

Thanks.

[toc] | [next] | [standalone]

#1576754

On 07/02/2017 20:58, Radim Krčmář wrote:
>> -	local_irq_disable();
>> +	if (kvm_lapic_enabled(vcpu)) {
>> +		/*
>> +		 * This handles the case where a posted interrupt was
>> +		 * notified with kvm_vcpu_kick.
>> +		 */
>> +		if (kvm_x86_ops->sync_pir_to_irr)
>> +			kvm_x86_ops->sync_pir_to_irr(vcpu);
> Hm, this is not working well when nesting while L1 has assigned devices:
> if the posted interrupt arrives just before local_irq_disable(), then
> we'll just enter L2 instead of doing a nested VM exit (in case we have
> interrupt exiting).
> 
> And after reading the code a bit, I think we allow posted interrupts in
> L2 while L1 has assigned devices that use posted interrupts, and that it
> doesn't work.

So you mean the interrupt is delivered to L2?  The fix would be to wrap
L2 entry and exit with some subset of pi_pre_block/pi_post_block.

Paolo

[toc] | [prev] | [next] | [standalone]

#1577679

2017-02-08 17:23+0100, Paolo Bonzini:
> On 07/02/2017 20:58, Radim Krčmář wrote:
>>> -	local_irq_disable();
>>> +	if (kvm_lapic_enabled(vcpu)) {
>>> +		/*
>>> +		 * This handles the case where a posted interrupt was
>>> +		 * notified with kvm_vcpu_kick.
>>> +		 */
>>> +		if (kvm_x86_ops->sync_pir_to_irr)
>>> +			kvm_x86_ops->sync_pir_to_irr(vcpu);
>> Hm, this is not working well when nesting while L1 has assigned devices:
>> if the posted interrupt arrives just before local_irq_disable(), then
>> we'll just enter L2 instead of doing a nested VM exit (in case we have
>> interrupt exiting).
>> 
>> And after reading the code a bit, I think we allow posted interrupts in
>> L2 while L1 has assigned devices that use posted interrupts, and that it
>> doesn't work.
> 
> So you mean the interrupt is delivered to L2?  The fix would be to wrap
> L2 entry and exit with some subset of pi_pre_block/pi_post_block.

I hope not, as their PI strucutres are separate, so we'd be just
delaying the interrupt injection to L1.  The CPU running L2 guest will
notice a posted notification, but its PIR.ON will/might not be set.
L1's PIR.ON will be set, but no-one is going to care until the next VM
exit.

I'll add some unit tests to check that I understood the bug correctly.

Changing the notification vector for L2 would be an ok solution.
We'd reserve a new vector in L0 and check L1's interrupts.  If it were
targetting a VCPU that is currently in L2 with a notification vector
configured for L2, we'd translate that vector into the notification
vector we set for L2 -- L1 could then post interrupts to L2 without a VM
exit.  And "posted" interrupts for L1 while in L2 would trigger a VM
exit, because the notification vector would be different.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web