Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1345889 > unrolled thread

[PATCH 0/3] KVM: Fix lost IRQ acks for RTC

Back to article view | Back to linux.kernel

Contents

  [PATCH 0/3] KVM: Fix lost IRQ acks for RTC Joerg Roedel <joro@8bytes.org> - 2016-02-29 16:10 +0100
    [PATCH 2/3] kvm: x86: Track irq vectors in ioapic->rtc_status.dest_map Joerg Roedel <joro@8bytes.org> - 2016-02-29 16:10 +0100
    [PATCH 3/3] kvm: x86: Check dest_map->vector to match eoi signals for rtc Joerg Roedel <joro@8bytes.org> - 2016-02-29 16:10 +0100
    Re: [PATCH 0/3] KVM: Fix lost IRQ acks for RTC Paolo Bonzini <pbonzini@redhat.com> - 2016-02-29 16:20 +0100
      Re: [PATCH 0/3] KVM: Fix lost IRQ acks for RTC Joerg Roedel <joro@8bytes.org> - 2016-02-29 16:40 +0100
        Re: [PATCH 0/3] KVM: Fix lost IRQ acks for RTC Steve Rutherford <srutherford@google.com> - 2016-03-02 01:30 +0100

#1345889 — [PATCH 0/3] KVM: Fix lost IRQ acks for RTC

Hi,

here is a small patch-set to fix a race condition which
happens when an RTC-IRQ is migrated to another VCPU while it
is being handled by the guest.

The RTC-EOI handling in KVM requires that all sent interrupt
messages to the VCPUs need to be acked before another
RTC-IRQ can be sent. When an EOI signal from the guest is
lost, it will never see an RTC interrupt again (until it
reboots).

This is easily reproducible with a Linux guest executing
this loop:

	$ while true;do time hwclock --show --test --debug;done

When the guest has multiple vcpus and the RTC-IRQ is
regularily migrated (e.g. by irqbalance), the race condition
will be hit after some time and the hwclock tool will fail
with:

	select() to /dev/rtc to wait for clock tick timed out...synchronization failed

The race condition happens because of the way the EOI
backtracking between local APIC and IOAPIC works in KVM. The
destination VCPU and vector is part of the IOAPIC state.
When the guest sends an EOI to the local APIC the vector is
matched against the destinations stored in the IOAPIC and
ACKed there too if it matches.

The problem begins when a VCPU handles an RTC interrupt and
at the same time another VCPU migrates the RTC-IRQ away from
that VCPU. This updates the IOAPIC state in KVM to
the new destination, so that the EOI sent from the first
VCPU does not match anymore in the IOAPIC, hence losing the
RTC-EOI.

This patch-set fixes the race-condition by adding explicit
back-tracking information for RTC-IRQs. The rtc_status
struct already holds a dest_map bitmap to store which VCPUs
receveived an RTC-IRQ. This is extended to also hold the
vector that was sent to this VCPU.

This information is then used to match EOI signals from the
guest to the RTC. This explicit back-tracking fixes the
issue.

Regards,

	Joerg

Joerg Roedel (3):
  kvm: x86: Convert ioapic->rtc_status.dest_map to a struct
  kvm: x86: Track irq vectors in ioapic->rtc_status.dest_map
  kvm: x86: Check dest_map->vector to match eoi signals for rtc

 arch/x86/kvm/ioapic.c   | 30 +++++++++++++++++++++---------
 arch/x86/kvm/ioapic.h   | 17 +++++++++++++++--
 arch/x86/kvm/irq_comm.c |  2 +-
 arch/x86/kvm/lapic.c    | 14 ++++++++------
 arch/x86/kvm/lapic.h    |  7 +++++--
 5 files changed, 50 insertions(+), 20 deletions(-)

-- 
1.9.1

[toc] | [next] | [standalone]

#1345891 — [PATCH 2/3] kvm: x86: Track irq vectors in ioapic->rtc_status.dest_map

From: Joerg Roedel <jroedel@suse.de>

This allows backtracking later in case the rtc irq has been
moved to another vcpu/vector.

Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
 arch/x86/kvm/ioapic.h | 7 +++++++
 arch/x86/kvm/lapic.c  | 4 +++-
 2 files changed, 10 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kvm/ioapic.h b/arch/x86/kvm/ioapic.h
index af72989..7d2692a 100644
--- a/arch/x86/kvm/ioapic.h
+++ b/arch/x86/kvm/ioapic.h
@@ -41,7 +41,14 @@ struct kvm_vcpu;
 #endif
 
 struct dest_map {
+	/* vcpu bitmap where IRQ has been sent */
 	DECLARE_BITMAP(map, KVM_MAX_VCPUS);
+
+	/*
+	 * Vector sent to a given vcpu, only valid when
+	 * the vcpu's bit in map is set
+	 */
+	u8 vectors[KVM_MAX_VCPUS];
 };
 
 
diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c
index 5b78c97..5ea7ef0 100644
--- a/arch/x86/kvm/lapic.c
+++ b/arch/x86/kvm/lapic.c
@@ -839,8 +839,10 @@ static int __apic_accept_irq(struct kvm_lapic *apic, int delivery_mode,
 
 		result = 1;
 
-		if (dest_map)
+		if (dest_map) {
 			__set_bit(vcpu->vcpu_id, dest_map->map);
+			dest_map->vectors[vcpu->vcpu_id] = vector;
+		}
 
 		if (apic_test_vector(vector, apic->regs + APIC_TMR) != !!trig_mode) {
 			if (trig_mode)
-- 
1.9.1

[toc] | [prev] | [next] | [standalone]

#1345893 — [PATCH 3/3] kvm: x86: Check dest_map->vector to match eoi signals for rtc

From: Joerg Roedel <jroedel@suse.de>

Using the vector stored at interrupt delivery makes the eoi
matching safe agains irq migration in the ioapic.

Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
 arch/x86/kvm/ioapic.c | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)

diff --git a/arch/x86/kvm/ioapic.c b/arch/x86/kvm/ioapic.c
index f2c9906..9db4709 100644
--- a/arch/x86/kvm/ioapic.c
+++ b/arch/x86/kvm/ioapic.c
@@ -237,10 +237,17 @@ static void kvm_ioapic_inject_all(struct kvm_ioapic *ioapic, unsigned long irr)
 void kvm_ioapic_scan_entry(struct kvm_vcpu *vcpu, ulong *ioapic_handled_vectors)
 {
 	struct kvm_ioapic *ioapic = vcpu->kvm->arch.vioapic;
+	struct dest_map *dest_map = &ioapic->rtc_status.dest_map;
 	union kvm_ioapic_redirect_entry *e;
 	int index;
 
 	spin_lock(&ioapic->lock);
+
+	/* Make sure we see any missing RTC EOI */
+	if (test_bit(vcpu->vcpu_id, dest_map->map))
+		__set_bit(dest_map->vectors[vcpu->vcpu_id],
+			  ioapic_handled_vectors);
+
 	for (index = 0; index < IOAPIC_NUM_PINS; index++) {
 		e = &ioapic->redirtbl[index];
 		if (e->fields.trig_mode == IOAPIC_LEVEL_TRIG ||
@@ -408,8 +415,14 @@ static void kvm_ioapic_eoi_inject_work(struct work_struct *work)
 static void __kvm_ioapic_update_eoi(struct kvm_vcpu *vcpu,
 			struct kvm_ioapic *ioapic, int vector, int trigger_mode)
 {
-	int i;
+	struct dest_map *dest_map = &ioapic->rtc_status.dest_map;
 	struct kvm_lapic *apic = vcpu->arch.apic;
+	int i;
+
+	/* RTC special handling */
+	if (test_bit(vcpu->vcpu_id, dest_map->map) &&
+	    vector == dest_map->vectors[vcpu->vcpu_id])
+		rtc_irq_eoi(ioapic, vcpu);
 
 	for (i = 0; i < IOAPIC_NUM_PINS; i++) {
 		union kvm_ioapic_redirect_entry *ent = &ioapic->redirtbl[i];
@@ -417,8 +430,6 @@ static void __kvm_ioapic_update_eoi(struct kvm_vcpu *vcpu,
 		if (ent->fields.vector != vector)
 			continue;
 
-		if (i == RTC_GSI)
-			rtc_irq_eoi(ioapic, vcpu);
 		/*
 		 * We are dropping lock while calling ack notifiers because ack
 		 * notifier callbacks for assigned devices call into IOAPIC
-- 
1.9.1

[toc] | [prev] | [next] | [standalone]

#1345896

On 29/02/2016 16:04, Joerg Roedel wrote:
> Hi,
> 
> here is a small patch-set to fix a race condition which
> happens when an RTC-IRQ is migrated to another VCPU while it
> is being handled by the guest.
> 
> The RTC-EOI handling in KVM requires that all sent interrupt
> messages to the VCPUs need to be acked before another
> RTC-IRQ can be sent. When an EOI signal from the guest is
> lost, it will never see an RTC interrupt again (until it
> reboots).
> 
> This is easily reproducible with a Linux guest executing
> this loop:
> 
> 	$ while true;do time hwclock --show --test --debug;done
> 
> When the guest has multiple vcpus and the RTC-IRQ is
> regularily migrated (e.g. by irqbalance), the race condition
> will be hit after some time and the hwclock tool will fail
> with:
> 
> 	select() to /dev/rtc to wait for clock tick timed out...synchronization failed
> 
> The race condition happens because of the way the EOI
> backtracking between local APIC and IOAPIC works in KVM. The
> destination VCPU and vector is part of the IOAPIC state.
> When the guest sends an EOI to the local APIC the vector is
> matched against the destinations stored in the IOAPIC and
> ACKed there too if it matches.
> 
> The problem begins when a VCPU handles an RTC interrupt and
> at the same time another VCPU migrates the RTC-IRQ away from
> that VCPU. This updates the IOAPIC state in KVM to
> the new destination, so that the EOI sent from the first
> VCPU does not match anymore in the IOAPIC, hence losing the
> RTC-EOI.
> 
> This patch-set fixes the race-condition by adding explicit
> back-tracking information for RTC-IRQs. The rtc_status
> struct already holds a dest_map bitmap to store which VCPUs
> receveived an RTC-IRQ. This is extended to also hold the
> vector that was sent to this VCPU.
> 
> This information is then used to match EOI signals from the
> guest to the RTC. This explicit back-tracking fixes the
> issue.
> 
> Regards,

Nice patches, really.  Ok to wait until 4.6?

Paolo

[toc] | [prev] | [next] | [standalone]

#1345914

On Mon, Feb 29, 2016 at 04:12:42PM +0100, Paolo Bonzini wrote:
> > This information is then used to match EOI signals from the
> > guest to the RTC. This explicit back-tracking fixes the
> > issue.
> 
> Nice patches, really.  Ok to wait until 4.6?

Thanks. Putting them into v4.6 is fine for me.


	Joerg

[toc] | [prev] | [next] | [standalone]

#1347259

This issue seems generic to level triggered interrupts as well as RTC
interrupts. It looks like KVM hacks around the issue with level
triggered interrupts by clearing the remote IRR when an IRQ is
reconfigured. Seems like an (admittedly lossy) way to handle this
issue with the RTC-IRQ would be to follow the lead of level-triggered
interrupts, and clear the pending EOIs when reconfiguring the RTC-IRQ.

[Given that we are already talking about this, this could be viewed as
a good time to go back and fix the issues with the remote IRR in the
IOAPIC.]

On Mon, Feb 29, 2016 at 7:30 AM, Joerg Roedel <joro@8bytes.org> wrote:
> On Mon, Feb 29, 2016 at 04:12:42PM +0100, Paolo Bonzini wrote:
>> > This information is then used to match EOI signals from the
>> > guest to the RTC. This explicit back-tracking fixes the
>> > issue.
>>
>> Nice patches, really.  Ok to wait until 4.6?
>
> Thanks. Putting them into v4.6 is fine for me.
>
>
>         Joerg
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web