[PATCH v3 57/59] KVM: arm/arm64: GICv4: Theory of operations

From	Marc Zyngier <marc.zyngier@arm.com>
Newsgroups	linux.kernel
Subject	[PATCH v3 57/59] KVM: arm/arm64: GICv4: Theory of operations
Date	2017-07-31 19:40 +0200
Message-ID	<u9mPF-H2-47@gated-at.bofh.it> (permalink)
References	<u9mFX-DO-3@gated-at.bofh.it>
Organization	linux.* mail to news gateway

Show all headers | View raw

Yet another braindump so I can free some cells...

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 virt/kvm/arm/vgic/vgic-v4.c | 68 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 68 insertions(+)

diff --git a/virt/kvm/arm/vgic/vgic-v4.c b/virt/kvm/arm/vgic/vgic-v4.c
index 0a8deefbcf1c..0c002d2be620 100644
--- a/virt/kvm/arm/vgic/vgic-v4.c
+++ b/virt/kvm/arm/vgic/vgic-v4.c
@@ -22,6 +22,74 @@
 
 #include "vgic.h"
 
+/*
+ * How KVM uses GICv4 (insert rude comments here):
+ *
+ * The vgic-v4 layer acts as a bridge between several entities:
+ * - The GICv4 ITS representation offered by the ITS driver
+ * - VFIO, which is in charge of the PCI endpoint
+ * - The virtual ITS, which is the only thing the guest sees
+ *
+ * The configuration of VLPIs is triggered by a callback from VFIO,
+ * instructing KVM that a PCI device has been configured to deliver
+ * MSIs to a vITS.
+ *
+ * kvm_vgic_v4_set_forwarding() is thus called with the routing entry,
+ * and this is used to find the corresponding vITS data structures
+ * (ITS instance, device, event and irq) using a process that is
+ * extremely similar to the injection of an MSI.
+ *
+ * At this stage, we can link the guest's view of an LPI (uniquely
+ * identified by the routing entry) and the host irq, using the GICv4
+ * driver mapping operation. Should the mapping succeed, we've then
+ * successfully upgraded the guest's LPI to a VLPI. We can then start
+ * with updating GICv4's view of the property table and generating an
+ * INValidation in order to kickstart the delivery of this VLPI to the
+ * guest directly, without software intervention. Well, almost.
+ *
+ * When the PCI endpoint is deconfigured, this operation is reversed
+ * with VFIO calling kvm_vgic_v4_unset_forwarding().
+ *
+ * Once the VLPI has been mapped, it needs to follow any change the
+ * guest performs on its LPI through the vITS. For that, a number of
+ * command handlers have hooks to communicate these changes to the HW:
+ * - Any invalidation triggers a call to its_prop_update_vlpi()
+ * - The INT command results in a irq_set_irqchip_state(), which
+ *   generates an INT on the corresponding VLPI.
+ * - The CLEAR command results in a irq_set_irqchip_state(), which
+ *   generates an CLEAR on the corresponding VLPI.
+ * - DISCARD translates into an unmap, similar to a call to
+ *   kvm_vgic_v4_unset_forwarding().
+ * - MOVI is translated by an update of the existing mapping, changing
+ *   the target vcpu, resulting in a VMOVI being generated.
+ * - MOVALL is translated by a string of mapping updates (similar to
+ *   the handling of MOVI). MOVALL is horrible.
+ *
+ * Note that a DISCARD/MAPTI sequence emitted from the guest without
+ * reprogramming the PCI endpoint after MAPTI does not result in a
+ * VLPI being mapped, as there is no callback from VFIO (the guest
+ * will get the interrupt via the normal SW injection). Fixing this is
+ * not trivial, and requires some horrible messing with the VFIO
+ * internals. Not fun. Don't do that.
+ *
+ * Then there is the scheduling. Each time a vcpu is about to run on a
+ * physical CPU, KVM must tell the corresponding redistributor about
+ * it. And if we've migrated our vcpu from one CPU to another, we must
+ * tell the ITS (so that the messages reach the right redistributor).
+ * This is done in two steps: first issue a irq_set_affinity() on the
+ * irq corresponding to the vcpu, then call its_schedule_vpe(). You
+ * must be in a non-preemptible context. On exit, another call to
+ * its_schedule_vpe() tells the redistributor that we're done with the
+ * vcpu.
+ *
+ * Finally, the doorbell handling: Each vcpu is allocated an interrupt
+ * which will fire each time a VLPI is made pending whilst the vcpu is
+ * not running. Each time the vcpu gets blocked, the doorbell
+ * interrupt gets enabled. When the vcpu is unblocked (for whatever
+ * reason), the doorbell interrupt is disabled. This behaviour is
+ * pretty similar to that of the backgroud timer.
+ */
+
 static irqreturn_t vgic_v4_doorbell_handler(int irq, void *info)
 {
 	struct kvm_vcpu *vcpu = info;
-- 
2.11.0

Thread

[PATCH v3 00/59] irqchip: KVM: Add support for GICv4 Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 19:30 +0200
  [PATCH v3 10/59] irqchip/gic-v3-its: Split out pending table allocation Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 19:30 +0200
  [PATCH v3 37/59] KVM: arm/arm64: vgic: Move kvm_vgic_destroy call around Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 19:30 +0200
  [PATCH v3 12/59] irqchip/gic-v3-its: Generalize device table allocation Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 19:30 +0200
  [PATCH v3 55/59] KVM: arm/arm64: GICv4: Enable VLPI support Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 19:30 +0200
  [PATCH v3 14/59] irqchip/gic-v4: Add management structure definitions Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 19:30 +0200
  [PATCH v3 54/59] KVM: arm/arm64: GICv4: Enable virtual cpuif if VLPIs can be delivered Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 19:30 +0200
  [PATCH v3 45/59] KVM: arm/arm64: GICv4: Handle CLEAR applied to a VLPI Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 19:30 +0200
  [PATCH v3 49/59] KVM: arm/arm64: GICv4: Propagate VLPI properties at map time Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 19:30 +0200
  [PATCH v3 59/59] irqchip/gic-v3-its: Workaround Huawei D05 redistributor addressing Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 19:30 +0200
  [PATCH v3 47/59] KVM: arm/arm64: GICv4: Propagate property updates to VLPIs Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 19:30 +0200
  [PATCH v3 05/59] irqchip/gic-v3-its: Add probing for VLPI properties Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 19:30 +0200
  [PATCH v3 29/59] irqchip/gic-v3-its: Set implementation defined bit to enable VLPIs Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 19:30 +0200
  [PATCH v3 33/59] irqchip/gic-v4: Add some basic documentation Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 19:30 +0200
  [PATCH v3 35/59] irqchip/gic-v3: Advertise GICv4 support to KVM Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 19:30 +0200
  [PATCH v3 41/59] KVM: arm/arm64: GICv4: Wire mapping/unmapping of VLPIs in VFIO irq bypass Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 19:30 +0200
  [PATCH v3 58/59] irqchip/gic-v3-its: Pass its_node pointer to each command bulder Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 19:30 +0200
  [PATCH v3 39/59] KVM: arm/arm64: GICv4: Add property field and per-VM predicate Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 19:30 +0200
  [PATCH v3 43/59] KVM: arm/arm64: GICv4: Unmap VLPI when freeing an LPI Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 19:30 +0200
  [PATCH v3 31/59] irqchip/gic-v4: Add VPE command interface Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 19:30 +0200
  [PATCH v3 24/59] irqchip/gic-v3-its: Add VPE invalidation hook Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 19:30 +0200
  [PATCH v3 19/59] irqchip/gic-v3-its: Add VPE domain infrastructure Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 19:30 +0200
  [PATCH v3 40/59] KVM: arm/arm64: GICv4: Add init/teardown of the per-VM vPE irq domain Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 19:30 +0200
  [PATCH v3 52/59] KVM: arm/arm64: GICv4: Use the doorbell interrupt as an unblocking source Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 19:40 +0200
  [PATCH v3 53/59] KVM: arm/arm64: GICv4: Hook vPE scheduling into vgic flush/sync Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 19:40 +0200
  [PATCH v3 44/59] KVM: arm/arm64: GICv4: Handle MOVI applied to a VLPI Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 19:40 +0200
  [PATCH v3 48/59] KVM: arm/arm64: GICv4: Handle INVALL applied to a vPE Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 19:40 +0200
  [PATCH v3 51/59] KVM: arm/arm64: GICv4: Add doorbell interrupt handling Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 19:40 +0200
    Re: [PATCH v3 51/59] KVM: arm/arm64: GICv4: Add doorbell interrupt  handling Marc Zyngier <marc.zyngier@arm.com> - 2017-08-04 09:50 +0200
  [PATCH v3 42/59] KVM: arm/arm64: GICv4: Handle INT command applied to a VLPI Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 19:40 +0200
  [PATCH v3 46/59] KVM: arm/arm64: GICv4: Handle MOVALL applied to a vPE Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 19:40 +0200
  [PATCH v3 56/59] KVM: arm/arm64: GICv4: Prevent heterogenous systems from using GICv4 Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 19:40 +0200
  [PATCH v3 50/59] KVM: arm/arm64: GICv4: Use pending_last as a scheduling hint Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 19:40 +0200
  [PATCH v3 57/59] KVM: arm/arm64: GICv4: Theory of operations Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 19:40 +0200
  [PATCH v3 38/59] KVM: arm/arm64: vITS: Add MSI translation helpers Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 20:00 +0200
  [PATCH v3 32/59] irqchip/gic-v4: Add VLPI configuration interface Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 20:00 +0200
  [PATCH v3 08/59] irqchip/gic-v3-its: Split out property table allocation Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 20:00 +0200
  [PATCH v3 30/59] irqchip/gic-v4: Add per-VM VPE domain creation Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 20:00 +0200
  [PATCH v3 36/59] KVM: arm: Select ARM_GIC_V3 and ARM_GIC_V3_ITS Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 20:00 +0200
  [PATCH v3 09/59] irqchip/gic-v3-its: Allow use of indirect VCPU tables Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 20:00 +0200
  [PATCH v3 13/59] irqchip/gic-v3-its: Generalize LPI configuration Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 20:00 +0200
  [PATCH v3 34/59] irqchip/gic-v4: Enable low-level GICv4 operations Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 20:00 +0200
  [PATCH v3 22/59] irqchip/gic-v3-its: Add VPENDBASER/VPROPBASER accessors Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 20:00 +0200
  [PATCH v3 15/59] irqchip/gic-v3-its: Add GICv4 ITS command definitions Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 20:00 +0200
  [PATCH v3 27/59] irqchip/gic-v3-its: Support VPE doorbell invalidation even when !DirectLPI Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 20:00 +0200
  [PATCH v3 26/59] irqchip/gic-v3-its: Add VPE interrupt masking Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 20:00 +0200
  [PATCH v3 16/59] irqchip/gic-v3-its: Add VLPI configuration hook Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 20:00 +0200
  [PATCH v3 28/59] irqchip/gic-v3-its: Allow doorbell interrupts to be injected/cleared Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 20:00 +0200
  [PATCH v3 06/59] irqchip/gic-v3-its: Macro-ize its_send_single_command Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 20:00 +0200
  [PATCH v3 25/59] irqchip/gic-v3-its: Add VPE affinity changes Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 20:00 +0200
  [PATCH v3 17/59] irqchip/gic-v3-its: Add VLPI map/unmap operations Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 20:00 +0200
  [PATCH v3 11/59] irqchip/gic-v3-its: Rework LPI freeing Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 20:00 +0200
  [PATCH v3 20/59] irqchip/gic-v3-its: Add VPE irq domain allocation/teardown Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 20:00 +0200
  [PATCH v3 18/59] irqchip/gic-v3-its: Add VLPI configuration handling Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 20:00 +0200
  [PATCH v3 23/59] irqchip/gic-v3-its: Add VPE scheduling Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 20:00 +0200
  [PATCH v3 02/59] irqchip/gic-v3: Add redistributor iterator Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 20:10 +0200
  [PATCH v3 01/59] genirq: Let irq_set_vcpu_affinity() iterate over hierarchy Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 20:10 +0200
  [PATCH v3 07/59] irqchip/gic-v3-its: Implement irq_set_irqchip_state for pending state Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 20:10 +0200
  [PATCH v3 03/59] irqchip/gic-v3: Add VLPI/DirectLPI discovery Marc Zyngier <marc.zyngier@arm.com> - 2017-07-31 20:10 +0200

csiph-web