Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1433119 > unrolled thread

[PATCH linux 0/8] xen: pvhvm: support bootup on secondary vCPUs

Back to article view | Back to linux.kernel

Contents

  [PATCH linux 0/8] xen: pvhvm: support bootup on secondary vCPUs Vitaly Kuznetsov <vkuznets@redhat.com> - 2016-06-28 18:50 +0200
    [PATCH linux 5/8] xen/events: use xen_vcpu_id mapping in events_base Vitaly Kuznetsov <vkuznets@redhat.com> - 2016-06-28 19:00 +0200
    [PATCH linux 7/8] xen/evtchn: use xen_vcpu_id mapping Vitaly Kuznetsov <vkuznets@redhat.com> - 2016-06-28 19:00 +0200
    [PATCH linux 4/8] x86/xen: use xen_vcpu_id mapping when pointing vcpu_info to the shared_info page Vitaly Kuznetsov <vkuznets@redhat.com> - 2016-06-28 19:00 +0200
      Re: [Xen-devel] [PATCH linux 4/8] x86/xen: use xen_vcpu_id mapping  when pointing vcpu_info to the shared_info page David Vrabel <david.vrabel@citrix.com> - 2016-06-29 14:40 +0200
    [PATCH linux 8/8] xen/pvhvm: run xen_vcpu_setup() for the boot CPU Vitaly Kuznetsov <vkuznets@redhat.com> - 2016-06-28 19:00 +0200
    [PATCH linux 6/8] xen/events: fifo: use xen_vcpu_id mapping Vitaly Kuznetsov <vkuznets@redhat.com> - 2016-06-28 19:00 +0200
    Re: [PATCH linux 0/8] xen: pvhvm: support bootup on secondary vCPUs David Vrabel <david.vrabel@citrix.com> - 2016-06-28 19:40 +0200
      Re: [PATCH linux 0/8] xen: pvhvm: support bootup on secondary vCPUs Vitaly Kuznetsov <vkuznets@redhat.com> - 2016-06-29 11:20 +0200
        Re: [PATCH linux 0/8] xen: pvhvm: support bootup on secondary vCPUs David Vrabel <david.vrabel@citrix.com> - 2016-06-29 14:40 +0200
          Re: [PATCH linux 0/8] xen: pvhvm: support bootup on secondary vCPUs Vitaly Kuznetsov <vkuznets@redhat.com> - 2016-07-01 11:50 +0200

#1433119 — [PATCH linux 0/8] xen: pvhvm: support bootup on secondary vCPUs

It may happen that Xen's and Linux's ideas of vCPU id diverge. In
particular, when we crash on a secondary vCPU we may want to do kdump
and unlike plain kexec where we do migrate_to_reboot_cpu() we try booting
on the vCPU which crashed. This doesn't work very well for PVHVM guests as
we have a number of hypercalls where we pass vCPU id as a parameter. These
hypercalls either fail or do something unexpected. To solve the issue we
need to have a mapping between Linux's and Xen's vCPU ids.

This series solves the issue for x86 PVHVM guests. PV guests don't (and
probably won't) support kdump so I always assume Xen's vCPU id == Linux's
vCPU id. ARM guests will probably need to get proper mapping once we start
supporting kexec/kdump there.

Vitaly Kuznetsov (8):
  x86/xen: update cpuid.h from Xen-4.7
  xen: introduce xen_vcpu_id mapping
  x86/xen: use xen_vcpu_id mapping for HYPERVISOR_vcpu_op
  x86/xen: use xen_vcpu_id mapping when pointing vcpu_info to the
    shared_info page
  xen/events: use xen_vcpu_id mapping in events_base
  xen/events: fifo: use xen_vcpu_id mapping
  xen/evtchn: use xen_vcpu_id mapping
  xen/pvhvm: run xen_vcpu_setup() for the boot CPU

 arch/arm/xen/enlighten.c             | 10 ++++++++++
 arch/x86/include/asm/xen/cpuid.h     |  5 ++++-
 arch/x86/include/asm/xen/hypercall.h | 11 ++++++++++-
 arch/x86/xen/enlighten.c             | 35 ++++++++++++++++++++++++++++-------
 arch/x86/xen/smp.c                   |  7 +++++++
 arch/x86/xen/xen-ops.h               |  1 +
 drivers/xen/events/events_base.c     | 10 +++++-----
 drivers/xen/events/events_fifo.c     |  2 +-
 drivers/xen/evtchn.c                 |  2 +-
 include/xen/xen-ops.h                |  1 +
 10 files changed, 68 insertions(+), 16 deletions(-)

-- 
2.5.5

[toc] | [next] | [standalone]

#1433120 — [PATCH linux 5/8] xen/events: use xen_vcpu_id mapping in events_base

EVTCHNOP_bind_ipi and EVTCHNOP_bind_virq pass vCPU id as a parameter and
Xen's idea of vCPU id should be used. Use the newly introduced xen_vcpu_id
mapping to convert it from Linux's id.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
 drivers/xen/events/events_base.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/drivers/xen/events/events_base.c b/drivers/xen/events/events_base.c
index 71d49a9..73b8b65 100644
--- a/drivers/xen/events/events_base.c
+++ b/drivers/xen/events/events_base.c
@@ -895,7 +895,7 @@ static int bind_ipi_to_irq(unsigned int ipi, unsigned int cpu)
 		irq_set_chip_and_handler_name(irq, &xen_percpu_chip,
 					      handle_percpu_irq, "ipi");
 
-		bind_ipi.vcpu = cpu;
+		bind_ipi.vcpu = per_cpu(xen_vcpu_id, cpu);
 		if (HYPERVISOR_event_channel_op(EVTCHNOP_bind_ipi,
 						&bind_ipi) != 0)
 			BUG();
@@ -991,7 +991,7 @@ int bind_virq_to_irq(unsigned int virq, unsigned int cpu, bool percpu)
 						      handle_edge_irq, "virq");
 
 		bind_virq.virq = virq;
-		bind_virq.vcpu = cpu;
+		bind_virq.vcpu = per_cpu(xen_vcpu_id, cpu);
 		ret = HYPERVISOR_event_channel_op(EVTCHNOP_bind_virq,
 						&bind_virq);
 		if (ret == 0)
@@ -1318,7 +1318,7 @@ static int rebind_irq_to_cpu(unsigned irq, unsigned tcpu)
 
 	/* Send future instances of this interrupt to other vcpu. */
 	bind_vcpu.port = evtchn;
-	bind_vcpu.vcpu = tcpu;
+	bind_vcpu.vcpu = per_cpu(xen_vcpu_id, tcpu);
 
 	/*
 	 * Mask the event while changing the VCPU binding to prevent
@@ -1458,7 +1458,7 @@ static void restore_cpu_virqs(unsigned int cpu)
 
 		/* Get a new binding from Xen. */
 		bind_virq.virq = virq;
-		bind_virq.vcpu = cpu;
+		bind_virq.vcpu = per_cpu(xen_vcpu_id, cpu);
 		if (HYPERVISOR_event_channel_op(EVTCHNOP_bind_virq,
 						&bind_virq) != 0)
 			BUG();
@@ -1482,7 +1482,7 @@ static void restore_cpu_ipis(unsigned int cpu)
 		BUG_ON(ipi_from_irq(irq) != ipi);
 
 		/* Get a new binding from Xen. */
-		bind_ipi.vcpu = cpu;
+		bind_ipi.vcpu = per_cpu(xen_vcpu_id, cpu);
 		if (HYPERVISOR_event_channel_op(EVTCHNOP_bind_ipi,
 						&bind_ipi) != 0)
 			BUG();
-- 
2.5.5

[toc] | [prev] | [next] | [standalone]

#1433122 — [PATCH linux 7/8] xen/evtchn: use xen_vcpu_id mapping

Use the newly introduced xen_vcpu_id mapping to get Xen's idea of vCPU id
for CPU0.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
 drivers/xen/evtchn.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/xen/evtchn.c b/drivers/xen/evtchn.c
index f4edd6df..0fc6be4 100644
--- a/drivers/xen/evtchn.c
+++ b/drivers/xen/evtchn.c
@@ -448,7 +448,7 @@ static long evtchn_ioctl(struct file *file,
 			break;
 
 		bind_virq.virq = bind.virq;
-		bind_virq.vcpu = 0;
+		bind_virq.vcpu = per_cpu(xen_vcpu_id, 0);
 		rc = HYPERVISOR_event_channel_op(EVTCHNOP_bind_virq,
 						 &bind_virq);
 		if (rc != 0)
-- 
2.5.5

[toc] | [prev] | [next] | [standalone]

#1433125 — [PATCH linux 4/8] x86/xen: use xen_vcpu_id mapping when pointing vcpu_info to the shared_info page

shared_info page has space for 32 vcpu info slots for first 32 vCPUs but
these are the first 32 vCPUs from Xen's perspective and we should map them
accordingly with the newly introduced xen_vcpu_id mapping.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
 arch/x86/xen/enlighten.c | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)

diff --git a/arch/x86/xen/enlighten.c b/arch/x86/xen/enlighten.c
index 69f4c0c..1596626 100644
--- a/arch/x86/xen/enlighten.c
+++ b/arch/x86/xen/enlighten.c
@@ -189,6 +189,7 @@ static void xen_vcpu_setup(int cpu)
 	struct vcpu_register_vcpu_info info;
 	int err;
 	struct vcpu_info *vcpup;
+	int xen_cpu = per_cpu(xen_vcpu_id, cpu);
 
 	BUG_ON(HYPERVISOR_shared_info == &xen_dummy_shared_info);
 
@@ -207,8 +208,9 @@ static void xen_vcpu_setup(int cpu)
 		if (per_cpu(xen_vcpu, cpu) == &per_cpu(xen_vcpu_info, cpu))
 			return;
 	}
-	if (cpu < MAX_VIRT_CPUS)
-		per_cpu(xen_vcpu,cpu) = &HYPERVISOR_shared_info->vcpu_info[cpu];
+	if (xen_cpu < MAX_VIRT_CPUS)
+		per_cpu(xen_vcpu, cpu) =
+			&HYPERVISOR_shared_info->vcpu_info[xen_cpu];
 
 	if (!have_vcpu_info_placement) {
 		if (cpu >= MAX_VIRT_CPUS)
@@ -1752,7 +1754,7 @@ asmlinkage __visible void __init xen_start_kernel(void)
 
 void __ref xen_hvm_init_shared_info(void)
 {
-	int cpu;
+	int cpu, xen_cpu;
 	struct xen_add_to_physmap xatp;
 	static struct shared_info *shared_info_page = 0;
 
@@ -1777,10 +1779,13 @@ void __ref xen_hvm_init_shared_info(void)
 	 * online but xen_hvm_init_shared_info is run at resume time too and
 	 * in that case multiple vcpus might be online. */
 	for_each_online_cpu(cpu) {
+		xen_cpu = per_cpu(xen_vcpu_id, cpu);
+
 		/* Leave it to be NULL. */
-		if (cpu >= MAX_VIRT_CPUS)
+		if (xen_cpu >= MAX_VIRT_CPUS)
 			continue;
-		per_cpu(xen_vcpu, cpu) = &HYPERVISOR_shared_info->vcpu_info[cpu];
+		per_cpu(xen_vcpu, cpu) =
+			&HYPERVISOR_shared_info->vcpu_info[xen_cpu];
 	}
 }
 
-- 
2.5.5

[toc] | [prev] | [next] | [standalone]

#1433706 — Re: [Xen-devel] [PATCH linux 4/8] x86/xen: use xen_vcpu_id mapping when pointing vcpu_info to the shared_info page

On 28/06/16 17:47, Vitaly Kuznetsov wrote:
> shared_info page has space for 32 vcpu info slots for first 32 vCPUs but
> these are the first 32 vCPUs from Xen's perspective and we should map them
> accordingly with the newly introduced xen_vcpu_id mapping.
> 
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> ---
>  arch/x86/xen/enlighten.c | 15 ++++++++++-----
>  1 file changed, 10 insertions(+), 5 deletions(-)
> 
> diff --git a/arch/x86/xen/enlighten.c b/arch/x86/xen/enlighten.c
> index 69f4c0c..1596626 100644
> --- a/arch/x86/xen/enlighten.c
> +++ b/arch/x86/xen/enlighten.c
> @@ -189,6 +189,7 @@ static void xen_vcpu_setup(int cpu)
>  	struct vcpu_register_vcpu_info info;
>  	int err;
>  	struct vcpu_info *vcpup;
> +	int xen_cpu = per_cpu(xen_vcpu_id, cpu);

I think there should be a

static inline int xen_vcpu_nr(int cpu)
{
	return per_cpu(xen_vcpu_id, cpu);
}

helper function.

David

[toc] | [prev] | [next] | [standalone]

#1433127 — [PATCH linux 8/8] xen/pvhvm: run xen_vcpu_setup() for the boot CPU

Historically we didn't call VCPUOP_register_vcpu_info for CPU0 for PVHVM
guests (while we had it for PV and ARM guests). This is usually fine as we
can use vcpu info in the sared_info page but when we try booting on a vCPU
with Xen's vCPU id > 31 (e.g. when we try to kdump after crashing on this
CPU) we're not able to boot. Switch to always doing
VCPUOP_register_vcpu_info for the boot CPU.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
 arch/x86/xen/enlighten.c | 2 +-
 arch/x86/xen/smp.c       | 7 +++++++
 arch/x86/xen/xen-ops.h   | 1 +
 3 files changed, 9 insertions(+), 1 deletion(-)

diff --git a/arch/x86/xen/enlighten.c b/arch/x86/xen/enlighten.c
index 1596626..0be29a0 100644
--- a/arch/x86/xen/enlighten.c
+++ b/arch/x86/xen/enlighten.c
@@ -184,7 +184,7 @@ static void clamp_max_cpus(void)
 #endif
 }
 
-static void xen_vcpu_setup(int cpu)
+void xen_vcpu_setup(int cpu)
 {
 	struct vcpu_register_vcpu_info info;
 	int err;
diff --git a/arch/x86/xen/smp.c b/arch/x86/xen/smp.c
index 719cf29..b9b31a7 100644
--- a/arch/x86/xen/smp.c
+++ b/arch/x86/xen/smp.c
@@ -322,6 +322,13 @@ static void __init xen_smp_prepare_boot_cpu(void)
 		xen_filter_cpu_maps();
 		xen_setup_vcpu_info_placement();
 	}
+
+	/*
+	 * Setup vcpu_info for boot CPU.
+	 */
+	if (xen_hvm_domain())
+		xen_vcpu_setup(0);
+
 	/*
 	 * The alternative logic (which patches the unlock/lock) runs before
 	 * the smp bootup up code is activated. Hence we need to set this up
diff --git a/arch/x86/xen/xen-ops.h b/arch/x86/xen/xen-ops.h
index 4140b07..3cbce3b 100644
--- a/arch/x86/xen/xen-ops.h
+++ b/arch/x86/xen/xen-ops.h
@@ -76,6 +76,7 @@ irqreturn_t xen_debug_interrupt(int irq, void *dev_id);
 
 bool xen_vcpu_stolen(int vcpu);
 
+void xen_vcpu_setup(int cpu);
 void xen_setup_vcpu_info_placement(void);
 
 #ifdef CONFIG_SMP
-- 
2.5.5

[toc] | [prev] | [next] | [standalone]

#1433130 — [PATCH linux 6/8] xen/events: fifo: use xen_vcpu_id mapping

EVTCHNOP_init_control has vCPU id as a parameter and Xen's idea of vCPU id
should be used. Use the newly introduced xen_vcpu_id mapping to convert
it from Linux's id.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
 drivers/xen/events/events_fifo.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/xen/events/events_fifo.c b/drivers/xen/events/events_fifo.c
index 9289a17..e3406cd 100644
--- a/drivers/xen/events/events_fifo.c
+++ b/drivers/xen/events/events_fifo.c
@@ -113,7 +113,7 @@ static int init_control_block(int cpu,
 
 	init_control.control_gfn = virt_to_gfn(control_block);
 	init_control.offset      = 0;
-	init_control.vcpu        = cpu;
+	init_control.vcpu        = per_cpu(xen_vcpu_id, cpu);
 
 	return HYPERVISOR_event_channel_op(EVTCHNOP_init_control, &init_control);
 }
-- 
2.5.5

[toc] | [prev] | [next] | [standalone]

#1433173

On 28/06/16 17:47, Vitaly Kuznetsov wrote:
> It may happen that Xen's and Linux's ideas of vCPU id diverge. In
> particular, when we crash on a secondary vCPU we may want to do kdump
> and unlike plain kexec where we do migrate_to_reboot_cpu() we try booting
> on the vCPU which crashed. This doesn't work very well for PVHVM guests as
> we have a number of hypercalls where we pass vCPU id as a parameter. These
> hypercalls either fail or do something unexpected. To solve the issue we
> need to have a mapping between Linux's and Xen's vCPU ids.

Could the soft-reboot hypercall (optionally) return on vcpu 0?

David

[toc] | [prev] | [next] | [standalone]

#1433599

David Vrabel <david.vrabel@citrix.com> writes:

> On 28/06/16 17:47, Vitaly Kuznetsov wrote:
>> It may happen that Xen's and Linux's ideas of vCPU id diverge. In
>> particular, when we crash on a secondary vCPU we may want to do kdump
>> and unlike plain kexec where we do migrate_to_reboot_cpu() we try booting
>> on the vCPU which crashed. This doesn't work very well for PVHVM guests as
>> we have a number of hypercalls where we pass vCPU id as a parameter. These
>> hypercalls either fail or do something unexpected. To solve the issue we
>> need to have a mapping between Linux's and Xen's vCPU ids.
>
> Could the soft-reboot hypercall (optionally) return on vcpu 0?
>

In theory, yes, I think we can re-arrange vCPUs inside the hypervisor so
Linux will get them in the natural order after soft reset.

-- 
  Vitaly

[toc] | [prev] | [next] | [standalone]

#1433708

On 29/06/16 10:16, Vitaly Kuznetsov wrote:
> David Vrabel <david.vrabel@citrix.com> writes:
> 
>> On 28/06/16 17:47, Vitaly Kuznetsov wrote:
>>> It may happen that Xen's and Linux's ideas of vCPU id diverge. In
>>> particular, when we crash on a secondary vCPU we may want to do kdump
>>> and unlike plain kexec where we do migrate_to_reboot_cpu() we try booting
>>> on the vCPU which crashed. This doesn't work very well for PVHVM guests as
>>> we have a number of hypercalls where we pass vCPU id as a parameter. These
>>> hypercalls either fail or do something unexpected. To solve the issue we
>>> need to have a mapping between Linux's and Xen's vCPU ids.
>>
>> Could the soft-reboot hypercall (optionally) return on vcpu 0?
>>
> 
> In theory, yes, I think we can re-arrange vCPUs inside the hypervisor so
> Linux will get them in the natural order after soft reset.

The series is straight forwards and the concept of the guest having to
map its idea of CPU to VCPU is fine, so unless you think a hypervisor
based solution is better we can take this series once it's fixed up.

David

[toc] | [prev] | [next] | [standalone]

#1435219

David Vrabel <david.vrabel@citrix.com> writes:

> On 29/06/16 10:16, Vitaly Kuznetsov wrote:
>> David Vrabel <david.vrabel@citrix.com> writes:
>> 
>>> On 28/06/16 17:47, Vitaly Kuznetsov wrote:
>>>> It may happen that Xen's and Linux's ideas of vCPU id diverge. In
>>>> particular, when we crash on a secondary vCPU we may want to do kdump
>>>> and unlike plain kexec where we do migrate_to_reboot_cpu() we try booting
>>>> on the vCPU which crashed. This doesn't work very well for PVHVM guests as
>>>> we have a number of hypercalls where we pass vCPU id as a parameter. These
>>>> hypercalls either fail or do something unexpected. To solve the issue we
>>>> need to have a mapping between Linux's and Xen's vCPU ids.
>>>
>>> Could the soft-reboot hypercall (optionally) return on vcpu 0?
>>>
>> 
>> In theory, yes, I think we can re-arrange vCPUs inside the hypervisor so
>> Linux will get them in the natural order after soft reset.
>
> The series is straight forwards and the concept of the guest having to
> map its idea of CPU to VCPU is fine, so unless you think a hypervisor
> based solution is better we can take this series once it's fixed up.

I'm afraid that kdump is not the only case where the mapping may come
handy. Linux CPU ids are not guaranteed to always match the order of
CPUs in MADT, e.g. if we fail to add a CPU ids for the rest will get
shifted. This could easily be emulated by adding 'disable_cpu_apicid='
kernel parameter. I didn't try but I'm pretty sure that disabling any
CPU will break guest bootup. That said, I think maintaining the
Xen-Linux vCPU id mapping is preferred till we introduce an option to
address vCPUs in hypercalls by some other ids (vLAPIC id, for example).

-- 
  Vitaly

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web