Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1513823 > unrolled thread

[RFC PATCH] perf/x86/intel/rapl: avoid access unallocate memory

Back to article view | Back to linux.kernel

Contents

  [RFC PATCH] perf/x86/intel/rapl: avoid access unallocate memory Sebastian Andrzej Siewior <bigeasy@linutronix.de> - 2016-11-02 13:30 +0100
    Re: [RFC PATCH] perf/x86/intel/rapl: avoid access unallocate memory "Charles (Chas) Williams" <ciwillia@brocade.com> - 2016-11-02 23:50 +0100
      Re: [RFC PATCH] perf/x86/intel/rapl: avoid access unallocate memory Sebastian Andrzej Siewior <bigeasy@linutronix.de> - 2016-11-03 18:50 +0100
        Re: [RFC PATCH] perf/x86/intel/rapl: avoid access unallocate memory "Charles (Chas) Williams" <ciwillia@brocade.com> - 2016-11-04 13:30 +0100
          Re: [RFC PATCH] perf/x86/intel/rapl: avoid access unallocate memory Sebastian Andrzej Siewior <bigeasy@linutronix.de> - 2016-11-04 19:10 +0100
            Re: [RFC PATCH] perf/x86/intel/rapl: avoid access unallocate memory "Charles (Chas) Williams" <ciwillia@brocade.com> - 2016-11-04 21:50 +0100
              Re: [RFC PATCH] perf/x86/intel/rapl: avoid access unallocate memory Sebastian Andrzej Siewior <bigeasy@linutronix.de> - 2016-11-04 22:00 +0100
      Re: [RFC PATCH] perf/x86/intel/rapl: avoid access unallocate  memory Thomas Gleixner <tglx@linutronix.de> - 2016-11-07 17:30 +0100
        Re: [RFC PATCH] perf/x86/intel/rapl: avoid access unallocate memory "Charles (Chas) Williams" <ciwillia@brocade.com> - 2016-11-07 18:10 +0100
          Re: [RFC PATCH] perf/x86/intel/rapl: avoid access unallocate  memory Thomas Gleixner <tglx@linutronix.de> - 2016-11-07 21:30 +0100
            Re: [RFC PATCH] perf/x86/intel/rapl: avoid access unallocate memory "Charles (Chas) Williams" <ciwillia@brocade.com> - 2016-11-08 15:30 +0100
              Re: [RFC PATCH] perf/x86/intel/rapl: avoid access unallocate  memory Thomas Gleixner <tglx@linutronix.de> - 2016-11-08 15:40 +0100
                Re: [RFC PATCH] perf/x86/intel/rapl: avoid access unallocate memory "Charles (Chas) Williams" <ciwillia@brocade.com> - 2016-11-08 16:00 +0100
                  Re: [RFC PATCH] perf/x86/intel/rapl: avoid access unallocate  memory Thomas Gleixner <tglx@linutronix.de> - 2016-11-08 17:30 +0100
                    [PATCH] x86/cpuid: Deal with broken firmware once more Thomas Gleixner <tglx@linutronix.de> - 2016-11-09 16:40 +0100
                      Re: [PATCH] x86/cpuid: Deal with broken firmware once more Thomas Gleixner <tglx@linutronix.de> - 2016-11-09 16:50 +0100
                      Re: [PATCH] x86/cpuid: Deal with broken firmware once more Peter Zijlstra <peterz@infradead.org> - 2016-11-09 17:10 +0100
                        Re: [PATCH] x86/cpuid: Deal with broken firmware once more "Charles (Chas) Williams" <ciwillia@brocade.com> - 2016-11-09 17:40 +0100
                          Re: [PATCH] x86/cpuid: Deal with broken firmware once more Thomas Gleixner <tglx@linutronix.de> - 2016-11-09 19:50 +0100
                      Re: [PATCH] x86/cpuid: Deal with broken firmware once more "Charles (Chas) Williams" <ciwillia@brocade.com> - 2016-11-09 19:20 +0100
                      [tip:x86/urgent] x86/cpu: Deal with broken firmware (VMWare/XEN) tip-bot for Thomas Gleixner <tipbot@zytor.com> - 2016-11-09 21:30 +0100
                        Re: [tip:x86/urgent] x86/cpu: Deal with broken firmware (VMWare/XEN) Alok Kataria <akataria@vmware.com> - 2016-11-11 07:00 +0100
                      Re: [PATCH] x86/cpuid: Deal with broken firmware once more "M. Vefa Bicakci" <m.v.b@runbox.com> - 2016-11-10 05:00 +0100
                        Re: [PATCH] x86/cpuid: Deal with broken firmware once more "Charles (Chas) Williams" <ciwillia@brocade.com> - 2016-11-10 12:00 +0100
                          Re: [PATCH] x86/cpuid: Deal with broken firmware once more Thomas Gleixner <tglx@linutronix.de> - 2016-11-10 12:20 +0100
                        Re: [PATCH] x86/cpuid: Deal with broken firmware once more Thomas Gleixner <tglx@linutronix.de> - 2016-11-10 12:20 +0100
                          Re: [PATCH] x86/cpuid: Deal with broken firmware once more Peter Zijlstra <peterz@infradead.org> - 2016-11-10 12:50 +0100
                          Re: [PATCH] x86/cpuid: Deal with broken firmware once more Boris Ostrovsky <boris.ostrovsky@oracle.com> - 2016-11-10 15:10 +0100
                            Re: [PATCH] x86/cpuid: Deal with broken firmware once more "Charles (Chas) Williams" <ciwillia@brocade.com> - 2016-11-10 16:10 +0100
                              Re: [PATCH] x86/cpuid: Deal with broken firmware once more Boris Ostrovsky <boris.ostrovsky@oracle.com> - 2016-11-10 16:40 +0100
                                Re: [PATCH] x86/cpuid: Deal with broken firmware once more Sebastian Andrzej Siewior <bigeasy@linutronix.de> - 2016-11-10 17:00 +0100
                                Re: [PATCH] x86/cpuid: Deal with broken firmware once more Thomas Gleixner <tglx@linutronix.de> - 2016-11-10 18:20 +0100
                            Re: [PATCH] x86/cpuid: Deal with broken firmware once more Thomas Gleixner <tglx@linutronix.de> - 2016-11-10 16:20 +0100
                              Re: [PATCH] x86/cpuid: Deal with broken firmware once more Boris Ostrovsky <boris.ostrovsky@oracle.com> - 2016-11-10 16:40 +0100
                                Re: [PATCH] x86/cpuid: Deal with broken firmware once more Thomas Gleixner <tglx@linutronix.de> - 2016-11-10 18:20 +0100
                                  Re: [PATCH] x86/cpuid: Deal with broken firmware once more Boris Ostrovsky <boris.ostrovsky@oracle.com> - 2016-11-10 19:10 +0100

Page 2 of 2 — ← Prev page 1 [2]

#1518460 — [tip:x86/urgent] x86/cpu: Deal with broken firmware (VMWare/XEN)

Commit-ID:  d49597fd3bc7d9534de55e9256767f073be1b33a
Gitweb:     http://git.kernel.org/tip/d49597fd3bc7d9534de55e9256767f073be1b33a
Author:     Thomas Gleixner <tglx@linutronix.de>
AuthorDate: Wed, 9 Nov 2016 16:35:51 +0100
Committer:  Thomas Gleixner <tglx@linutronix.de>
CommitDate: Wed, 9 Nov 2016 21:05:01 +0100

x86/cpu: Deal with broken firmware (VMWare/XEN)

Both ACPI and MP specifications require that the APIC id in the respective
tables must be the same as the APIC id in CPUID.

The kernel retrieves the physical package id from the APIC id during the
ACPI/MP table scan and builds the physical to logical package map. The
physical package id which is used after a CPU comes up is retrieved from
CPUID. So we rely on ACPI/MP tables and CPUID agreeing in that respect.

There exist VMware and XEN implementations which violate the spec. As a
result the physical to logical package map, which relies on the ACPI/MP
tables does not work on those systems, because the CPUID initialized
physical package id does not match the firmware id. This causes system
crashes and malfunction due to invalid package mappings.

The only way to cure this is to sanitize the physical package id after the
CPUID enumeration and yell when the APIC ids are different. Fix up the
initial APIC id, which is fine as it is only used printout purposes.

If the physical package IDs differ yell and use the package information
from the ACPI/MP tables so the existing logical package map just works.

Chas provided the resulting dmesg output for his affected 4 virtual
sockets, 1 core per socket VM:

[Firmware Bug]: CPU1: APIC id mismatch. Firmware: 1 CPUID: 2
[Firmware Bug]: CPU1: Using firmware package id 1 instead of 2
....

Reported-and-tested-by: "Charles (Chas) Williams" <ciwillia@brocade.com>,
Reported-by: M. Vefa Bicakci <m.v.b@runbox.com>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Alok Kataria <akataria@vmware.com>
Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: #4.6+ <stable@vger,kernel.org>
Link: http://lkml.kernel.org/r/alpine.DEB.2.20.1611091613540.3501@nanos
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/kernel/cpu/common.c | 32 ++++++++++++++++++++++++++++++--
 1 file changed, 30 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
index 9bd910a..cc9e980 100644
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -979,6 +979,35 @@ static void x86_init_cache_qos(struct cpuinfo_x86 *c)
 }
 
 /*
+ * The physical to logical package id mapping is initialized from the
+ * acpi/mptables information. Make sure that CPUID actually agrees with
+ * that.
+ */
+static void sanitize_package_id(struct cpuinfo_x86 *c)
+{
+#ifdef CONFIG_SMP
+	unsigned int pkg, apicid, cpu = smp_processor_id();
+
+	apicid = apic->cpu_present_to_apicid(cpu);
+	pkg = apicid >> boot_cpu_data.x86_coreid_bits;
+
+	if (apicid != c->initial_apicid) {
+		pr_err(FW_BUG "CPU%u: APIC id mismatch. Firmware: %x CPUID: %x\n",
+		       cpu, apicid, c->initial_apicid);
+		c->initial_apicid = apicid;
+	}
+	if (pkg != c->phys_proc_id) {
+		pr_err(FW_BUG "CPU%u: Using firmware package id %u instead of %u\n",
+		       cpu, pkg, c->phys_proc_id);
+		c->phys_proc_id = pkg;
+	}
+	c->logical_proc_id = topology_phys_to_logical_pkg(pkg);
+#else
+	c->logical_proc_id = 0;
+#endif
+}
+
+/*
  * This does the hard work of actually picking apart the CPU stuff...
  */
 static void identify_cpu(struct cpuinfo_x86 *c)
@@ -1103,8 +1132,7 @@ static void identify_cpu(struct cpuinfo_x86 *c)
 #ifdef CONFIG_NUMA
 	numa_add_cpu(smp_processor_id());
 #endif
-	/* The boot/hotplug time assigment got cleared, restore it */
-	c->logical_proc_id = topology_phys_to_logical_pkg(c->phys_proc_id);
+	sanitize_package_id(c);
 }
 
 /*

[toc] | [prev] | [next] | [standalone]

#1519584 — Re: [tip:x86/urgent] x86/cpu: Deal with broken firmware (VMWare/XEN)

Hi Thomas, 

On Wed, 2016-11-09 at 12:27 -0800, tip-bot for Thomas Gleixner wrote:
> Commit-ID:  d49597fd3bc7d9534de55e9256767f073be1b33a
> Gitweb:     https://urldefense.proofpoint.com/v2/url?u=http-3A__git.kernel.org_tip_d49597fd3bc7d9534de55e9256767f073be1b33a&d=CwIDaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=2AkLWShm6V8Nuu8ZZ-80Flo6y0XxCGmO1xrsAeRArAE&m=WBsB4JFr-Dct0um4Kf8QAxC7w6p-Mlk3H-LwItQJ7Fw&s=qI64vSH3y6q8wJhcqpI4dXYma-i1RTtlxgKwKwhFWWo&e= 
> Author:     Thomas Gleixner <tglx@linutronix.de>
> AuthorDate: Wed, 9 Nov 2016 16:35:51 +0100
> Committer:  Thomas Gleixner <tglx@linutronix.de>
> CommitDate: Wed, 9 Nov 2016 21:05:01 +0100
> 
> x86/cpu: Deal with broken firmware (VMWare/XEN)
> 
> Both ACPI and MP specifications require that the APIC id in the respective
> tables must be the same as the APIC id in CPUID.
> 
> The kernel retrieves the physical package id from the APIC id during the
> ACPI/MP table scan and builds the physical to logical package map. The
> physical package id which is used after a CPU comes up is retrieved from
> CPUID. So we rely on ACPI/MP tables and CPUID agreeing in that respect.
> 
> There exist VMware and XEN implementations which violate the spec. As a
> result the physical to logical package map, which relies on the ACPI/MP
> tables does not work on those systems, because the CPUID initialized
> physical package id does not match the firmware id. This causes system
> crashes and malfunction due to invalid package mappings.

For documentation purpose let me note that, VMware VMs running at
virtual hardware version 9 and above don't have this ACPI/MP and CPUID
divergence on the package id. So not everyone will see this issue on
their VMs, this bug is limited to folks running at virtual hardware
version 8 and prior.

It's good that we can workaround the platform bug for those VMs, thanks
for adding these checks.

Alok

> 
> The only way to cure this is to sanitize the physical package id after the
> CPUID enumeration and yell when the APIC ids are different. Fix up the
> initial APIC id, which is fine as it is only used printout purposes.
> 
> If the physical package IDs differ yell and use the package information
> from the ACPI/MP tables so the existing logical package map just works.
> 
> Chas provided the resulting dmesg output for his affected 4 virtual
> sockets, 1 core per socket VM:
> 
> [Firmware Bug]: CPU1: APIC id mismatch. Firmware: 1 CPUID: 2
> [Firmware Bug]: CPU1: Using firmware package id 1 instead of 2
> ....
> 
> Reported-and-tested-by: "Charles (Chas) Williams" <ciwillia@brocade.com>,
> Reported-by: M. Vefa Bicakci <m.v.b@runbox.com>
> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
> Cc: Borislav Petkov <bp@alien8.de>
> Cc: Alok Kataria <akataria@vmware.com>
> Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
> Cc: #4.6+ <stable@vger,kernel.org>
> Link: https://urldefense.proofpoint.com/v2/url?u=http-3A__lkml.kernel.org_r_alpine.DEB.2.20.1611091613540.3501-40nanos&d=CwIDaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=2AkLWShm6V8Nuu8ZZ-80Flo6y0XxCGmO1xrsAeRArAE&m=WBsB4JFr-Dct0um4Kf8QAxC7w6p-Mlk3H-LwItQJ7Fw&s=HNQMGUrw_s6Mc_oyREBnD4TrUjERbLcH1viAZr-aFPY&e= 
> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
> ---
>  arch/x86/kernel/cpu/common.c | 32 ++++++++++++++++++++++++++++++--
>  1 file changed, 30 insertions(+), 2 deletions(-)
> 
> diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
> index 9bd910a..cc9e980 100644
> --- a/arch/x86/kernel/cpu/common.c
> +++ b/arch/x86/kernel/cpu/common.c
> @@ -979,6 +979,35 @@ static void x86_init_cache_qos(struct cpuinfo_x86 *c)
>  }
>  
>  /*
> + * The physical to logical package id mapping is initialized from the
> + * acpi/mptables information. Make sure that CPUID actually agrees with
> + * that.
> + */
> +static void sanitize_package_id(struct cpuinfo_x86 *c)
> +{
> +#ifdef CONFIG_SMP
> +	unsigned int pkg, apicid, cpu = smp_processor_id();
> +
> +	apicid = apic->cpu_present_to_apicid(cpu);
> +	pkg = apicid >> boot_cpu_data.x86_coreid_bits;
> +
> +	if (apicid != c->initial_apicid) {
> +		pr_err(FW_BUG "CPU%u: APIC id mismatch. Firmware: %x CPUID: %x\n",
> +		       cpu, apicid, c->initial_apicid);
> +		c->initial_apicid = apicid;
> +	}
> +	if (pkg != c->phys_proc_id) {
> +		pr_err(FW_BUG "CPU%u: Using firmware package id %u instead of %u\n",
> +		       cpu, pkg, c->phys_proc_id);
> +		c->phys_proc_id = pkg;
> +	}
> +	c->logical_proc_id = topology_phys_to_logical_pkg(pkg);
> +#else
> +	c->logical_proc_id = 0;
> +#endif
> +}
> +
> +/*
>   * This does the hard work of actually picking apart the CPU stuff...
>   */
>  static void identify_cpu(struct cpuinfo_x86 *c)
> @@ -1103,8 +1132,7 @@ static void identify_cpu(struct cpuinfo_x86 *c)
>  #ifdef CONFIG_NUMA
>  	numa_add_cpu(smp_processor_id());
>  #endif
> -	/* The boot/hotplug time assigment got cleared, restore it */
> -	c->logical_proc_id = topology_phys_to_logical_pkg(c->phys_proc_id);
> +	sanitize_package_id(c);
>  }
>  
>  /*

[toc] | [prev] | [next] | [standalone]

#1518678 — Re: [PATCH] x86/cpuid: Deal with broken firmware once more

On 11/09/2016 06:35 PM, Thomas Gleixner wrote:
> Both ACPI and MP specifications require that the APIC id in the respective
> tables must be the same as the APIC id in CPUID. 
> 
> The kernel retrieves the physical package id from the APIC id during the
> ACPI/MP table scan and builds the physical to logical package map.
> 
> There exist Virtualbox and Xen implementations which violate the spec. As a
> result the physical to logical package map, which relies on the ACPI/MP
> tables does not work on those systems, because the CPUID initialized
> physical package id does not match the firmware id. This causes system
> crashes and malfunction due to invalid package mappings.
> 
> The only way to cure this is to sanitize the physical package id after the
> CPUID enumeration and yell when the APIC ids are different. If the physical
> package IDs differ use the package information from the ACPI/MP tables so
> the existing logical package map just works.
> 
> Reported-by: "Charles (Chas) Williams" <ciwillia@brocade.com>,
> Reported-by: M. Vefa Bicakci <m.v.b@runbox.com>
> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>

Hello Thomas and Sebastian,

Sorry for the delay in reporting what I have found out and for the delay in testing this patch, which has been due to my health.

I have found that your patch unfortunately does not improve the situation for me. Here is an excerpt obtained from the dmesg of a kernel compiled with this patch *as well as* Sebastian's patch:

=== 8< ===
[    0.002561] CPU: Physical Processor ID: 0

[    0.002566] CPU: Processor Core ID: 0

[    0.002572] [Firmware Bug]: CPU0: APIC id mismatch. Firmware: ffff CPUID: 2

[    0.002577] [Firmware Bug]: CPU0: Using firmware package id 4095 instead of 0

[    0.002586] Last level iTLB entries: 4KB 512, 2MB 8, 4MB 8

[    0.002591] Last level dTLB entries: 4KB 512, 2MB 32, 4MB 32, 1GB 0

[    0.033319] ftrace: allocating 28753 entries in 113 pages

[    0.040121] cpu 0 spinlock event irq 1

[    0.040145] smpboot: Max logical packages: 1

[    0.040155] VPMU disabled by hypervisor.

[    0.040181] Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only.

[    0.047050] NMI watchdog: disabled (cpu0): hardware events not enabled

[    0.047065] NMI watchdog: Shutting down hard lockup detector on all cpus

[    0.052015] installing Xen timer for CPU 1

[    0.052074] SMP alternatives: switching to SMP code

[    0.002000] Disabled fast string operations

[    0.002000] [Firmware Bug]: CPU1: APIC id mismatch. Firmware: ffff CPUID: 2

[    0.002000] [Firmware Bug]: CPU1: Using firmware package id 4095 instead of 0

[    0.002000] smpboot: APIC(ffff) Converting physical 4095 to logical package 0

[    0.078061] cpu 1 spinlock event irq 13

...
[    0.216404] Freeing initrd memory: 4340K (ffff880001fa7000 - ffff8800023e4000)

[    0.216487] RAPL PMU: rapl pmu error: max package: 1 but CPU0 belongs to 65535

[    0.217572] futex hash table entries: 512 (order: 3, 32768 bytes)

[    0.218293] Initialise system trusted keyrings

...
[    0.216404] Freeing initrd memory: 4340K (ffff880001fa7000 - ffff8800023e4000)

[    0.216487] RAPL PMU: rapl pmu error: max package: 1 but CPU0 belongs to 65535

[    0.217572] futex hash table entries: 512 (order: 3, 32768 bytes)

...
[    2.974474] intel_rapl: Found RAPL domain package

[    2.974489] intel_rapl: Found RAPL domain core

[    2.974498] intel_rapl: Found RAPL domain uncore

[    2.974518] intel_rapl: RAPL package 4095 domain package locked by BIOS

=== >8 ===

As you can see, your patch unfortunately does not correct the issue with the virtual CPU package identifiers in Xen-based virtual machines using para-virtualization.

In summary, the root cause of the issue for me appears to be that the boot-up code in the init_apic_mappings function switches the APIC 'ops' structure from Xen's 'ops' structure to the no-op ops structure. Due to this, the smp_init_package_map uses the no-op APIC ops structure's cpu_present_to_to_apicid function, even though it should use the corresponding method from Xen's APIC ops structure (i.e., xen_cpu_present_to_apicid).

Here is a dmesg excerpt with a kernel patched with Sebastian's patch (and some debugging code), exhibiting the issue I have just explained: (Note the 'switched to apic NOOP' line.)

=== 8< ===
(early) [    0.000000] SFI: Simple Firmware Interface v0.81 http://simplefirmware.org
(early) [    0.000000] smpboot: Allowing 2 CPUs, 0 hotplug CPUs
(early) [    0.000000] No local APIC present
(early) [    0.000000] APIC: disable apic facility
(early) [    0.000000] APIC: switched to apic NOOP
(early) [    0.000000] e820: [mem 0xfa000000-0xffffffff] available for PCI devices
(early) [    0.000000] Booting paravirtualized kernel on Xen
...
[    0.034082] mvb: kernel_init_freeable:1007: About to call smp_prepare_cpus...
[    0.034123] cpu 0 spinlock event irq 1
[    0.034138] smpboot: mvb: smp_init_package_map:372: max_physical_pkg_id, after set: 4096
[    0.034146] mvb: __default_cpu_present_to_apicid:612: Returning 65535! mps_cpu: 0, nr_cpu_ids: 2, cpu_present(mps_cpu): 1
[    0.034155] smpboot: mvb: smp_init_package_map:379: apicid: 65535, apic_id_valid(apicid):0
[    0.034162] smpboot: Max logical packages: 1
[    0.034169] VPMU disabled by hypervisor.
[    0.034187] Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only.
[    0.041131] NMI watchdog: disabled (cpu0): hardware events not enabled
[    0.041142] NMI watchdog: Shutting down hard lockup detector on all cpus
[    0.046024] installing Xen timer for CPU 1
[    0.046121] SMP alternatives: switching to SMP code
[    0.002000] Disabled fast string operations
[    0.002000] mvb: detect_ht: CPU has hyper-threading capability
[    0.002000] mvb: CPU: Physical Processor ID: 0
[    0.002000] mvb: CPU: Processor Core ID: 0
[    0.002000] mvb: identify_cpu:1112: c: ffff880013b0a040, c->logical_proc_id: 65535
[    0.002000] mvb: __default_cpu_present_to_apicid:612: Returning 65535! mps_cpu: 1, nr_cpu_ids: 2, cpu_present(mps_cpu): 1
[    0.002000] smpboot: mvb: topology_update_package_map:270: cpu: 1, pkg: 4095
[    0.002000] smpboot: APIC(ffff) Converting physical 4095 to logical package 0
[    0.002000] smpboot: mvb: topology_update_package_map:305: cpu: 1, cpu_data(cpu).logical_proc_id: 0
...
[    0.266540] RAPL PMU: mvb: init_rapl_pmus:686: rapl pmu: max package: 1
[    0.266547] RAPL PMU: mvb: rapl pmu: CPU0 (ffff880013a0a040) belongs to package ID 65535.
[    0.266559] RAPL PMU: mvb: rapl pmu: Package ID >= max package for CPU0 (ffff880013a0a040). This is an error.
[    0.266569] RAPL PMU: mvb: rapl pmu: CPU1 (ffff880013b0a040) belongs to package ID 0.
=== >8 ===

Through some debugging, last week I came up with the patch at the end of this e-mail, which does correct the issue for me. Once again, I am sorry for reporting this too late.

With a kernel built with the patch below, Sebastian's patch and some debugging code, I obtain the following dmesg output, which appears to be correct and does not emit RAPL-related errors during boot-up:

=== 8< ===
[    0.032083] mvb: kernel_init_freeable:1007: About to call smp_prepare_cpus...

[    0.032106] cpu 0 spinlock event irq 1

[    0.032119] smpboot: mvb: smp_init_package_map:372: max_physical_pkg_id, after set: 4096

[    0.032127] mvb: xen_cpu_present_to_apicid:151: cpu: 0, ret: 0

[    0.032132] smpboot: mvb: topology_update_package_map:270: cpu: 0, pkg: 0

[    0.032137] smpboot: APIC(0) Converting physical 0 to logical package 0

[    0.032142] smpboot: mvb: topology_update_package_map:305: cpu: 0, cpu_data(cpu).logical_proc_id: 0

[    0.032149] smpboot: mvb: smp_init_package_map:385: topology_update_package_map successful.

[    0.032155] smpboot: Max logical packages: 1

[    0.032161] VPMU disabled by hypervisor.

[    0.032177] Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only.

[    0.039061] NMI watchdog: disabled (cpu0): hardware events not enabled

[    0.039102] NMI watchdog: Shutting down hard lockup detector on all cpus

[    0.045048] installing Xen timer for CPU 1

[    0.045145] SMP alternatives: switching to SMP code

[    0.002000] Disabled fast string operations

[    0.002000] mvb: detect_ht: CPU has hyper-threading capability

[    0.002000] mvb: CPU: Physical Processor ID: 0

[    0.002000] mvb: CPU: Processor Core ID: 1

[    0.002000] mvb: identify_cpu:1112: c: ffff88001790a040, c->logical_proc_id: 0

[    0.002000] mvb: xen_cpu_present_to_apicid:151: cpu: 1, ret: 0

[    0.002000] smpboot: mvb: topology_update_package_map:270: cpu: 1, pkg: 0

[    0.002000] smpboot: mvb: topology_update_package_map:305: cpu: 1, cpu_data(cpu).logical_proc_id: 0

[    0.067149] cpu 1 spinlock event irq 13

...
[    0.208395] RAPL PMU: mvb: init_rapl_pmus:686: rapl pmu: max package: 1

[    0.208400] RAPL PMU: mvb: rapl pmu: CPU0 (ffff88001780a040) belongs to package ID 0.

[    0.208409] RAPL PMU: mvb: rapl pmu: CPU1 (ffff88001790a040) belongs to package ID 0.

[    0.208491] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 163840 ms ovfl timer

[    0.208507] RAPL PMU: hw unit of domain pp0-core 2^-16 Joules

[    0.208512] RAPL PMU: hw unit of domain package 2^-16 Joules

[    0.208517] RAPL PMU: hw unit of domain pp1-gpu 2^-16 Joules

[    0.209083] futex hash table entries: 512 (order: 3, 32768 bytes)

=== >8 ===

My patch may not be a very elegant correction for the issue at hand, unfortunately (and it probably does not comply with the kernel patch submission guidelines).

In addition, I think Sebastian's patch should be included in the mainline kernel, so that potential bugs do not bring down the kernel with a RAPL-related oops at boot-up time with virtualization.

Please note that I will need to be offline for most of the day today (my current timezone is UTC+03:00), and as a result my responses to your replies will most likely be a bit late.

Thank you,

Vefa


From b7097820285ab6a8588879969d74c56d890d4fd4 Mon Sep 17 00:00:00 2001
From: "M. Vefa Bicakci" <m.v.b@runbox.com>
Date: Fri, 4 Nov 2016 10:01:19 +0300
Subject: [PATCH] Do not reset apic to apic_noop with Xen

---
 arch/x86/include/asm/apic.h | 3 +++
 arch/x86/kernel/apic/apic.c | 7 +++++--
 arch/x86/xen/apic.c         | 4 +++-
 3 files changed, 11 insertions(+), 3 deletions(-)

diff --git a/arch/x86/include/asm/apic.h b/arch/x86/include/asm/apic.h
index b3d4c042e610..8c37580b7eb7 100644
--- a/arch/x86/include/asm/apic.h
+++ b/arch/x86/include/asm/apic.h
@@ -359,6 +359,9 @@ struct apic {
  */
 extern struct apic *apic;
 
+/* Indicates whether a hypervisor has already set 'apic'. */
+extern int apic_set_by_hypervisor;
+
 /*
  * APIC drivers are probed based on how they are listed in the .apicdrivers
  * section. So the order is important and enforced by the ordering
diff --git a/arch/x86/kernel/apic/apic.c b/arch/x86/kernel/apic/apic.c
index 076c315cdf18..a3a1d4570acf 100644
--- a/arch/x86/kernel/apic/apic.c
+++ b/arch/x86/kernel/apic/apic.c
@@ -172,6 +172,8 @@ unsigned long mp_lapic_addr;
 int disable_apic;
 /* Disable local APIC timer from the kernel commandline or via dmi quirk */
 static int disable_apic_timer __initdata;
+/* Indicates whether a hypervisor has already set 'apic'. */
+int apic_set_by_hypervisor;
 /* Local APIC timer works in C2 */
 int local_apic_timer_c2_ok;
 EXPORT_SYMBOL_GPL(local_apic_timer_c2_ok);
@@ -1788,8 +1790,9 @@ void __init init_apic_mappings(void)
 		return;
 	}
 
-	/* If no local APIC can be found return early */
-	if (!smp_found_config && detect_init_APIC()) {
+	if (apic_set_by_hypervisor) {
+		/* Hypervisor has already taken care of the APIC. */
+	} else if (!smp_found_config && detect_init_APIC()) {
 		/* lets NOP'ify apic operations */
 		pr_info("APIC: disable apic facility\n");
 		apic_disable();
diff --git a/arch/x86/xen/apic.c b/arch/x86/xen/apic.c
index 5ac792a1d419..b79a003c13a5 100644
--- a/arch/x86/xen/apic.c
+++ b/arch/x86/xen/apic.c
@@ -229,8 +229,10 @@ void __init xen_init_apic(void)
 	x86_io_apic_ops.read = xen_io_apic_read;
 	/* On PV guests the APIC CPUID bit is disabled so none of the
 	 * routines end up executing. */
-	if (!xen_initial_domain())
+	if (!xen_initial_domain()) {
 		apic = &xen_pv_apic;
+		apic_set_by_hypervisor = 1;
+	}
 
 	x86_platform.apic_post_init = xen_apic_check;
 }
-- 
2.5.5

[toc] | [prev] | [next] | [standalone]

#1518851 — Re: [PATCH] x86/cpuid: Deal with broken firmware once more

On 11/09/2016 10:57 PM, M. Vefa Bicakci wrote:
> [    0.002000] mvb: CPU: Physical Processor ID: 0
> [    0.002000] mvb: CPU: Processor Core ID: 0
> [    0.002000] mvb: identify_cpu:1112: c: ffff880013b0a040, c->logical_proc_id: 65535
> [    0.002000] mvb: __default_cpu_present_to_apicid:612: Returning 65535! mps_cpu: 1, nr_cpu_ids: 2, cpu_present(mps_cpu): 1
> [    0.002000] smpboot: mvb: topology_update_package_map:270: cpu: 1, pkg: 4095
> [    0.002000] smpboot: APIC(ffff) Converting physical 4095 to logical package 0
> [    0.002000] smpboot: mvb: topology_update_package_map:305: cpu: 1, cpu_data(cpu).logical_proc_id: 0

This seems strange.  0xffff is BAD_APICID.  Why didn't this fail here:

         for_each_present_cpu(cpu) {
                 unsigned int apicid = apic->cpu_present_to_apicid(cpu);

                 if (apicid == BAD_APICID || !apic->apic_id_valid(apicid))  <<<<<<<<<<
                         continue;
                 if (!topology_update_package_map(apicid, cpu))
                         continue;

topology_update_package_map() should never have been called?

[toc] | [prev] | [next] | [standalone]

#1518875 — Re: [PATCH] x86/cpuid: Deal with broken firmware once more

On Thu, 10 Nov 2016, Charles (Chas) Williams wrote:
> On 11/09/2016 10:57 PM, M. Vefa Bicakci wrote:
> > [    0.002000] mvb: CPU: Physical Processor ID: 0
> > [    0.002000] mvb: CPU: Processor Core ID: 0
> > [    0.002000] mvb: identify_cpu:1112: c: ffff880013b0a040,
> > c->logical_proc_id: 65535
> > [    0.002000] mvb: __default_cpu_present_to_apicid:612: Returning 65535!
> > mps_cpu: 1, nr_cpu_ids: 2, cpu_present(mps_cpu): 1
> > [    0.002000] smpboot: mvb: topology_update_package_map:270: cpu: 1, pkg:
> > 4095
> > [    0.002000] smpboot: APIC(ffff) Converting physical 4095 to logical
> > package 0
> > [    0.002000] smpboot: mvb: topology_update_package_map:305: cpu: 1,
> > cpu_data(cpu).logical_proc_id: 0
> 
> This seems strange.  0xffff is BAD_APICID.  Why didn't this fail here:
> 
>         for_each_present_cpu(cpu) {
>                 unsigned int apicid = apic->cpu_present_to_apicid(cpu);
> 
>                 if (apicid == BAD_APICID || !apic->apic_id_valid(apicid))
> <<<<<<<<<<
>                         continue;
>                 if (!topology_update_package_map(apicid, cpu))
>                         continue;
> 
> topology_update_package_map() should never have been called?

That's right. Looking into it ....
 

[toc] | [prev] | [next] | [standalone]

#1518874 — Re: [PATCH] x86/cpuid: Deal with broken firmware once more

On Thu, 10 Nov 2016, M. Vefa Bicakci wrote:

> I have found that your patch unfortunately does not improve the situation
> for me. Here is an excerpt obtained from the dmesg of a kernel compiled
> with this patch *as well as* Sebastian's patch:

> [    0.002561] CPU: Physical Processor ID: 0
> [    0.002566] CPU: Processor Core ID: 0
> [    0.002572] [Firmware Bug]: CPU0: APIC id mismatch. Firmware: ffff CPUID: 2

So apic->cpu_present_to_apicid() gives us a completely bogus APIC id which
translates to a bogus package id. And looking at the XEN code:

   xen_pv_apic.cpu_present_to_apicid = xen_cpu_present_to_apicid,

and xen_cpu_present_to_apicid does:

static int xen_cpu_present_to_apicid(int cpu)
{
        if (cpu_present(cpu))
                return xen_get_apic_id(xen_apic_read(APIC_ID));
        else
                return BAD_APICID;
}

So independent of which present CPU we query we get just some random
information, in the above case we get BAD_APICID from xen_apic_read() not
from the else path as this CPU _IS_ present.

What's so wrong with storing the fricking firmware supplied APICid as
everybody else does and report it back when queried?

This damned attitude of we just hack the code into submission and let
everybody else deal with the outcoming is utterly annoying.

Thanks,

	tglx

[toc] | [prev] | [next] | [standalone]

#1518905 — Re: [PATCH] x86/cpuid: Deal with broken firmware once more

On Thu, Nov 10, 2016 at 12:13:04PM +0100, Thomas Gleixner wrote:
> What's so wrong with storing the fricking firmware supplied APICid as
> everybody else does and report it back when queried?
> 
> This damned attitude of we just hack the code into submission and let
> everybody else deal with the outcoming is utterly annoying.

At some point I'd recommend just marking things BROKEN and getting on
with life. There's only so much crap you can deal with.

What Xen does here is completely insane.

[toc] | [prev] | [next] | [standalone]

#1518996 — Re: [PATCH] x86/cpuid: Deal with broken firmware once more

On 11/10/2016 06:13 AM, Thomas Gleixner wrote:
> On Thu, 10 Nov 2016, M. Vefa Bicakci wrote:
>
>> I have found that your patch unfortunately does not improve the situation
>> for me. Here is an excerpt obtained from the dmesg of a kernel compiled
>> with this patch *as well as* Sebastian's patch:
>> [    0.002561] CPU: Physical Processor ID: 0
>> [    0.002566] CPU: Processor Core ID: 0
>> [    0.002572] [Firmware Bug]: CPU0: APIC id mismatch. Firmware: ffff CPUID: 2
> So apic->cpu_present_to_apicid() gives us a completely bogus APIC id which
> translates to a bogus package id. And looking at the XEN code:
>
>    xen_pv_apic.cpu_present_to_apicid = xen_cpu_present_to_apicid,
>
> and xen_cpu_present_to_apicid does:
>
> static int xen_cpu_present_to_apicid(int cpu)
> {
>         if (cpu_present(cpu))
>                 return xen_get_apic_id(xen_apic_read(APIC_ID));
>         else
>                 return BAD_APICID;
> }
>
> So independent of which present CPU we query we get just some random
> information, in the above case we get BAD_APICID from xen_apic_read() not
> from the else path as this CPU _IS_ present.
>
> What's so wrong with storing the fricking firmware supplied APICid as
> everybody else does and report it back when queried?

By firmware you mean ACPI? It is most likely not available to PV guests.
How about returning cpu_data(cpu).initial_apicid?

And what was the original problem?

-boris

>
> This damned attitude of we just hack the code into submission and let
> everybody else deal with the outcoming is utterly annoying.
>
> Thanks,
>
> 	tglx

[toc] | [prev] | [next] | [standalone]

#1519046 — Re: [PATCH] x86/cpuid: Deal with broken firmware once more

On 11/10/2016 09:02 AM, Boris Ostrovsky wrote:
> On 11/10/2016 06:13 AM, Thomas Gleixner wrote:
>> On Thu, 10 Nov 2016, M. Vefa Bicakci wrote:
>>
>>> I have found that your patch unfortunately does not improve the situation
>>> for me. Here is an excerpt obtained from the dmesg of a kernel compiled
>>> with this patch *as well as* Sebastian's patch:
>>> [    0.002561] CPU: Physical Processor ID: 0
>>> [    0.002566] CPU: Processor Core ID: 0
>>> [    0.002572] [Firmware Bug]: CPU0: APIC id mismatch. Firmware: ffff CPUID: 2
>> So apic->cpu_present_to_apicid() gives us a completely bogus APIC id which
>> translates to a bogus package id. And looking at the XEN code:
>>
>>    xen_pv_apic.cpu_present_to_apicid = xen_cpu_present_to_apicid,
>>
>> and xen_cpu_present_to_apicid does:
>>
>> static int xen_cpu_present_to_apicid(int cpu)
>> {
>>         if (cpu_present(cpu))
>>                 return xen_get_apic_id(xen_apic_read(APIC_ID));
>>         else
>>                 return BAD_APICID;
>> }
>>
>> So independent of which present CPU we query we get just some random
>> information, in the above case we get BAD_APICID from xen_apic_read() not
>> from the else path as this CPU _IS_ present.
>>
>> What's so wrong with storing the fricking firmware supplied APICid as
>> everybody else does and report it back when queried?
>
> By firmware you mean ACPI? It is most likely not available to PV guests.
> How about returning cpu_data(cpu).initial_apicid?
>
> And what was the original problem?

The original issue I found was that VMware was returning a different set
of APIC id's in the ACPI tables than what it advertised on the CPU's.

http://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1266716.html

>>
>> This damned attitude of we just hack the code into submission and let
>> everybody else deal with the outcoming is utterly annoying.
>>
>> Thanks,
>>
>> 	tglx
>
>

[toc] | [prev] | [next] | [standalone]

#1519062 — Re: [PATCH] x86/cpuid: Deal with broken firmware once more

On 11/10/2016 10:05 AM, Charles (Chas) Williams wrote:
>
>
> On 11/10/2016 09:02 AM, Boris Ostrovsky wrote:
>> On 11/10/2016 06:13 AM, Thomas Gleixner wrote:
>>> On Thu, 10 Nov 2016, M. Vefa Bicakci wrote:
>>>
>>>> I have found that your patch unfortunately does not improve the
>>>> situation
>>>> for me. Here is an excerpt obtained from the dmesg of a kernel
>>>> compiled
>>>> with this patch *as well as* Sebastian's patch:
>>>> [    0.002561] CPU: Physical Processor ID: 0
>>>> [    0.002566] CPU: Processor Core ID: 0
>>>> [    0.002572] [Firmware Bug]: CPU0: APIC id mismatch. Firmware:
>>>> ffff CPUID: 2
>>> So apic->cpu_present_to_apicid() gives us a completely bogus APIC id
>>> which
>>> translates to a bogus package id. And looking at the XEN code:
>>>
>>>    xen_pv_apic.cpu_present_to_apicid = xen_cpu_present_to_apicid,
>>>
>>> and xen_cpu_present_to_apicid does:
>>>
>>> static int xen_cpu_present_to_apicid(int cpu)
>>> {
>>>         if (cpu_present(cpu))
>>>                 return xen_get_apic_id(xen_apic_read(APIC_ID));
>>>         else
>>>                 return BAD_APICID;
>>> }
>>>
>>> So independent of which present CPU we query we get just some random
>>> information, in the above case we get BAD_APICID from
>>> xen_apic_read() not
>>> from the else path as this CPU _IS_ present.
>>>
>>> What's so wrong with storing the fricking firmware supplied APICid as
>>> everybody else does and report it back when queried?
>>
>> By firmware you mean ACPI? It is most likely not available to PV guests.
>> How about returning cpu_data(cpu).initial_apicid?
>>
>> And what was the original problem?
>
> The original issue I found was that VMware was returning a different set
> of APIC id's in the ACPI tables than what it advertised on the CPU's.
>
> http://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1266716.html

For Xen, we recently added a6a198bc60e6 ("xen/x86: Update topology map
for PV VCPUs") to at least temporarily work around some topology map
problems that PV guests have with RAPL (which I think is what Vefa's
problem was).

-boris

[toc] | [prev] | [next] | [standalone]

#1519081 — Re: [PATCH] x86/cpuid: Deal with broken firmware once more

On 2016-11-10 10:31:20 [-0500], Boris Ostrovsky wrote:
> For Xen, we recently added a6a198bc60e6 ("xen/x86: Update topology map
> for PV VCPUs") to at least temporarily work around some topology map
> problems that PV guests have with RAPL (which I think is what Vefa's
> problem was).

I wouldn't blame it on RAPL. It is RAPL that exposes the wrong APICID
<-> CPU <-> Package mapping.

> -boris

Sebastian

[toc] | [prev] | [next] | [standalone]

#1519161 — Re: [PATCH] x86/cpuid: Deal with broken firmware once more

On Thu, 10 Nov 2016, Boris Ostrovsky wrote:
> On 11/10/2016 10:05 AM, Charles (Chas) Williams wrote:
> For Xen, we recently added a6a198bc60e6 ("xen/x86: Update topology map
> for PV VCPUs") to at least temporarily work around some topology map
> problems that PV guests have with RAPL (which I think is what Vefa's
> problem was).

RAPL is just the messenger and it could be any other facility relying on
the package map information. The underlying root cause is something
else. See the other mail.

Thanks,

	tglx

[toc] | [prev] | [next] | [standalone]

#1519047 — Re: [PATCH] x86/cpuid: Deal with broken firmware once more

On Thu, 10 Nov 2016, Boris Ostrovsky wrote:
> On 11/10/2016 06:13 AM, Thomas Gleixner wrote:
> > On Thu, 10 Nov 2016, M. Vefa Bicakci wrote:
> >
> >> I have found that your patch unfortunately does not improve the situation
> >> for me. Here is an excerpt obtained from the dmesg of a kernel compiled
> >> with this patch *as well as* Sebastian's patch:
> >> [    0.002561] CPU: Physical Processor ID: 0
> >> [    0.002566] CPU: Processor Core ID: 0
> >> [    0.002572] [Firmware Bug]: CPU0: APIC id mismatch. Firmware: ffff CPUID: 2
> > So apic->cpu_present_to_apicid() gives us a completely bogus APIC id which
> > translates to a bogus package id. And looking at the XEN code:
> >
> >    xen_pv_apic.cpu_present_to_apicid = xen_cpu_present_to_apicid,
> >
> > and xen_cpu_present_to_apicid does:
> >
> > static int xen_cpu_present_to_apicid(int cpu)
> > {
> >         if (cpu_present(cpu))
> >                 return xen_get_apic_id(xen_apic_read(APIC_ID));
> >         else
> >                 return BAD_APICID;
> > }
> >
> > So independent of which present CPU we query we get just some random
> > information, in the above case we get BAD_APICID from xen_apic_read() not
> > from the else path as this CPU _IS_ present.
> >
> > What's so wrong with storing the fricking firmware supplied APICid as
> > everybody else does and report it back when queried?
> 
> By firmware you mean ACPI? It is most likely not available to PV guests.

You either have to provide ACPI or MP tables. And either of those has to
provide the intial APIC ids for the CPUs. They are supposed to match the
IDs which are in the CPUID leafs.

> How about returning cpu_data(cpu).initial_apicid?

For a not yet brought up CPU. That's what the initial ACPI/MP table
enumeration is for.

Thanks,

	tglx

[toc] | [prev] | [next] | [standalone]

#1519065 — Re: [PATCH] x86/cpuid: Deal with broken firmware once more

On 11/10/2016 10:12 AM, Thomas Gleixner wrote:
> On Thu, 10 Nov 2016, Boris Ostrovsky wrote:
>> On 11/10/2016 06:13 AM, Thomas Gleixner wrote:
>>> On Thu, 10 Nov 2016, M. Vefa Bicakci wrote:
>>>
>>>> I have found that your patch unfortunately does not improve the situation
>>>> for me. Here is an excerpt obtained from the dmesg of a kernel compiled
>>>> with this patch *as well as* Sebastian's patch:
>>>> [    0.002561] CPU: Physical Processor ID: 0
>>>> [    0.002566] CPU: Processor Core ID: 0
>>>> [    0.002572] [Firmware Bug]: CPU0: APIC id mismatch. Firmware: ffff CPUID: 2
>>> So apic->cpu_present_to_apicid() gives us a completely bogus APIC id which
>>> translates to a bogus package id. And looking at the XEN code:
>>>
>>>    xen_pv_apic.cpu_present_to_apicid = xen_cpu_present_to_apicid,
>>>
>>> and xen_cpu_present_to_apicid does:
>>>
>>> static int xen_cpu_present_to_apicid(int cpu)
>>> {
>>>         if (cpu_present(cpu))
>>>                 return xen_get_apic_id(xen_apic_read(APIC_ID));
>>>         else
>>>                 return BAD_APICID;
>>> }
>>>
>>> So independent of which present CPU we query we get just some random
>>> information, in the above case we get BAD_APICID from xen_apic_read() not
>>> from the else path as this CPU _IS_ present.
>>>
>>> What's so wrong with storing the fricking firmware supplied APICid as
>>> everybody else does and report it back when queried?
>> By firmware you mean ACPI? It is most likely not available to PV guests.
> You either have to provide ACPI or MP tables. And either of those has to
> provide the intial APIC ids for the CPUs. They are supposed to match the
> IDs which are in the CPUID leafs.
>
>> How about returning cpu_data(cpu).initial_apicid?
> For a not yet brought up CPU. That's what the initial ACPI/MP table
> enumeration is for.

Unfortunately PV guests have neither. So we may need to emulate
something in xen_cpu_present_to_apicid().

-boris

(+xen-devel)

[toc] | [prev] | [next] | [standalone]

#1519162 — Re: [PATCH] x86/cpuid: Deal with broken firmware once more

On Thu, 10 Nov 2016, Boris Ostrovsky wrote:
> On 11/10/2016 10:12 AM, Thomas Gleixner wrote:
> > On Thu, 10 Nov 2016, Boris Ostrovsky wrote:
> >> By firmware you mean ACPI? It is most likely not available to PV guests.
> > You either have to provide ACPI or MP tables. And either of those has to
> > provide the intial APIC ids for the CPUs. They are supposed to match the
> > IDs which are in the CPUID leafs.
> >
> >> How about returning cpu_data(cpu).initial_apicid?
> > For a not yet brought up CPU. That's what the initial ACPI/MP table
> > enumeration is for.
> 
> Unfortunately PV guests have neither. So we may need to emulate
> something in xen_cpu_present_to_apicid().

SFI does the same thing and according to the dmesg which was posted, this
is using SFI. We also have devicetree based boot concept which provides the
APICids in the CPU enumeration at boot time in a way which the whole x86
machinery is expecting.

So what kind of APICid is XEN handing in via SFI? None, or just an invalid
one?

Thanks,

	tglx

[toc] | [prev] | [next] | [standalone]

#1519254 — Re: [PATCH] x86/cpuid: Deal with broken firmware once more

On 11/10/2016 12:13 PM, Thomas Gleixner wrote:
> On Thu, 10 Nov 2016, Boris Ostrovsky wrote:
>> On 11/10/2016 10:12 AM, Thomas Gleixner wrote:
>>> On Thu, 10 Nov 2016, Boris Ostrovsky wrote:
>>>> By firmware you mean ACPI? It is most likely not available to PV guests.
>>> You either have to provide ACPI or MP tables. And either of those has to
>>> provide the intial APIC ids for the CPUs. They are supposed to match the
>>> IDs which are in the CPUID leafs.
>>>
>>>> How about returning cpu_data(cpu).initial_apicid?
>>> For a not yet brought up CPU. That's what the initial ACPI/MP table
>>> enumeration is for.
>> Unfortunately PV guests have neither. So we may need to emulate
>> something in xen_cpu_present_to_apicid().
> SFI does the same thing and according to the dmesg which was posted, this
> is using SFI. We also have devicetree based boot concept which provides the
> APICids in the CPU enumeration at boot time in a way which the whole x86
> machinery is expecting.
>
> So what kind of APICid is XEN handing in via SFI? None, or just an invalid
> one?

None, SFI is not used at all. Or device tree for that matter.

The (PV) guest currently assumes APIC ID (i.e. APIC[0x20]) to always be
zero (we just fixed a bug where CPUID may return an
inconsistent/non-zero initial APIC ID value, but that's a different
issue I think). There is no topology, really, everything is flat.

So this obviously is rather, ahem, fragile and we've been fixing bugs
there, especially with introduction of package map (a6a198bc60e6 that I
mentioned in the other message is one such example). There is work
scheduled on the hypervisor side to properly report initial APIC IDs in
CPUID and that may help with correctly dealing with this on Linux side.

(Longer term we would like to move to PVH where we will have "true" APIC
emulation, with ACPI, just like in HVM guests. You can do your part by
helping: http://marc.info/?l=linux-kernel&m=147791731414152&w=3  ;-))


-boris

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | linux.kernel

csiph-web