Groups > linux.kernel > #1601754 > unrolled thread

[PATCH v5 untested] kvm: better MWAIT emulation for guests

Started by	"Michael S. Tsirkin" <mst@redhat.com>
First post	2017-03-15 22:30 +0100
Last post	2017-03-27 15:40 +0200
Articles	20 on this page of 43 — 7 participants

Back to article view | Back to linux.kernel

  [PATCH v5 untested] kvm: better MWAIT emulation for guests "Michael S. Tsirkin" <mst@redhat.com> - 2017-03-15 22:30 +0100
    Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Gabriel L. Somlo" <gsomlo@gmail.com> - 2017-03-16 00:40 +0100
      Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Michael S. Tsirkin" <mst@redhat.com> - 2017-03-16 00:50 +0100
        Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Gabriel L. Somlo" <gsomlo@gmail.com> - 2017-03-16 14:30 +0100
          Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests Radim Krčmář <rkrcmar@redhat.com> - 2017-03-16 15:10 +0100
            Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Gabriel L. Somlo" <gsomlo@gmail.com> - 2017-03-16 17:00 +0100
              Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests Radim Krčmář <rkrcmar@redhat.com> - 2017-03-16 17:00 +0100
                Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Gabriel L. Somlo" <gsomlo@gmail.com> - 2017-03-16 17:30 +0100
          Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Michael S. Tsirkin" <mst@redhat.com> - 2017-03-16 15:10 +0100
            Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Gabriel L. Somlo" <gsomlo@gmail.com> - 2017-03-16 16:00 +0100
              Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Michael S. Tsirkin" <mst@redhat.com> - 2017-03-16 16:30 +0100
              Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests Radim Krčmář <rkrcmar@redhat.com> - 2017-03-16 16:40 +0100
                Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests Radim Krčmář <rkrcmar@redhat.com> - 2017-03-16 17:10 +0100
                  Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Gabriel L. Somlo" <gsomlo@gmail.com> - 2017-03-16 17:50 +0100
                    Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Michael S. Tsirkin" <mst@redhat.com> - 2017-03-16 18:30 +0100
                      Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Gabriel L. Somlo" <gsomlo@gmail.com> - 2017-03-16 18:50 +0100
                        Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Michael S. Tsirkin" <mst@redhat.com> - 2017-03-16 19:40 +0100
                          Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Michael S. Tsirkin" <mst@redhat.com> - 2017-03-16 20:30 +0100
                            Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Gabriel L. Somlo" <gsomlo@gmail.com> - 2017-03-16 21:20 +0100
                              Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Gabriel L. Somlo" <gsomlo@gmail.com> - 2017-03-16 22:20 +0100
                                Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Michael S. Tsirkin" <mst@redhat.com> - 2017-03-17 03:10 +0100
                                  Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Gabriel L. Somlo" <gsomlo@gmail.com> - 2017-03-17 14:40 +0100
                                    Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Michael S. Tsirkin" <mst@redhat.com> - 2017-03-21 04:30 +0100
                                      Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests Radim Krčmář <rkrcmar@redhat.com> - 2017-03-21 18:10 +0100
                                        Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests Nadav Amit <nadav.amit@gmail.com> - 2017-03-21 18:40 +0100
                                          Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests Radim Krčmář <rkrcmar@redhat.com> - 2017-03-21 20:30 +0100
                                            Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests Gabriel Somlo <gsomlo@gmail.com> - 2017-03-22 00:00 +0100
                                              Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests Nadav Amit <nadav.amit@gmail.com> - 2017-03-22 01:10 +0100
                                                Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Michael S. Tsirkin" <mst@redhat.com> - 2017-03-22 14:40 +0100
                                                  Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Michael S. Tsirkin" <mst@redhat.com> - 2017-03-22 15:20 +0100
                                                  Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Gabriel L. Somlo" <gsomlo@gmail.com> - 2017-03-22 15:20 +0100
                          Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Gabriel L. Somlo" <gsomlo@gmail.com> - 2017-03-16 20:30 +0100
                    Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests Radim Krčmář <rkrcmar@redhat.com> - 2017-03-16 18:30 +0100
                      Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Gabriel L. Somlo" <gsomlo@gmail.com> - 2017-03-16 18:40 +0100
                Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Gabriel L. Somlo" <gsomlo@gmail.com> - 2017-03-16 17:20 +0100
                  Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Michael S. Tsirkin" <mst@redhat.com> - 2017-03-16 17:50 +0100
                    Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Gabriel L. Somlo" <gsomlo@gmail.com> - 2017-03-16 18:00 +0100
                      Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Gabriel L. Somlo" <gsomlo@gmail.com> - 2017-03-16 18:00 +0100
                        Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Michael S. Tsirkin" <mst@redhat.com> - 2017-03-16 18:20 +0100
                          Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests Radim Krčmář <rkrcmar@redhat.com> - 2017-03-16 18:40 +0100
    Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests Joerg Roedel <joro@8bytes.org> - 2017-03-21 17:20 +0100
      Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Michael S. Tsirkin" <mst@redhat.com> - 2017-03-21 19:50 +0100
    Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests Alexander Graf <agraf@suse.de> - 2017-03-27 15:40 +0200

Page 1 of 3 [1] 2 3 Next page →

#1601754 — [PATCH v5 untested] kvm: better MWAIT emulation for guests

From	"Michael S. Tsirkin" <mst@redhat.com>
Date	2017-03-15 22:30 +0100
Subject	[PATCH v5 untested] kvm: better MWAIT emulation for guests
Message-ID	<tloEy-50F-19@gated-at.bofh.it>

Guests running Mac OS 5, 6, and 7 (Leopard through Lion) have a problem:
unless explicitly provided with kernel command line argument
"idlehalt=0" they'd implicitly assume MONITOR and MWAIT availability,
without checking CPUID.

We currently emulate that as a NOP but on VMX we can do better: let
guest stop the CPU until timer, IPI or memory change.  CPU will be busy
but that isn't any worse than a NOP emulation.

Note that mwait within guests is not the same as on real hardware
because halt causes an exit while mwait doesn't.  For this reason it
might not be a good idea to use the regular MWAIT flag in CPUID to
signal this capability.  Add a flag in the hypervisor leaf instead.

Additionally, we add a capability for QEMU - e.g. if it knows there's an
isolated CPU dedicated for the VCPU it can set the standard MWAIT flag
to improve guest behaviour.

Reported-by: "Gabriel L. Somlo" <gsomlo@gmail.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---

This is for Gabriel's testing only. A bit rushed so untested.

 Documentation/virtual/kvm/api.txt    |  9 +++++++++
 Documentation/virtual/kvm/cpuid.txt  |  6 ++++++
 arch/x86/include/uapi/asm/kvm_para.h |  1 +
 arch/x86/kvm/cpuid.c                 |  3 +++
 arch/x86/kvm/svm.c                   |  2 --
 arch/x86/kvm/vmx.c                   |  6 ++++--
 arch/x86/kvm/x86.c                   |  3 +++
 arch/x86/kvm/x86.h                   | 28 ++++++++++++++++++++++++++++
 include/uapi/linux/kvm.h             |  1 +
 9 files changed, 55 insertions(+), 4 deletions(-)

diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
index 3c248f7..6ee2e43 100644
--- a/Documentation/virtual/kvm/api.txt
+++ b/Documentation/virtual/kvm/api.txt
@@ -4147,3 +4147,12 @@ This capability, if KVM_CHECK_EXTENSION indicates that it is
 available, means that that the kernel can support guests using the
 hashed page table MMU defined in Power ISA V3.00 (as implemented in
 the POWER9 processor), including in-memory segment tables.
+
+8.5 KVM_CAP_X86_GUEST_MWAIT
+
+Architectures: x86
+
+This capability indicates that guest using memory monotoring instructions
+(MWAIT/MWAITX) to stop the virtual CPU will not cause a VM exit.  As such time
+spent while virtual CPU is halted in this way will then be accounted for as
+guest running time on the host (as opposed to e.g. HLT).
diff --git a/Documentation/virtual/kvm/cpuid.txt b/Documentation/virtual/kvm/cpuid.txt
index 3c65feb..04c201c 100644
--- a/Documentation/virtual/kvm/cpuid.txt
+++ b/Documentation/virtual/kvm/cpuid.txt
@@ -54,6 +54,12 @@ KVM_FEATURE_PV_UNHALT              ||     7 || guest checks this feature bit
                                    ||       || before enabling paravirtualized
                                    ||       || spinlock support.
 ------------------------------------------------------------------------------
+KVM_FEATURE_MWAIT                  ||     8 || guest can use monitor/mwait
+                                   ||       || to halt the VCPU without exits,
+                                   ||       || time spent while halted in this
+                                   ||       || way is accounted for on host as
+                                   ||       || VCPU run time.
+------------------------------------------------------------------------------
 KVM_FEATURE_CLOCKSOURCE_STABLE_BIT ||    24 || host will warn if no guest-side
                                    ||       || per-cpu warps are expected in
                                    ||       || kvmclock.
diff --git a/arch/x86/include/uapi/asm/kvm_para.h b/arch/x86/include/uapi/asm/kvm_para.h
index cff0bb6..9cc77a7 100644
--- a/arch/x86/include/uapi/asm/kvm_para.h
+++ b/arch/x86/include/uapi/asm/kvm_para.h
@@ -24,6 +24,7 @@
 #define KVM_FEATURE_STEAL_TIME		5
 #define KVM_FEATURE_PV_EOI		6
 #define KVM_FEATURE_PV_UNHALT		7
+#define KVM_FEATURE_MWAIT		8
 
 /* The last 8 bits are used to indicate how to interpret the flags field
  * in pvclock structure. If no bits are set, all flags are ignored.
diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c
index efde6cc..5638102 100644
--- a/arch/x86/kvm/cpuid.c
+++ b/arch/x86/kvm/cpuid.c
@@ -594,6 +594,9 @@ static inline int __do_cpuid_ent(struct kvm_cpuid_entry2 *entry, u32 function,
 		if (sched_info_on())
 			entry->eax |= (1 << KVM_FEATURE_STEAL_TIME);
 
+		if (kvm_mwait_in_guest())
+			entry->eax |= (1 << KVM_FEATURE_MWAIT);
+
 		entry->ebx = 0;
 		entry->ecx = 0;
 		entry->edx = 0;
diff --git a/arch/x86/kvm/svm.c b/arch/x86/kvm/svm.c
index d1efe2c..18e53bc 100644
--- a/arch/x86/kvm/svm.c
+++ b/arch/x86/kvm/svm.c
@@ -1198,8 +1198,6 @@ static void init_vmcb(struct vcpu_svm *svm)
 	set_intercept(svm, INTERCEPT_CLGI);
 	set_intercept(svm, INTERCEPT_SKINIT);
 	set_intercept(svm, INTERCEPT_WBINVD);
-	set_intercept(svm, INTERCEPT_MONITOR);
-	set_intercept(svm, INTERCEPT_MWAIT);
 	set_intercept(svm, INTERCEPT_XSETBV);
 
 	control->iopm_base_pa = iopm_base;
diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c
index 98e82ee..ea0c96a 100644
--- a/arch/x86/kvm/vmx.c
+++ b/arch/x86/kvm/vmx.c
@@ -3547,11 +3547,13 @@ static __init int setup_vmcs_config(struct vmcs_config *vmcs_conf)
 	      CPU_BASED_USE_IO_BITMAPS |
 	      CPU_BASED_MOV_DR_EXITING |
 	      CPU_BASED_USE_TSC_OFFSETING |
-	      CPU_BASED_MWAIT_EXITING |
-	      CPU_BASED_MONITOR_EXITING |
 	      CPU_BASED_INVLPG_EXITING |
 	      CPU_BASED_RDPMC_EXITING;
 
+	if (!kvm_mwait_in_guest())
+		min |= CPU_BASED_MWAIT_EXITING |
+			CPU_BASED_MONITOR_EXITING;
+
 	opt = CPU_BASED_TPR_SHADOW |
 	      CPU_BASED_USE_MSR_BITMAPS |
 	      CPU_BASED_ACTIVATE_SECONDARY_CONTROLS;
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 1faf620..8c74fff 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -2684,6 +2684,9 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_ADJUST_CLOCK:
 		r = KVM_CLOCK_TSC_STABLE;
 		break;
+	case KVM_CAP_X86_GUEST_MWAIT:
+		r = kvm_mwait_in_guest();
+		break;
 	case KVM_CAP_X86_SMM:
 		/* SMBASE is usually relocated above 1M on modern chipsets,
 		 * and SMM handlers might indeed rely on 4G segment limits,
diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h
index e8ff3e4..a2d8964 100644
--- a/arch/x86/kvm/x86.h
+++ b/arch/x86/kvm/x86.h
@@ -1,6 +1,8 @@
 #ifndef ARCH_X86_KVM_X86_H
 #define ARCH_X86_KVM_X86_H
 
+#include <asm/processor.h>
+#include <asm/mwait.h>
 #include <linux/kvm_host.h>
 #include <asm/pvclock.h>
 #include "kvm_cache_regs.h"
@@ -212,4 +214,30 @@ static inline u64 nsec_to_cycles(struct kvm_vcpu *vcpu, u64 nsec)
 	    __rem;						\
 	 })
 
+static inline bool kvm_mwait_in_guest(void)
+{
+	unsigned int eax, ebx, ecx, edx;
+
+	if (!cpu_has(&boot_cpu_data, X86_FEATURE_MWAIT))
+		return false;
+
+	if (boot_cpu_data.x86_vendor != X86_VENDOR_INTEL)
+		return false;
+
+	/*
+	 * Intel CPUs without CPUID5_ECX_INTERRUPT_BREAK are problematic as
+	 * they would allow guest to stop the CPU completely by disabling
+	 * interrupts then invoking MWAIT.
+	 */
+	if (boot_cpu_data.cpuid_level < CPUID_MWAIT_LEAF)
+		return false;
+
+	cpuid(CPUID_MWAIT_LEAF, &eax, &ebx, &ecx, &edx);
+
+	if (!(ecx & CPUID5_ECX_INTERRUPT_BREAK))
+		return false;
+
+	return true;
+}
+
 #endif
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index f51d508..8b6bc06 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -883,6 +883,7 @@ struct kvm_ppc_resize_hpt {
 #define KVM_CAP_PPC_MMU_RADIX 134
 #define KVM_CAP_PPC_MMU_HASH_V3 135
 #define KVM_CAP_IMMEDIATE_EXIT 136
+#define KVM_CAP_X86_GUEST_MWAIT 137
 
 #ifdef KVM_CAP_IRQ_ROUTING
 
-- 
MST

[toc] | [next] | [standalone]

#1601811

From	"Gabriel L. Somlo" <gsomlo@gmail.com>
Date	2017-03-16 00:40 +0100
Message-ID	<tlqGm-6pL-13@gated-at.bofh.it>
In reply to	#1601754

On Wed, Mar 15, 2017 at 11:22:18PM +0200, Michael S. Tsirkin wrote:
> Guests running Mac OS 5, 6, and 7 (Leopard through Lion) have a problem:
> unless explicitly provided with kernel command line argument
> "idlehalt=0" they'd implicitly assume MONITOR and MWAIT availability,
> without checking CPUID.
> 
> We currently emulate that as a NOP but on VMX we can do better: let
> guest stop the CPU until timer, IPI or memory change.  CPU will be busy
> but that isn't any worse than a NOP emulation.
> 
> Note that mwait within guests is not the same as on real hardware
> because halt causes an exit while mwait doesn't.  For this reason it
> might not be a good idea to use the regular MWAIT flag in CPUID to
> signal this capability.  Add a flag in the hypervisor leaf instead.
> 
> Additionally, we add a capability for QEMU - e.g. if it knows there's an
> isolated CPU dedicated for the VCPU it can set the standard MWAIT flag
> to improve guest behaviour.

Same behavior (on the mac pro 1,1 running F22 with custom-compiled
kernel from kvm git master, plus this patch on top).

The OS X 10.7 kernel hangs (or at least progresses extremely slowly)
on boot, does not bring up guest graphical interface within the first
10 minutes that I waited for it. That, in contrast with the default
nop-based emulation where the guest comes up within 30 seconds.

I will run another round of tests on a newer Mac (4-year-old macbook
air) and report back tomorrow.

Going off on a tangent, why would encouraging otherwise well-behaved
guests (like linux ones, for example) to use MWAIT be desirable to
begin with ? Is it a matter of minimizing the overhead associated with
exiting and re-entering L1 ? Because if so, AFAIR staying inside L1 and
running guest-mode MWAIT in a tight loop will actually waste the host
CPU without the opportunity to yield to some other L0 thread. Sorry if
I fell into the middle of an ongoing conversation on this and missed
most of the relevant context, in which case please feel free to ignore
me... :)

Thanks,
--G

> 
> Reported-by: "Gabriel L. Somlo" <gsomlo@gmail.com>
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
> 
> This is for Gabriel's testing only. A bit rushed so untested.
> 
>  Documentation/virtual/kvm/api.txt    |  9 +++++++++
>  Documentation/virtual/kvm/cpuid.txt  |  6 ++++++
>  arch/x86/include/uapi/asm/kvm_para.h |  1 +
>  arch/x86/kvm/cpuid.c                 |  3 +++
>  arch/x86/kvm/svm.c                   |  2 --
>  arch/x86/kvm/vmx.c                   |  6 ++++--
>  arch/x86/kvm/x86.c                   |  3 +++
>  arch/x86/kvm/x86.h                   | 28 ++++++++++++++++++++++++++++
>  include/uapi/linux/kvm.h             |  1 +
>  9 files changed, 55 insertions(+), 4 deletions(-)
> 
> diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
> index 3c248f7..6ee2e43 100644
> --- a/Documentation/virtual/kvm/api.txt
> +++ b/Documentation/virtual/kvm/api.txt
> @@ -4147,3 +4147,12 @@ This capability, if KVM_CHECK_EXTENSION indicates that it is
>  available, means that that the kernel can support guests using the
>  hashed page table MMU defined in Power ISA V3.00 (as implemented in
>  the POWER9 processor), including in-memory segment tables.
> +
> +8.5 KVM_CAP_X86_GUEST_MWAIT
> +
> +Architectures: x86
> +
> +This capability indicates that guest using memory monotoring instructions
> +(MWAIT/MWAITX) to stop the virtual CPU will not cause a VM exit.  As such time
> +spent while virtual CPU is halted in this way will then be accounted for as
> +guest running time on the host (as opposed to e.g. HLT).
> diff --git a/Documentation/virtual/kvm/cpuid.txt b/Documentation/virtual/kvm/cpuid.txt
> index 3c65feb..04c201c 100644
> --- a/Documentation/virtual/kvm/cpuid.txt
> +++ b/Documentation/virtual/kvm/cpuid.txt
> @@ -54,6 +54,12 @@ KVM_FEATURE_PV_UNHALT              ||     7 || guest checks this feature bit
>                                     ||       || before enabling paravirtualized
>                                     ||       || spinlock support.
>  ------------------------------------------------------------------------------
> +KVM_FEATURE_MWAIT                  ||     8 || guest can use monitor/mwait
> +                                   ||       || to halt the VCPU without exits,
> +                                   ||       || time spent while halted in this
> +                                   ||       || way is accounted for on host as
> +                                   ||       || VCPU run time.
> +------------------------------------------------------------------------------
>  KVM_FEATURE_CLOCKSOURCE_STABLE_BIT ||    24 || host will warn if no guest-side
>                                     ||       || per-cpu warps are expected in
>                                     ||       || kvmclock.
> diff --git a/arch/x86/include/uapi/asm/kvm_para.h b/arch/x86/include/uapi/asm/kvm_para.h
> index cff0bb6..9cc77a7 100644
> --- a/arch/x86/include/uapi/asm/kvm_para.h
> +++ b/arch/x86/include/uapi/asm/kvm_para.h
> @@ -24,6 +24,7 @@
>  #define KVM_FEATURE_STEAL_TIME		5
>  #define KVM_FEATURE_PV_EOI		6
>  #define KVM_FEATURE_PV_UNHALT		7
> +#define KVM_FEATURE_MWAIT		8
>  
>  /* The last 8 bits are used to indicate how to interpret the flags field
>   * in pvclock structure. If no bits are set, all flags are ignored.
> diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c
> index efde6cc..5638102 100644
> --- a/arch/x86/kvm/cpuid.c
> +++ b/arch/x86/kvm/cpuid.c
> @@ -594,6 +594,9 @@ static inline int __do_cpuid_ent(struct kvm_cpuid_entry2 *entry, u32 function,
>  		if (sched_info_on())
>  			entry->eax |= (1 << KVM_FEATURE_STEAL_TIME);
>  
> +		if (kvm_mwait_in_guest())
> +			entry->eax |= (1 << KVM_FEATURE_MWAIT);
> +
>  		entry->ebx = 0;
>  		entry->ecx = 0;
>  		entry->edx = 0;
> diff --git a/arch/x86/kvm/svm.c b/arch/x86/kvm/svm.c
> index d1efe2c..18e53bc 100644
> --- a/arch/x86/kvm/svm.c
> +++ b/arch/x86/kvm/svm.c
> @@ -1198,8 +1198,6 @@ static void init_vmcb(struct vcpu_svm *svm)
>  	set_intercept(svm, INTERCEPT_CLGI);
>  	set_intercept(svm, INTERCEPT_SKINIT);
>  	set_intercept(svm, INTERCEPT_WBINVD);
> -	set_intercept(svm, INTERCEPT_MONITOR);
> -	set_intercept(svm, INTERCEPT_MWAIT);
>  	set_intercept(svm, INTERCEPT_XSETBV);
>  
>  	control->iopm_base_pa = iopm_base;
> diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c
> index 98e82ee..ea0c96a 100644
> --- a/arch/x86/kvm/vmx.c
> +++ b/arch/x86/kvm/vmx.c
> @@ -3547,11 +3547,13 @@ static __init int setup_vmcs_config(struct vmcs_config *vmcs_conf)
>  	      CPU_BASED_USE_IO_BITMAPS |
>  	      CPU_BASED_MOV_DR_EXITING |
>  	      CPU_BASED_USE_TSC_OFFSETING |
> -	      CPU_BASED_MWAIT_EXITING |
> -	      CPU_BASED_MONITOR_EXITING |
>  	      CPU_BASED_INVLPG_EXITING |
>  	      CPU_BASED_RDPMC_EXITING;
>  
> +	if (!kvm_mwait_in_guest())
> +		min |= CPU_BASED_MWAIT_EXITING |
> +			CPU_BASED_MONITOR_EXITING;
> +
>  	opt = CPU_BASED_TPR_SHADOW |
>  	      CPU_BASED_USE_MSR_BITMAPS |
>  	      CPU_BASED_ACTIVATE_SECONDARY_CONTROLS;
> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> index 1faf620..8c74fff 100644
> --- a/arch/x86/kvm/x86.c
> +++ b/arch/x86/kvm/x86.c
> @@ -2684,6 +2684,9 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
>  	case KVM_CAP_ADJUST_CLOCK:
>  		r = KVM_CLOCK_TSC_STABLE;
>  		break;
> +	case KVM_CAP_X86_GUEST_MWAIT:
> +		r = kvm_mwait_in_guest();
> +		break;
>  	case KVM_CAP_X86_SMM:
>  		/* SMBASE is usually relocated above 1M on modern chipsets,
>  		 * and SMM handlers might indeed rely on 4G segment limits,
> diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h
> index e8ff3e4..a2d8964 100644
> --- a/arch/x86/kvm/x86.h
> +++ b/arch/x86/kvm/x86.h
> @@ -1,6 +1,8 @@
>  #ifndef ARCH_X86_KVM_X86_H
>  #define ARCH_X86_KVM_X86_H
>  
> +#include <asm/processor.h>
> +#include <asm/mwait.h>
>  #include <linux/kvm_host.h>
>  #include <asm/pvclock.h>
>  #include "kvm_cache_regs.h"
> @@ -212,4 +214,30 @@ static inline u64 nsec_to_cycles(struct kvm_vcpu *vcpu, u64 nsec)
>  	    __rem;						\
>  	 })
>  
> +static inline bool kvm_mwait_in_guest(void)
> +{
> +	unsigned int eax, ebx, ecx, edx;
> +
> +	if (!cpu_has(&boot_cpu_data, X86_FEATURE_MWAIT))
> +		return false;
> +
> +	if (boot_cpu_data.x86_vendor != X86_VENDOR_INTEL)
> +		return false;
> +
> +	/*
> +	 * Intel CPUs without CPUID5_ECX_INTERRUPT_BREAK are problematic as
> +	 * they would allow guest to stop the CPU completely by disabling
> +	 * interrupts then invoking MWAIT.
> +	 */
> +	if (boot_cpu_data.cpuid_level < CPUID_MWAIT_LEAF)
> +		return false;
> +
> +	cpuid(CPUID_MWAIT_LEAF, &eax, &ebx, &ecx, &edx);
> +
> +	if (!(ecx & CPUID5_ECX_INTERRUPT_BREAK))
> +		return false;
> +
> +	return true;
> +}
> +
>  #endif
> diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
> index f51d508..8b6bc06 100644
> --- a/include/uapi/linux/kvm.h
> +++ b/include/uapi/linux/kvm.h
> @@ -883,6 +883,7 @@ struct kvm_ppc_resize_hpt {
>  #define KVM_CAP_PPC_MMU_RADIX 134
>  #define KVM_CAP_PPC_MMU_HASH_V3 135
>  #define KVM_CAP_IMMEDIATE_EXIT 136
> +#define KVM_CAP_X86_GUEST_MWAIT 137
>  
>  #ifdef KVM_CAP_IRQ_ROUTING
>  
> -- 
> MST

[toc] | [prev] | [next] | [standalone]

#1601814

From	"Michael S. Tsirkin" <mst@redhat.com>
Date	2017-03-16 00:50 +0100
Message-ID	<tlqQ2-6uq-7@gated-at.bofh.it>
In reply to	#1601811

On Wed, Mar 15, 2017 at 07:35:34PM -0400, Gabriel L. Somlo wrote:
> On Wed, Mar 15, 2017 at 11:22:18PM +0200, Michael S. Tsirkin wrote:
> > Guests running Mac OS 5, 6, and 7 (Leopard through Lion) have a problem:
> > unless explicitly provided with kernel command line argument
> > "idlehalt=0" they'd implicitly assume MONITOR and MWAIT availability,
> > without checking CPUID.
> > 
> > We currently emulate that as a NOP but on VMX we can do better: let
> > guest stop the CPU until timer, IPI or memory change.  CPU will be busy
> > but that isn't any worse than a NOP emulation.
> > 
> > Note that mwait within guests is not the same as on real hardware
> > because halt causes an exit while mwait doesn't.  For this reason it
> > might not be a good idea to use the regular MWAIT flag in CPUID to
> > signal this capability.  Add a flag in the hypervisor leaf instead.
> > 
> > Additionally, we add a capability for QEMU - e.g. if it knows there's an
> > isolated CPU dedicated for the VCPU it can set the standard MWAIT flag
> > to improve guest behaviour.
> 
> Same behavior (on the mac pro 1,1 running F22 with custom-compiled
> kernel from kvm git master, plus this patch on top).
> 
> The OS X 10.7 kernel hangs (or at least progresses extremely slowly)
> on boot, does not bring up guest graphical interface within the first
> 10 minutes that I waited for it. That, in contrast with the default
> nop-based emulation where the guest comes up within 30 seconds.


Thanks a lot, meanwhile I'll try to write a unit-test and experiment
with various behaviours.

> I will run another round of tests on a newer Mac (4-year-old macbook
> air) and report back tomorrow.
> 
> Going off on a tangent, why would encouraging otherwise well-behaved
> guests (like linux ones, for example) to use MWAIT be desirable to
> begin with ? Is it a matter of minimizing the overhead associated with
> exiting and re-entering L1 ? Because if so, AFAIR staying inside L1 and
> running guest-mode MWAIT in a tight loop will actually waste the host
> CPU without the opportunity to yield to some other L0 thread. Sorry if
> I fell into the middle of an ongoing conversation on this and missed
> most of the relevant context, in which case please feel free to ignore
> me... :)
> 
> Thanks,
> --G

It's just some experiments I'm running, I'm not ready to describe it
yet. I thought this part might be useful to at least some guests, so
trying to upstream it right now.

-- 
MST

[toc] | [prev] | [next] | [standalone]

#1602269

From	"Gabriel L. Somlo" <gsomlo@gmail.com>
Date	2017-03-16 14:30 +0100
Message-ID	<tlDDz-7mM-3@gated-at.bofh.it>
In reply to	#1601814

On Thu, Mar 16, 2017 at 01:41:28AM +0200, Michael S. Tsirkin wrote:
> On Wed, Mar 15, 2017 at 07:35:34PM -0400, Gabriel L. Somlo wrote:
> > On Wed, Mar 15, 2017 at 11:22:18PM +0200, Michael S. Tsirkin wrote:
> > > Guests running Mac OS 5, 6, and 7 (Leopard through Lion) have a problem:
> > > unless explicitly provided with kernel command line argument
> > > "idlehalt=0" they'd implicitly assume MONITOR and MWAIT availability,
> > > without checking CPUID.
> > > 
> > > We currently emulate that as a NOP but on VMX we can do better: let
> > > guest stop the CPU until timer, IPI or memory change.  CPU will be busy
> > > but that isn't any worse than a NOP emulation.
> > > 
> > > Note that mwait within guests is not the same as on real hardware
> > > because halt causes an exit while mwait doesn't.  For this reason it
> > > might not be a good idea to use the regular MWAIT flag in CPUID to
> > > signal this capability.  Add a flag in the hypervisor leaf instead.
> > > 
> > > Additionally, we add a capability for QEMU - e.g. if it knows there's an
> > > isolated CPU dedicated for the VCPU it can set the standard MWAIT flag
> > > to improve guest behaviour.
> > 
> > Same behavior (on the mac pro 1,1 running F22 with custom-compiled
> > kernel from kvm git master, plus this patch on top).
> > 
> > The OS X 10.7 kernel hangs (or at least progresses extremely slowly)
> > on boot, does not bring up guest graphical interface within the first
> > 10 minutes that I waited for it. That, in contrast with the default
> > nop-based emulation where the guest comes up within 30 seconds.
> 
> 
> Thanks a lot, meanwhile I'll try to write a unit-test and experiment
> with various behaviours.
> 
> > I will run another round of tests on a newer Mac (4-year-old macbook
> > air) and report back tomorrow.
> > 
> > Going off on a tangent, why would encouraging otherwise well-behaved
> > guests (like linux ones, for example) to use MWAIT be desirable to
> > begin with ? Is it a matter of minimizing the overhead associated with
> > exiting and re-entering L1 ? Because if so, AFAIR staying inside L1 and
> > running guest-mode MWAIT in a tight loop will actually waste the host
> > CPU without the opportunity to yield to some other L0 thread. Sorry if
> > I fell into the middle of an ongoing conversation on this and missed
> > most of the relevant context, in which case please feel free to ignore
> > me... :)
> > 
> > Thanks,
> > --G
> 
> It's just some experiments I'm running, I'm not ready to describe it
> yet. I thought this part might be useful to at least some guests, so
> trying to upstream it right now.

OK, so on a macbook air running F25 and the latest kvm git master plus
your v5 patch (4.11.0-rc2+), things appear to work.

host-side cpuid output:
eax=0x000040 ebx=0x000040 ecx=0x000003 edx=0x021120

guest-side cpuid output:
eax=00000000 ebx=00000000 ecx=0x000003 edx=00000000

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 42
model name	: Intel(R) Core(TM) i7-2677M CPU @ 1.80GHz
stepping	: 7
microcode	: 0x29
cpu MHz		: 1157.849
cache size	: 4096 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 2
apicid		: 3
initial apicid	: 3
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm tpr_shadow vnmi flexpriority ept vpid xsaveopt dtherm ida arat pln pts
bugs		:
bogomips	: 3604.68
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management:

After studying your patch a bit more carefully (sorry, it's crazy
around here right now :) ) I realized you're simply trying to
(selectively) decide when to exit L1 and emulate as NOP vs. when to
just allow L1 to execute MONITOR & MWAIT natively.

Is that right ? Because if so, the issues I saw on my MacPro1,1 are
weird and inexplicable, given that allowing L>=1 to run MONITOR/MWAIT
natively was one of the options Alex Graf and Rene Rebe used back in
the very early days of OS X on QEMU, at the time I got involved with
that project. Here's part of an out of tree patch against 3.4 which did
just that, and worked as far as I remember on *any* MWAIT capable
intel chip I had access to back in 2010:

##############################################################################
# 99-mwait.patch.kvm-kmod (Rene Rebe <rene@exactcode.de>) 2010-04-27
##############################################################################
diff -pNarU5 linux-3.4/arch/x86/kvm/cpuid.c linux-3.4-mac/arch/x86/kvm/cpuid.c
--- linux-3.4/arch/x86/kvm/cpuid.c	2012-05-20 18:29:13.000000000 -0400
+++ linux-3.4-mac/arch/x86/kvm/cpuid.c	2012-10-09 11:42:59.921215750 -0400
@@ -222,11 +222,11 @@ static int do_cpuid_ent(struct kvm_cpuid
 		f_nx | 0 /* Reserved */ | F(MMXEXT) | F(MMX) |
 		F(FXSR) | F(FXSR_OPT) | f_gbpages | f_rdtscp |
 		0 /* Reserved */ | f_lm | F(3DNOWEXT) | F(3DNOW);
 	/* cpuid 1.ecx */
 	const u32 kvm_supported_word4_x86_features =
-		F(XMM3) | F(PCLMULQDQ) | 0 /* DTES64, MONITOR */ |
+		F(XMM3) | F(PCLMULQDQ) | F(MWAIT) /* DTES64, MONITOR */ |
 		0 /* DS-CPL, VMX, SMX, EST */ |
 		0 /* TM2 */ | F(SSSE3) | 0 /* CNXT-ID */ | 0 /* Reserved */ |
 		F(FMA) | F(CX16) | 0 /* xTPR Update, PDCM */ |
 		0 /* Reserved, DCA */ | F(XMM4_1) |
 		F(XMM4_2) | F(X2APIC) | F(MOVBE) | F(POPCNT) |
diff -pNarU5 linux-3.4/arch/x86/kvm/svm.c linux-3.4-mac/arch/x86/kvm/svm.c
--- linux-3.4/arch/x86/kvm/svm.c	2012-05-20 18:29:13.000000000 -0400
+++ linux-3.4-mac/arch/x86/kvm/svm.c	2012-10-09 11:44:41.598997481 -0400
@@ -1102,12 +1102,10 @@ static void init_vmcb(struct vcpu_svm *s
 	set_intercept(svm, INTERCEPT_VMSAVE);
 	set_intercept(svm, INTERCEPT_STGI);
 	set_intercept(svm, INTERCEPT_CLGI);
 	set_intercept(svm, INTERCEPT_SKINIT);
 	set_intercept(svm, INTERCEPT_WBINVD);
-	set_intercept(svm, INTERCEPT_MONITOR);
-	set_intercept(svm, INTERCEPT_MWAIT);
 	set_intercept(svm, INTERCEPT_XSETBV);
 
 	control->iopm_base_pa = iopm_base;
 	control->msrpm_base_pa = __pa(svm->msrpm);
 	control->int_ctl = V_INTR_MASKING_MASK;
diff -pNarU5 linux-3.4/arch/x86/kvm/vmx.c linux-3.4-mac/arch/x86/kvm/vmx.c
--- linux-3.4/arch/x86/kvm/vmx.c	2012-05-20 18:29:13.000000000 -0400
+++ linux-3.4-mac/arch/x86/kvm/vmx.c	2012-10-09 11:42:59.925215977 -0400
@@ -1938,11 +1938,11 @@ static __init void nested_vmx_setup_ctls
 		nested_vmx_procbased_ctls_low, nested_vmx_procbased_ctls_high);
 	nested_vmx_procbased_ctls_low = 0;
 	nested_vmx_procbased_ctls_high &=
 		CPU_BASED_VIRTUAL_INTR_PENDING | CPU_BASED_USE_TSC_OFFSETING |
 		CPU_BASED_HLT_EXITING | CPU_BASED_INVLPG_EXITING |
-		CPU_BASED_MWAIT_EXITING | CPU_BASED_CR3_LOAD_EXITING |
+		CPU_BASED_CR3_LOAD_EXITING |
 		CPU_BASED_CR3_STORE_EXITING |
 #ifdef CONFIG_X86_64
 		CPU_BASED_CR8_LOAD_EXITING | CPU_BASED_CR8_STORE_EXITING |
 #endif
 		CPU_BASED_MOV_DR_EXITING | CPU_BASED_UNCOND_IO_EXITING |
@@ -2404,12 +2404,10 @@ static __init int setup_vmcs_config(stru
 	      CPU_BASED_CR3_LOAD_EXITING |
 	      CPU_BASED_CR3_STORE_EXITING |
 	      CPU_BASED_USE_IO_BITMAPS |
 	      CPU_BASED_MOV_DR_EXITING |
 	      CPU_BASED_USE_TSC_OFFSETING |
-	      CPU_BASED_MWAIT_EXITING |
-	      CPU_BASED_MONITOR_EXITING |
 	      CPU_BASED_INVLPG_EXITING |
 	      CPU_BASED_RDPMC_EXITING;
 
 	opt = CPU_BASED_TPR_SHADOW |
 	      CPU_BASED_USE_MSR_BITMAPS |

If all you're trying to do is (selectively) revert to this behavior,
that "shouldn't" mess it up for the MacPro either, so I'm thoroughly
confused at this point :)

Back in 2010, running MWAIT in L>=1  behaved 100% exactly like a NOP,
didn't power down the physical CPU, just immediately moved on to the
next instruction. As such, there was no power saving and no
opportunity to yield to another L0 thread either, unlike with NOP
emulation at L0.

Did that change on newer Intel chips (i.e., is guest-mode MWAIT now
doing something smarter than just acting as a guest-mode NOP) ?

Thanks,
--Gabriel

[toc] | [prev] | [next] | [standalone]

#1602309

From	Radim Krčmář <rkrcmar@redhat.com>
Date	2017-03-16 15:10 +0100
Message-ID	<tlEgi-7RT-17@gated-at.bofh.it>
In reply to	#1602269

2017-03-16 09:24-0400, Gabriel L. Somlo:
> On Thu, Mar 16, 2017 at 01:41:28AM +0200, Michael S. Tsirkin wrote:
> > On Wed, Mar 15, 2017 at 07:35:34PM -0400, Gabriel L. Somlo wrote:
> > > On Wed, Mar 15, 2017 at 11:22:18PM +0200, Michael S. Tsirkin wrote:
> > > > Guests running Mac OS 5, 6, and 7 (Leopard through Lion) have a problem:
> > > > unless explicitly provided with kernel command line argument
> > > > "idlehalt=0" they'd implicitly assume MONITOR and MWAIT availability,
> > > > without checking CPUID.
> > > > 
> > > > We currently emulate that as a NOP but on VMX we can do better: let
> > > > guest stop the CPU until timer, IPI or memory change.  CPU will be busy
> > > > but that isn't any worse than a NOP emulation.
> > > > 
> > > > Note that mwait within guests is not the same as on real hardware
> > > > because halt causes an exit while mwait doesn't.  For this reason it
> > > > might not be a good idea to use the regular MWAIT flag in CPUID to
> > > > signal this capability.  Add a flag in the hypervisor leaf instead.
> > > > 
> > > > Additionally, we add a capability for QEMU - e.g. if it knows there's an
> > > > isolated CPU dedicated for the VCPU it can set the standard MWAIT flag
> > > > to improve guest behaviour.
> > > 
> > > Same behavior (on the mac pro 1,1 running F22 with custom-compiled
> > > kernel from kvm git master, plus this patch on top).
> > > 
> > > The OS X 10.7 kernel hangs (or at least progresses extremely slowly)
> > > on boot, does not bring up guest graphical interface within the first
> > > 10 minutes that I waited for it. That, in contrast with the default
> > > nop-based emulation where the guest comes up within 30 seconds.
> > 
> > 
> > Thanks a lot, meanwhile I'll try to write a unit-test and experiment
> > with various behaviours.
> > 
> > > I will run another round of tests on a newer Mac (4-year-old macbook
> > > air) and report back tomorrow.
> > > 
> > > Going off on a tangent, why would encouraging otherwise well-behaved
> > > guests (like linux ones, for example) to use MWAIT be desirable to
> > > begin with ? Is it a matter of minimizing the overhead associated with
> > > exiting and re-entering L1 ? Because if so, AFAIR staying inside L1 and
> > > running guest-mode MWAIT in a tight loop will actually waste the host
> > > CPU without the opportunity to yield to some other L0 thread. Sorry if
> > > I fell into the middle of an ongoing conversation on this and missed
> > > most of the relevant context, in which case please feel free to ignore
> > > me... :)
> > > 
> > > Thanks,
> > > --G
> > 
> > It's just some experiments I'm running, I'm not ready to describe it
> > yet. I thought this part might be useful to at least some guests, so
> > trying to upstream it right now.
> 
> OK, so on a macbook air running F25 and the latest kvm git master plus
> your v5 patch (4.11.0-rc2+), things appear to work.
> 
> host-side cpuid output:
> eax=0x000040 ebx=0x000040 ecx=0x000003 edx=0x021120
> 
> guest-side cpuid output:
> eax=00000000 ebx=00000000 ecx=0x000003 edx=00000000
> 
> processor	: 3
> vendor_id	: GenuineIntel
> cpu family	: 6
> model		: 42
> model name	: Intel(R) Core(TM) i7-2677M CPU @ 1.80GHz
> stepping	: 7
> microcode	: 0x29
> cpu MHz		: 1157.849
> cache size	: 4096 KB
> physical id	: 0
> siblings	: 4
> core id		: 1
> cpu cores	: 2
> apicid		: 3
> initial apicid	: 3
> fpu		: yes
> fpu_exception	: yes
> cpuid level	: 13
> wp		: yes
> flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm tpr_shadow vnmi flexpriority ept vpid xsaveopt dtherm ida arat pln pts
> bugs		:
> bogomips	: 3604.68
> clflush size	: 64
> cache_alignment	: 64
> address sizes	: 36 bits physical, 48 bits virtual
> power management:
> 
> After studying your patch a bit more carefully (sorry, it's crazy
> around here right now :) ) I realized you're simply trying to
> (selectively) decide when to exit L1 and emulate as NOP vs. when to
> just allow L1 to execute MONITOR & MWAIT natively.
> 
> Is that right ? Because if so, the issues I saw on my MacPro1,1 are
> weird and inexplicable, given that allowing L>=1 to run MONITOR/MWAIT
> natively was one of the options Alex Graf and Rene Rebe used back in
> the very early days of OS X on QEMU, at the time I got involved with
> that project. Here's part of an out of tree patch against 3.4 which did
> just that, and worked as far as I remember on *any* MWAIT capable
> intel chip I had access to back in 2010:
> 
> ##############################################################################
> # 99-mwait.patch.kvm-kmod (Rene Rebe <rene@exactcode.de>) 2010-04-27
> ##############################################################################
> diff -pNarU5 linux-3.4/arch/x86/kvm/cpuid.c linux-3.4-mac/arch/x86/kvm/cpuid.c
> --- linux-3.4/arch/x86/kvm/cpuid.c	2012-05-20 18:29:13.000000000 -0400
> +++ linux-3.4-mac/arch/x86/kvm/cpuid.c	2012-10-09 11:42:59.921215750 -0400
> @@ -222,11 +222,11 @@ static int do_cpuid_ent(struct kvm_cpuid
>  		f_nx | 0 /* Reserved */ | F(MMXEXT) | F(MMX) |
>  		F(FXSR) | F(FXSR_OPT) | f_gbpages | f_rdtscp |
>  		0 /* Reserved */ | f_lm | F(3DNOWEXT) | F(3DNOW);
>  	/* cpuid 1.ecx */
>  	const u32 kvm_supported_word4_x86_features =
> -		F(XMM3) | F(PCLMULQDQ) | 0 /* DTES64, MONITOR */ |
> +		F(XMM3) | F(PCLMULQDQ) | F(MWAIT) /* DTES64, MONITOR */ |
>  		0 /* DS-CPL, VMX, SMX, EST */ |
>  		0 /* TM2 */ | F(SSSE3) | 0 /* CNXT-ID */ | 0 /* Reserved */ |
>  		F(FMA) | F(CX16) | 0 /* xTPR Update, PDCM */ |
>  		0 /* Reserved, DCA */ | F(XMM4_1) |
>  		F(XMM4_2) | F(X2APIC) | F(MOVBE) | F(POPCNT) |
> diff -pNarU5 linux-3.4/arch/x86/kvm/svm.c linux-3.4-mac/arch/x86/kvm/svm.c
> --- linux-3.4/arch/x86/kvm/svm.c	2012-05-20 18:29:13.000000000 -0400
> +++ linux-3.4-mac/arch/x86/kvm/svm.c	2012-10-09 11:44:41.598997481 -0400
> @@ -1102,12 +1102,10 @@ static void init_vmcb(struct vcpu_svm *s
>  	set_intercept(svm, INTERCEPT_VMSAVE);
>  	set_intercept(svm, INTERCEPT_STGI);
>  	set_intercept(svm, INTERCEPT_CLGI);
>  	set_intercept(svm, INTERCEPT_SKINIT);
>  	set_intercept(svm, INTERCEPT_WBINVD);
> -	set_intercept(svm, INTERCEPT_MONITOR);
> -	set_intercept(svm, INTERCEPT_MWAIT);
>  	set_intercept(svm, INTERCEPT_XSETBV);
>  
>  	control->iopm_base_pa = iopm_base;
>  	control->msrpm_base_pa = __pa(svm->msrpm);
>  	control->int_ctl = V_INTR_MASKING_MASK;
> diff -pNarU5 linux-3.4/arch/x86/kvm/vmx.c linux-3.4-mac/arch/x86/kvm/vmx.c
> --- linux-3.4/arch/x86/kvm/vmx.c	2012-05-20 18:29:13.000000000 -0400
> +++ linux-3.4-mac/arch/x86/kvm/vmx.c	2012-10-09 11:42:59.925215977 -0400
> @@ -1938,11 +1938,11 @@ static __init void nested_vmx_setup_ctls
>  		nested_vmx_procbased_ctls_low, nested_vmx_procbased_ctls_high);
>  	nested_vmx_procbased_ctls_low = 0;
>  	nested_vmx_procbased_ctls_high &=
>  		CPU_BASED_VIRTUAL_INTR_PENDING | CPU_BASED_USE_TSC_OFFSETING |
>  		CPU_BASED_HLT_EXITING | CPU_BASED_INVLPG_EXITING |
> -		CPU_BASED_MWAIT_EXITING | CPU_BASED_CR3_LOAD_EXITING |
> +		CPU_BASED_CR3_LOAD_EXITING |
>  		CPU_BASED_CR3_STORE_EXITING |
>  #ifdef CONFIG_X86_64
>  		CPU_BASED_CR8_LOAD_EXITING | CPU_BASED_CR8_STORE_EXITING |
>  #endif
>  		CPU_BASED_MOV_DR_EXITING | CPU_BASED_UNCOND_IO_EXITING |
> @@ -2404,12 +2404,10 @@ static __init int setup_vmcs_config(stru
>  	      CPU_BASED_CR3_LOAD_EXITING |
>  	      CPU_BASED_CR3_STORE_EXITING |
>  	      CPU_BASED_USE_IO_BITMAPS |
>  	      CPU_BASED_MOV_DR_EXITING |
>  	      CPU_BASED_USE_TSC_OFFSETING |
> -	      CPU_BASED_MWAIT_EXITING |
> -	      CPU_BASED_MONITOR_EXITING |
>  	      CPU_BASED_INVLPG_EXITING |
>  	      CPU_BASED_RDPMC_EXITING;
>  
>  	opt = CPU_BASED_TPR_SHADOW |
>  	      CPU_BASED_USE_MSR_BITMAPS |
> 
> If all you're trying to do is (selectively) revert to this behavior,
> that "shouldn't" mess it up for the MacPro either, so I'm thoroughly
> confused at this point :)
> 
> Back in 2010, running MWAIT in L>=1  behaved 100% exactly like a NOP,
> didn't power down the physical CPU, just immediately moved on to the
> next instruction. As such, there was no power saving and no
> opportunity to yield to another L0 thread either, unlike with NOP
> emulation at L0.
> 
> Did that change on newer Intel chips (i.e., is guest-mode MWAIT now
> doing something smarter than just acting as a guest-mode NOP) ?

Probably, MWAIT in new intel chips enters power saving mode normally.

If hardware-executed MWAIT acted as a NOP in your old chip, then that
shouldn't be a problem either ...  Maybe OS X gets confused into doing
something really dumb because we do not expose the MONITOR/MWAIT feature
bit correctly.

Can you try this QEMU patch on the old hardware?

diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index 7aa762245a54..4b112e12188a 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -2764,10 +2764,7 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count,
         break;
     case 5:
         /* mwait info: needed for Core compatibility */
-        *eax = 0; /* Smallest monitor-line size in bytes */
-        *ebx = 0; /* Largest monitor-line size in bytes */
-        *ecx = CPUID_MWAIT_EMX | CPUID_MWAIT_IBE;
-        *edx = 0;
+        host_cpuid(index, 0, eax, ebx, ecx, edx);
         break;
     case 6:
         /* Thermal and Power Leaf */
diff --git a/target/i386/kvm.c b/target/i386/kvm.c
index 55865dbee0aa..1eb78291b093 100644
--- a/target/i386/kvm.c
+++ b/target/i386/kvm.c
@@ -360,6 +360,7 @@ uint32_t kvm_arch_get_supported_cpuid(KVMState *s, uint32_t function,
         if (!kvm_irqchip_in_kernel()) {
             ret &= ~CPUID_EXT_X2APIC;
         }
+        ret |= CPUID_EXT_MONITOR;
     } else if (function == 6 && reg == R_EAX) {
         ret |= CPUID_6_EAX_ARAT; /* safe to allow because of emulated APIC */
     } else if (function == 7 && index == 0 && reg == R_EBX) {


Thanks.

[toc] | [prev] | [next] | [standalone]

#1602590

From	"Gabriel L. Somlo" <gsomlo@gmail.com>
Date	2017-03-16 17:00 +0100
Message-ID	<tlFYJ-td-15@gated-at.bofh.it>
In reply to	#1602309

On Thu, Mar 16, 2017 at 03:08:07PM +0100, Radim Krčmář wrote:
> 2017-03-16 09:24-0400, Gabriel L. Somlo:
> > On Thu, Mar 16, 2017 at 01:41:28AM +0200, Michael S. Tsirkin wrote:
> > > On Wed, Mar 15, 2017 at 07:35:34PM -0400, Gabriel L. Somlo wrote:
> > > > On Wed, Mar 15, 2017 at 11:22:18PM +0200, Michael S. Tsirkin wrote:
> > > > > Guests running Mac OS 5, 6, and 7 (Leopard through Lion) have a problem:
> > > > > unless explicitly provided with kernel command line argument
> > > > > "idlehalt=0" they'd implicitly assume MONITOR and MWAIT availability,
> > > > > without checking CPUID.
> > > > > 
> > > > > We currently emulate that as a NOP but on VMX we can do better: let
> > > > > guest stop the CPU until timer, IPI or memory change.  CPU will be busy
> > > > > but that isn't any worse than a NOP emulation.
> > > > > 
> > > > > Note that mwait within guests is not the same as on real hardware
> > > > > because halt causes an exit while mwait doesn't.  For this reason it
> > > > > might not be a good idea to use the regular MWAIT flag in CPUID to
> > > > > signal this capability.  Add a flag in the hypervisor leaf instead.
> > > > > 
> > > > > Additionally, we add a capability for QEMU - e.g. if it knows there's an
> > > > > isolated CPU dedicated for the VCPU it can set the standard MWAIT flag
> > > > > to improve guest behaviour.
> > > > 
> > > > Same behavior (on the mac pro 1,1 running F22 with custom-compiled
> > > > kernel from kvm git master, plus this patch on top).
> > > > 
> > > > The OS X 10.7 kernel hangs (or at least progresses extremely slowly)
> > > > on boot, does not bring up guest graphical interface within the first
> > > > 10 minutes that I waited for it. That, in contrast with the default
> > > > nop-based emulation where the guest comes up within 30 seconds.
> > > 
> > > 
> > > Thanks a lot, meanwhile I'll try to write a unit-test and experiment
> > > with various behaviours.
> > > 
> > > > I will run another round of tests on a newer Mac (4-year-old macbook
> > > > air) and report back tomorrow.
> > > > 
> > > > Going off on a tangent, why would encouraging otherwise well-behaved
> > > > guests (like linux ones, for example) to use MWAIT be desirable to
> > > > begin with ? Is it a matter of minimizing the overhead associated with
> > > > exiting and re-entering L1 ? Because if so, AFAIR staying inside L1 and
> > > > running guest-mode MWAIT in a tight loop will actually waste the host
> > > > CPU without the opportunity to yield to some other L0 thread. Sorry if
> > > > I fell into the middle of an ongoing conversation on this and missed
> > > > most of the relevant context, in which case please feel free to ignore
> > > > me... :)
> > > > 
> > > > Thanks,
> > > > --G
> > > 
> > > It's just some experiments I'm running, I'm not ready to describe it
> > > yet. I thought this part might be useful to at least some guests, so
> > > trying to upstream it right now.
> > 
> > OK, so on a macbook air running F25 and the latest kvm git master plus
> > your v5 patch (4.11.0-rc2+), things appear to work.
> > 
> > host-side cpuid output:
> > eax=0x000040 ebx=0x000040 ecx=0x000003 edx=0x021120
> > 
> > guest-side cpuid output:
> > eax=00000000 ebx=00000000 ecx=0x000003 edx=00000000
> > 
> > processor	: 3
> > vendor_id	: GenuineIntel
> > cpu family	: 6
> > model		: 42
> > model name	: Intel(R) Core(TM) i7-2677M CPU @ 1.80GHz
> > stepping	: 7
> > microcode	: 0x29
> > cpu MHz		: 1157.849
> > cache size	: 4096 KB
> > physical id	: 0
> > siblings	: 4
> > core id		: 1
> > cpu cores	: 2
> > apicid		: 3
> > initial apicid	: 3
> > fpu		: yes
> > fpu_exception	: yes
> > cpuid level	: 13
> > wp		: yes
> > flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm tpr_shadow vnmi flexpriority ept vpid xsaveopt dtherm ida arat pln pts
> > bugs		:
> > bogomips	: 3604.68
> > clflush size	: 64
> > cache_alignment	: 64
> > address sizes	: 36 bits physical, 48 bits virtual
> > power management:
> > 
> > After studying your patch a bit more carefully (sorry, it's crazy
> > around here right now :) ) I realized you're simply trying to
> > (selectively) decide when to exit L1 and emulate as NOP vs. when to
> > just allow L1 to execute MONITOR & MWAIT natively.
> > 
> > Is that right ? Because if so, the issues I saw on my MacPro1,1 are
> > weird and inexplicable, given that allowing L>=1 to run MONITOR/MWAIT
> > natively was one of the options Alex Graf and Rene Rebe used back in
> > the very early days of OS X on QEMU, at the time I got involved with
> > that project. Here's part of an out of tree patch against 3.4 which did
> > just that, and worked as far as I remember on *any* MWAIT capable
> > intel chip I had access to back in 2010:
> > 
> > ##############################################################################
> > # 99-mwait.patch.kvm-kmod (Rene Rebe <rene@exactcode.de>) 2010-04-27
> > ##############################################################################
> > diff -pNarU5 linux-3.4/arch/x86/kvm/cpuid.c linux-3.4-mac/arch/x86/kvm/cpuid.c
> > --- linux-3.4/arch/x86/kvm/cpuid.c	2012-05-20 18:29:13.000000000 -0400
> > +++ linux-3.4-mac/arch/x86/kvm/cpuid.c	2012-10-09 11:42:59.921215750 -0400
> > @@ -222,11 +222,11 @@ static int do_cpuid_ent(struct kvm_cpuid
> >  		f_nx | 0 /* Reserved */ | F(MMXEXT) | F(MMX) |
> >  		F(FXSR) | F(FXSR_OPT) | f_gbpages | f_rdtscp |
> >  		0 /* Reserved */ | f_lm | F(3DNOWEXT) | F(3DNOW);
> >  	/* cpuid 1.ecx */
> >  	const u32 kvm_supported_word4_x86_features =
> > -		F(XMM3) | F(PCLMULQDQ) | 0 /* DTES64, MONITOR */ |
> > +		F(XMM3) | F(PCLMULQDQ) | F(MWAIT) /* DTES64, MONITOR */ |
> >  		0 /* DS-CPL, VMX, SMX, EST */ |
> >  		0 /* TM2 */ | F(SSSE3) | 0 /* CNXT-ID */ | 0 /* Reserved */ |
> >  		F(FMA) | F(CX16) | 0 /* xTPR Update, PDCM */ |
> >  		0 /* Reserved, DCA */ | F(XMM4_1) |
> >  		F(XMM4_2) | F(X2APIC) | F(MOVBE) | F(POPCNT) |
> > diff -pNarU5 linux-3.4/arch/x86/kvm/svm.c linux-3.4-mac/arch/x86/kvm/svm.c
> > --- linux-3.4/arch/x86/kvm/svm.c	2012-05-20 18:29:13.000000000 -0400
> > +++ linux-3.4-mac/arch/x86/kvm/svm.c	2012-10-09 11:44:41.598997481 -0400
> > @@ -1102,12 +1102,10 @@ static void init_vmcb(struct vcpu_svm *s
> >  	set_intercept(svm, INTERCEPT_VMSAVE);
> >  	set_intercept(svm, INTERCEPT_STGI);
> >  	set_intercept(svm, INTERCEPT_CLGI);
> >  	set_intercept(svm, INTERCEPT_SKINIT);
> >  	set_intercept(svm, INTERCEPT_WBINVD);
> > -	set_intercept(svm, INTERCEPT_MONITOR);
> > -	set_intercept(svm, INTERCEPT_MWAIT);
> >  	set_intercept(svm, INTERCEPT_XSETBV);
> >  
> >  	control->iopm_base_pa = iopm_base;
> >  	control->msrpm_base_pa = __pa(svm->msrpm);
> >  	control->int_ctl = V_INTR_MASKING_MASK;
> > diff -pNarU5 linux-3.4/arch/x86/kvm/vmx.c linux-3.4-mac/arch/x86/kvm/vmx.c
> > --- linux-3.4/arch/x86/kvm/vmx.c	2012-05-20 18:29:13.000000000 -0400
> > +++ linux-3.4-mac/arch/x86/kvm/vmx.c	2012-10-09 11:42:59.925215977 -0400
> > @@ -1938,11 +1938,11 @@ static __init void nested_vmx_setup_ctls
> >  		nested_vmx_procbased_ctls_low, nested_vmx_procbased_ctls_high);
> >  	nested_vmx_procbased_ctls_low = 0;
> >  	nested_vmx_procbased_ctls_high &=
> >  		CPU_BASED_VIRTUAL_INTR_PENDING | CPU_BASED_USE_TSC_OFFSETING |
> >  		CPU_BASED_HLT_EXITING | CPU_BASED_INVLPG_EXITING |
> > -		CPU_BASED_MWAIT_EXITING | CPU_BASED_CR3_LOAD_EXITING |
> > +		CPU_BASED_CR3_LOAD_EXITING |
> >  		CPU_BASED_CR3_STORE_EXITING |
> >  #ifdef CONFIG_X86_64
> >  		CPU_BASED_CR8_LOAD_EXITING | CPU_BASED_CR8_STORE_EXITING |
> >  #endif
> >  		CPU_BASED_MOV_DR_EXITING | CPU_BASED_UNCOND_IO_EXITING |
> > @@ -2404,12 +2404,10 @@ static __init int setup_vmcs_config(stru
> >  	      CPU_BASED_CR3_LOAD_EXITING |
> >  	      CPU_BASED_CR3_STORE_EXITING |
> >  	      CPU_BASED_USE_IO_BITMAPS |
> >  	      CPU_BASED_MOV_DR_EXITING |
> >  	      CPU_BASED_USE_TSC_OFFSETING |
> > -	      CPU_BASED_MWAIT_EXITING |
> > -	      CPU_BASED_MONITOR_EXITING |
> >  	      CPU_BASED_INVLPG_EXITING |
> >  	      CPU_BASED_RDPMC_EXITING;
> >  
> >  	opt = CPU_BASED_TPR_SHADOW |
> >  	      CPU_BASED_USE_MSR_BITMAPS |
> > 
> > If all you're trying to do is (selectively) revert to this behavior,
> > that "shouldn't" mess it up for the MacPro either, so I'm thoroughly
> > confused at this point :)
> > 
> > Back in 2010, running MWAIT in L>=1  behaved 100% exactly like a NOP,
> > didn't power down the physical CPU, just immediately moved on to the
> > next instruction. As such, there was no power saving and no
> > opportunity to yield to another L0 thread either, unlike with NOP
> > emulation at L0.
> > 
> > Did that change on newer Intel chips (i.e., is guest-mode MWAIT now
> > doing something smarter than just acting as a guest-mode NOP) ?
> 
> Probably, MWAIT in new intel chips enters power saving mode normally.
> 
> If hardware-executed MWAIT acted as a NOP in your old chip, then that
> shouldn't be a problem either ...  Maybe OS X gets confused into doing
> something really dumb because we do not expose the MONITOR/MWAIT feature
> bit correctly.
> 
> Can you try this QEMU patch on the old hardware?
> 
> diff --git a/target/i386/cpu.c b/target/i386/cpu.c
> index 7aa762245a54..4b112e12188a 100644
> --- a/target/i386/cpu.c
> +++ b/target/i386/cpu.c
> @@ -2764,10 +2764,7 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count,
>          break;
>      case 5:
>          /* mwait info: needed for Core compatibility */
> -        *eax = 0; /* Smallest monitor-line size in bytes */
> -        *ebx = 0; /* Largest monitor-line size in bytes */
> -        *ecx = CPUID_MWAIT_EMX | CPUID_MWAIT_IBE;
> -        *edx = 0;
> +        host_cpuid(index, 0, eax, ebx, ecx, edx);
>          break;
>      case 6:
>          /* Thermal and Power Leaf */
> diff --git a/target/i386/kvm.c b/target/i386/kvm.c
> index 55865dbee0aa..1eb78291b093 100644
> --- a/target/i386/kvm.c
> +++ b/target/i386/kvm.c
> @@ -360,6 +360,7 @@ uint32_t kvm_arch_get_supported_cpuid(KVMState *s, uint32_t function,
>          if (!kvm_irqchip_in_kernel()) {
>              ret &= ~CPUID_EXT_X2APIC;
>          }
> +        ret |= CPUID_EXT_MONITOR;
>      } else if (function == 6 && reg == R_EAX) {
>          ret |= CPUID_6_EAX_ARAT; /* safe to allow because of emulated APIC */
>      } else if (function == 7 && index == 0 && reg == R_EBX) {
> 
> 
> Thanks.

No change, still hangs on boot.

Thanks,
--G

[toc] | [prev] | [next] | [standalone]

#1602591

From	Radim Krčmář <rkrcmar@redhat.com>
Date	2017-03-16 17:00 +0100
Message-ID	<tlFYK-td-17@gated-at.bofh.it>
In reply to	#1602590

2017-03-16 11:44-0400, Gabriel L. Somlo:
> On Thu, Mar 16, 2017 at 03:08:07PM +0100, Radim Krčmář wrote:
>> 2017-03-16 09:24-0400, Gabriel L. Somlo:
>> > On Thu, Mar 16, 2017 at 01:41:28AM +0200, Michael S. Tsirkin wrote:
>> > > On Wed, Mar 15, 2017 at 07:35:34PM -0400, Gabriel L. Somlo wrote:
>> > > > On Wed, Mar 15, 2017 at 11:22:18PM +0200, Michael S. Tsirkin wrote:
>> > > > > Guests running Mac OS 5, 6, and 7 (Leopard through Lion) have a problem:
>> > > > > unless explicitly provided with kernel command line argument
>> > > > > "idlehalt=0" they'd implicitly assume MONITOR and MWAIT availability,
>> > > > > without checking CPUID.
>> > > > > 
>> > > > > We currently emulate that as a NOP but on VMX we can do better: let
>> > > > > guest stop the CPU until timer, IPI or memory change.  CPU will be busy
>> > > > > but that isn't any worse than a NOP emulation.
>> > > > > 
>> > > > > Note that mwait within guests is not the same as on real hardware
>> > > > > because halt causes an exit while mwait doesn't.  For this reason it
>> > > > > might not be a good idea to use the regular MWAIT flag in CPUID to
>> > > > > signal this capability.  Add a flag in the hypervisor leaf instead.
>> > > > > 
>> > > > > Additionally, we add a capability for QEMU - e.g. if it knows there's an
>> > > > > isolated CPU dedicated for the VCPU it can set the standard MWAIT flag
>> > > > > to improve guest behaviour.
>> > > > 
>> > > > Same behavior (on the mac pro 1,1 running F22 with custom-compiled
>> > > > kernel from kvm git master, plus this patch on top).
>> > > > 
>> > > > The OS X 10.7 kernel hangs (or at least progresses extremely slowly)
>> > > > on boot, does not bring up guest graphical interface within the first
>> > > > 10 minutes that I waited for it. That, in contrast with the default
>> > > > nop-based emulation where the guest comes up within 30 seconds.
>> > > 
>> > > 
>> > > Thanks a lot, meanwhile I'll try to write a unit-test and experiment
>> > > with various behaviours.
>> > > 
>> > > > I will run another round of tests on a newer Mac (4-year-old macbook
>> > > > air) and report back tomorrow.
>> > > > 
>> > > > Going off on a tangent, why would encouraging otherwise well-behaved
>> > > > guests (like linux ones, for example) to use MWAIT be desirable to
>> > > > begin with ? Is it a matter of minimizing the overhead associated with
>> > > > exiting and re-entering L1 ? Because if so, AFAIR staying inside L1 and
>> > > > running guest-mode MWAIT in a tight loop will actually waste the host
>> > > > CPU without the opportunity to yield to some other L0 thread. Sorry if
>> > > > I fell into the middle of an ongoing conversation on this and missed
>> > > > most of the relevant context, in which case please feel free to ignore
>> > > > me... :)
>> > > > 
>> > > > Thanks,
>> > > > --G
>> > > 
>> > > It's just some experiments I'm running, I'm not ready to describe it
>> > > yet. I thought this part might be useful to at least some guests, so
>> > > trying to upstream it right now.
>> > 
>> > OK, so on a macbook air running F25 and the latest kvm git master plus
>> > your v5 patch (4.11.0-rc2+), things appear to work.
>> > 
>> > host-side cpuid output:
>> > eax=0x000040 ebx=0x000040 ecx=0x000003 edx=0x021120
>> > 
>> > guest-side cpuid output:
>> > eax=00000000 ebx=00000000 ecx=0x000003 edx=00000000
>> > 
>> > processor	: 3
>> > vendor_id	: GenuineIntel
>> > cpu family	: 6
>> > model		: 42
>> > model name	: Intel(R) Core(TM) i7-2677M CPU @ 1.80GHz
>> > stepping	: 7
>> > microcode	: 0x29
>> > cpu MHz		: 1157.849
>> > cache size	: 4096 KB
>> > physical id	: 0
>> > siblings	: 4
>> > core id		: 1
>> > cpu cores	: 2
>> > apicid		: 3
>> > initial apicid	: 3
>> > fpu		: yes
>> > fpu_exception	: yes
>> > cpuid level	: 13
>> > wp		: yes
>> > flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm tpr_shadow vnmi flexpriority ept vpid xsaveopt dtherm ida arat pln pts
>> > bugs		:
>> > bogomips	: 3604.68
>> > clflush size	: 64
>> > cache_alignment	: 64
>> > address sizes	: 36 bits physical, 48 bits virtual
>> > power management:
>> > 
>> > After studying your patch a bit more carefully (sorry, it's crazy
>> > around here right now :) ) I realized you're simply trying to
>> > (selectively) decide when to exit L1 and emulate as NOP vs. when to
>> > just allow L1 to execute MONITOR & MWAIT natively.
>> > 
>> > Is that right ? Because if so, the issues I saw on my MacPro1,1 are
>> > weird and inexplicable, given that allowing L>=1 to run MONITOR/MWAIT
>> > natively was one of the options Alex Graf and Rene Rebe used back in
>> > the very early days of OS X on QEMU, at the time I got involved with
>> > that project. Here's part of an out of tree patch against 3.4 which did
>> > just that, and worked as far as I remember on *any* MWAIT capable
>> > intel chip I had access to back in 2010:
>> > 
>> > ##############################################################################
>> > # 99-mwait.patch.kvm-kmod (Rene Rebe <rene@exactcode.de>) 2010-04-27
>> > ##############################################################################
>> > diff -pNarU5 linux-3.4/arch/x86/kvm/cpuid.c linux-3.4-mac/arch/x86/kvm/cpuid.c
>> > --- linux-3.4/arch/x86/kvm/cpuid.c	2012-05-20 18:29:13.000000000 -0400
>> > +++ linux-3.4-mac/arch/x86/kvm/cpuid.c	2012-10-09 11:42:59.921215750 -0400
>> > @@ -222,11 +222,11 @@ static int do_cpuid_ent(struct kvm_cpuid
>> >  		f_nx | 0 /* Reserved */ | F(MMXEXT) | F(MMX) |
>> >  		F(FXSR) | F(FXSR_OPT) | f_gbpages | f_rdtscp |
>> >  		0 /* Reserved */ | f_lm | F(3DNOWEXT) | F(3DNOW);
>> >  	/* cpuid 1.ecx */
>> >  	const u32 kvm_supported_word4_x86_features =
>> > -		F(XMM3) | F(PCLMULQDQ) | 0 /* DTES64, MONITOR */ |
>> > +		F(XMM3) | F(PCLMULQDQ) | F(MWAIT) /* DTES64, MONITOR */ |
>> >  		0 /* DS-CPL, VMX, SMX, EST */ |
>> >  		0 /* TM2 */ | F(SSSE3) | 0 /* CNXT-ID */ | 0 /* Reserved */ |
>> >  		F(FMA) | F(CX16) | 0 /* xTPR Update, PDCM */ |
>> >  		0 /* Reserved, DCA */ | F(XMM4_1) |
>> >  		F(XMM4_2) | F(X2APIC) | F(MOVBE) | F(POPCNT) |
>> > diff -pNarU5 linux-3.4/arch/x86/kvm/svm.c linux-3.4-mac/arch/x86/kvm/svm.c
>> > --- linux-3.4/arch/x86/kvm/svm.c	2012-05-20 18:29:13.000000000 -0400
>> > +++ linux-3.4-mac/arch/x86/kvm/svm.c	2012-10-09 11:44:41.598997481 -0400
>> > @@ -1102,12 +1102,10 @@ static void init_vmcb(struct vcpu_svm *s
>> >  	set_intercept(svm, INTERCEPT_VMSAVE);
>> >  	set_intercept(svm, INTERCEPT_STGI);
>> >  	set_intercept(svm, INTERCEPT_CLGI);
>> >  	set_intercept(svm, INTERCEPT_SKINIT);
>> >  	set_intercept(svm, INTERCEPT_WBINVD);
>> > -	set_intercept(svm, INTERCEPT_MONITOR);
>> > -	set_intercept(svm, INTERCEPT_MWAIT);
>> >  	set_intercept(svm, INTERCEPT_XSETBV);
>> >  
>> >  	control->iopm_base_pa = iopm_base;
>> >  	control->msrpm_base_pa = __pa(svm->msrpm);
>> >  	control->int_ctl = V_INTR_MASKING_MASK;
>> > diff -pNarU5 linux-3.4/arch/x86/kvm/vmx.c linux-3.4-mac/arch/x86/kvm/vmx.c
>> > --- linux-3.4/arch/x86/kvm/vmx.c	2012-05-20 18:29:13.000000000 -0400
>> > +++ linux-3.4-mac/arch/x86/kvm/vmx.c	2012-10-09 11:42:59.925215977 -0400
>> > @@ -1938,11 +1938,11 @@ static __init void nested_vmx_setup_ctls
>> >  		nested_vmx_procbased_ctls_low, nested_vmx_procbased_ctls_high);
>> >  	nested_vmx_procbased_ctls_low = 0;
>> >  	nested_vmx_procbased_ctls_high &=
>> >  		CPU_BASED_VIRTUAL_INTR_PENDING | CPU_BASED_USE_TSC_OFFSETING |
>> >  		CPU_BASED_HLT_EXITING | CPU_BASED_INVLPG_EXITING |
>> > -		CPU_BASED_MWAIT_EXITING | CPU_BASED_CR3_LOAD_EXITING |
>> > +		CPU_BASED_CR3_LOAD_EXITING |
>> >  		CPU_BASED_CR3_STORE_EXITING |
>> >  #ifdef CONFIG_X86_64
>> >  		CPU_BASED_CR8_LOAD_EXITING | CPU_BASED_CR8_STORE_EXITING |
>> >  #endif
>> >  		CPU_BASED_MOV_DR_EXITING | CPU_BASED_UNCOND_IO_EXITING |
>> > @@ -2404,12 +2404,10 @@ static __init int setup_vmcs_config(stru
>> >  	      CPU_BASED_CR3_LOAD_EXITING |
>> >  	      CPU_BASED_CR3_STORE_EXITING |
>> >  	      CPU_BASED_USE_IO_BITMAPS |
>> >  	      CPU_BASED_MOV_DR_EXITING |
>> >  	      CPU_BASED_USE_TSC_OFFSETING |
>> > -	      CPU_BASED_MWAIT_EXITING |
>> > -	      CPU_BASED_MONITOR_EXITING |
>> >  	      CPU_BASED_INVLPG_EXITING |
>> >  	      CPU_BASED_RDPMC_EXITING;
>> >  
>> >  	opt = CPU_BASED_TPR_SHADOW |
>> >  	      CPU_BASED_USE_MSR_BITMAPS |
>> > 
>> > If all you're trying to do is (selectively) revert to this behavior,
>> > that "shouldn't" mess it up for the MacPro either, so I'm thoroughly
>> > confused at this point :)
>> > 
>> > Back in 2010, running MWAIT in L>=1  behaved 100% exactly like a NOP,
>> > didn't power down the physical CPU, just immediately moved on to the
>> > next instruction. As such, there was no power saving and no
>> > opportunity to yield to another L0 thread either, unlike with NOP
>> > emulation at L0.
>> > 
>> > Did that change on newer Intel chips (i.e., is guest-mode MWAIT now
>> > doing something smarter than just acting as a guest-mode NOP) ?
>> 
>> Probably, MWAIT in new intel chips enters power saving mode normally.
>> 
>> If hardware-executed MWAIT acted as a NOP in your old chip, then that
>> shouldn't be a problem either ...  Maybe OS X gets confused into doing
>> something really dumb because we do not expose the MONITOR/MWAIT feature
>> bit correctly.
>> 
>> Can you try this QEMU patch on the old hardware?
>> 
>> diff --git a/target/i386/cpu.c b/target/i386/cpu.c
>> index 7aa762245a54..4b112e12188a 100644
>> --- a/target/i386/cpu.c
>> +++ b/target/i386/cpu.c
>> @@ -2764,10 +2764,7 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count,
>>          break;
>>      case 5:
>>          /* mwait info: needed for Core compatibility */
>> -        *eax = 0; /* Smallest monitor-line size in bytes */
>> -        *ebx = 0; /* Largest monitor-line size in bytes */
>> -        *ecx = CPUID_MWAIT_EMX | CPUID_MWAIT_IBE;
>> -        *edx = 0;
>> +        host_cpuid(index, 0, eax, ebx, ecx, edx);
>>          break;
>>      case 6:
>>          /* Thermal and Power Leaf */
>> diff --git a/target/i386/kvm.c b/target/i386/kvm.c
>> index 55865dbee0aa..1eb78291b093 100644
>> --- a/target/i386/kvm.c
>> +++ b/target/i386/kvm.c
>> @@ -360,6 +360,7 @@ uint32_t kvm_arch_get_supported_cpuid(KVMState *s, uint32_t function,
>>          if (!kvm_irqchip_in_kernel()) {
>>              ret &= ~CPUID_EXT_X2APIC;
>>          }
>> +        ret |= CPUID_EXT_MONITOR;
>>      } else if (function == 6 && reg == R_EAX) {
>>          ret |= CPUID_6_EAX_ARAT; /* safe to allow because of emulated APIC */
>>      } else if (function == 7 && index == 0 && reg == R_EBX) {
>> 
>> 
>> Thanks.
> 
> No change, still hangs on boot.

Hm, also with '-cpu host'?
(I forgot that the CPUID_EXT_MONITOR isn't visible in the guest
 otherwise ...)

Thanks.

[toc] | [prev] | [next] | [standalone]

#1602626

From	"Gabriel L. Somlo" <gsomlo@gmail.com>
Date	2017-03-16 17:30 +0100
Message-ID	<tlGrL-X3-3@gated-at.bofh.it>
In reply to	#1602591

On Thu, Mar 16, 2017 at 04:54:06PM +0100, Radim Krčmář wrote:
> 2017-03-16 11:44-0400, Gabriel L. Somlo:
> > On Thu, Mar 16, 2017 at 03:08:07PM +0100, Radim Krčmář wrote:
> >> 2017-03-16 09:24-0400, Gabriel L. Somlo:
> >> > On Thu, Mar 16, 2017 at 01:41:28AM +0200, Michael S. Tsirkin wrote:
> >> > > On Wed, Mar 15, 2017 at 07:35:34PM -0400, Gabriel L. Somlo wrote:
> >> > > > On Wed, Mar 15, 2017 at 11:22:18PM +0200, Michael S. Tsirkin wrote:
> >> > > > > Guests running Mac OS 5, 6, and 7 (Leopard through Lion) have a problem:
> >> > > > > unless explicitly provided with kernel command line argument
> >> > > > > "idlehalt=0" they'd implicitly assume MONITOR and MWAIT availability,
> >> > > > > without checking CPUID.
> >> > > > > 
> >> > > > > We currently emulate that as a NOP but on VMX we can do better: let
> >> > > > > guest stop the CPU until timer, IPI or memory change.  CPU will be busy
> >> > > > > but that isn't any worse than a NOP emulation.
> >> > > > > 
> >> > > > > Note that mwait within guests is not the same as on real hardware
> >> > > > > because halt causes an exit while mwait doesn't.  For this reason it
> >> > > > > might not be a good idea to use the regular MWAIT flag in CPUID to
> >> > > > > signal this capability.  Add a flag in the hypervisor leaf instead.
> >> > > > > 
> >> > > > > Additionally, we add a capability for QEMU - e.g. if it knows there's an
> >> > > > > isolated CPU dedicated for the VCPU it can set the standard MWAIT flag
> >> > > > > to improve guest behaviour.
> >> > > > 
> >> > > > Same behavior (on the mac pro 1,1 running F22 with custom-compiled
> >> > > > kernel from kvm git master, plus this patch on top).
> >> > > > 
> >> > > > The OS X 10.7 kernel hangs (or at least progresses extremely slowly)
> >> > > > on boot, does not bring up guest graphical interface within the first
> >> > > > 10 minutes that I waited for it. That, in contrast with the default
> >> > > > nop-based emulation where the guest comes up within 30 seconds.
> >> > > 
> >> > > 
> >> > > Thanks a lot, meanwhile I'll try to write a unit-test and experiment
> >> > > with various behaviours.
> >> > > 
> >> > > > I will run another round of tests on a newer Mac (4-year-old macbook
> >> > > > air) and report back tomorrow.
> >> > > > 
> >> > > > Going off on a tangent, why would encouraging otherwise well-behaved
> >> > > > guests (like linux ones, for example) to use MWAIT be desirable to
> >> > > > begin with ? Is it a matter of minimizing the overhead associated with
> >> > > > exiting and re-entering L1 ? Because if so, AFAIR staying inside L1 and
> >> > > > running guest-mode MWAIT in a tight loop will actually waste the host
> >> > > > CPU without the opportunity to yield to some other L0 thread. Sorry if
> >> > > > I fell into the middle of an ongoing conversation on this and missed
> >> > > > most of the relevant context, in which case please feel free to ignore
> >> > > > me... :)
> >> > > > 
> >> > > > Thanks,
> >> > > > --G
> >> > > 
> >> > > It's just some experiments I'm running, I'm not ready to describe it
> >> > > yet. I thought this part might be useful to at least some guests, so
> >> > > trying to upstream it right now.
> >> > 
> >> > OK, so on a macbook air running F25 and the latest kvm git master plus
> >> > your v5 patch (4.11.0-rc2+), things appear to work.
> >> > 
> >> > host-side cpuid output:
> >> > eax=0x000040 ebx=0x000040 ecx=0x000003 edx=0x021120
> >> > 
> >> > guest-side cpuid output:
> >> > eax=00000000 ebx=00000000 ecx=0x000003 edx=00000000
> >> > 
> >> > processor	: 3
> >> > vendor_id	: GenuineIntel
> >> > cpu family	: 6
> >> > model		: 42
> >> > model name	: Intel(R) Core(TM) i7-2677M CPU @ 1.80GHz
> >> > stepping	: 7
> >> > microcode	: 0x29
> >> > cpu MHz		: 1157.849
> >> > cache size	: 4096 KB
> >> > physical id	: 0
> >> > siblings	: 4
> >> > core id		: 1
> >> > cpu cores	: 2
> >> > apicid		: 3
> >> > initial apicid	: 3
> >> > fpu		: yes
> >> > fpu_exception	: yes
> >> > cpuid level	: 13
> >> > wp		: yes
> >> > flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm tpr_shadow vnmi flexpriority ept vpid xsaveopt dtherm ida arat pln pts
> >> > bugs		:
> >> > bogomips	: 3604.68
> >> > clflush size	: 64
> >> > cache_alignment	: 64
> >> > address sizes	: 36 bits physical, 48 bits virtual
> >> > power management:
> >> > 
> >> > After studying your patch a bit more carefully (sorry, it's crazy
> >> > around here right now :) ) I realized you're simply trying to
> >> > (selectively) decide when to exit L1 and emulate as NOP vs. when to
> >> > just allow L1 to execute MONITOR & MWAIT natively.
> >> > 
> >> > Is that right ? Because if so, the issues I saw on my MacPro1,1 are
> >> > weird and inexplicable, given that allowing L>=1 to run MONITOR/MWAIT
> >> > natively was one of the options Alex Graf and Rene Rebe used back in
> >> > the very early days of OS X on QEMU, at the time I got involved with
> >> > that project. Here's part of an out of tree patch against 3.4 which did
> >> > just that, and worked as far as I remember on *any* MWAIT capable
> >> > intel chip I had access to back in 2010:
> >> > 
> >> > ##############################################################################
> >> > # 99-mwait.patch.kvm-kmod (Rene Rebe <rene@exactcode.de>) 2010-04-27
> >> > ##############################################################################
> >> > diff -pNarU5 linux-3.4/arch/x86/kvm/cpuid.c linux-3.4-mac/arch/x86/kvm/cpuid.c
> >> > --- linux-3.4/arch/x86/kvm/cpuid.c	2012-05-20 18:29:13.000000000 -0400
> >> > +++ linux-3.4-mac/arch/x86/kvm/cpuid.c	2012-10-09 11:42:59.921215750 -0400
> >> > @@ -222,11 +222,11 @@ static int do_cpuid_ent(struct kvm_cpuid
> >> >  		f_nx | 0 /* Reserved */ | F(MMXEXT) | F(MMX) |
> >> >  		F(FXSR) | F(FXSR_OPT) | f_gbpages | f_rdtscp |
> >> >  		0 /* Reserved */ | f_lm | F(3DNOWEXT) | F(3DNOW);
> >> >  	/* cpuid 1.ecx */
> >> >  	const u32 kvm_supported_word4_x86_features =
> >> > -		F(XMM3) | F(PCLMULQDQ) | 0 /* DTES64, MONITOR */ |
> >> > +		F(XMM3) | F(PCLMULQDQ) | F(MWAIT) /* DTES64, MONITOR */ |
> >> >  		0 /* DS-CPL, VMX, SMX, EST */ |
> >> >  		0 /* TM2 */ | F(SSSE3) | 0 /* CNXT-ID */ | 0 /* Reserved */ |
> >> >  		F(FMA) | F(CX16) | 0 /* xTPR Update, PDCM */ |
> >> >  		0 /* Reserved, DCA */ | F(XMM4_1) |
> >> >  		F(XMM4_2) | F(X2APIC) | F(MOVBE) | F(POPCNT) |
> >> > diff -pNarU5 linux-3.4/arch/x86/kvm/svm.c linux-3.4-mac/arch/x86/kvm/svm.c
> >> > --- linux-3.4/arch/x86/kvm/svm.c	2012-05-20 18:29:13.000000000 -0400
> >> > +++ linux-3.4-mac/arch/x86/kvm/svm.c	2012-10-09 11:44:41.598997481 -0400
> >> > @@ -1102,12 +1102,10 @@ static void init_vmcb(struct vcpu_svm *s
> >> >  	set_intercept(svm, INTERCEPT_VMSAVE);
> >> >  	set_intercept(svm, INTERCEPT_STGI);
> >> >  	set_intercept(svm, INTERCEPT_CLGI);
> >> >  	set_intercept(svm, INTERCEPT_SKINIT);
> >> >  	set_intercept(svm, INTERCEPT_WBINVD);
> >> > -	set_intercept(svm, INTERCEPT_MONITOR);
> >> > -	set_intercept(svm, INTERCEPT_MWAIT);
> >> >  	set_intercept(svm, INTERCEPT_XSETBV);
> >> >  
> >> >  	control->iopm_base_pa = iopm_base;
> >> >  	control->msrpm_base_pa = __pa(svm->msrpm);
> >> >  	control->int_ctl = V_INTR_MASKING_MASK;
> >> > diff -pNarU5 linux-3.4/arch/x86/kvm/vmx.c linux-3.4-mac/arch/x86/kvm/vmx.c
> >> > --- linux-3.4/arch/x86/kvm/vmx.c	2012-05-20 18:29:13.000000000 -0400
> >> > +++ linux-3.4-mac/arch/x86/kvm/vmx.c	2012-10-09 11:42:59.925215977 -0400
> >> > @@ -1938,11 +1938,11 @@ static __init void nested_vmx_setup_ctls
> >> >  		nested_vmx_procbased_ctls_low, nested_vmx_procbased_ctls_high);
> >> >  	nested_vmx_procbased_ctls_low = 0;
> >> >  	nested_vmx_procbased_ctls_high &=
> >> >  		CPU_BASED_VIRTUAL_INTR_PENDING | CPU_BASED_USE_TSC_OFFSETING |
> >> >  		CPU_BASED_HLT_EXITING | CPU_BASED_INVLPG_EXITING |
> >> > -		CPU_BASED_MWAIT_EXITING | CPU_BASED_CR3_LOAD_EXITING |
> >> > +		CPU_BASED_CR3_LOAD_EXITING |
> >> >  		CPU_BASED_CR3_STORE_EXITING |
> >> >  #ifdef CONFIG_X86_64
> >> >  		CPU_BASED_CR8_LOAD_EXITING | CPU_BASED_CR8_STORE_EXITING |
> >> >  #endif
> >> >  		CPU_BASED_MOV_DR_EXITING | CPU_BASED_UNCOND_IO_EXITING |
> >> > @@ -2404,12 +2404,10 @@ static __init int setup_vmcs_config(stru
> >> >  	      CPU_BASED_CR3_LOAD_EXITING |
> >> >  	      CPU_BASED_CR3_STORE_EXITING |
> >> >  	      CPU_BASED_USE_IO_BITMAPS |
> >> >  	      CPU_BASED_MOV_DR_EXITING |
> >> >  	      CPU_BASED_USE_TSC_OFFSETING |
> >> > -	      CPU_BASED_MWAIT_EXITING |
> >> > -	      CPU_BASED_MONITOR_EXITING |
> >> >  	      CPU_BASED_INVLPG_EXITING |
> >> >  	      CPU_BASED_RDPMC_EXITING;
> >> >  
> >> >  	opt = CPU_BASED_TPR_SHADOW |
> >> >  	      CPU_BASED_USE_MSR_BITMAPS |
> >> > 
> >> > If all you're trying to do is (selectively) revert to this behavior,
> >> > that "shouldn't" mess it up for the MacPro either, so I'm thoroughly
> >> > confused at this point :)
> >> > 
> >> > Back in 2010, running MWAIT in L>=1  behaved 100% exactly like a NOP,
> >> > didn't power down the physical CPU, just immediately moved on to the
> >> > next instruction. As such, there was no power saving and no
> >> > opportunity to yield to another L0 thread either, unlike with NOP
> >> > emulation at L0.
> >> > 
> >> > Did that change on newer Intel chips (i.e., is guest-mode MWAIT now
> >> > doing something smarter than just acting as a guest-mode NOP) ?
> >> 
> >> Probably, MWAIT in new intel chips enters power saving mode normally.
> >> 
> >> If hardware-executed MWAIT acted as a NOP in your old chip, then that
> >> shouldn't be a problem either ...  Maybe OS X gets confused into doing
> >> something really dumb because we do not expose the MONITOR/MWAIT feature
> >> bit correctly.
> >> 
> >> Can you try this QEMU patch on the old hardware?
> >> 
> >> diff --git a/target/i386/cpu.c b/target/i386/cpu.c
> >> index 7aa762245a54..4b112e12188a 100644
> >> --- a/target/i386/cpu.c
> >> +++ b/target/i386/cpu.c
> >> @@ -2764,10 +2764,7 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count,
> >>          break;
> >>      case 5:
> >>          /* mwait info: needed for Core compatibility */
> >> -        *eax = 0; /* Smallest monitor-line size in bytes */
> >> -        *ebx = 0; /* Largest monitor-line size in bytes */
> >> -        *ecx = CPUID_MWAIT_EMX | CPUID_MWAIT_IBE;
> >> -        *edx = 0;
> >> +        host_cpuid(index, 0, eax, ebx, ecx, edx);
> >>          break;
> >>      case 6:
> >>          /* Thermal and Power Leaf */
> >> diff --git a/target/i386/kvm.c b/target/i386/kvm.c
> >> index 55865dbee0aa..1eb78291b093 100644
> >> --- a/target/i386/kvm.c
> >> +++ b/target/i386/kvm.c
> >> @@ -360,6 +360,7 @@ uint32_t kvm_arch_get_supported_cpuid(KVMState *s, uint32_t function,
> >>          if (!kvm_irqchip_in_kernel()) {
> >>              ret &= ~CPUID_EXT_X2APIC;
> >>          }
> >> +        ret |= CPUID_EXT_MONITOR;
> >>      } else if (function == 6 && reg == R_EAX) {
> >>          ret |= CPUID_6_EAX_ARAT; /* safe to allow because of emulated APIC */
> >>      } else if (function == 7 && index == 0 && reg == R_EBX) {
> >> 
> >> 
> >> Thanks.
> > 
> > No change, still hangs on boot.
> 
> Hm, also with '-cpu host'?
> (I forgot that the CPUID_EXT_MONITOR isn't visible in the guest
>  otherwise ...)

Yeah, managed to get it started with '-cpu host', but same behavior.
Maybe that version of Xeon really was braindamaged in some way, and
never would have worked with L1 MWAIT regardless.

I only ever used that machine after the emulate-as-nop patch made it
into KVM (commit 87c0057), so I honestly can't say whether it ever
worked with MWAIT run natively at L1...

Thanks,
--Gabriel

[toc] | [prev] | [next] | [standalone]

#1602312

From	"Michael S. Tsirkin" <mst@redhat.com>
Date	2017-03-16 15:10 +0100
Message-ID	<tlEgi-7RT-27@gated-at.bofh.it>
In reply to	#1602269

On Thu, Mar 16, 2017 at 09:24:27AM -0400, Gabriel L. Somlo wrote:
> After studying your patch a bit more carefully (sorry, it's crazy
> around here right now :) ) I realized you're simply trying to
> (selectively) decide when to exit L1 and emulate as NOP vs. when to
> just allow L1 to execute MONITOR & MWAIT natively.
> 
> Is that right ? Because if so, the issues I saw on my MacPro1,1 are
> weird and inexplicable, given that allowing L>=1 to run MONITOR/MWAIT
> natively was one of the options Alex Graf and Rene Rebe used back in
> the very early days of OS X on QEMU, at the time I got involved with
> that project. Here's part of an out of tree patch against 3.4 which did
> just that, and worked as far as I remember on *any* MWAIT capable
> intel chip I had access to back in 2010:
> 
> ##############################################################################
> # 99-mwait.patch.kvm-kmod (Rene Rebe <rene@exactcode.de>) 2010-04-27
> ##############################################################################
> diff -pNarU5 linux-3.4/arch/x86/kvm/cpuid.c linux-3.4-mac/arch/x86/kvm/cpuid.c
> --- linux-3.4/arch/x86/kvm/cpuid.c	2012-05-20 18:29:13.000000000 -0400
> +++ linux-3.4-mac/arch/x86/kvm/cpuid.c	2012-10-09 11:42:59.921215750 -0400
> @@ -222,11 +222,11 @@ static int do_cpuid_ent(struct kvm_cpuid
>  		f_nx | 0 /* Reserved */ | F(MMXEXT) | F(MMX) |
>  		F(FXSR) | F(FXSR_OPT) | f_gbpages | f_rdtscp |
>  		0 /* Reserved */ | f_lm | F(3DNOWEXT) | F(3DNOW);
>  	/* cpuid 1.ecx */
>  	const u32 kvm_supported_word4_x86_features =
> -		F(XMM3) | F(PCLMULQDQ) | 0 /* DTES64, MONITOR */ |
> +		F(XMM3) | F(PCLMULQDQ) | F(MWAIT) /* DTES64, MONITOR */ |
>  		0 /* DS-CPL, VMX, SMX, EST */ |
>  		0 /* TM2 */ | F(SSSE3) | 0 /* CNXT-ID */ | 0 /* Reserved */ |
>  		F(FMA) | F(CX16) | 0 /* xTPR Update, PDCM */ |
>  		0 /* Reserved, DCA */ | F(XMM4_1) |
>  		F(XMM4_2) | F(X2APIC) | F(MOVBE) | F(POPCNT) |
> diff -pNarU5 linux-3.4/arch/x86/kvm/svm.c linux-3.4-mac/arch/x86/kvm/svm.c
> --- linux-3.4/arch/x86/kvm/svm.c	2012-05-20 18:29:13.000000000 -0400
> +++ linux-3.4-mac/arch/x86/kvm/svm.c	2012-10-09 11:44:41.598997481 -0400
> @@ -1102,12 +1102,10 @@ static void init_vmcb(struct vcpu_svm *s
>  	set_intercept(svm, INTERCEPT_VMSAVE);
>  	set_intercept(svm, INTERCEPT_STGI);
>  	set_intercept(svm, INTERCEPT_CLGI);
>  	set_intercept(svm, INTERCEPT_SKINIT);
>  	set_intercept(svm, INTERCEPT_WBINVD);
> -	set_intercept(svm, INTERCEPT_MONITOR);
> -	set_intercept(svm, INTERCEPT_MWAIT);
>  	set_intercept(svm, INTERCEPT_XSETBV);
>  
>  	control->iopm_base_pa = iopm_base;
>  	control->msrpm_base_pa = __pa(svm->msrpm);
>  	control->int_ctl = V_INTR_MASKING_MASK;
> diff -pNarU5 linux-3.4/arch/x86/kvm/vmx.c linux-3.4-mac/arch/x86/kvm/vmx.c
> --- linux-3.4/arch/x86/kvm/vmx.c	2012-05-20 18:29:13.000000000 -0400
> +++ linux-3.4-mac/arch/x86/kvm/vmx.c	2012-10-09 11:42:59.925215977 -0400
> @@ -1938,11 +1938,11 @@ static __init void nested_vmx_setup_ctls
>  		nested_vmx_procbased_ctls_low, nested_vmx_procbased_ctls_high);
>  	nested_vmx_procbased_ctls_low = 0;
>  	nested_vmx_procbased_ctls_high &=
>  		CPU_BASED_VIRTUAL_INTR_PENDING | CPU_BASED_USE_TSC_OFFSETING |
>  		CPU_BASED_HLT_EXITING | CPU_BASED_INVLPG_EXITING |
> -		CPU_BASED_MWAIT_EXITING | CPU_BASED_CR3_LOAD_EXITING |
> +		CPU_BASED_CR3_LOAD_EXITING |
>  		CPU_BASED_CR3_STORE_EXITING |
>  #ifdef CONFIG_X86_64
>  		CPU_BASED_CR8_LOAD_EXITING | CPU_BASED_CR8_STORE_EXITING |
>  #endif
>  		CPU_BASED_MOV_DR_EXITING | CPU_BASED_UNCOND_IO_EXITING |
> @@ -2404,12 +2404,10 @@ static __init int setup_vmcs_config(stru
>  	      CPU_BASED_CR3_LOAD_EXITING |
>  	      CPU_BASED_CR3_STORE_EXITING |
>  	      CPU_BASED_USE_IO_BITMAPS |
>  	      CPU_BASED_MOV_DR_EXITING |
>  	      CPU_BASED_USE_TSC_OFFSETING |
> -	      CPU_BASED_MWAIT_EXITING |
> -	      CPU_BASED_MONITOR_EXITING |
>  	      CPU_BASED_INVLPG_EXITING |
>  	      CPU_BASED_RDPMC_EXITING;
>  
>  	opt = CPU_BASED_TPR_SHADOW |
>  	      CPU_BASED_USE_MSR_BITMAPS |
> 
> If all you're trying to do is (selectively) revert to this behavior,
> that "shouldn't" mess it up for the MacPro either, so I'm thoroughly
> confused at this point :)

Yes.  Me too. Want to try that other patch and see what happens?

> Back in 2010, running MWAIT in L>=1  behaved 100% exactly like a NOP,
> didn't power down the physical CPU, just immediately moved on to the
> next instruction. As such, there was no power saving and no
> opportunity to yield to another L0 thread either, unlike with NOP
> emulation at L0.
> 
> Did that change on newer Intel chips (i.e., is guest-mode MWAIT now
> doing something smarter than just acting as a guest-mode NOP) ?
> 
> Thanks,
> --Gabriel

Interesting.  What it seems to say is this:

MWAIT. Behavior of the MWAIT instruction (which always causes an invalid-
opcode exception—#UD—if CPL > 0) is determined by the setting of the “MWAIT
exiting” VM-execution control:
— If the “MWAIT exiting” VM-execution control is 1, MWAIT causes a VM exit
(see Section 22.1.3).
— If the “MWAIT exiting” VM-execution control is 0, MWAIT operates normally if
any of the following is true: (1) the “interrupt-window exiting” VM-execution
control is 0; (2) ECX[0] is 0; or (3) RFLAGS.IF = 1.
— If the “MWAIT exiting” VM-execution control is 0, the “interrupt-window
exiting” VM-execution control is 1, ECX[0] = 1, and RFLAGS.IF = 0, MWAIT
does not cause the processor to enter an implementation-dependent
optimized state; instead, control passes to the instruction following the
MWAIT instruction.


And since interrupt-window exiting is 0 most of the time for KVM,
I would expect MWAIT to behave normally.


-- 
MST

[toc] | [prev] | [next] | [standalone]

#1602427

From	"Gabriel L. Somlo" <gsomlo@gmail.com>
Date	2017-03-16 16:00 +0100
Message-ID	<tlF2H-8g9-27@gated-at.bofh.it>
In reply to	#1602312

On Thu, Mar 16, 2017 at 04:04:12PM +0200, Michael S. Tsirkin wrote:
> On Thu, Mar 16, 2017 at 09:24:27AM -0400, Gabriel L. Somlo wrote:
> > After studying your patch a bit more carefully (sorry, it's crazy
> > around here right now :) ) I realized you're simply trying to
> > (selectively) decide when to exit L1 and emulate as NOP vs. when to
> > just allow L1 to execute MONITOR & MWAIT natively.
> > 
> > Is that right ? Because if so, the issues I saw on my MacPro1,1 are
> > weird and inexplicable, given that allowing L>=1 to run MONITOR/MWAIT
> > natively was one of the options Alex Graf and Rene Rebe used back in
> > the very early days of OS X on QEMU, at the time I got involved with
> > that project. Here's part of an out of tree patch against 3.4 which did
> > just that, and worked as far as I remember on *any* MWAIT capable
> > intel chip I had access to back in 2010:
> > 
> > ##############################################################################
> > # 99-mwait.patch.kvm-kmod (Rene Rebe <rene@exactcode.de>) 2010-04-27
> > ##############################################################################
> > diff -pNarU5 linux-3.4/arch/x86/kvm/cpuid.c linux-3.4-mac/arch/x86/kvm/cpuid.c
> > --- linux-3.4/arch/x86/kvm/cpuid.c	2012-05-20 18:29:13.000000000 -0400
> > +++ linux-3.4-mac/arch/x86/kvm/cpuid.c	2012-10-09 11:42:59.921215750 -0400
> > @@ -222,11 +222,11 @@ static int do_cpuid_ent(struct kvm_cpuid
> >  		f_nx | 0 /* Reserved */ | F(MMXEXT) | F(MMX) |
> >  		F(FXSR) | F(FXSR_OPT) | f_gbpages | f_rdtscp |
> >  		0 /* Reserved */ | f_lm | F(3DNOWEXT) | F(3DNOW);
> >  	/* cpuid 1.ecx */
> >  	const u32 kvm_supported_word4_x86_features =
> > -		F(XMM3) | F(PCLMULQDQ) | 0 /* DTES64, MONITOR */ |
> > +		F(XMM3) | F(PCLMULQDQ) | F(MWAIT) /* DTES64, MONITOR */ |
> >  		0 /* DS-CPL, VMX, SMX, EST */ |
> >  		0 /* TM2 */ | F(SSSE3) | 0 /* CNXT-ID */ | 0 /* Reserved */ |
> >  		F(FMA) | F(CX16) | 0 /* xTPR Update, PDCM */ |
> >  		0 /* Reserved, DCA */ | F(XMM4_1) |
> >  		F(XMM4_2) | F(X2APIC) | F(MOVBE) | F(POPCNT) |
> > diff -pNarU5 linux-3.4/arch/x86/kvm/svm.c linux-3.4-mac/arch/x86/kvm/svm.c
> > --- linux-3.4/arch/x86/kvm/svm.c	2012-05-20 18:29:13.000000000 -0400
> > +++ linux-3.4-mac/arch/x86/kvm/svm.c	2012-10-09 11:44:41.598997481 -0400
> > @@ -1102,12 +1102,10 @@ static void init_vmcb(struct vcpu_svm *s
> >  	set_intercept(svm, INTERCEPT_VMSAVE);
> >  	set_intercept(svm, INTERCEPT_STGI);
> >  	set_intercept(svm, INTERCEPT_CLGI);
> >  	set_intercept(svm, INTERCEPT_SKINIT);
> >  	set_intercept(svm, INTERCEPT_WBINVD);
> > -	set_intercept(svm, INTERCEPT_MONITOR);
> > -	set_intercept(svm, INTERCEPT_MWAIT);
> >  	set_intercept(svm, INTERCEPT_XSETBV);
> >  
> >  	control->iopm_base_pa = iopm_base;
> >  	control->msrpm_base_pa = __pa(svm->msrpm);
> >  	control->int_ctl = V_INTR_MASKING_MASK;
> > diff -pNarU5 linux-3.4/arch/x86/kvm/vmx.c linux-3.4-mac/arch/x86/kvm/vmx.c
> > --- linux-3.4/arch/x86/kvm/vmx.c	2012-05-20 18:29:13.000000000 -0400
> > +++ linux-3.4-mac/arch/x86/kvm/vmx.c	2012-10-09 11:42:59.925215977 -0400
> > @@ -1938,11 +1938,11 @@ static __init void nested_vmx_setup_ctls
> >  		nested_vmx_procbased_ctls_low, nested_vmx_procbased_ctls_high);
> >  	nested_vmx_procbased_ctls_low = 0;
> >  	nested_vmx_procbased_ctls_high &=
> >  		CPU_BASED_VIRTUAL_INTR_PENDING | CPU_BASED_USE_TSC_OFFSETING |
> >  		CPU_BASED_HLT_EXITING | CPU_BASED_INVLPG_EXITING |
> > -		CPU_BASED_MWAIT_EXITING | CPU_BASED_CR3_LOAD_EXITING |
> > +		CPU_BASED_CR3_LOAD_EXITING |
> >  		CPU_BASED_CR3_STORE_EXITING |
> >  #ifdef CONFIG_X86_64
> >  		CPU_BASED_CR8_LOAD_EXITING | CPU_BASED_CR8_STORE_EXITING |
> >  #endif
> >  		CPU_BASED_MOV_DR_EXITING | CPU_BASED_UNCOND_IO_EXITING |
> > @@ -2404,12 +2404,10 @@ static __init int setup_vmcs_config(stru
> >  	      CPU_BASED_CR3_LOAD_EXITING |
> >  	      CPU_BASED_CR3_STORE_EXITING |
> >  	      CPU_BASED_USE_IO_BITMAPS |
> >  	      CPU_BASED_MOV_DR_EXITING |
> >  	      CPU_BASED_USE_TSC_OFFSETING |
> > -	      CPU_BASED_MWAIT_EXITING |
> > -	      CPU_BASED_MONITOR_EXITING |
> >  	      CPU_BASED_INVLPG_EXITING |
> >  	      CPU_BASED_RDPMC_EXITING;
> >  
> >  	opt = CPU_BASED_TPR_SHADOW |
> >  	      CPU_BASED_USE_MSR_BITMAPS |
> > 
> > If all you're trying to do is (selectively) revert to this behavior,
> > that "shouldn't" mess it up for the MacPro either, so I'm thoroughly
> > confused at this point :)
> 
> Yes.  Me too. Want to try that other patch and see what happens?

You mean the old 3.4 patch against current KVM ? I'll try to do that,
might take me a while :)

> > Back in 2010, running MWAIT in L>=1  behaved 100% exactly like a NOP,
> > didn't power down the physical CPU, just immediately moved on to the
> > next instruction. As such, there was no power saving and no
> > opportunity to yield to another L0 thread either, unlike with NOP
> > emulation at L0.
> > 
> > Did that change on newer Intel chips (i.e., is guest-mode MWAIT now
> > doing something smarter than just acting as a guest-mode NOP) ?
> > 
> > Thanks,
> > --Gabriel
> 
> Interesting.  What it seems to say is this:
> 
> MWAIT. Behavior of the MWAIT instruction (which always causes an invalid-
> opcode exception—#UD—if CPL > 0) is determined by the setting of the “MWAIT
> exiting” VM-execution control:
> — If the “MWAIT exiting” VM-execution control is 1, MWAIT causes a VM exit
> (see Section 22.1.3).
> — If the “MWAIT exiting” VM-execution control is 0, MWAIT operates normally if
> any of the following is true: (1) the “interrupt-window exiting” VM-execution
> control is 0; (2) ECX[0] is 0; or (3) RFLAGS.IF = 1.
> — If the “MWAIT exiting” VM-execution control is 0, the “interrupt-window
> exiting” VM-execution control is 1, ECX[0] = 1, and RFLAGS.IF = 0, MWAIT
> does not cause the processor to enter an implementation-dependent
> optimized state; instead, control passes to the instruction following the
> MWAIT instruction.
> 
> 
> And since interrupt-window exiting is 0 most of the time for KVM,
> I would expect MWAIT to behave normally.

The intel manual said the same thing back in 2010 as well. However,
regardless of how any flags were set, interrupt-window exiting or not,
"normal" L1 MWAIT behavior was that it woke up immediately regardless.
Remember, never going to sleep is still correct ("normal" ?) behavior
per the ISA definition of MWAIT :)

Also, when I tested your patch on the macbook air (where it worked),
not only was the host reporting 400% CPU for qemu (which is to be
expected), but the thermal fan/cooling thing also shifted up into high
gear, which means the physical CPU got hot, which it shouldn't have if
the guest-mode MWAIT actually did put the host CPU into low power.

So at least on this 4-year-old core-I7 chip, the story Intel tells in
its manual still doesn't check out. I could never get any
clarification on what they mean by "operates normally" :)

[toc] | [prev] | [next] | [standalone]

#1602525

From	"Michael S. Tsirkin" <mst@redhat.com>
Date	2017-03-16 16:30 +0100
Message-ID	<tlFvI-gv-25@gated-at.bofh.it>
In reply to	#1602427

On Thu, Mar 16, 2017 at 10:58:20AM -0400, Gabriel L. Somlo wrote:
> On Thu, Mar 16, 2017 at 04:04:12PM +0200, Michael S. Tsirkin wrote:
> > On Thu, Mar 16, 2017 at 09:24:27AM -0400, Gabriel L. Somlo wrote:
> > > After studying your patch a bit more carefully (sorry, it's crazy
> > > around here right now :) ) I realized you're simply trying to
> > > (selectively) decide when to exit L1 and emulate as NOP vs. when to
> > > just allow L1 to execute MONITOR & MWAIT natively.
> > > 
> > > Is that right ? Because if so, the issues I saw on my MacPro1,1 are
> > > weird and inexplicable, given that allowing L>=1 to run MONITOR/MWAIT
> > > natively was one of the options Alex Graf and Rene Rebe used back in
> > > the very early days of OS X on QEMU, at the time I got involved with
> > > that project. Here's part of an out of tree patch against 3.4 which did
> > > just that, and worked as far as I remember on *any* MWAIT capable
> > > intel chip I had access to back in 2010:
> > > 
> > > ##############################################################################
> > > # 99-mwait.patch.kvm-kmod (Rene Rebe <rene@exactcode.de>) 2010-04-27
> > > ##############################################################################
> > > diff -pNarU5 linux-3.4/arch/x86/kvm/cpuid.c linux-3.4-mac/arch/x86/kvm/cpuid.c
> > > --- linux-3.4/arch/x86/kvm/cpuid.c	2012-05-20 18:29:13.000000000 -0400
> > > +++ linux-3.4-mac/arch/x86/kvm/cpuid.c	2012-10-09 11:42:59.921215750 -0400
> > > @@ -222,11 +222,11 @@ static int do_cpuid_ent(struct kvm_cpuid
> > >  		f_nx | 0 /* Reserved */ | F(MMXEXT) | F(MMX) |
> > >  		F(FXSR) | F(FXSR_OPT) | f_gbpages | f_rdtscp |
> > >  		0 /* Reserved */ | f_lm | F(3DNOWEXT) | F(3DNOW);
> > >  	/* cpuid 1.ecx */
> > >  	const u32 kvm_supported_word4_x86_features =
> > > -		F(XMM3) | F(PCLMULQDQ) | 0 /* DTES64, MONITOR */ |
> > > +		F(XMM3) | F(PCLMULQDQ) | F(MWAIT) /* DTES64, MONITOR */ |
> > >  		0 /* DS-CPL, VMX, SMX, EST */ |
> > >  		0 /* TM2 */ | F(SSSE3) | 0 /* CNXT-ID */ | 0 /* Reserved */ |
> > >  		F(FMA) | F(CX16) | 0 /* xTPR Update, PDCM */ |
> > >  		0 /* Reserved, DCA */ | F(XMM4_1) |
> > >  		F(XMM4_2) | F(X2APIC) | F(MOVBE) | F(POPCNT) |
> > > diff -pNarU5 linux-3.4/arch/x86/kvm/svm.c linux-3.4-mac/arch/x86/kvm/svm.c
> > > --- linux-3.4/arch/x86/kvm/svm.c	2012-05-20 18:29:13.000000000 -0400
> > > +++ linux-3.4-mac/arch/x86/kvm/svm.c	2012-10-09 11:44:41.598997481 -0400
> > > @@ -1102,12 +1102,10 @@ static void init_vmcb(struct vcpu_svm *s
> > >  	set_intercept(svm, INTERCEPT_VMSAVE);
> > >  	set_intercept(svm, INTERCEPT_STGI);
> > >  	set_intercept(svm, INTERCEPT_CLGI);
> > >  	set_intercept(svm, INTERCEPT_SKINIT);
> > >  	set_intercept(svm, INTERCEPT_WBINVD);
> > > -	set_intercept(svm, INTERCEPT_MONITOR);
> > > -	set_intercept(svm, INTERCEPT_MWAIT);
> > >  	set_intercept(svm, INTERCEPT_XSETBV);
> > >  
> > >  	control->iopm_base_pa = iopm_base;
> > >  	control->msrpm_base_pa = __pa(svm->msrpm);
> > >  	control->int_ctl = V_INTR_MASKING_MASK;
> > > diff -pNarU5 linux-3.4/arch/x86/kvm/vmx.c linux-3.4-mac/arch/x86/kvm/vmx.c
> > > --- linux-3.4/arch/x86/kvm/vmx.c	2012-05-20 18:29:13.000000000 -0400
> > > +++ linux-3.4-mac/arch/x86/kvm/vmx.c	2012-10-09 11:42:59.925215977 -0400
> > > @@ -1938,11 +1938,11 @@ static __init void nested_vmx_setup_ctls
> > >  		nested_vmx_procbased_ctls_low, nested_vmx_procbased_ctls_high);
> > >  	nested_vmx_procbased_ctls_low = 0;
> > >  	nested_vmx_procbased_ctls_high &=
> > >  		CPU_BASED_VIRTUAL_INTR_PENDING | CPU_BASED_USE_TSC_OFFSETING |
> > >  		CPU_BASED_HLT_EXITING | CPU_BASED_INVLPG_EXITING |
> > > -		CPU_BASED_MWAIT_EXITING | CPU_BASED_CR3_LOAD_EXITING |
> > > +		CPU_BASED_CR3_LOAD_EXITING |
> > >  		CPU_BASED_CR3_STORE_EXITING |
> > >  #ifdef CONFIG_X86_64
> > >  		CPU_BASED_CR8_LOAD_EXITING | CPU_BASED_CR8_STORE_EXITING |
> > >  #endif
> > >  		CPU_BASED_MOV_DR_EXITING | CPU_BASED_UNCOND_IO_EXITING |
> > > @@ -2404,12 +2404,10 @@ static __init int setup_vmcs_config(stru
> > >  	      CPU_BASED_CR3_LOAD_EXITING |
> > >  	      CPU_BASED_CR3_STORE_EXITING |
> > >  	      CPU_BASED_USE_IO_BITMAPS |
> > >  	      CPU_BASED_MOV_DR_EXITING |
> > >  	      CPU_BASED_USE_TSC_OFFSETING |
> > > -	      CPU_BASED_MWAIT_EXITING |
> > > -	      CPU_BASED_MONITOR_EXITING |
> > >  	      CPU_BASED_INVLPG_EXITING |
> > >  	      CPU_BASED_RDPMC_EXITING;
> > >  
> > >  	opt = CPU_BASED_TPR_SHADOW |
> > >  	      CPU_BASED_USE_MSR_BITMAPS |
> > > 
> > > If all you're trying to do is (selectively) revert to this behavior,
> > > that "shouldn't" mess it up for the MacPro either, so I'm thoroughly
> > > confused at this point :)
> > 
> > Yes.  Me too. Want to try that other patch and see what happens?
> 
> You mean the old 3.4 patch against current KVM ? I'll try to do that,
> might take me a while :)

I can rebase them for you if you send me a link.

> > > Back in 2010, running MWAIT in L>=1  behaved 100% exactly like a NOP,
> > > didn't power down the physical CPU, just immediately moved on to the
> > > next instruction. As such, there was no power saving and no
> > > opportunity to yield to another L0 thread either, unlike with NOP
> > > emulation at L0.
> > > 
> > > Did that change on newer Intel chips (i.e., is guest-mode MWAIT now
> > > doing something smarter than just acting as a guest-mode NOP) ?
> > > 
> > > Thanks,
> > > --Gabriel
> > 
> > Interesting.  What it seems to say is this:
> > 
> > MWAIT. Behavior of the MWAIT instruction (which always causes an invalid-
> > opcode exception—#UD—if CPL > 0) is determined by the setting of the “MWAIT
> > exiting” VM-execution control:
> > — If the “MWAIT exiting” VM-execution control is 1, MWAIT causes a VM exit
> > (see Section 22.1.3).
> > — If the “MWAIT exiting” VM-execution control is 0, MWAIT operates normally if
> > any of the following is true: (1) the “interrupt-window exiting” VM-execution
> > control is 0; (2) ECX[0] is 0; or (3) RFLAGS.IF = 1.
> > — If the “MWAIT exiting” VM-execution control is 0, the “interrupt-window
> > exiting” VM-execution control is 1, ECX[0] = 1, and RFLAGS.IF = 0, MWAIT
> > does not cause the processor to enter an implementation-dependent
> > optimized state; instead, control passes to the instruction following the
> > MWAIT instruction.
> > 
> > 
> > And since interrupt-window exiting is 0 most of the time for KVM,
> > I would expect MWAIT to behave normally.
> 
> The intel manual said the same thing back in 2010 as well. However,
> regardless of how any flags were set, interrupt-window exiting or not,
> "normal" L1 MWAIT behavior was that it woke up immediately regardless.
> Remember, never going to sleep is still correct ("normal" ?) behavior
> per the ISA definition of MWAIT :)
> 
> Also, when I tested your patch on the macbook air (where it worked),
> not only was the host reporting 400% CPU for qemu (which is to be
> expected), but the thermal fan/cooling thing also shifted up into high
> gear, which means the physical CPU got hot, which it shouldn't have if
> the guest-mode MWAIT actually did put the host CPU into low power.

Does same happen with NOP btw?

> So at least on this 4-year-old core-I7 chip, the story Intel tells in
> its manual still doesn't check out. I could never get any
> clarification on what they mean by "operates normally" :)

It could be Mac OS sets ECX[0] = 1 and RFLAGS.IF = 0.

-- 
MST

[toc] | [prev] | [next] | [standalone]

#1602553

From	Radim Krčmář <rkrcmar@redhat.com>
Date	2017-03-16 16:40 +0100
Message-ID	<tlFFo-lI-27@gated-at.bofh.it>
In reply to	#1602427

2017-03-16 10:58-0400, Gabriel L. Somlo:
> On Thu, Mar 16, 2017 at 04:04:12PM +0200, Michael S. Tsirkin wrote:
> > On Thu, Mar 16, 2017 at 09:24:27AM -0400, Gabriel L. Somlo wrote:
> > > After studying your patch a bit more carefully (sorry, it's crazy
> > > around here right now :) ) I realized you're simply trying to
> > > (selectively) decide when to exit L1 and emulate as NOP vs. when to
> > > just allow L1 to execute MONITOR & MWAIT natively.
> > > 
> > > Is that right ? Because if so, the issues I saw on my MacPro1,1 are
> > > weird and inexplicable, given that allowing L>=1 to run MONITOR/MWAIT
> > > natively was one of the options Alex Graf and Rene Rebe used back in
> > > the very early days of OS X on QEMU, at the time I got involved with
> > > that project. Here's part of an out of tree patch against 3.4 which did
> > > just that, and worked as far as I remember on *any* MWAIT capable
> > > intel chip I had access to back in 2010:
> > > 
> > > ##############################################################################
> > > # 99-mwait.patch.kvm-kmod (Rene Rebe <rene@exactcode.de>) 2010-04-27
> > > ##############################################################################
> > > diff -pNarU5 linux-3.4/arch/x86/kvm/cpuid.c linux-3.4-mac/arch/x86/kvm/cpuid.c
> > > --- linux-3.4/arch/x86/kvm/cpuid.c	2012-05-20 18:29:13.000000000 -0400
> > > +++ linux-3.4-mac/arch/x86/kvm/cpuid.c	2012-10-09 11:42:59.921215750 -0400
> > > @@ -222,11 +222,11 @@ static int do_cpuid_ent(struct kvm_cpuid
> > >  		f_nx | 0 /* Reserved */ | F(MMXEXT) | F(MMX) |
> > >  		F(FXSR) | F(FXSR_OPT) | f_gbpages | f_rdtscp |
> > >  		0 /* Reserved */ | f_lm | F(3DNOWEXT) | F(3DNOW);
> > >  	/* cpuid 1.ecx */
> > >  	const u32 kvm_supported_word4_x86_features =
> > > -		F(XMM3) | F(PCLMULQDQ) | 0 /* DTES64, MONITOR */ |
> > > +		F(XMM3) | F(PCLMULQDQ) | F(MWAIT) /* DTES64, MONITOR */ |
> > >  		0 /* DS-CPL, VMX, SMX, EST */ |
> > >  		0 /* TM2 */ | F(SSSE3) | 0 /* CNXT-ID */ | 0 /* Reserved */ |
> > >  		F(FMA) | F(CX16) | 0 /* xTPR Update, PDCM */ |
> > >  		0 /* Reserved, DCA */ | F(XMM4_1) |
> > >  		F(XMM4_2) | F(X2APIC) | F(MOVBE) | F(POPCNT) |
> > > diff -pNarU5 linux-3.4/arch/x86/kvm/svm.c linux-3.4-mac/arch/x86/kvm/svm.c
> > > --- linux-3.4/arch/x86/kvm/svm.c	2012-05-20 18:29:13.000000000 -0400
> > > +++ linux-3.4-mac/arch/x86/kvm/svm.c	2012-10-09 11:44:41.598997481 -0400
> > > @@ -1102,12 +1102,10 @@ static void init_vmcb(struct vcpu_svm *s
> > >  	set_intercept(svm, INTERCEPT_VMSAVE);
> > >  	set_intercept(svm, INTERCEPT_STGI);
> > >  	set_intercept(svm, INTERCEPT_CLGI);
> > >  	set_intercept(svm, INTERCEPT_SKINIT);
> > >  	set_intercept(svm, INTERCEPT_WBINVD);
> > > -	set_intercept(svm, INTERCEPT_MONITOR);
> > > -	set_intercept(svm, INTERCEPT_MWAIT);
> > >  	set_intercept(svm, INTERCEPT_XSETBV);
> > >  
> > >  	control->iopm_base_pa = iopm_base;
> > >  	control->msrpm_base_pa = __pa(svm->msrpm);
> > >  	control->int_ctl = V_INTR_MASKING_MASK;
> > > diff -pNarU5 linux-3.4/arch/x86/kvm/vmx.c linux-3.4-mac/arch/x86/kvm/vmx.c
> > > --- linux-3.4/arch/x86/kvm/vmx.c	2012-05-20 18:29:13.000000000 -0400
> > > +++ linux-3.4-mac/arch/x86/kvm/vmx.c	2012-10-09 11:42:59.925215977 -0400
> > > @@ -1938,11 +1938,11 @@ static __init void nested_vmx_setup_ctls
> > >  		nested_vmx_procbased_ctls_low, nested_vmx_procbased_ctls_high);
> > >  	nested_vmx_procbased_ctls_low = 0;
> > >  	nested_vmx_procbased_ctls_high &=
> > >  		CPU_BASED_VIRTUAL_INTR_PENDING | CPU_BASED_USE_TSC_OFFSETING |
> > >  		CPU_BASED_HLT_EXITING | CPU_BASED_INVLPG_EXITING |
> > > -		CPU_BASED_MWAIT_EXITING | CPU_BASED_CR3_LOAD_EXITING |
> > > +		CPU_BASED_CR3_LOAD_EXITING |
> > >  		CPU_BASED_CR3_STORE_EXITING |
> > >  #ifdef CONFIG_X86_64
> > >  		CPU_BASED_CR8_LOAD_EXITING | CPU_BASED_CR8_STORE_EXITING |
> > >  #endif
> > >  		CPU_BASED_MOV_DR_EXITING | CPU_BASED_UNCOND_IO_EXITING |
> > > @@ -2404,12 +2404,10 @@ static __init int setup_vmcs_config(stru
> > >  	      CPU_BASED_CR3_LOAD_EXITING |
> > >  	      CPU_BASED_CR3_STORE_EXITING |
> > >  	      CPU_BASED_USE_IO_BITMAPS |
> > >  	      CPU_BASED_MOV_DR_EXITING |
> > >  	      CPU_BASED_USE_TSC_OFFSETING |
> > > -	      CPU_BASED_MWAIT_EXITING |
> > > -	      CPU_BASED_MONITOR_EXITING |
> > >  	      CPU_BASED_INVLPG_EXITING |
> > >  	      CPU_BASED_RDPMC_EXITING;
> > >  
> > >  	opt = CPU_BASED_TPR_SHADOW |
> > >  	      CPU_BASED_USE_MSR_BITMAPS |
> > > 
> > > If all you're trying to do is (selectively) revert to this behavior,
> > > that "shouldn't" mess it up for the MacPro either, so I'm thoroughly
> > > confused at this point :)
> > 
> > Yes.  Me too. Want to try that other patch and see what happens?
> 
> You mean the old 3.4 patch against current KVM ? I'll try to do that,
> might take me a while :)

Michael's patch already did most of that, you just need to add

diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c
index efde6cc50875..b12f07d4ce17 100644
--- a/arch/x86/kvm/cpuid.c
+++ b/arch/x86/kvm/cpuid.c
@@ -348,7 +348,7 @@ static inline int __do_cpuid_ent(struct kvm_cpuid_entry2 *entry, u32 function,
 	const u32 kvm_cpuid_1_ecx_x86_features =
 		/* NOTE: MONITOR (and MWAIT) are emulated as NOP,
 		 * but *not* advertised to guests via CPUID ! */
-		F(XMM3) | F(PCLMULQDQ) | 0 /* DTES64, MONITOR */ |
+		F(XMM3) | F(PCLMULQDQ) | F(MWAIT) /* DTES64, MONITOR */ |
 		0 /* DS-CPL, VMX, SMX, EST */ |
 		0 /* TM2 */ | F(SSSE3) | 0 /* CNXT-ID */ | 0 /* Reserved */ |
 		F(FMA) | F(CX16) | 0 /* xTPR Update, PDCM */ |

Note: this will never be upstream, because mwait isn't what we want by
default. :)

>> > Back in 2010, running MWAIT in L>=1  behaved 100% exactly like a NOP,
>> > didn't power down the physical CPU, just immediately moved on to the
>> > next instruction. As such, there was no power saving and no
>> > opportunity to yield to another L0 thread either, unlike with NOP
>> > emulation at L0.
>> > 
>> > Did that change on newer Intel chips (i.e., is guest-mode MWAIT now
>> > doing something smarter than just acting as a guest-mode NOP) ?
>> > 
>> > Thanks,
>> > --Gabriel
>> 
>> Interesting.  What it seems to say is this:
>> 
>> MWAIT. Behavior of the MWAIT instruction (which always causes an invalid-
>> opcode exception—#UD—if CPL > 0) is determined by the setting of the “MWAIT
>> exiting” VM-execution control:
>> — If the “MWAIT exiting” VM-execution control is 1, MWAIT causes a VM exit
>> (see Section 22.1.3).
>> — If the “MWAIT exiting” VM-execution control is 0, MWAIT operates normally if
>> any of the following is true: (1) the “interrupt-window exiting” VM-execution
>> control is 0; (2) ECX[0] is 0; or (3) RFLAGS.IF = 1.
>> — If the “MWAIT exiting” VM-execution control is 0, the “interrupt-window
>> exiting” VM-execution control is 1, ECX[0] = 1, and RFLAGS.IF = 0, MWAIT
>> does not cause the processor to enter an implementation-dependent
>> optimized state; instead, control passes to the instruction following the
>> MWAIT instruction.
>> 
>> 
>> And since interrupt-window exiting is 0 most of the time for KVM,
>> I would expect MWAIT to behave normally.
> 
> The intel manual said the same thing back in 2010 as well. However,
> regardless of how any flags were set, interrupt-window exiting or not,
> "normal" L1 MWAIT behavior was that it woke up immediately regardless.
> Remember, never going to sleep is still correct ("normal" ?) behavior
> per the ISA definition of MWAIT :)

I'll write a simple kvm-unit-test to better understand why it is broken
for you ...

> Also, when I tested your patch on the macbook air (where it worked),
> not only was the host reporting 400% CPU for qemu (which is to be
> expected), but the thermal fan/cooling thing also shifted up into high
> gear, which means the physical CPU got hot, which it shouldn't have if
> the guest-mode MWAIT actually did put the host CPU into low power.

I tested MWAIT with basically the same kernel patch and the qemu patch
with Linux guest on Haswell and Nehalem.  Running the guest took 100% of
the host CPUs, but it still had the same temperature as when the host
was idle.

That reminds me that you to pass '-cpu host' for QEMU reasons.

[toc] | [prev] | [next] | [standalone]

#1602605

From	Radim Krčmář <rkrcmar@redhat.com>
Date	2017-03-16 17:10 +0100
Message-ID	<tlG8q-LZ-23@gated-at.bofh.it>
In reply to	#1602553

2017-03-16 16:35+0100, Radim Krčmář:
> 2017-03-16 10:58-0400, Gabriel L. Somlo:
>> The intel manual said the same thing back in 2010 as well. However,
>> regardless of how any flags were set, interrupt-window exiting or not,
>> "normal" L1 MWAIT behavior was that it woke up immediately regardless.
>> Remember, never going to sleep is still correct ("normal" ?) behavior
>> per the ISA definition of MWAIT :)
> 
> I'll write a simple kvm-unit-test to better understand why it is broken
> for you ...

Please get git://git.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git

and try this, thanks!

---8<---
x86/mwait: crappy test

`./configure && make` to build it, then follow the comment in code to
try few cases.

---
 x86/Makefile.common |  1 +
 x86/mwait.c         | 41 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 42 insertions(+)
 create mode 100644 x86/mwait.c

diff --git a/x86/Makefile.common b/x86/Makefile.common
index 1dad18ba26e1..1e708a6acd39 100644
--- a/x86/Makefile.common
+++ b/x86/Makefile.common
@@ -46,6 +46,7 @@ tests-common = $(TEST_DIR)/vmexit.flat $(TEST_DIR)/tsc.flat \
                $(TEST_DIR)/tsc_adjust.flat $(TEST_DIR)/asyncpf.flat \
                $(TEST_DIR)/init.flat $(TEST_DIR)/smap.flat \
                $(TEST_DIR)/hyperv_synic.flat $(TEST_DIR)/hyperv_stimer.flat \
+               $(TEST_DIR)/mwait.flat \
 
 ifdef API
 tests-common += api/api-sample
diff --git a/x86/mwait.c b/x86/mwait.c
new file mode 100644
index 000000000000..c21dab5cc97d
--- /dev/null
+++ b/x86/mwait.c
@@ -0,0 +1,41 @@
+#include "vm.h"
+
+#define TARGET_RESUMES 10000
+volatile unsigned page[4096 / 4];
+
+/*
+ * Execute
+ *   time TIMEOUT=20 ./x86-run x86/mwait.flat -append '0 1 1'
+ * (first two arguments are eax and ecx for MWAIT, the third is FLAGS.IF bit)
+ * I assume you have 1000 Hz scheduler, so the test should take about 10
+ * seconds to run if mwait works (host timer interrupts will kick mwait).
+ *
+ * If you get far less, then mwait is just nop, as in the case of
+ *
+ *   time TIMEOUT=20 ./x86-run x86/mwait.flat -append '0 1 0'
+ *
+ * All other combinations of arguments should take 10 seconds.
+ * Getting killed by the TIMEOUT most likely means that you have different HZ,
+ * but could also be a bug ...
+ */
+int main(int argc, char **argv)
+{
+	uint32_t eax = atol(argv[1]);
+	uint32_t ecx = atol(argv[2]);
+	bool sti = atol(argv[3]);
+	unsigned resumes = 0;
+
+	if (sti)
+		asm volatile ("sti");
+	else
+		asm volatile ("cli");
+
+	while (resumes < TARGET_RESUMES) {
+		asm volatile("monitor" :: "a" (page), "c" (0), "d" (0));
+		asm volatile("mwait" :: "a" (eax), "c" (ecx));
+		resumes++;
+	}
+
+	report("resumed from mwait %u times", resumes == TARGET_RESUMES, resumes);
+	return report_summary();
+}
-- 
2.11.0

[toc] | [prev] | [next] | [standalone]

#1602643

From	"Gabriel L. Somlo" <gsomlo@gmail.com>
Date	2017-03-16 17:50 +0100
Message-ID	<tlGL8-14c-9@gated-at.bofh.it>
In reply to	#1602605

On Thu, Mar 16, 2017 at 05:01:58PM +0100, Radim Krčmář wrote:
> 2017-03-16 16:35+0100, Radim Krčmář:
> > 2017-03-16 10:58-0400, Gabriel L. Somlo:
> >> The intel manual said the same thing back in 2010 as well. However,
> >> regardless of how any flags were set, interrupt-window exiting or not,
> >> "normal" L1 MWAIT behavior was that it woke up immediately regardless.
> >> Remember, never going to sleep is still correct ("normal" ?) behavior
> >> per the ISA definition of MWAIT :)
> > 
> > I'll write a simple kvm-unit-test to better understand why it is broken
> > for you ...
> 
> Please get git://git.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git
> 
> and try this, thanks!
> 
> ---8<---
> x86/mwait: crappy test
> 
> `./configure && make` to build it, then follow the comment in code to
> try few cases.

kvm-unit-tests]$ time TIMEOUT=20 ./x86-run x86/mwait.flat -append '0 1 1'
timeout -k 1s --foreground 20 qemu-kvm -nodefaults -enable-kvm -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -vnc none -serial stdio -device pci-testdev -kernel x86/mwait.flat -append 0 1 1
enabling apic
PASS: resumed from mwait 10000 times
SUMMARY: 1 tests

real    0m10.564s
user    0m10.339s
sys     0m0.225s


and

kvm-unit-tests]$ time TIMEOUT=20 ./x86-run x86/mwait.flat -append '0 1 0'
timeout -k 1s --foreground 20 qemu-kvm -nodefaults -enable-kvm -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -vnc none -serial stdio -device pci-testdev -kernel x86/mwait.flat -append 0 1 0
enabling apic
PASS: resumed from mwait 10000 times
SUMMARY: 1 tests

real    0m0.746s
user    0m0.555s
sys     0m0.200s

Both of these with Michael's v5 patch applied, on the MacPro1,1.

Similar behavior (0 1 1 takes 10 seconds, 0 1 0 returns immediately)
on the macbook air.

If I revert to the original (nop-emulated MWAIT) kvm source, I get
both versions to return immediately.

HTH,
--Gabriel



> 
> ---
>  x86/Makefile.common |  1 +
>  x86/mwait.c         | 41 +++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 42 insertions(+)
>  create mode 100644 x86/mwait.c
> 
> diff --git a/x86/Makefile.common b/x86/Makefile.common
> index 1dad18ba26e1..1e708a6acd39 100644
> --- a/x86/Makefile.common
> +++ b/x86/Makefile.common
> @@ -46,6 +46,7 @@ tests-common = $(TEST_DIR)/vmexit.flat $(TEST_DIR)/tsc.flat \
>                 $(TEST_DIR)/tsc_adjust.flat $(TEST_DIR)/asyncpf.flat \
>                 $(TEST_DIR)/init.flat $(TEST_DIR)/smap.flat \
>                 $(TEST_DIR)/hyperv_synic.flat $(TEST_DIR)/hyperv_stimer.flat \
> +               $(TEST_DIR)/mwait.flat \
>  
>  ifdef API
>  tests-common += api/api-sample
> diff --git a/x86/mwait.c b/x86/mwait.c
> new file mode 100644
> index 000000000000..c21dab5cc97d
> --- /dev/null
> +++ b/x86/mwait.c
> @@ -0,0 +1,41 @@
> +#include "vm.h"
> +
> +#define TARGET_RESUMES 10000
> +volatile unsigned page[4096 / 4];
> +
> +/*
> + * Execute
> + *   time TIMEOUT=20 ./x86-run x86/mwait.flat -append '0 1 1'
> + * (first two arguments are eax and ecx for MWAIT, the third is FLAGS.IF bit)
> + * I assume you have 1000 Hz scheduler, so the test should take about 10
> + * seconds to run if mwait works (host timer interrupts will kick mwait).
> + *
> + * If you get far less, then mwait is just nop, as in the case of
> + *
> + *   time TIMEOUT=20 ./x86-run x86/mwait.flat -append '0 1 0'
> + *
> + * All other combinations of arguments should take 10 seconds.
> + * Getting killed by the TIMEOUT most likely means that you have different HZ,
> + * but could also be a bug ...
> + */
> +int main(int argc, char **argv)
> +{
> +	uint32_t eax = atol(argv[1]);
> +	uint32_t ecx = atol(argv[2]);
> +	bool sti = atol(argv[3]);
> +	unsigned resumes = 0;
> +
> +	if (sti)
> +		asm volatile ("sti");
> +	else
> +		asm volatile ("cli");
> +
> +	while (resumes < TARGET_RESUMES) {
> +		asm volatile("monitor" :: "a" (page), "c" (0), "d" (0));
> +		asm volatile("mwait" :: "a" (eax), "c" (ecx));
> +		resumes++;
> +	}
> +
> +	report("resumed from mwait %u times", resumes == TARGET_RESUMES, resumes);
> +	return report_summary();
> +}
> -- 
> 2.11.0
>

[toc] | [prev] | [next] | [standalone]

#1602692

From	"Michael S. Tsirkin" <mst@redhat.com>
Date	2017-03-16 18:30 +0100
Message-ID	<tlHnR-1ze-37@gated-at.bofh.it>
In reply to	#1602643

On Thu, Mar 16, 2017 at 12:47:50PM -0400, Gabriel L. Somlo wrote:
> On Thu, Mar 16, 2017 at 05:01:58PM +0100, Radim Krčmář wrote:
> > 2017-03-16 16:35+0100, Radim Krčmář:
> > > 2017-03-16 10:58-0400, Gabriel L. Somlo:
> > >> The intel manual said the same thing back in 2010 as well. However,
> > >> regardless of how any flags were set, interrupt-window exiting or not,
> > >> "normal" L1 MWAIT behavior was that it woke up immediately regardless.
> > >> Remember, never going to sleep is still correct ("normal" ?) behavior
> > >> per the ISA definition of MWAIT :)
> > > 
> > > I'll write a simple kvm-unit-test to better understand why it is broken
> > > for you ...
> > 
> > Please get git://git.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git
> > 
> > and try this, thanks!
> > 
> > ---8<---
> > x86/mwait: crappy test
> > 
> > `./configure && make` to build it, then follow the comment in code to
> > try few cases.
> 
> kvm-unit-tests]$ time TIMEOUT=20 ./x86-run x86/mwait.flat -append '0 1 1'
> timeout -k 1s --foreground 20 qemu-kvm -nodefaults -enable-kvm -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -vnc none -serial stdio -device pci-testdev -kernel x86/mwait.flat -append 0 1 1
> enabling apic
> PASS: resumed from mwait 10000 times
> SUMMARY: 1 tests
> 
> real    0m10.564s
> user    0m10.339s
> sys     0m0.225s
> 
> 
> and
> 
> kvm-unit-tests]$ time TIMEOUT=20 ./x86-run x86/mwait.flat -append '0 1 0'
> timeout -k 1s --foreground 20 qemu-kvm -nodefaults -enable-kvm -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -vnc none -serial stdio -device pci-testdev -kernel x86/mwait.flat -append 0 1 0
> enabling apic
> PASS: resumed from mwait 10000 times
> SUMMARY: 1 tests
> 
> real    0m0.746s
> user    0m0.555s
> sys     0m0.200s
> 
> Both of these with Michael's v5 patch applied, on the MacPro1,1.

Would it make sense to try to set ECX to 0? 0 0 1 and 0 0 0.


> Similar behavior (0 1 1 takes 10 seconds, 0 1 0 returns immediately)
> on the macbook air.
> 
> If I revert to the original (nop-emulated MWAIT) kvm source, I get
> both versions to return immediately.
> 
> HTH,
> --Gabriel
> 
> 
> 
> > 
> > ---
> >  x86/Makefile.common |  1 +
> >  x86/mwait.c         | 41 +++++++++++++++++++++++++++++++++++++++++
> >  2 files changed, 42 insertions(+)
> >  create mode 100644 x86/mwait.c
> > 
> > diff --git a/x86/Makefile.common b/x86/Makefile.common
> > index 1dad18ba26e1..1e708a6acd39 100644
> > --- a/x86/Makefile.common
> > +++ b/x86/Makefile.common
> > @@ -46,6 +46,7 @@ tests-common = $(TEST_DIR)/vmexit.flat $(TEST_DIR)/tsc.flat \
> >                 $(TEST_DIR)/tsc_adjust.flat $(TEST_DIR)/asyncpf.flat \
> >                 $(TEST_DIR)/init.flat $(TEST_DIR)/smap.flat \
> >                 $(TEST_DIR)/hyperv_synic.flat $(TEST_DIR)/hyperv_stimer.flat \
> > +               $(TEST_DIR)/mwait.flat \
> >  
> >  ifdef API
> >  tests-common += api/api-sample
> > diff --git a/x86/mwait.c b/x86/mwait.c
> > new file mode 100644
> > index 000000000000..c21dab5cc97d
> > --- /dev/null
> > +++ b/x86/mwait.c
> > @@ -0,0 +1,41 @@
> > +#include "vm.h"
> > +
> > +#define TARGET_RESUMES 10000
> > +volatile unsigned page[4096 / 4];
> > +
> > +/*
> > + * Execute
> > + *   time TIMEOUT=20 ./x86-run x86/mwait.flat -append '0 1 1'
> > + * (first two arguments are eax and ecx for MWAIT, the third is FLAGS.IF bit)
> > + * I assume you have 1000 Hz scheduler, so the test should take about 10
> > + * seconds to run if mwait works (host timer interrupts will kick mwait).
> > + *
> > + * If you get far less, then mwait is just nop, as in the case of
> > + *
> > + *   time TIMEOUT=20 ./x86-run x86/mwait.flat -append '0 1 0'
> > + *
> > + * All other combinations of arguments should take 10 seconds.
> > + * Getting killed by the TIMEOUT most likely means that you have different HZ,
> > + * but could also be a bug ...
> > + */
> > +int main(int argc, char **argv)
> > +{
> > +	uint32_t eax = atol(argv[1]);
> > +	uint32_t ecx = atol(argv[2]);
> > +	bool sti = atol(argv[3]);
> > +	unsigned resumes = 0;
> > +
> > +	if (sti)
> > +		asm volatile ("sti");
> > +	else
> > +		asm volatile ("cli");
> > +
> > +	while (resumes < TARGET_RESUMES) {
> > +		asm volatile("monitor" :: "a" (page), "c" (0), "d" (0));
> > +		asm volatile("mwait" :: "a" (eax), "c" (ecx));
> > +		resumes++;
> > +	}
> > +
> > +	report("resumed from mwait %u times", resumes == TARGET_RESUMES, resumes);
> > +	return report_summary();
> > +}
> > -- 
> > 2.11.0
> >

[toc] | [prev] | [next] | [standalone]

#1602716

From	"Gabriel L. Somlo" <gsomlo@gmail.com>
Date	2017-03-16 18:50 +0100
Message-ID	<tlHHd-1ID-37@gated-at.bofh.it>
In reply to	#1602692

On Thu, Mar 16, 2017 at 07:27:34PM +0200, Michael S. Tsirkin wrote:
> On Thu, Mar 16, 2017 at 12:47:50PM -0400, Gabriel L. Somlo wrote:
> > On Thu, Mar 16, 2017 at 05:01:58PM +0100, Radim Krčmář wrote:
> > > 2017-03-16 16:35+0100, Radim Krčmář:
> > > > 2017-03-16 10:58-0400, Gabriel L. Somlo:
> > > >> The intel manual said the same thing back in 2010 as well. However,
> > > >> regardless of how any flags were set, interrupt-window exiting or not,
> > > >> "normal" L1 MWAIT behavior was that it woke up immediately regardless.
> > > >> Remember, never going to sleep is still correct ("normal" ?) behavior
> > > >> per the ISA definition of MWAIT :)
> > > > 
> > > > I'll write a simple kvm-unit-test to better understand why it is broken
> > > > for you ...
> > > 
> > > Please get git://git.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git
> > > 
> > > and try this, thanks!
> > > 
> > > ---8<---
> > > x86/mwait: crappy test
> > > 
> > > `./configure && make` to build it, then follow the comment in code to
> > > try few cases.
> > 
> > kvm-unit-tests]$ time TIMEOUT=20 ./x86-run x86/mwait.flat -append '0 1 1'
> > timeout -k 1s --foreground 20 qemu-kvm -nodefaults -enable-kvm -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -vnc none -serial stdio -device pci-testdev -kernel x86/mwait.flat -append 0 1 1
> > enabling apic
> > PASS: resumed from mwait 10000 times
> > SUMMARY: 1 tests
> > 
> > real    0m10.564s
> > user    0m10.339s
> > sys     0m0.225s
> > 
> > 
> > and
> > 
> > kvm-unit-tests]$ time TIMEOUT=20 ./x86-run x86/mwait.flat -append '0 1 0'
> > timeout -k 1s --foreground 20 qemu-kvm -nodefaults -enable-kvm -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -vnc none -serial stdio -device pci-testdev -kernel x86/mwait.flat -append 0 1 0
> > enabling apic
> > PASS: resumed from mwait 10000 times
> > SUMMARY: 1 tests
> > 
> > real    0m0.746s
> > user    0m0.555s
> > sys     0m0.200s
> > 
> > Both of these with Michael's v5 patch applied, on the MacPro1,1.
> 
> Would it make sense to try to set ECX to 0? 0 0 1 and 0 0 0.

$ time TIMEOUT=20 ./x86-run x86/mwait.flat -append '0 0 1'
timeout -k 1s --foreground 20 qemu-kvm -nodefaults -enable-kvm -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -vnc none -serial stdio -device pci-testdev -kernel x86/mwait.flat -append 0 0 1
enabling apic
PASS: resumed from mwait 10000 times
SUMMARY: 1 tests

real    0m10.567s
user    0m10.367s
sys     0m0.210s


$ time TIMEOUT=20 ./x86-run x86/mwait.flat -append '0 0 0'
timeout -k 1s --foreground 20 qemu-kvm -nodefaults -enable-kvm -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -vnc none -serial stdio -device pci-testdev -kernel x86/mwait.flat -append 0 0 0
enabling apic
PASS: resumed from mwait 10000 times
SUMMARY: 1 tests

real    0m10.549s
user    0m10.352s
sys     0m0.206s

Both took 10 seconds.
 
> 
> > Similar behavior (0 1 1 takes 10 seconds, 0 1 0 returns immediately)
> > on the macbook air.
> > 
> > If I revert to the original (nop-emulated MWAIT) kvm source, I get
> > both versions to return immediately.
> > 
> > HTH,
> > --Gabriel
> > 
> > 
> > 
> > > 
> > > ---
> > >  x86/Makefile.common |  1 +
> > >  x86/mwait.c         | 41 +++++++++++++++++++++++++++++++++++++++++
> > >  2 files changed, 42 insertions(+)
> > >  create mode 100644 x86/mwait.c
> > > 
> > > diff --git a/x86/Makefile.common b/x86/Makefile.common
> > > index 1dad18ba26e1..1e708a6acd39 100644
> > > --- a/x86/Makefile.common
> > > +++ b/x86/Makefile.common
> > > @@ -46,6 +46,7 @@ tests-common = $(TEST_DIR)/vmexit.flat $(TEST_DIR)/tsc.flat \
> > >                 $(TEST_DIR)/tsc_adjust.flat $(TEST_DIR)/asyncpf.flat \
> > >                 $(TEST_DIR)/init.flat $(TEST_DIR)/smap.flat \
> > >                 $(TEST_DIR)/hyperv_synic.flat $(TEST_DIR)/hyperv_stimer.flat \
> > > +               $(TEST_DIR)/mwait.flat \
> > >  
> > >  ifdef API
> > >  tests-common += api/api-sample
> > > diff --git a/x86/mwait.c b/x86/mwait.c
> > > new file mode 100644
> > > index 000000000000..c21dab5cc97d
> > > --- /dev/null
> > > +++ b/x86/mwait.c
> > > @@ -0,0 +1,41 @@
> > > +#include "vm.h"
> > > +
> > > +#define TARGET_RESUMES 10000
> > > +volatile unsigned page[4096 / 4];
> > > +
> > > +/*
> > > + * Execute
> > > + *   time TIMEOUT=20 ./x86-run x86/mwait.flat -append '0 1 1'
> > > + * (first two arguments are eax and ecx for MWAIT, the third is FLAGS.IF bit)
> > > + * I assume you have 1000 Hz scheduler, so the test should take about 10
> > > + * seconds to run if mwait works (host timer interrupts will kick mwait).
> > > + *
> > > + * If you get far less, then mwait is just nop, as in the case of
> > > + *
> > > + *   time TIMEOUT=20 ./x86-run x86/mwait.flat -append '0 1 0'
> > > + *
> > > + * All other combinations of arguments should take 10 seconds.
> > > + * Getting killed by the TIMEOUT most likely means that you have different HZ,
> > > + * but could also be a bug ...
> > > + */
> > > +int main(int argc, char **argv)
> > > +{
> > > +	uint32_t eax = atol(argv[1]);
> > > +	uint32_t ecx = atol(argv[2]);
> > > +	bool sti = atol(argv[3]);
> > > +	unsigned resumes = 0;
> > > +
> > > +	if (sti)
> > > +		asm volatile ("sti");
> > > +	else
> > > +		asm volatile ("cli");
> > > +
> > > +	while (resumes < TARGET_RESUMES) {
> > > +		asm volatile("monitor" :: "a" (page), "c" (0), "d" (0));
> > > +		asm volatile("mwait" :: "a" (eax), "c" (ecx));
> > > +		resumes++;
> > > +	}
> > > +
> > > +	report("resumed from mwait %u times", resumes == TARGET_RESUMES, resumes);
> > > +	return report_summary();
> > > +}
> > > -- 
> > > 2.11.0
> > >

[toc] | [prev] | [next] | [standalone]

#1602758

From	"Michael S. Tsirkin" <mst@redhat.com>
Date	2017-03-16 19:40 +0100
Message-ID	<tlItA-2iT-11@gated-at.bofh.it>
In reply to	#1602716

Let's take a step back and try to figure out how is
mwait called. How about dumping code of VCPUs
around mwait?  gdb disa command will do this.

-- 
MST

[toc] | [prev] | [next] | [standalone]

#1602797

From	"Michael S. Tsirkin" <mst@redhat.com>
Date	2017-03-16 20:30 +0100
Message-ID	<tlJfY-2Un-19@gated-at.bofh.it>
In reply to	#1602758

On Thu, Mar 16, 2017 at 03:24:41PM -0400, Gabriel L. Somlo wrote:
> On Thu, Mar 16, 2017 at 08:29:32PM +0200, Michael S. Tsirkin wrote:
> > Let's take a step back and try to figure out how is
> > mwait called. How about dumping code of VCPUs
> > around mwait?  gdb disa command will do this.
> 
> Started guest with '-s', tried to attach from gdb with
> "target remote localhost:1234", got
> "remote 'g' packet reply is too long: <lengthy string of numbers>"

Try

set arch x86-64:x86-64


> Tried typing 'cont' in the qemu monitor, got os x to crash:
> 
> panic (cpu 1 caller 0xffffff7f813ff488): pmLock: waited too long, held
> by 0xffffff7f813eff65
> 
> Hmm, maybe that's where it keeps its monitor/mwait idle loop.
> Restarted the guest, tried this from monitor:
> 
> 	dump-guest-memory foobar 0xffffff7f813e0000 0x20000
> 
> Got "'dump-guest-memory' has failed: integer is for 32-bit values"
> 
> Hmmm... I have no idea what I'm doing anymore at this point... :)
> 
> --G

I think 0xffffff7f813ff488 is a PC.

-- 
MST

[toc] | [prev] | [next] | [standalone]

#1602824

From	"Gabriel L. Somlo" <gsomlo@gmail.com>
Date	2017-03-16 21:20 +0100
Message-ID	<tlK2m-3uf-11@gated-at.bofh.it>
In reply to	#1602797

On Thu, Mar 16, 2017 at 09:27:56PM +0200, Michael S. Tsirkin wrote:
> On Thu, Mar 16, 2017 at 03:24:41PM -0400, Gabriel L. Somlo wrote:
> > On Thu, Mar 16, 2017 at 08:29:32PM +0200, Michael S. Tsirkin wrote:
> > > Let's take a step back and try to figure out how is
> > > mwait called. How about dumping code of VCPUs
> > > around mwait?  gdb disa command will do this.
> > 
> > Started guest with '-s', tried to attach from gdb with
> > "target remote localhost:1234", got
> > "remote 'g' packet reply is too long: <lengthy string of numbers>"
> 
> Try
> 
> set arch x86-64:x86-64

'set architecture i386:x86-64:intel' is what worked for me;

Been rooting around for a while, can't find mwait or monitor :(

Guess I'll have to recompile KVM to actually issue an invalid opcode,
so OS X will print a panic message with the exact address :)

Stay tuned...
 
> 
> > Tried typing 'cont' in the qemu monitor, got os x to crash:
> > 
> > panic (cpu 1 caller 0xffffff7f813ff488): pmLock: waited too long, held
> > by 0xffffff7f813eff65
> > 
> > Hmm, maybe that's where it keeps its monitor/mwait idle loop.
> > Restarted the guest, tried this from monitor:
> > 
> > 	dump-guest-memory foobar 0xffffff7f813e0000 0x20000
> > 
> > Got "'dump-guest-memory' has failed: integer is for 32-bit values"
> > 
> > Hmmm... I have no idea what I'm doing anymore at this point... :)
> > 
> > --G
> 
> I think 0xffffff7f813ff488 is a PC.
> 
> -- 
> MST

[toc] | [prev] | [next] | [standalone]

#1602839

From	"Gabriel L. Somlo" <gsomlo@gmail.com>
Date	2017-03-16 22:20 +0100
Message-ID	<tlKYq-49V-19@gated-at.bofh.it>
In reply to	#1602824

On Thu, Mar 16, 2017 at 04:17:11PM -0400, Gabriel L. Somlo wrote:
> On Thu, Mar 16, 2017 at 09:27:56PM +0200, Michael S. Tsirkin wrote:
> > On Thu, Mar 16, 2017 at 03:24:41PM -0400, Gabriel L. Somlo wrote:
> > > On Thu, Mar 16, 2017 at 08:29:32PM +0200, Michael S. Tsirkin wrote:
> > > > Let's take a step back and try to figure out how is
> > > > mwait called. How about dumping code of VCPUs
> > > > around mwait?  gdb disa command will do this.
> > > 
> > > Started guest with '-s', tried to attach from gdb with
> > > "target remote localhost:1234", got
> > > "remote 'g' packet reply is too long: <lengthy string of numbers>"
> > 
> > Try
> > 
> > set arch x86-64:x86-64
> 
> 'set architecture i386:x86-64:intel' is what worked for me;
> 
> Been rooting around for a while, can't find mwait or monitor :(
> 
> Guess I'll have to recompile KVM to actually issue an invalid opcode,
> so OS X will print a panic message with the exact address :)
> 
> Stay tuned...

OK, so I found a few instances. The one closest to where a random
interrupt from gdb landed, was this one:

...
   0xffffff7f813ff379:  mov    0x90(%r15),%rax
   0xffffff7f813ff380:  mov    0x18(%rax),%rsi
   0xffffff7f813ff384:  xor    %ecx,%ecx
   0xffffff7f813ff386:  mov    %rsi,%rax
   0xffffff7f813ff389:  xor    %edx,%edx
   0xffffff7f813ff38b:  monitor %rax,%rcx,%rdx
   0xffffff7f813ff38e:  test   %r14,%r14
   0xffffff7f813ff391:  je     0xffffff7f813ff3ad
   0xffffff7f813ff393:  movq   $0x0,0x8(%r14)
   0xffffff7f813ff39b:  movl   $0x0,(%r14)
   0xffffff7f813ff3a2:  test   %ebx,%ebx
   0xffffff7f813ff3a4:  je     0xffffff7f813ff3b2
   0xffffff7f813ff3a6:  mfence 
   0xffffff7f813ff3a9:  wbinvd
   0xffffff7f813ff3ab:  jmp    0xffffff7f813ff3b2
   0xffffff7f813ff3ad:  cmpl   $0x0,(%rsi)
   0xffffff7f813ff3b0:  jne    0xffffff7f813ff3d6
   0xffffff7f813ff3b2:  mov    %r12d,%eax
   0xffffff7f813ff3b5:  imul   $0x148,%rax,%rax
   0xffffff7f813ff3bc:  lea    0x153bd(%rip),%rcx        # 0xffffff7f81414780
   0xffffff7f813ff3c3:  mov    (%rcx),%rcx 
   0xffffff7f813ff3c6:  mov    0x20(%rcx),%rcx
   0xffffff7f813ff3ca:  mov    0xc(%rcx,%rax,1),%eax
   0xffffff7f813ff3ce:  mov    $0x1,%ecx
   0xffffff7f813ff3d3:  mwait  %rax,%rcx
=> 0xffffff7f813ff3d6:  lfence
   0xffffff7f813ff3d9:  rdtsc  
   0xffffff7f813ff3db:  lfence 
   0xffffff7f813ff3de:  mov    %rax,%rbx
   0xffffff7f813ff3e1:  mov    %rdx,%r15
...

Also, there were a few more within the range occupied by
AppleIntelCPUPowerManagement.kext (which provides is the "smart"
idle loop used by OS X):

...
   0xffffff7f813f799a:  mov    0x90(%r15),%rax
   0xffffff7f813f79a1:  mov    0x18(%rax),%r15
   0xffffff7f813f79a5:  xor    %ecx,%ecx
   0xffffff7f813f79a7:  mov    %r15,%rax
   0xffffff7f813f79aa:  xor    %edx,%edx
   0xffffff7f813f79ac:  monitor %rax,%rcx,%rdx
   0xffffff7f813f79af:  mov    %r12d,%r12d
   0xffffff7f813f79b2:  imul   $0x148,%r12,%r13
   0xffffff7f813f79b9:  lea    0x1cdc0(%rip),%rax        # 0xffffff7f81414780
   0xffffff7f813f79c0:  mov    (%rax),%rax
   0xffffff7f813f79c3:  mov    0x20(%rax),%rcx
   0xffffff7f813f79c7:  testb  $0x10,0x2(%rcx,%r13,1)
   0xffffff7f813f79cd:  je     0xffffff7f813f79d5
   0xffffff7f813f79cf:  callq  *0x80(%rax)
   0xffffff7f813f79d5:  test   %r14,%r14
   0xffffff7f813f79d8:  je     0xffffff7f813f79f4
   0xffffff7f813f79da:  movq   $0x0,0x8(%r14)
   0xffffff7f813f79e2:  movl   $0x0,(%r14)
   0xffffff7f813f79e9:  test   %ebx,%ebx
   0xffffff7f813f79eb:  je     0xffffff7f813f79fa
   0xffffff7f813f79ed:  mfence  
   0xffffff7f813f79f0:  wbinvd 
   0xffffff7f813f79f2:  jmp    0xffffff7f813f79fa
   0xffffff7f813f79f4:  cmpl   $0x0,(%r15)
   0xffffff7f813f79f8:  jne    0xffffff7f813f7a15
   0xffffff7f813f79fa:  lea    0x1cd7f(%rip),%rax        # 0xffffff7f81414780
   0xffffff7f813f7a01:  mov    (%rax),%rax
   0xffffff7f813f7a04:  mov    0x20(%rax),%rax
   0xffffff7f813f7a08:  mov    0xc(%rax,%r13,1),%eax
   0xffffff7f813f7a0d:  mov    $0x1,%ecx
   0xffffff7f813f7a12:  mwait  %rax,%rcx
   0xffffff7f813f7a15:  lfence 
   0xffffff7f813f7a18:  rdtsc  
   0xffffff7f813f7a1a:  lfence 
   0xffffff7f813f7a1d:  mov    %rax,%rbx
   0xffffff7f813f7a20:  mov    %rdx,%r15
...

...
   0xffffff7f813f89c9:  xor    %ecx,%ecx
   0xffffff7f813f89cb:  mov    %r13,%rax
   0xffffff7f813f89ce:  xor    %edx,%edx
   0xffffff7f813f89d0:  monitor %rax,%rcx,%rdx
   0xffffff7f813f89d3:  mov    %r12d,%r15d
   0xffffff7f813f89d6:  imul   $0x148,%r15,%r12
   0xffffff7f813f89dd:  lea    0x1bd9c(%rip),%rax        # 0xffffff7f81414780
   0xffffff7f813f89e4:  mov    (%rax),%rax
   0xffffff7f813f89e7:  mov    0x20(%rax),%rcx
   0xffffff7f813f89eb:  testb  $0x10,0x2(%rcx,%r12,1)
   0xffffff7f813f89f1:  je     0xffffff7f813f89f9
   0xffffff7f813f89f3:  callq  *0x80(%rax)
   0xffffff7f813f89f9:  test   %r14,%r14
   0xffffff7f813f89fc:  je     0xffffff7f813f8a18
   0xffffff7f813f89fe:  movq   $0x0,0x8(%r14)
   0xffffff7f813f8a06:  movl   $0x0,(%r14)
   0xffffff7f813f8a0d:  test   %ebx,%ebx
   0xffffff7f813f8a0f:  je     0xffffff7f813f8a1f
   0xffffff7f813f8a11:  mfence 
   0xffffff7f813f8a14:  wbinvd 
   0xffffff7f813f8a16:  jmp    0xffffff7f813f8a1f
   0xffffff7f813f8a18:  cmpl   $0x0,0x0(%r13)
   0xffffff7f813f8a1d:  jne    0xffffff7f813f8a3a
   0xffffff7f813f8a1f:  lea    0x1bd5a(%rip),%rax        # 0xffffff7f81414780
   0xffffff7f813f8a26:  mov    (%rax),%rax
   0xffffff7f813f8a29:  mov    0x20(%rax),%rax
   0xffffff7f813f8a2d:  mov    0xc(%rax,%r12,1),%eax
   0xffffff7f813f8a32:  mov    $0x1,%ecx
   0xffffff7f813f8a37:  mwait  %rax,%rcx
   0xffffff7f813f8a3a:  lfence 
   0xffffff7f813f8a3d:  rdtsc  
   0xffffff7f813f8a3f:  lfence  
   0xffffff7f813f8a42:  mov    %rax,%rbx
   0xffffff7f813f8a45:  mov    %rdx,%r12
   0xffffff7f813f8a48:  shl    $0x20,%r12
...

...
   0xffffff7f81401c10:  mov    %r13,%rax
   0xffffff7f81401c13:  xor    %edx,%edx
   0xffffff7f81401c15:  monitor %rax,%rcx,%rdx
   0xffffff7f81401c18:  mov    %r12d,%r15d
   0xffffff7f81401c1b:  imul   $0x148,%r15,%r12
   0xffffff7f81401c22:  lea    0x12b57(%rip),%rax        # 0xffffff7f81414780
   0xffffff7f81401c29:  mov    (%rax),%rax
   0xffffff7f81401c2c:  mov    0x20(%rax),%rcx
   0xffffff7f81401c30:  testb  $0x10,0x2(%rcx,%r12,1)
   0xffffff7f81401c36:  je     0xffffff7f81401c3e
   0xffffff7f81401c38:  callq  *0x80(%rax)
   0xffffff7f81401c3e:  test   %r14,%r14
   0xffffff7f81401c41:  je     0xffffff7f81401c5d
   0xffffff7f81401c43:  movq   $0x0,0x8(%r14)
   0xffffff7f81401c4b:  movl   $0x0,(%r14)
   0xffffff7f81401c52:  test   %ebx,%ebx
   0xffffff7f81401c54:  je     0xffffff7f81401c64
   0xffffff7f81401c56:  mfence 
   0xffffff7f81401c59:  wbinvd  
   0xffffff7f81401c5b:  jmp    0xffffff7f81401c64
   0xffffff7f81401c5d:  cmpl   $0x0,0x0(%r13)
   0xffffff7f81401c62:  jne    0xffffff7f81401c7f
   0xffffff7f81401c64:  lea    0x12b15(%rip),%rax        # 0xffffff7f81414780
   0xffffff7f81401c6b:  mov    (%rax),%rax
   0xffffff7f81401c6e:  mov    0x20(%rax),%rax
   0xffffff7f81401c72:  mov    0xc(%rax,%r12,1),%eax
   0xffffff7f81401c77:  mov    $0x1,%ecx
   0xffffff7f81401c7c:  mwait  %rax,%rcx
   0xffffff7f81401c7f:  lfence 
   0xffffff7f81401c82:  rdtsc  
   0xffffff7f81401c84:  lfence 
   0xffffff7f81401c87:  mov    %rax,%rbx
   0xffffff7f81401c8a:  mov    %rdx,%r12
   0xffffff7f81401c8d:  shl    $0x20,%r12
   0xffffff7f81401c91:  lea    0xaf1c(%rip),%rax        # 0xffffff7f8140cbb4
   0xffffff7f81401c98:  testb  $0x1,(%rax)
...

If that's not enough context, I can email you the whole 'script'
output I collected...

HTH,
--Gabriel

[toc] | [prev] | [next] | [standalone]

Page 1 of 3 [1] 2 3 Next page →

csiph-web

[PATCH v5 untested] kvm: better MWAIT emulation for guests

Contents

#1601754 — [PATCH v5 untested] kvm: better MWAIT emulation for guests

#1601811

#1601814

#1602269

#1602309

#1602590

#1602591

#1602626

#1602312

#1602427

#1602525

#1602553

#1602605

#1602643

#1602692

#1602716

#1602758

#1602797

#1602824

#1602839