Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1601754 > unrolled thread
| Started by | "Michael S. Tsirkin" <mst@redhat.com> |
|---|---|
| First post | 2017-03-15 22:30 +0100 |
| Last post | 2017-04-04 15:20 +0200 |
| Articles | 12 on this page of 52 — 8 participants |
Back to article view | Back to linux.kernel
[PATCH v5 untested] kvm: better MWAIT emulation for guests "Michael S. Tsirkin" <mst@redhat.com> - 2017-03-15 22:30 +0100
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Gabriel L. Somlo" <gsomlo@gmail.com> - 2017-03-16 00:40 +0100
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Michael S. Tsirkin" <mst@redhat.com> - 2017-03-16 00:50 +0100
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Gabriel L. Somlo" <gsomlo@gmail.com> - 2017-03-16 14:30 +0100
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests Radim Krčmář <rkrcmar@redhat.com> - 2017-03-16 15:10 +0100
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Gabriel L. Somlo" <gsomlo@gmail.com> - 2017-03-16 17:00 +0100
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests Radim Krčmář <rkrcmar@redhat.com> - 2017-03-16 17:00 +0100
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Gabriel L. Somlo" <gsomlo@gmail.com> - 2017-03-16 17:30 +0100
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Michael S. Tsirkin" <mst@redhat.com> - 2017-03-16 15:10 +0100
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Gabriel L. Somlo" <gsomlo@gmail.com> - 2017-03-16 16:00 +0100
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Michael S. Tsirkin" <mst@redhat.com> - 2017-03-16 16:30 +0100
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests Radim Krčmář <rkrcmar@redhat.com> - 2017-03-16 16:40 +0100
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests Radim Krčmář <rkrcmar@redhat.com> - 2017-03-16 17:10 +0100
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Gabriel L. Somlo" <gsomlo@gmail.com> - 2017-03-16 17:50 +0100
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Michael S. Tsirkin" <mst@redhat.com> - 2017-03-16 18:30 +0100
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Gabriel L. Somlo" <gsomlo@gmail.com> - 2017-03-16 18:50 +0100
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Michael S. Tsirkin" <mst@redhat.com> - 2017-03-16 19:40 +0100
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Michael S. Tsirkin" <mst@redhat.com> - 2017-03-16 20:30 +0100
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Gabriel L. Somlo" <gsomlo@gmail.com> - 2017-03-16 21:20 +0100
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Gabriel L. Somlo" <gsomlo@gmail.com> - 2017-03-16 22:20 +0100
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Michael S. Tsirkin" <mst@redhat.com> - 2017-03-17 03:10 +0100
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Gabriel L. Somlo" <gsomlo@gmail.com> - 2017-03-17 14:40 +0100
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Michael S. Tsirkin" <mst@redhat.com> - 2017-03-21 04:30 +0100
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests Radim Krčmář <rkrcmar@redhat.com> - 2017-03-21 18:10 +0100
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests Nadav Amit <nadav.amit@gmail.com> - 2017-03-21 18:40 +0100
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests Radim Krčmář <rkrcmar@redhat.com> - 2017-03-21 20:30 +0100
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests Gabriel Somlo <gsomlo@gmail.com> - 2017-03-22 00:00 +0100
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests Nadav Amit <nadav.amit@gmail.com> - 2017-03-22 01:10 +0100
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Michael S. Tsirkin" <mst@redhat.com> - 2017-03-22 14:40 +0100
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Michael S. Tsirkin" <mst@redhat.com> - 2017-03-22 15:20 +0100
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Gabriel L. Somlo" <gsomlo@gmail.com> - 2017-03-22 15:20 +0100
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Gabriel L. Somlo" <gsomlo@gmail.com> - 2017-03-16 20:30 +0100
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests Radim Krčmář <rkrcmar@redhat.com> - 2017-03-16 18:30 +0100
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Gabriel L. Somlo" <gsomlo@gmail.com> - 2017-03-16 18:40 +0100
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Gabriel L. Somlo" <gsomlo@gmail.com> - 2017-03-16 17:20 +0100
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Michael S. Tsirkin" <mst@redhat.com> - 2017-03-16 17:50 +0100
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Gabriel L. Somlo" <gsomlo@gmail.com> - 2017-03-16 18:00 +0100
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Gabriel L. Somlo" <gsomlo@gmail.com> - 2017-03-16 18:00 +0100
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Michael S. Tsirkin" <mst@redhat.com> - 2017-03-16 18:20 +0100
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests Radim Krčmář <rkrcmar@redhat.com> - 2017-03-16 18:40 +0100
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests Joerg Roedel <joro@8bytes.org> - 2017-03-21 17:20 +0100
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests "Michael S. Tsirkin" <mst@redhat.com> - 2017-03-21 19:50 +0100
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests Alexander Graf <agraf@suse.de> - 2017-03-27 15:40 +0200
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests Radim Krčmář <rkrcmar@redhat.com> - 2017-03-28 16:40 +0200
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests Jim Mattson <jmattson@google.com> - 2017-03-28 22:40 +0200
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests Radim Krčmář <rkrcmar@redhat.com> - 2017-03-29 14:20 +0200
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests Alexander Graf <agraf@suse.de> - 2017-04-03 12:10 +0200
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests Radim Krčmář <rkrcmar@redhat.com> - 2017-04-04 14:40 +0200
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests Alexander Graf <agraf@suse.de> - 2017-04-04 15:00 +0200
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests Alexander Graf <agraf@suse.de> - 2017-04-04 15:20 +0200
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests Radim Krčmář <rkrcmar@redhat.com> - 2017-04-04 15:50 +0200
Re: [PATCH v5 untested] kvm: better MWAIT emulation for guests Radim Krčmář <rkrcmar@redhat.com> - 2017-04-04 15:20 +0200
Page 3 of 3 — ← Prev page 1 2 [3]
| From | Joerg Roedel <joro@8bytes.org> |
|---|---|
| Date | 2017-03-21 17:20 +0100 |
| Message-ID | <tnuFQ-5IV-37@gated-at.bofh.it> |
| In reply to | #1601754 |
On Wed, Mar 15, 2017 at 11:22:18PM +0200, Michael S. Tsirkin wrote: > diff --git a/arch/x86/kvm/svm.c b/arch/x86/kvm/svm.c > index d1efe2c..18e53bc 100644 > --- a/arch/x86/kvm/svm.c > +++ b/arch/x86/kvm/svm.c > @@ -1198,8 +1198,6 @@ static void init_vmcb(struct vcpu_svm *svm) > set_intercept(svm, INTERCEPT_CLGI); > set_intercept(svm, INTERCEPT_SKINIT); > set_intercept(svm, INTERCEPT_WBINVD); > - set_intercept(svm, INTERCEPT_MONITOR); > - set_intercept(svm, INTERCEPT_MWAIT); Why do you remove the intercepts for AMD? The new kvm_mwait_in_guest() function will always return false on AMD anyway, and on Intel you re-add the intercepts for !kvm_mwait_in_guest(). Joerg
[toc] | [prev] | [next] | [standalone]
| From | "Michael S. Tsirkin" <mst@redhat.com> |
|---|---|
| Date | 2017-03-21 19:50 +0100 |
| Message-ID | <tnx11-7bE-35@gated-at.bofh.it> |
| In reply to | #1605773 |
On Tue, Mar 21, 2017 at 05:16:32PM +0100, Joerg Roedel wrote: > On Wed, Mar 15, 2017 at 11:22:18PM +0200, Michael S. Tsirkin wrote: > > diff --git a/arch/x86/kvm/svm.c b/arch/x86/kvm/svm.c > > index d1efe2c..18e53bc 100644 > > --- a/arch/x86/kvm/svm.c > > +++ b/arch/x86/kvm/svm.c > > @@ -1198,8 +1198,6 @@ static void init_vmcb(struct vcpu_svm *svm) > > set_intercept(svm, INTERCEPT_CLGI); > > set_intercept(svm, INTERCEPT_SKINIT); > > set_intercept(svm, INTERCEPT_WBINVD); > > - set_intercept(svm, INTERCEPT_MONITOR); > > - set_intercept(svm, INTERCEPT_MWAIT); > > Why do you remove the intercepts for AMD? The new kvm_mwait_in_guest() > function will always return false on AMD anyway, I think that's a bug and I should fix it to return true there. > and on Intel you re-add > the intercepts for !kvm_mwait_in_guest(). > > > Joerg Does AMD need some work-around similar to CPUID5_ECX_INTERRUPT_BREAK? That's why we have kvm_mwait_in_guest ... -- MST
[toc] | [prev] | [next] | [standalone]
| From | Alexander Graf <agraf@suse.de> |
|---|---|
| Date | 2017-03-27 15:40 +0200 |
| Message-ID | <tpD2h-kn-21@gated-at.bofh.it> |
| In reply to | #1601754 |
On 15/03/2017 22:22, Michael S. Tsirkin wrote: > Guests running Mac OS 5, 6, and 7 (Leopard through Lion) have a problem: > unless explicitly provided with kernel command line argument > "idlehalt=0" they'd implicitly assume MONITOR and MWAIT availability, > without checking CPUID. > > We currently emulate that as a NOP but on VMX we can do better: let > guest stop the CPU until timer, IPI or memory change. CPU will be busy > but that isn't any worse than a NOP emulation. > > Note that mwait within guests is not the same as on real hardware > because halt causes an exit while mwait doesn't. For this reason it > might not be a good idea to use the regular MWAIT flag in CPUID to > signal this capability. Add a flag in the hypervisor leaf instead. So imagine we had proper MWAIT emulation capabilities based on page faults. In that case, we could do something as fancy as Treat MWAIT as pass-through by default Have a per-vcpu monitor timer 10 times a second in the background that checks which instruction we're in If we're in mwait for the last - say - 1 second, switch to emulated MWAIT, if $IP was in non-mwait within that time, reset counter. Or instead maybe just reuse the adapter hlt logic? Either way, with that we should be able to get super low latency IPIs running while still maintaining some sanity on systems which don't have dedicated CPUs for workloads. And we wouldn't need guest modifications, which is a great plus. So older guests (and Windows?) could benefit from mwait as well. Alex
[toc] | [prev] | [next] | [standalone]
| From | Radim Krčmář <rkrcmar@redhat.com> |
|---|---|
| Date | 2017-03-28 16:40 +0200 |
| Message-ID | <tq0rT-vR-13@gated-at.bofh.it> |
| In reply to | #1609837 |
2017-03-27 15:34+0200, Alexander Graf: > On 15/03/2017 22:22, Michael S. Tsirkin wrote: >> Guests running Mac OS 5, 6, and 7 (Leopard through Lion) have a problem: >> unless explicitly provided with kernel command line argument >> "idlehalt=0" they'd implicitly assume MONITOR and MWAIT availability, >> without checking CPUID. >> >> We currently emulate that as a NOP but on VMX we can do better: let >> guest stop the CPU until timer, IPI or memory change. CPU will be busy >> but that isn't any worse than a NOP emulation. >> >> Note that mwait within guests is not the same as on real hardware >> because halt causes an exit while mwait doesn't. For this reason it >> might not be a good idea to use the regular MWAIT flag in CPUID to >> signal this capability. Add a flag in the hypervisor leaf instead. > > So imagine we had proper MWAIT emulation capabilities based on page faults. > In that case, we could do something as fancy as > > Treat MWAIT as pass-through by default > > Have a per-vcpu monitor timer 10 times a second in the background that > checks which instruction we're in > > If we're in mwait for the last - say - 1 second, switch to emulated MWAIT, > if $IP was in non-mwait within that time, reset counter. Or we could reuse external interrupts for sampling. Exits trigerred by them would check for current instruction (probably would be best to limit just to timer tick) and a sufficient ratio (> 0?) of other exits would imply that MWAIT is not used. > Or instead maybe just reuse the adapter hlt logic? Emulated MWAIT is very similar to emulated HLT, so reusing the logic makes sense. We would just add new wakeup methods. > Either way, with that we should be able to get super low latency IPIs > running while still maintaining some sanity on systems which don't have > dedicated CPUs for workloads. > > And we wouldn't need guest modifications, which is a great plus. So older > guests (and Windows?) could benefit from mwait as well. There is no need guest modifications -- it could be exposed as standard MWAIT feature to the guest, with responsibilities for guest/host-impact on the user. I think that the page-fault based MWAIT would require paravirt if it should be enabled by default, because of performance concerns: Enabling write protection on a page needs a VM exit on all other VCPUs when beginning monitoring (to reload page permissions and prevent missed writes). We'd want to keep trapping writes to the page all the time because toggling is slow, but this could regress performance for an OS that has other data accessed by other VCPUs in that page. No current interface can tell the guest that it should reserve the whole page instead of what CPUID[5] says and that writes to the monitored page are not "cheap", but can trigger a VM exit ... And before we disable MWAIT exiting by default, we also have to understand the old OS X on core 2 bug from Gabriel.
[toc] | [prev] | [next] | [standalone]
| From | Jim Mattson <jmattson@google.com> |
|---|---|
| Date | 2017-03-28 22:40 +0200 |
| Message-ID | <tq64h-4CD-5@gated-at.bofh.it> |
| In reply to | #1611031 |
On Tue, Mar 28, 2017 at 7:28 AM, Radim Krčmář <rkrcmar@redhat.com> wrote: > 2017-03-27 15:34+0200, Alexander Graf: >> On 15/03/2017 22:22, Michael S. Tsirkin wrote: >>> Guests running Mac OS 5, 6, and 7 (Leopard through Lion) have a problem: >>> unless explicitly provided with kernel command line argument >>> "idlehalt=0" they'd implicitly assume MONITOR and MWAIT availability, >>> without checking CPUID. >>> >>> We currently emulate that as a NOP but on VMX we can do better: let >>> guest stop the CPU until timer, IPI or memory change. CPU will be busy >>> but that isn't any worse than a NOP emulation. >>> >>> Note that mwait within guests is not the same as on real hardware >>> because halt causes an exit while mwait doesn't. For this reason it >>> might not be a good idea to use the regular MWAIT flag in CPUID to >>> signal this capability. Add a flag in the hypervisor leaf instead. >> >> So imagine we had proper MWAIT emulation capabilities based on page faults. >> In that case, we could do something as fancy as >> >> Treat MWAIT as pass-through by default >> >> Have a per-vcpu monitor timer 10 times a second in the background that >> checks which instruction we're in >> >> If we're in mwait for the last - say - 1 second, switch to emulated MWAIT, >> if $IP was in non-mwait within that time, reset counter. > > Or we could reuse external interrupts for sampling. Exits trigerred by > them would check for current instruction (probably would be best to > limit just to timer tick) and a sufficient ratio (> 0?) of other exits > would imply that MWAIT is not used. > >> Or instead maybe just reuse the adapter hlt logic? > > Emulated MWAIT is very similar to emulated HLT, so reusing the logic > makes sense. We would just add new wakeup methods. > >> Either way, with that we should be able to get super low latency IPIs >> running while still maintaining some sanity on systems which don't have >> dedicated CPUs for workloads. >> >> And we wouldn't need guest modifications, which is a great plus. So older >> guests (and Windows?) could benefit from mwait as well. > > There is no need guest modifications -- it could be exposed as standard > MWAIT feature to the guest, with responsibilities for guest/host-impact > on the user. > > I think that the page-fault based MWAIT would require paravirt if it > should be enabled by default, because of performance concerns: > Enabling write protection on a page needs a VM exit on all other VCPUs > when beginning monitoring (to reload page permissions and prevent missed > writes). > We'd want to keep trapping writes to the page all the time because > toggling is slow, but this could regress performance for an OS that has > other data accessed by other VCPUs in that page. > No current interface can tell the guest that it should reserve the whole > page instead of what CPUID[5] says and that writes to the monitored page > are not "cheap", but can trigger a VM exit ... CPUID.05H:EBX is supposed to address the false sharing issue. IIRC, VMware Fusion reports 64 in CPUID.05H:EAX and 4096 in CPUID.05H:EBX when running Mac OS X guests. Per Intel's SDM volume 3, section 8.10.5, "To avoid false wake-ups; use the largest monitor line size to pad the data structure used to monitor writes. Software must make sure that beyond the data structure, no unrelated data variable exists in the triggering area for MWAIT. A pad may be needed to avoid this situation." Unfortunately, most operating systems do not follow this advice. > > And before we disable MWAIT exiting by default, we also have to > understand the old OS X on core 2 bug from Gabriel.
[toc] | [prev] | [next] | [standalone]
| From | Radim Krčmář <rkrcmar@redhat.com> |
|---|---|
| Date | 2017-03-29 14:20 +0200 |
| Message-ID | <tqkJX-6K0-7@gated-at.bofh.it> |
| In reply to | #1611344 |
2017-03-28 13:35-0700, Jim Mattson: > On Tue, Mar 28, 2017 at 7:28 AM, Radim Krčmář <rkrcmar@redhat.com> wrote: >> 2017-03-27 15:34+0200, Alexander Graf: >>> On 15/03/2017 22:22, Michael S. Tsirkin wrote: >>>> Guests running Mac OS 5, 6, and 7 (Leopard through Lion) have a problem: >>>> unless explicitly provided with kernel command line argument >>>> "idlehalt=0" they'd implicitly assume MONITOR and MWAIT availability, >>>> without checking CPUID. >>>> >>>> We currently emulate that as a NOP but on VMX we can do better: let >>>> guest stop the CPU until timer, IPI or memory change. CPU will be busy >>>> but that isn't any worse than a NOP emulation. >>>> >>>> Note that mwait within guests is not the same as on real hardware >>>> because halt causes an exit while mwait doesn't. For this reason it >>>> might not be a good idea to use the regular MWAIT flag in CPUID to >>>> signal this capability. Add a flag in the hypervisor leaf instead. >>> >>> So imagine we had proper MWAIT emulation capabilities based on page faults. >>> In that case, we could do something as fancy as >>> >>> Treat MWAIT as pass-through by default >>> >>> Have a per-vcpu monitor timer 10 times a second in the background that >>> checks which instruction we're in >>> >>> If we're in mwait for the last - say - 1 second, switch to emulated MWAIT, >>> if $IP was in non-mwait within that time, reset counter. >> >> Or we could reuse external interrupts for sampling. Exits trigerred by >> them would check for current instruction (probably would be best to >> limit just to timer tick) and a sufficient ratio (> 0?) of other exits >> would imply that MWAIT is not used. >> >>> Or instead maybe just reuse the adapter hlt logic? >> >> Emulated MWAIT is very similar to emulated HLT, so reusing the logic >> makes sense. We would just add new wakeup methods. >> >>> Either way, with that we should be able to get super low latency IPIs >>> running while still maintaining some sanity on systems which don't have >>> dedicated CPUs for workloads. >>> >>> And we wouldn't need guest modifications, which is a great plus. So older >>> guests (and Windows?) could benefit from mwait as well. >> >> There is no need guest modifications -- it could be exposed as standard >> MWAIT feature to the guest, with responsibilities for guest/host-impact >> on the user. >> >> I think that the page-fault based MWAIT would require paravirt if it >> should be enabled by default, because of performance concerns: >> Enabling write protection on a page needs a VM exit on all other VCPUs >> when beginning monitoring (to reload page permissions and prevent missed >> writes). >> We'd want to keep trapping writes to the page all the time because >> toggling is slow, but this could regress performance for an OS that has >> other data accessed by other VCPUs in that page. >> No current interface can tell the guest that it should reserve the whole >> page instead of what CPUID[5] says and that writes to the monitored page >> are not "cheap", but can trigger a VM exit ... > > CPUID.05H:EBX is supposed to address the false sharing issue. IIRC, > VMware Fusion reports 64 in CPUID.05H:EAX and 4096 in CPUID.05H:EBX > when running Mac OS X guests. Per Intel's SDM volume 3, section > 8.10.5, "To avoid false wake-ups; use the largest monitor line size to > pad the data structure used to monitor writes. Software must make sure > that beyond the data structure, no unrelated data variable exists in > the triggering area for MWAIT. A pad may be needed to avoid this > situation." Unfortunately, most operating systems do not follow this > advice. Right, EBX provides what we need to expose that the whole page is monitored, thanks! > Unfortunately, most operating systems do not follow this > advice. Yeah ... KVM could add yet another heuristic to drop MWAIT emulation and use hardware if there were many traps while the target was not MWAITING, it's getting over-complicated, though :/
[toc] | [prev] | [next] | [standalone]
| From | Alexander Graf <agraf@suse.de> |
|---|---|
| Date | 2017-04-03 12:10 +0200 |
| Message-ID | <ts75T-5t7-5@gated-at.bofh.it> |
| In reply to | #1611881 |
On 03/29/2017 02:11 PM, Radim Krčmář wrote: > 2017-03-28 13:35-0700, Jim Mattson: >> On Tue, Mar 28, 2017 at 7:28 AM, Radim Krčmář <rkrcmar@redhat.com> wrote: >>> 2017-03-27 15:34+0200, Alexander Graf: >>>> On 15/03/2017 22:22, Michael S. Tsirkin wrote: >>>>> Guests running Mac OS 5, 6, and 7 (Leopard through Lion) have a problem: >>>>> unless explicitly provided with kernel command line argument >>>>> "idlehalt=0" they'd implicitly assume MONITOR and MWAIT availability, >>>>> without checking CPUID. >>>>> >>>>> We currently emulate that as a NOP but on VMX we can do better: let >>>>> guest stop the CPU until timer, IPI or memory change. CPU will be busy >>>>> but that isn't any worse than a NOP emulation. >>>>> >>>>> Note that mwait within guests is not the same as on real hardware >>>>> because halt causes an exit while mwait doesn't. For this reason it >>>>> might not be a good idea to use the regular MWAIT flag in CPUID to >>>>> signal this capability. Add a flag in the hypervisor leaf instead. >>>> So imagine we had proper MWAIT emulation capabilities based on page faults. >>>> In that case, we could do something as fancy as >>>> >>>> Treat MWAIT as pass-through by default >>>> >>>> Have a per-vcpu monitor timer 10 times a second in the background that >>>> checks which instruction we're in >>>> >>>> If we're in mwait for the last - say - 1 second, switch to emulated MWAIT, >>>> if $IP was in non-mwait within that time, reset counter. >>> Or we could reuse external interrupts for sampling. Exits trigerred by >>> them would check for current instruction (probably would be best to >>> limit just to timer tick) and a sufficient ratio (> 0?) of other exits >>> would imply that MWAIT is not used. >>> >>>> Or instead maybe just reuse the adapter hlt logic? >>> Emulated MWAIT is very similar to emulated HLT, so reusing the logic >>> makes sense. We would just add new wakeup methods. >>> >>>> Either way, with that we should be able to get super low latency IPIs >>>> running while still maintaining some sanity on systems which don't have >>>> dedicated CPUs for workloads. >>>> >>>> And we wouldn't need guest modifications, which is a great plus. So older >>>> guests (and Windows?) could benefit from mwait as well. >>> There is no need guest modifications -- it could be exposed as standard >>> MWAIT feature to the guest, with responsibilities for guest/host-impact >>> on the user. >>> >>> I think that the page-fault based MWAIT would require paravirt if it >>> should be enabled by default, because of performance concerns: >>> Enabling write protection on a page needs a VM exit on all other VCPUs >>> when beginning monitoring (to reload page permissions and prevent missed >>> writes). >>> We'd want to keep trapping writes to the page all the time because >>> toggling is slow, but this could regress performance for an OS that has >>> other data accessed by other VCPUs in that page. >>> No current interface can tell the guest that it should reserve the whole >>> page instead of what CPUID[5] says and that writes to the monitored page >>> are not "cheap", but can trigger a VM exit ... >> CPUID.05H:EBX is supposed to address the false sharing issue. IIRC, >> VMware Fusion reports 64 in CPUID.05H:EAX and 4096 in CPUID.05H:EBX >> when running Mac OS X guests. Per Intel's SDM volume 3, section >> 8.10.5, "To avoid false wake-ups; use the largest monitor line size to >> pad the data structure used to monitor writes. Software must make sure >> that beyond the data structure, no unrelated data variable exists in >> the triggering area for MWAIT. A pad may be needed to avoid this >> situation." Unfortunately, most operating systems do not follow this >> advice. > Right, EBX provides what we need to expose that the whole page is > monitored, thanks! So coming back to the original patch, is there anything that should keep us from exposing MWAIT straight into the guest at all times? Alex
[toc] | [prev] | [next] | [standalone]
| From | Radim Krčmář <rkrcmar@redhat.com> |
|---|---|
| Date | 2017-04-04 14:40 +0200 |
| Message-ID | <tsvUD-4Rz-27@gated-at.bofh.it> |
| In reply to | #1615051 |
2017-04-03 12:04+0200, Alexander Graf: > On 03/29/2017 02:11 PM, Radim Krčmář wrote: >> 2017-03-28 13:35-0700, Jim Mattson: >> > On Tue, Mar 28, 2017 at 7:28 AM, Radim Krčmář <rkrcmar@redhat.com> wrote: >> > > 2017-03-27 15:34+0200, Alexander Graf: >> > > > On 15/03/2017 22:22, Michael S. Tsirkin wrote: >> > > > > Guests running Mac OS 5, 6, and 7 (Leopard through Lion) have a problem: >> > > > > unless explicitly provided with kernel command line argument >> > > > > "idlehalt=0" they'd implicitly assume MONITOR and MWAIT availability, >> > > > > without checking CPUID. >> > > > > >> > > > > We currently emulate that as a NOP but on VMX we can do better: let >> > > > > guest stop the CPU until timer, IPI or memory change. CPU will be busy >> > > > > but that isn't any worse than a NOP emulation. >> > > > > >> > > > > Note that mwait within guests is not the same as on real hardware >> > > > > because halt causes an exit while mwait doesn't. For this reason it >> > > > > might not be a good idea to use the regular MWAIT flag in CPUID to >> > > > > signal this capability. Add a flag in the hypervisor leaf instead. >> > > > So imagine we had proper MWAIT emulation capabilities based on page faults. >> > > > In that case, we could do something as fancy as >> > > > >> > > > Treat MWAIT as pass-through by default >> > > > >> > > > Have a per-vcpu monitor timer 10 times a second in the background that >> > > > checks which instruction we're in >> > > > >> > > > If we're in mwait for the last - say - 1 second, switch to emulated MWAIT, >> > > > if $IP was in non-mwait within that time, reset counter. >> > > Or we could reuse external interrupts for sampling. Exits trigerred by >> > > them would check for current instruction (probably would be best to >> > > limit just to timer tick) and a sufficient ratio (> 0?) of other exits >> > > would imply that MWAIT is not used. >> > > >> > > > Or instead maybe just reuse the adapter hlt logic? >> > > Emulated MWAIT is very similar to emulated HLT, so reusing the logic >> > > makes sense. We would just add new wakeup methods. >> > > >> > > > Either way, with that we should be able to get super low latency IPIs >> > > > running while still maintaining some sanity on systems which don't have >> > > > dedicated CPUs for workloads. >> > > > >> > > > And we wouldn't need guest modifications, which is a great plus. So older >> > > > guests (and Windows?) could benefit from mwait as well. >> > > There is no need guest modifications -- it could be exposed as standard >> > > MWAIT feature to the guest, with responsibilities for guest/host-impact >> > > on the user. >> > > >> > > I think that the page-fault based MWAIT would require paravirt if it >> > > should be enabled by default, because of performance concerns: >> > > Enabling write protection on a page needs a VM exit on all other VCPUs >> > > when beginning monitoring (to reload page permissions and prevent missed >> > > writes). >> > > We'd want to keep trapping writes to the page all the time because >> > > toggling is slow, but this could regress performance for an OS that has >> > > other data accessed by other VCPUs in that page. >> > > No current interface can tell the guest that it should reserve the whole >> > > page instead of what CPUID[5] says and that writes to the monitored page >> > > are not "cheap", but can trigger a VM exit ... >> > CPUID.05H:EBX is supposed to address the false sharing issue. IIRC, >> > VMware Fusion reports 64 in CPUID.05H:EAX and 4096 in CPUID.05H:EBX >> > when running Mac OS X guests. Per Intel's SDM volume 3, section >> > 8.10.5, "To avoid false wake-ups; use the largest monitor line size to >> > pad the data structure used to monitor writes. Software must make sure >> > that beyond the data structure, no unrelated data variable exists in >> > the triggering area for MWAIT. A pad may be needed to avoid this >> > situation." Unfortunately, most operating systems do not follow this >> > advice. >> Right, EBX provides what we need to expose that the whole page is >> monitored, thanks! > > So coming back to the original patch, is there anything that should keep us > from exposing MWAIT straight into the guest at all times? Just minor issues: * OS X on Core 2 fails for unknown reason if we disable the instruction trapping, which is an argument against doing it by default * idling guests would consume host CPU, which is a significant change in behavior and shouldn't be done without userspace's involvement I think the best compromise is to add a capability for the MWAIT VM-exit controls and let userspace expose MWAIT if it wishes to. Will send a patch.
[toc] | [prev] | [next] | [standalone]
| From | Alexander Graf <agraf@suse.de> |
|---|---|
| Date | 2017-04-04 15:00 +0200 |
| Message-ID | <tswdX-50q-3@gated-at.bofh.it> |
| In reply to | #1615986 |
On 04/04/2017 02:39 PM, Radim Krčmář wrote: > 2017-04-03 12:04+0200, Alexander Graf: >> On 03/29/2017 02:11 PM, Radim Krčmář wrote: >>> 2017-03-28 13:35-0700, Jim Mattson: >>>> On Tue, Mar 28, 2017 at 7:28 AM, Radim Krčmář <rkrcmar@redhat.com> wrote: >>>>> 2017-03-27 15:34+0200, Alexander Graf: >>>>>> On 15/03/2017 22:22, Michael S. Tsirkin wrote: >>>>>>> Guests running Mac OS 5, 6, and 7 (Leopard through Lion) have a problem: >>>>>>> unless explicitly provided with kernel command line argument >>>>>>> "idlehalt=0" they'd implicitly assume MONITOR and MWAIT availability, >>>>>>> without checking CPUID. >>>>>>> >>>>>>> We currently emulate that as a NOP but on VMX we can do better: let >>>>>>> guest stop the CPU until timer, IPI or memory change. CPU will be busy >>>>>>> but that isn't any worse than a NOP emulation. >>>>>>> >>>>>>> Note that mwait within guests is not the same as on real hardware >>>>>>> because halt causes an exit while mwait doesn't. For this reason it >>>>>>> might not be a good idea to use the regular MWAIT flag in CPUID to >>>>>>> signal this capability. Add a flag in the hypervisor leaf instead. >>>>>> So imagine we had proper MWAIT emulation capabilities based on page faults. >>>>>> In that case, we could do something as fancy as >>>>>> >>>>>> Treat MWAIT as pass-through by default >>>>>> >>>>>> Have a per-vcpu monitor timer 10 times a second in the background that >>>>>> checks which instruction we're in >>>>>> >>>>>> If we're in mwait for the last - say - 1 second, switch to emulated MWAIT, >>>>>> if $IP was in non-mwait within that time, reset counter. >>>>> Or we could reuse external interrupts for sampling. Exits trigerred by >>>>> them would check for current instruction (probably would be best to >>>>> limit just to timer tick) and a sufficient ratio (> 0?) of other exits >>>>> would imply that MWAIT is not used. >>>>> >>>>>> Or instead maybe just reuse the adapter hlt logic? >>>>> Emulated MWAIT is very similar to emulated HLT, so reusing the logic >>>>> makes sense. We would just add new wakeup methods. >>>>> >>>>>> Either way, with that we should be able to get super low latency IPIs >>>>>> running while still maintaining some sanity on systems which don't have >>>>>> dedicated CPUs for workloads. >>>>>> >>>>>> And we wouldn't need guest modifications, which is a great plus. So older >>>>>> guests (and Windows?) could benefit from mwait as well. >>>>> There is no need guest modifications -- it could be exposed as standard >>>>> MWAIT feature to the guest, with responsibilities for guest/host-impact >>>>> on the user. >>>>> >>>>> I think that the page-fault based MWAIT would require paravirt if it >>>>> should be enabled by default, because of performance concerns: >>>>> Enabling write protection on a page needs a VM exit on all other VCPUs >>>>> when beginning monitoring (to reload page permissions and prevent missed >>>>> writes). >>>>> We'd want to keep trapping writes to the page all the time because >>>>> toggling is slow, but this could regress performance for an OS that has >>>>> other data accessed by other VCPUs in that page. >>>>> No current interface can tell the guest that it should reserve the whole >>>>> page instead of what CPUID[5] says and that writes to the monitored page >>>>> are not "cheap", but can trigger a VM exit ... >>>> CPUID.05H:EBX is supposed to address the false sharing issue. IIRC, >>>> VMware Fusion reports 64 in CPUID.05H:EAX and 4096 in CPUID.05H:EBX >>>> when running Mac OS X guests. Per Intel's SDM volume 3, section >>>> 8.10.5, "To avoid false wake-ups; use the largest monitor line size to >>>> pad the data structure used to monitor writes. Software must make sure >>>> that beyond the data structure, no unrelated data variable exists in >>>> the triggering area for MWAIT. A pad may be needed to avoid this >>>> situation." Unfortunately, most operating systems do not follow this >>>> advice. >>> Right, EBX provides what we need to expose that the whole page is >>> monitored, thanks! >> So coming back to the original patch, is there anything that should keep us >> from exposing MWAIT straight into the guest at all times? > Just minor issues: > * OS X on Core 2 fails for unknown reason if we disable the instruction > trapping, which is an argument against doing it by default So for that we should try and see if changing the exposed CPUID MWAIT leaf helps. Currently we return 0/0 which is pretty bogus and might be the reason OSX fails. > * idling guests would consume host CPU, which is a significant change > in behavior and shouldn't be done without userspace's involvement That's the same as today, as idling guests with MWAIT would also today end up in a NOP emulated loop. Please bear in mind that I do not advocate to expose the MWAIT CPUID flag. This is only for the instruction trap. > I think the best compromise is to add a capability for the MWAIT VM-exit > controls and let userspace expose MWAIT if it wishes to. > Will send a patch. Please see my patch to force enable CPUID bits ;). Alex
[toc] | [prev] | [next] | [standalone]
| From | Alexander Graf <agraf@suse.de> |
|---|---|
| Date | 2017-04-04 15:20 +0200 |
| Message-ID | <tswxj-5mj-3@gated-at.bofh.it> |
| In reply to | #1616000 |
On 04/04/2017 03:13 PM, Radim Krčmář wrote: > 2017-04-04 14:51+0200, Alexander Graf: >> On 04/04/2017 02:39 PM, Radim Krčmář wrote: >>> 2017-04-03 12:04+0200, Alexander Graf: >>>> So coming back to the original patch, is there anything that should keep us >>>> from exposing MWAIT straight into the guest at all times? >>> Just minor issues: >>> * OS X on Core 2 fails for unknown reason if we disable the instruction >>> trapping, which is an argument against doing it by default >> So for that we should try and see if changing the exposed CPUID MWAIT leaf >> helps. Currently we return 0/0 which is pretty bogus and might be the reason >> OSX fails. > We have tried to pass host's CPUID MWAIT leaf and it still failed: > https://www.spinics.net/lists/kvm/msg146686.html > > I wouldn't mind breaking that particular combination of OS X and > hardware, but I'm worried to do it because we don't understand why it > broke, so there could be more ... > >>> * idling guests would consume host CPU, which is a significant change >>> in behavior and shouldn't be done without userspace's involvement >> That's the same as today, as idling guests with MWAIT would also today end >> up in a NOP emulated loop. >> >> Please bear in mind that I do not advocate to expose the MWAIT CPUID flag. >> This is only for the instruction trap. > Ah, makes sense. > >>> I think the best compromise is to add a capability for the MWAIT VM-exit >>> controls and let userspace expose MWAIT if it wishes to. >>> Will send a patch. >> Please see my patch to force enable CPUID bits ;). > Nice. MWAIT could also use setting of arbitrary values for its leaf, > but a generic interface for that would probably look clunky on the > command line ... I think we should have an interface similar to smbios for that eventually. Something where you can explicitly set arbitrary CPUID leaf information using leaf specific syntax. There are more leafs where it would make sense - cache topology for example. Alex
[toc] | [prev] | [next] | [standalone]
| From | Radim Krčmář <rkrcmar@redhat.com> |
|---|---|
| Date | 2017-04-04 15:50 +0200 |
| Message-ID | <tsx0m-5yd-25@gated-at.bofh.it> |
| In reply to | #1616012 |
[Cc qemu-devel as we've gone off-topic] 2017-04-04 15:15+0200, Alexander Graf: > On 04/04/2017 03:13 PM, Radim Krčmář wrote: >> 2017-04-04 14:51+0200, Alexander Graf: >> > Please see my patch to force enable CPUID bits ;). >> Nice. MWAIT could also use setting of arbitrary values for its leaf, >> but a generic interface for that would probably look clunky on the >> command line ... > > > I think we should have an interface similar to smbios for that eventually. > Something where you can explicitly set arbitrary CPUID leaf information > using leaf specific syntax. There are more leafs where it would make sense - > cache topology for example. Right, separating cpuid from -cpu makes it bearable, like -cpuid leaf=%x[,subleaf=%x][,eax=%x][,ebx=%x][,ecx=%x][,edx=%x] And Having multiple interfaces for the same thing would result in some corner case decisions ... I think QEMU should check that feature flags specified flags specified by -cpu are not cleared by -cpuid. I'm not sure if setters like "|=" and "&=~" would be beneficial in some cases.
[toc] | [prev] | [next] | [standalone]
| From | Radim Krčmář <rkrcmar@redhat.com> |
|---|---|
| Date | 2017-04-04 15:20 +0200 |
| Message-ID | <tswxj-5mj-5@gated-at.bofh.it> |
| In reply to | #1616000 |
2017-04-04 14:51+0200, Alexander Graf: > On 04/04/2017 02:39 PM, Radim Krčmář wrote: >> 2017-04-03 12:04+0200, Alexander Graf: >> > So coming back to the original patch, is there anything that should keep us >> > from exposing MWAIT straight into the guest at all times? >> Just minor issues: >> * OS X on Core 2 fails for unknown reason if we disable the instruction >> trapping, which is an argument against doing it by default > > So for that we should try and see if changing the exposed CPUID MWAIT leaf > helps. Currently we return 0/0 which is pretty bogus and might be the reason > OSX fails. We have tried to pass host's CPUID MWAIT leaf and it still failed: https://www.spinics.net/lists/kvm/msg146686.html I wouldn't mind breaking that particular combination of OS X and hardware, but I'm worried to do it because we don't understand why it broke, so there could be more ... >> * idling guests would consume host CPU, which is a significant change >> in behavior and shouldn't be done without userspace's involvement > > That's the same as today, as idling guests with MWAIT would also today end > up in a NOP emulated loop. > > Please bear in mind that I do not advocate to expose the MWAIT CPUID flag. > This is only for the instruction trap. Ah, makes sense. >> I think the best compromise is to add a capability for the MWAIT VM-exit >> controls and let userspace expose MWAIT if it wishes to. >> Will send a patch. > > Please see my patch to force enable CPUID bits ;). Nice. MWAIT could also use setting of arbitrary values for its leaf, but a generic interface for that would probably look clunky on the command line ...
[toc] | [prev] | [standalone]
Page 3 of 3 — ← Prev page 1 2 [3]
Back to top | Article view | linux.kernel
csiph-web