Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]


Groups > linux.kernel > #1681007 > unrolled thread

Re: [PATCH 2/2] x86/idle: use dynamic halt poll

Started byRadim Krčmář <rkrcmar@redhat.com>
First post2017-07-04 16:20 +0200
Last post2017-07-17 15:00 +0200
Articles 7 — 4 participants

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.


Contents

  Re: [PATCH 2/2] x86/idle: use dynamic halt poll Radim Krčmář <rkrcmar@redhat.com> - 2017-07-04 16:20 +0200
    Re: [PATCH 2/2] x86/idle: use dynamic halt poll Thomas Gleixner <tglx@linutronix.de> - 2017-07-04 17:00 +0200
    Re: [PATCH 2/2] x86/idle: use dynamic halt poll Yang Zhang <yang.zhang.wz@gmail.com> - 2017-07-13 14:00 +0200
      Re: [PATCH 2/2] x86/idle: use dynamic halt poll Alexander Graf <agraf@suse.de> - 2017-07-14 11:40 +0200
        Re: [PATCH 2/2] x86/idle: use dynamic halt poll Yang Zhang <yang.zhang.wz@gmail.com> - 2017-07-17 11:30 +0200
          Re: [PATCH 2/2] x86/idle: use dynamic halt poll Alexander Graf <agraf@suse.de> - 2017-07-17 12:00 +0200
            Re: [PATCH 2/2] x86/idle: use dynamic halt poll Yang Zhang <yang.zhang.wz@gmail.com> - 2017-07-17 15:00 +0200

#1681007 — Re: [PATCH 2/2] x86/idle: use dynamic halt poll

FromRadim Krčmář <rkrcmar@redhat.com>
Date2017-07-04 16:20 +0200
SubjectRe: [PATCH 2/2] x86/idle: use dynamic halt poll
Message-ID<tZwQi-Ro-19@gated-at.bofh.it>
2017-07-03 17:28+0800, Yang Zhang:
> The background is that we(Alibaba Cloud) do get more and more complaints
> from our customers in both KVM and Xen compare to bare-mental.After
> investigations, the root cause is known to us: big cost in message passing
> workload(David show it in KVM forum 2015)
> 
> A typical message workload like below:
> vcpu 0                             vcpu 1
> 1. send ipi                     2.  doing hlt
> 3. go into idle                 4.  receive ipi and wake up from hlt
> 5. write APIC time twice        6.  write APIC time twice to
>    to stop sched timer              reprogram sched timer

One write is enough to disable/re-enable the APIC timer -- why does
Linux use two?

> 7. doing hlt                    8.  handle task and send ipi to
>                                     vcpu 0
> 9. same to 4.                   10. same to 3
> 
> One transaction will introduce about 12 vmexits(2 hlt and 10 msr write). The
> cost of such vmexits will degrades performance severely.

Yeah, sounds like too much ... I understood that there are

  IPI from 1 to 2
  4 * APIC timer
  IPI from 2 to 1

which adds to 6 MSR writes -- what are the other 4?

>                                                          Linux kernel
> already provide idle=poll to mitigate the trend. But it only eliminates the
> IPI and hlt vmexit. It has nothing to do with start/stop sched timer. A
> compromise would be to turn off NOHZ kernel, but it is not the default
> config for new distributions. Same for halt-poll in KVM, it only solve the
> cost from schedule in/out in host and can not help such workload much.
> 
> The purpose of this patch we want to improve current idle=poll mechanism to

Please aim to allow MWAIT instead of idle=poll -- MWAIT doesn't slow
down the sibling hyperthread.  MWAIT solves the IPI problem, but doesn't
get rid of the timer one.

> use dynamic polling and do poll before touch sched timer. It should not be a
> virtualization specific feature but seems bare mental have low cost to
> access the MSR. So i want to only enable it in VM. Though the idea below the
> patch may not so perfect to fit all conditions, it looks no worse than now.

It adds code to hot-paths (interrupt handlers) while trying to optimize
an idle-path, which is suspicious.

> How about we keep current implementation and i integrate the patch to
> para-virtualize part as Paolo suggested? We can continue discuss it and i
> will continue to refine it if anyone has better suggestions?

I think there is a nicer solution to avoid the expensive timer rewrite:
Linux uses one-shot APIC timers and getting the timer interrupt is about
as expensive as programming the timer, so the guest can keep the timer
armed, but not re-arm it after the expiration if the CPU is idle.

This should also mitigate the problem with short idle periods, but the
optimized window is anywhere between 0 to 1ms.

Do you see disadvantages of this combined with MWAIT?

Thanks.

[toc] | [next] | [standalone]


#1681029

FromThomas Gleixner <tglx@linutronix.de>
Date2017-07-04 17:00 +0200
Message-ID<tZxt0-1ij-11@gated-at.bofh.it>
In reply to#1681007

[Multipart message — attachments visible in raw view] — view raw

On Tue, 4 Jul 2017, Radim Krčmář wrote:
> I think there is a nicer solution to avoid the expensive timer rewrite:
> Linux uses one-shot APIC timers and getting the timer interrupt is about
> as expensive as programming the timer, so the guest can keep the timer
> armed, but not re-arm it after the expiration if the CPU is idle.

So much for the theory. The NOHZ logic has to reprogram the timer when the
next expiry is farther away than the next tick. Otherwise you wake up on
every idle entry once for nothing, which defeats the whole purpose of NOHZ
to not do that.

Thanks,

	tglx

[toc] | [prev] | [next] | [standalone]


#1686476

FromYang Zhang <yang.zhang.wz@gmail.com>
Date2017-07-13 14:00 +0200
Message-ID<u2KWK-58b-17@gated-at.bofh.it>
In reply to#1681007
On 2017/7/4 22:13, Radim Krčmář wrote:
> 2017-07-03 17:28+0800, Yang Zhang:
>> The background is that we(Alibaba Cloud) do get more and more complaints
>> from our customers in both KVM and Xen compare to bare-mental.After
>> investigations, the root cause is known to us: big cost in message passing
>> workload(David show it in KVM forum 2015)
>>
>> A typical message workload like below:
>> vcpu 0                             vcpu 1
>> 1. send ipi                     2.  doing hlt
>> 3. go into idle                 4.  receive ipi and wake up from hlt
>> 5. write APIC time twice        6.  write APIC time twice to
>>    to stop sched timer              reprogram sched timer
>
> One write is enough to disable/re-enable the APIC timer -- why does
> Linux use two?

One is to remove the timer and another one is to reprogram the timer. 
Normally, only one write to remove the timer.But in some cases, it will 
reprogram it.

>
>> 7. doing hlt                    8.  handle task and send ipi to
>>                                     vcpu 0
>> 9. same to 4.                   10. same to 3
>>
>> One transaction will introduce about 12 vmexits(2 hlt and 10 msr write). The
>> cost of such vmexits will degrades performance severely.
>
> Yeah, sounds like too much ... I understood that there are
>
>   IPI from 1 to 2
>   4 * APIC timer
>   IPI from 2 to 1
>
> which adds to 6 MSR writes -- what are the other 4?

In the worst case, each timer will touch APIC timer twice.So it will add 
additional 4 msr writse. But this is  not always true.

>
>>                                                          Linux kernel
>> already provide idle=poll to mitigate the trend. But it only eliminates the
>> IPI and hlt vmexit. It has nothing to do with start/stop sched timer. A
>> compromise would be to turn off NOHZ kernel, but it is not the default
>> config for new distributions. Same for halt-poll in KVM, it only solve the
>> cost from schedule in/out in host and can not help such workload much.
>>
>> The purpose of this patch we want to improve current idle=poll mechanism to
>
> Please aim to allow MWAIT instead of idle=poll -- MWAIT doesn't slow
> down the sibling hyperthread.  MWAIT solves the IPI problem, but doesn't
> get rid of the timer one.

Yes, i can try it. But MWAIT will not yield CPU, it only helps the 
sibling hyperthread as you mentioned.

>
>> use dynamic polling and do poll before touch sched timer. It should not be a
>> virtualization specific feature but seems bare mental have low cost to
>> access the MSR. So i want to only enable it in VM. Though the idea below the
>> patch may not so perfect to fit all conditions, it looks no worse than now.
>
> It adds code to hot-paths (interrupt handlers) while trying to optimize
> an idle-path, which is suspicious.
>
>> How about we keep current implementation and i integrate the patch to
>> para-virtualize part as Paolo suggested? We can continue discuss it and i
>> will continue to refine it if anyone has better suggestions?
>
> I think there is a nicer solution to avoid the expensive timer rewrite:
> Linux uses one-shot APIC timers and getting the timer interrupt is about
> as expensive as programming the timer, so the guest can keep the timer
> armed, but not re-arm it after the expiration if the CPU is idle.
>
> This should also mitigate the problem with short idle periods, but the
> optimized window is anywhere between 0 to 1ms.
>
> Do you see disadvantages of this combined with MWAIT?
>
> Thanks.
>


-- 
Yang
Alibaba Cloud Computing

[toc] | [prev] | [next] | [standalone]


#1687201

FromAlexander Graf <agraf@suse.de>
Date2017-07-14 11:40 +0200
Message-ID<u35eP-1tG-37@gated-at.bofh.it>
In reply to#1686476

On 13.07.17 13:49, Yang Zhang wrote:
> On 2017/7/4 22:13, Radim Krčmář wrote:
>> 2017-07-03 17:28+0800, Yang Zhang:
>>> The background is that we(Alibaba Cloud) do get more and more complaints
>>> from our customers in both KVM and Xen compare to bare-mental.After
>>> investigations, the root cause is known to us: big cost in message 
>>> passing
>>> workload(David show it in KVM forum 2015)
>>>
>>> A typical message workload like below:
>>> vcpu 0                             vcpu 1
>>> 1. send ipi                     2.  doing hlt
>>> 3. go into idle                 4.  receive ipi and wake up from hlt
>>> 5. write APIC time twice        6.  write APIC time twice to
>>>    to stop sched timer              reprogram sched timer
>>
>> One write is enough to disable/re-enable the APIC timer -- why does
>> Linux use two?
> 
> One is to remove the timer and another one is to reprogram the timer. 
> Normally, only one write to remove the timer.But in some cases, it will 
> reprogram it.
> 
>>
>>> 7. doing hlt                    8.  handle task and send ipi to
>>>                                     vcpu 0
>>> 9. same to 4.                   10. same to 3
>>>
>>> One transaction will introduce about 12 vmexits(2 hlt and 10 msr 
>>> write). The
>>> cost of such vmexits will degrades performance severely.
>>
>> Yeah, sounds like too much ... I understood that there are
>>
>>   IPI from 1 to 2
>>   4 * APIC timer
>>   IPI from 2 to 1
>>
>> which adds to 6 MSR writes -- what are the other 4?
> 
> In the worst case, each timer will touch APIC timer twice.So it will add 
> additional 4 msr writse. But this is  not always true.
> 
>>
>>>                                                          Linux kernel
>>> already provide idle=poll to mitigate the trend. But it only 
>>> eliminates the
>>> IPI and hlt vmexit. It has nothing to do with start/stop sched timer. A
>>> compromise would be to turn off NOHZ kernel, but it is not the default
>>> config for new distributions. Same for halt-poll in KVM, it only 
>>> solve the
>>> cost from schedule in/out in host and can not help such workload much.
>>>
>>> The purpose of this patch we want to improve current idle=poll 
>>> mechanism to
>>
>> Please aim to allow MWAIT instead of idle=poll -- MWAIT doesn't slow
>> down the sibling hyperthread.  MWAIT solves the IPI problem, but doesn't
>> get rid of the timer one.
> 
> Yes, i can try it. But MWAIT will not yield CPU, it only helps the 
> sibling hyperthread as you mentioned.

If you implement proper MWAIT emulation that conditionally gets en- or 
disabled depending on the same halt poll dynamics that we already have 
for in-host HLT handling, it will also yield the CPU.

As for the timer - are you sure the problem is really the overhead of 
the timer configuration, not the latency that it takes to actually fire 
the guest timer?

One major problem I see is that we configure the host hrtimer to fire at 
the point in time when the guest wants to see a timer event. But in a 
virtual environment, the point in time when we have to start switching 
to the VM really should be a bit *before* the guest wants to be woken 
up, as it takes quite some time to switch back into the VM context.


Alex

[toc] | [prev] | [next] | [standalone]


#1688813

FromYang Zhang <yang.zhang.wz@gmail.com>
Date2017-07-17 11:30 +0200
Message-ID<u4avM-3bA-9@gated-at.bofh.it>
In reply to#1687201
On 2017/7/14 17:37, Alexander Graf wrote:
>
>
> On 13.07.17 13:49, Yang Zhang wrote:
>> On 2017/7/4 22:13, Radim Krčmář wrote:
>>> 2017-07-03 17:28+0800, Yang Zhang:
>>>> The background is that we(Alibaba Cloud) do get more and more
>>>> complaints
>>>> from our customers in both KVM and Xen compare to bare-mental.After
>>>> investigations, the root cause is known to us: big cost in message
>>>> passing
>>>> workload(David show it in KVM forum 2015)
>>>>
>>>> A typical message workload like below:
>>>> vcpu 0                             vcpu 1
>>>> 1. send ipi                     2.  doing hlt
>>>> 3. go into idle                 4.  receive ipi and wake up from hlt
>>>> 5. write APIC time twice        6.  write APIC time twice to
>>>>    to stop sched timer              reprogram sched timer
>>>
>>> One write is enough to disable/re-enable the APIC timer -- why does
>>> Linux use two?
>>
>> One is to remove the timer and another one is to reprogram the timer.
>> Normally, only one write to remove the timer.But in some cases, it
>> will reprogram it.
>>
>>>
>>>> 7. doing hlt                    8.  handle task and send ipi to
>>>>                                     vcpu 0
>>>> 9. same to 4.                   10. same to 3
>>>>
>>>> One transaction will introduce about 12 vmexits(2 hlt and 10 msr
>>>> write). The
>>>> cost of such vmexits will degrades performance severely.
>>>
>>> Yeah, sounds like too much ... I understood that there are
>>>
>>>   IPI from 1 to 2
>>>   4 * APIC timer
>>>   IPI from 2 to 1
>>>
>>> which adds to 6 MSR writes -- what are the other 4?
>>
>> In the worst case, each timer will touch APIC timer twice.So it will
>> add additional 4 msr writse. But this is  not always true.
>>
>>>
>>>>                                                          Linux kernel
>>>> already provide idle=poll to mitigate the trend. But it only
>>>> eliminates the
>>>> IPI and hlt vmexit. It has nothing to do with start/stop sched timer. A
>>>> compromise would be to turn off NOHZ kernel, but it is not the default
>>>> config for new distributions. Same for halt-poll in KVM, it only
>>>> solve the
>>>> cost from schedule in/out in host and can not help such workload much.
>>>>
>>>> The purpose of this patch we want to improve current idle=poll
>>>> mechanism to
>>>
>>> Please aim to allow MWAIT instead of idle=poll -- MWAIT doesn't slow
>>> down the sibling hyperthread.  MWAIT solves the IPI problem, but doesn't
>>> get rid of the timer one.
>>
>> Yes, i can try it. But MWAIT will not yield CPU, it only helps the
>> sibling hyperthread as you mentioned.
>
> If you implement proper MWAIT emulation that conditionally gets en- or
> disabled depending on the same halt poll dynamics that we already have
> for in-host HLT handling, it will also yield the CPU.

It is hard to do . If we not intercept MWAIT instruction, there is no 
chance to wake up the CPU unless an interrupt arrived or a store to the 
address armed by MONITOR which is the same with idle=polling.

>
> As for the timer - are you sure the problem is really the overhead of
> the timer configuration, not the latency that it takes to actually fire
> the guest timer?

No, the main cost is introduced by vmexit, includes IPIs, Timer program, 
HLT. David detailed it in KVM forum, you can search "Message Passing 
Workloads in KVM" in google and the first link give the whole analysis 
of the problem.

>
> One major problem I see is that we configure the host hrtimer to fire at
> the point in time when the guest wants to see a timer event. But in a
> virtual environment, the point in time when we have to start switching
> to the VM really should be a bit *before* the guest wants to be woken
> up, as it takes quite some time to switch back into the VM context.
>
>
> Alex


-- 
Yang
Alibaba Cloud Computing

[toc] | [prev] | [next] | [standalone]


#1688839

FromAlexander Graf <agraf@suse.de>
Date2017-07-17 12:00 +0200
Message-ID<u4aYP-3lP-27@gated-at.bofh.it>
In reply to#1688813

On 17.07.17 11:26, Yang Zhang wrote:
> On 2017/7/14 17:37, Alexander Graf wrote:
>>
>>
>> On 13.07.17 13:49, Yang Zhang wrote:
>>> On 2017/7/4 22:13, Radim Krčmář wrote:
>>>> 2017-07-03 17:28+0800, Yang Zhang:
>>>>> The background is that we(Alibaba Cloud) do get more and more
>>>>> complaints
>>>>> from our customers in both KVM and Xen compare to bare-mental.After
>>>>> investigations, the root cause is known to us: big cost in message
>>>>> passing
>>>>> workload(David show it in KVM forum 2015)
>>>>>
>>>>> A typical message workload like below:
>>>>> vcpu 0                             vcpu 1
>>>>> 1. send ipi                     2.  doing hlt
>>>>> 3. go into idle                 4.  receive ipi and wake up from hlt
>>>>> 5. write APIC time twice        6.  write APIC time twice to
>>>>>    to stop sched timer              reprogram sched timer
>>>>
>>>> One write is enough to disable/re-enable the APIC timer -- why does
>>>> Linux use two?
>>>
>>> One is to remove the timer and another one is to reprogram the timer.
>>> Normally, only one write to remove the timer.But in some cases, it
>>> will reprogram it.
>>>
>>>>
>>>>> 7. doing hlt                    8.  handle task and send ipi to
>>>>>                                     vcpu 0
>>>>> 9. same to 4.                   10. same to 3
>>>>>
>>>>> One transaction will introduce about 12 vmexits(2 hlt and 10 msr
>>>>> write). The
>>>>> cost of such vmexits will degrades performance severely.
>>>>
>>>> Yeah, sounds like too much ... I understood that there are
>>>>
>>>>   IPI from 1 to 2
>>>>   4 * APIC timer
>>>>   IPI from 2 to 1
>>>>
>>>> which adds to 6 MSR writes -- what are the other 4?
>>>
>>> In the worst case, each timer will touch APIC timer twice.So it will
>>> add additional 4 msr writse. But this is  not always true.
>>>
>>>>
>>>>>                                                          Linux kernel
>>>>> already provide idle=poll to mitigate the trend. But it only
>>>>> eliminates the
>>>>> IPI and hlt vmexit. It has nothing to do with start/stop sched 
>>>>> timer. A
>>>>> compromise would be to turn off NOHZ kernel, but it is not the default
>>>>> config for new distributions. Same for halt-poll in KVM, it only
>>>>> solve the
>>>>> cost from schedule in/out in host and can not help such workload much.
>>>>>
>>>>> The purpose of this patch we want to improve current idle=poll
>>>>> mechanism to
>>>>
>>>> Please aim to allow MWAIT instead of idle=poll -- MWAIT doesn't slow
>>>> down the sibling hyperthread.  MWAIT solves the IPI problem, but 
>>>> doesn't
>>>> get rid of the timer one.
>>>
>>> Yes, i can try it. But MWAIT will not yield CPU, it only helps the
>>> sibling hyperthread as you mentioned.
>>
>> If you implement proper MWAIT emulation that conditionally gets en- or
>> disabled depending on the same halt poll dynamics that we already have
>> for in-host HLT handling, it will also yield the CPU.
> 
> It is hard to do . If we not intercept MWAIT instruction, there is no 
> chance to wake up the CPU unless an interrupt arrived or a store to the 
> address armed by MONITOR which is the same with idle=polling.

Yes, but you can reconfigure the VMCS/VMCB to trap on MWAIT or not trap 
on it. That's something that idle=polling does not give you at all - a 
guest vcpu will always use 100% CPU.

The only really tricky part is how to limit the effect of MONITOR on 
nested page table maintenance. But if we just set the MONITOR cache size 
to 4k, well behaved guests should ideally always give us the one same 
page for wakeup - which we can then leave marked as trapping.

> 
>>
>> As for the timer - are you sure the problem is really the overhead of
>> the timer configuration, not the latency that it takes to actually fire
>> the guest timer?
> 
> No, the main cost is introduced by vmexit, includes IPIs, Timer program, 
> HLT. David detailed it in KVM forum, you can search "Message Passing 
> Workloads in KVM" in google and the first link give the whole analysis 
> of the problem.

During time critical message passing you want to keep both vCPUs inside 
the guest, yes. That again is something that guest exposed MWAIT would 
buy you.

The problem is that overcommitting CPU is very expensive with anything 
that does not set the guests idle at all. And not everyone can afford to 
throw more CPUs at problems :).


Alex

[toc] | [prev] | [next] | [standalone]


#1689009

FromYang Zhang <yang.zhang.wz@gmail.com>
Date2017-07-17 15:00 +0200
Message-ID<u4dN0-57W-25@gated-at.bofh.it>
In reply to#1688839
On 2017/7/17 17:54, Alexander Graf wrote:
>
>
> On 17.07.17 11:26, Yang Zhang wrote:
>> On 2017/7/14 17:37, Alexander Graf wrote:
>>>
>>>
>>> On 13.07.17 13:49, Yang Zhang wrote:
>>>> On 2017/7/4 22:13, Radim Krčmář wrote:
>>>>> 2017-07-03 17:28+0800, Yang Zhang:
>>>>>> The background is that we(Alibaba Cloud) do get more and more
>>>>>> complaints
>>>>>> from our customers in both KVM and Xen compare to bare-mental.After
>>>>>> investigations, the root cause is known to us: big cost in message
>>>>>> passing
>>>>>> workload(David show it in KVM forum 2015)
>>>>>>
>>>>>> A typical message workload like below:
>>>>>> vcpu 0                             vcpu 1
>>>>>> 1. send ipi                     2.  doing hlt
>>>>>> 3. go into idle                 4.  receive ipi and wake up from hlt
>>>>>> 5. write APIC time twice        6.  write APIC time twice to
>>>>>>    to stop sched timer              reprogram sched timer
>>>>>
>>>>> One write is enough to disable/re-enable the APIC timer -- why does
>>>>> Linux use two?
>>>>
>>>> One is to remove the timer and another one is to reprogram the timer.
>>>> Normally, only one write to remove the timer.But in some cases, it
>>>> will reprogram it.
>>>>
>>>>>
>>>>>> 7. doing hlt                    8.  handle task and send ipi to
>>>>>>                                     vcpu 0
>>>>>> 9. same to 4.                   10. same to 3
>>>>>>
>>>>>> One transaction will introduce about 12 vmexits(2 hlt and 10 msr
>>>>>> write). The
>>>>>> cost of such vmexits will degrades performance severely.
>>>>>
>>>>> Yeah, sounds like too much ... I understood that there are
>>>>>
>>>>>   IPI from 1 to 2
>>>>>   4 * APIC timer
>>>>>   IPI from 2 to 1
>>>>>
>>>>> which adds to 6 MSR writes -- what are the other 4?
>>>>
>>>> In the worst case, each timer will touch APIC timer twice.So it will
>>>> add additional 4 msr writse. But this is  not always true.
>>>>
>>>>>
>>>>>>                                                          Linux kernel
>>>>>> already provide idle=poll to mitigate the trend. But it only
>>>>>> eliminates the
>>>>>> IPI and hlt vmexit. It has nothing to do with start/stop sched
>>>>>> timer. A
>>>>>> compromise would be to turn off NOHZ kernel, but it is not the
>>>>>> default
>>>>>> config for new distributions. Same for halt-poll in KVM, it only
>>>>>> solve the
>>>>>> cost from schedule in/out in host and can not help such workload
>>>>>> much.
>>>>>>
>>>>>> The purpose of this patch we want to improve current idle=poll
>>>>>> mechanism to
>>>>>
>>>>> Please aim to allow MWAIT instead of idle=poll -- MWAIT doesn't slow
>>>>> down the sibling hyperthread.  MWAIT solves the IPI problem, but
>>>>> doesn't
>>>>> get rid of the timer one.
>>>>
>>>> Yes, i can try it. But MWAIT will not yield CPU, it only helps the
>>>> sibling hyperthread as you mentioned.
>>>
>>> If you implement proper MWAIT emulation that conditionally gets en- or
>>> disabled depending on the same halt poll dynamics that we already have
>>> for in-host HLT handling, it will also yield the CPU.
>>
>> It is hard to do . If we not intercept MWAIT instruction, there is no
>> chance to wake up the CPU unless an interrupt arrived or a store to
>> the address armed by MONITOR which is the same with idle=polling.
>
> Yes, but you can reconfigure the VMCS/VMCB to trap on MWAIT or not trap
> on it. That's something that idle=polling does not give you at all - a
> guest vcpu will always use 100% CPU.

There are two things we need to figure out:
1. How and when to reconfigure the VMCS? Currently, all the knowledge 
are from guest, we don't know when to reconfigure it. Also, we cannot 
prevent guest from using MWAIT in other place if it see the feature.

2. If guest execute MWAIT without trap, since there is no way to set 
timeout for it, that would be a waste of CPU too.


>
> The only really tricky part is how to limit the effect of MONITOR on
> nested page table maintenance. But if we just set the MONITOR cache size
> to 4k, well behaved guests should ideally always give us the one same
> page for wakeup - which we can then leave marked as trapping.
>
>>
>>>
>>> As for the timer - are you sure the problem is really the overhead of
>>> the timer configuration, not the latency that it takes to actually fire
>>> the guest timer?
>>
>> No, the main cost is introduced by vmexit, includes IPIs, Timer
>> program, HLT. David detailed it in KVM forum, you can search "Message
>> Passing Workloads in KVM" in google and the first link give the whole
>> analysis of the problem.
>
> During time critical message passing you want to keep both vCPUs inside
> the guest, yes. That again is something that guest exposed MWAIT would
> buy you.

I think MWAIT only helps sibling hyper-threading case. But in real 
Cloud, hyper-threading is not always turning on, i.e. most products of 
Azure and some products of Alibaba Cloud. So it shouldn't be a big problem.

>
> The problem is that overcommitting CPU is very expensive with anything
> that does not set the guests idle at all. And not everyone can afford to
> throw more CPUs at problems :).

Agree, that's the reason why we choose dynamically halt polling. But on 
other side, the cloud vendor has the knowledge to control whether turn 
on it or not. The only problem is that there is no such way for us to do 
currently.

>
>
> Alex


-- 
Yang
Alibaba Cloud Computing

[toc] | [prev] | [standalone]


Back to top | Article view | linux.kernel


csiph-web