Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1722395 > unrolled thread

[RFC PATCH v2 0/7] x86/idle: add halt poll support

Started byYang Zhang <yang.zhang.wz@gmail.com>
First post2017-08-29 13:50 +0200
Last post2017-08-29 17:00 +0200
Articles 9 on this page of 29 — 11 participants

Back to article view | Back to linux.kernel

Contents

  [RFC PATCH v2 0/7] x86/idle: add halt poll support Yang Zhang <yang.zhang.wz@gmail.com> - 2017-08-29 13:50 +0200
    [RFC PATCH v2 3/7] sched/idle: Add poll before enter real idle path Yang Zhang <yang.zhang.wz@gmail.com> - 2017-08-29 13:50 +0200
      Re: [RFC PATCH v2 3/7] sched/idle: Add poll before enter real idle  path Peter Zijlstra <peterz@infradead.org> - 2017-08-29 14:50 +0200
        Re: [RFC PATCH v2 3/7] sched/idle: Add poll before enter real idle  path Quan Xu <quan.xu0@gmail.com> - 2017-09-01 08:00 +0200
      Re: [RFC PATCH v2 3/7] sched/idle: Add poll before enter real idle  path Borislav Petkov <bp@alien8.de> - 2017-08-29 16:40 +0200
        Re: [RFC PATCH v2 3/7] sched/idle: Add poll before enter real idle  path Quan Xu <quan.xu0@gmail.com> - 2017-09-01 09:00 +0200
    [RFC PATCH v2 1/7] x86/paravirt: Add pv_idle_ops to paravirt ops Yang Zhang <yang.zhang.wz@gmail.com> - 2017-08-29 13:50 +0200
      Re: [RFC PATCH v2 1/7] x86/paravirt: Add pv_idle_ops to paravirt ops Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> - 2017-08-29 16:10 +0200
        Re: [RFC PATCH v2 1/7] x86/paravirt: Add pv_idle_ops to paravirt ops Juergen Gross <jgross@suse.com> - 2017-08-30 09:40 +0200
        Re: [RFC PATCH v2 1/7] x86/paravirt: Add pv_idle_ops to paravirt ops Yang Zhang <yang.zhang.wz@gmail.com> - 2017-09-01 09:00 +0200
    [RFC PATCH v2 5/7] Documentation: Add three sysctls for smart idle poll Yang Zhang <yang.zhang.wz@gmail.com> - 2017-08-29 13:50 +0200
      Re: [RFC PATCH v2 5/7] Documentation: Add three sysctls for smart  idle poll "Luis R. Rodriguez" <mcgrof@kernel.org> - 2017-08-29 19:30 +0200
    [RFC PATCH v2 2/7] KVM guest: register kvm_idle_poll for pv_idle_ops Yang Zhang <yang.zhang.wz@gmail.com> - 2017-08-29 13:50 +0200
    [RFC PATCH v2 4/7] x86/paravirt: Add update in x86/paravirt pv_idle_ops Yang Zhang <yang.zhang.wz@gmail.com> - 2017-08-29 13:50 +0200
    [RFC PATCH v2 7/7] sched/idle: update poll time when wakeup from idle Yang Zhang <yang.zhang.wz@gmail.com> - 2017-08-29 13:50 +0200
      Re: [RFC PATCH v2 7/7] sched/idle: update poll time when wakeup from  idle Peter Zijlstra <peterz@infradead.org> - 2017-08-29 14:50 +0200
        Re: [RFC PATCH v2 7/7] sched/idle: update poll time when wakeup from  idle Yang Zhang <yang.zhang.wz@gmail.com> - 2017-09-01 09:40 +0200
    Re: [RFC PATCH v2 0/7] x86/idle: add halt poll support Alexander Graf <agraf@suse.de> - 2017-08-29 14:00 +0200
      Re: [RFC PATCH v2 0/7] x86/idle: add halt poll support Yang Zhang <yang.zhang.wz@gmail.com> - 2017-09-01 08:30 +0200
    Re: [RFC PATCH v2 0/7] x86/idle: add halt poll support Andi Kleen <andi@firstfloor.org> - 2017-08-29 15:10 +0200
    Re: [RFC PATCH v2 0/7] x86/idle: add halt poll support Wanpeng Li <kernellwp@gmail.com> - 2017-08-29 16:10 +0200
      Re: [RFC PATCH v2 0/7] x86/idle: add halt poll support Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> - 2017-08-29 16:30 +0200
      Re: [RFC PATCH v2 0/7] x86/idle: add halt poll support "Michael S. Tsirkin" <mst@redhat.com> - 2017-08-29 16:40 +0200
      Re: [RFC PATCH v2 0/7] x86/idle: add halt poll support Yang Zhang <yang.zhang.wz@gmail.com> - 2017-09-01 08:40 +0200
        Re: [RFC PATCH v2 0/7] x86/idle: add halt poll support Wanpeng Li <kernellwp@gmail.com> - 2017-09-01 09:00 +0200
      Re: [RFC PATCH v2 0/7] x86/idle: add halt poll support Yang Zhang <yang.zhang.wz@gmail.com> - 2017-09-01 08:50 +0200
        Re: [RFC PATCH v2 0/7] x86/idle: add halt poll support Wanpeng Li <kernellwp@gmail.com> - 2017-09-01 09:00 +0200
          Re: [RFC PATCH v2 0/7] x86/idle: add halt poll support Yang Zhang <yang.zhang.wz@gmail.com> - 2017-09-01 10:00 +0200
    Re: [RFC PATCH v2 0/7] x86/idle: add halt poll support "Michael S. Tsirkin" <mst@redhat.com> - 2017-08-29 17:00 +0200

Page 2 of 2 — ← Prev page 1 [2]

#1722517

FromWanpeng Li <kernellwp@gmail.com>
Date2017-08-29 16:10 +0200
Message-ID<ujPnk-3Ci-21@gated-at.bofh.it>
In reply to#1722395
> Here is the data we get when running benchmark netperf:
>
>    2. w/ patch:
>       halt_poll_threshold=10000 -- 15803.89 bits/s -- 159.5 %CPU
>       halt_poll_threshold=20000 -- 15899.04 bits/s -- 161.5 %CPU
>       halt_poll_threshold=30000 -- 15642.38 bits/s -- 161.8 %CPU
>       halt_poll_threshold=40000 -- 18040.76 bits/s -- 184.0 %CPU
>       halt_poll_threshold=50000 -- 18877.61 bits/s -- 197.3 %CPU
>
>    3. kvm dynamic poll
>       halt_poll_ns=10000 -- 15876.00 bits/s -- 172.2 %CPU
>       halt_poll_ns=20000 -- 15602.58 bits/s -- 185.4 %CPU
>       halt_poll_ns=30000 -- 15930.69 bits/s -- 194.4 %CPU
>       halt_poll_ns=40000 -- 16413.09 bits/s -- 195.3 %CPU
>       halt_poll_ns=50000 -- 16417.42 bits/s -- 196.3 %CPU
>

Actually I'm not sure how much sense it makes to introduce this pv
stuff and the duplicate adaptive halt-polling logic as what has
already been done in kvm w/o obvious benefit for real workload like
netperf. In addition, as you mentioned offline to me, enable both the
patchset and the adaptive halt-polling logic in kvm simultaneously can
result in more cpu power consumption. I remembered that David from
Google mentioned that Windows Event Objects can get 2x latency
improvement in KVM FORUM, which means that the adaptive halt-polling
in kvm should be enabled by default. So if the windows guests and
linux guests are mixed on the same host, then this patchset will
result in more cpu power consumption if the customer enable the
polling in the linux guest. Anyway, if the patchset is finally
acceptable by maintainer, I will introduce the generic adaptive
halt-polling framework in kvm to avoid the duplicate logic.

Regards,
Wanpeng Li

[toc] | [prev] | [next] | [standalone]

#1722534

FromKonrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Date2017-08-29 16:30 +0200
Message-ID<ujPGG-3J0-15@gated-at.bofh.it>
In reply to#1722517
On Tue, Aug 29, 2017 at 10:02:15PM +0800, Wanpeng Li wrote:
> > Here is the data we get when running benchmark netperf:
> >
> >    2. w/ patch:
> >       halt_poll_threshold=10000 -- 15803.89 bits/s -- 159.5 %CPU
> >       halt_poll_threshold=20000 -- 15899.04 bits/s -- 161.5 %CPU
> >       halt_poll_threshold=30000 -- 15642.38 bits/s -- 161.8 %CPU
> >       halt_poll_threshold=40000 -- 18040.76 bits/s -- 184.0 %CPU
> >       halt_poll_threshold=50000 -- 18877.61 bits/s -- 197.3 %CPU
> >
> >    3. kvm dynamic poll
> >       halt_poll_ns=10000 -- 15876.00 bits/s -- 172.2 %CPU
> >       halt_poll_ns=20000 -- 15602.58 bits/s -- 185.4 %CPU
> >       halt_poll_ns=30000 -- 15930.69 bits/s -- 194.4 %CPU
> >       halt_poll_ns=40000 -- 16413.09 bits/s -- 195.3 %CPU
> >       halt_poll_ns=50000 -- 16417.42 bits/s -- 196.3 %CPU
> >
> 
> Actually I'm not sure how much sense it makes to introduce this pv
> stuff and the duplicate adaptive halt-polling logic as what has
> already been done in kvm w/o obvious benefit for real workload like
> netperf. In addition, as you mentioned offline to me, enable both the

"real workload like netperf"? 

That is not a real workload. That is a synthetic one.

> patchset and the adaptive halt-polling logic in kvm simultaneously can
> result in more cpu power consumption. I remembered that David from
> Google mentioned that Windows Event Objects can get 2x latency
> improvement in KVM FORUM, which means that the adaptive halt-polling
> in kvm should be enabled by default. So if the windows guests and
> linux guests are mixed on the same host, then this patchset will
> result in more cpu power consumption if the customer enable the
> polling in the linux guest. Anyway, if the patchset is finally

More CPU power consumption sounds as a bad idea, does it not?

> acceptable by maintainer, I will introduce the generic adaptive
> halt-polling framework in kvm to avoid the duplicate logic.
> 
> Regards,
> Wanpeng Li

[toc] | [prev] | [next] | [standalone]

#1722540

From"Michael S. Tsirkin" <mst@redhat.com>
Date2017-08-29 16:40 +0200
Message-ID<ujPQl-3Mg-3@gated-at.bofh.it>
In reply to#1722517
On Tue, Aug 29, 2017 at 10:02:15PM +0800, Wanpeng Li wrote:
> Actually I'm not sure how much sense it makes to introduce this pv
> stuff and the duplicate adaptive halt-polling logic as what has
> already been done in kvm w/o obvious benefit for real workload like
> netperf.

In fact, I would really like to better understand why does the polling
in kvm even help.  Switching to the idle task is supposed to be really
cheap as you are not losing context.  In case of e.g. network polling
you gain the interrupt latency, but in case of kvm it's just an IPI
which is converted to a memory write when using mwait. Is mwait more
costly than commonly thought? Or is the idle driver too agressive in
putting the CPU into deep sleep?

I think this analysis is something that would benefit
bare-metal/containers as well.

-- 
MST

[toc] | [prev] | [next] | [standalone]

#1724804

FromYang Zhang <yang.zhang.wz@gmail.com>
Date2017-09-01 08:40 +0200
Message-ID<ukNMt-5l-7@gated-at.bofh.it>
In reply to#1722517
On 2017/8/29 22:02, Wanpeng Li wrote:
>> Here is the data we get when running benchmark netperf:
>>
>>     2. w/ patch:
>>        halt_poll_threshold=10000 -- 15803.89 bits/s -- 159.5 %CPU
>>        halt_poll_threshold=20000 -- 15899.04 bits/s -- 161.5 %CPU
>>        halt_poll_threshold=30000 -- 15642.38 bits/s -- 161.8 %CPU
>>        halt_poll_threshold=40000 -- 18040.76 bits/s -- 184.0 %CPU
>>        halt_poll_threshold=50000 -- 18877.61 bits/s -- 197.3 %CPU
>>
>>     3. kvm dynamic poll
>>        halt_poll_ns=10000 -- 15876.00 bits/s -- 172.2 %CPU
>>        halt_poll_ns=20000 -- 15602.58 bits/s -- 185.4 %CPU
>>        halt_poll_ns=30000 -- 15930.69 bits/s -- 194.4 %CPU
>>        halt_poll_ns=40000 -- 16413.09 bits/s -- 195.3 %CPU
>>        halt_poll_ns=50000 -- 16417.42 bits/s -- 196.3 %CPU
>>
> 
> Actually I'm not sure how much sense it makes to introduce this pv
> stuff and the duplicate adaptive halt-polling logic as what has
> already been done in kvm w/o obvious benefit for real workload like > netperf. In addition, as you mentioned offline to me, enable both the
> patchset and the adaptive halt-polling logic in kvm simultaneously can
> result in more cpu power consumption. I remembered that David from

If we use poll in KVM side, it will consume more cpu than in guest side. 
If use both two, then we can get the same performance as only enable 
guest side poll but it will cost more cpu because of poll KVM side. It 
means we should disable KVM side poll since it cannot give much 
improvement than  guest side except consume more cpu and large latency.

> Google mentioned that Windows Event Objects can get 2x latency
> improvement in KVM FORUM, which means that the adaptive halt-polling
> in kvm should be enabled by default. So if the windows guests and
> linux guests are mixed on the same host, then this patchset will
> result in more cpu power consumption if the customer enable the
> polling in the linux guest. Anyway, if the patchset is finally
> acceptable by maintainer, I will introduce the generic adaptive
> halt-polling framework in kvm to avoid the duplicate logic.

We will add more conditions than the current algorithm in future. But 
it's ok to use the one copy currently, we will do it in next version.


-- 
Yang
Alibaba Cloud Computing

[toc] | [prev] | [next] | [standalone]

#1724812

FromWanpeng Li <kernellwp@gmail.com>
Date2017-09-01 09:00 +0200
Message-ID<ukO5P-dZ-5@gated-at.bofh.it>
In reply to#1724804
2017-09-01 14:32 GMT+08:00 Yang Zhang <yang.zhang.wz@gmail.com>:
> On 2017/8/29 22:02, Wanpeng Li wrote:
>>>
>>> Here is the data we get when running benchmark netperf:
>>>
>>>     2. w/ patch:
>>>        halt_poll_threshold=10000 -- 15803.89 bits/s -- 159.5 %CPU
>>>        halt_poll_threshold=20000 -- 15899.04 bits/s -- 161.5 %CPU
>>>        halt_poll_threshold=30000 -- 15642.38 bits/s -- 161.8 %CPU
>>>        halt_poll_threshold=40000 -- 18040.76 bits/s -- 184.0 %CPU
>>>        halt_poll_threshold=50000 -- 18877.61 bits/s -- 197.3 %CPU
>>>
>>>     3. kvm dynamic poll
>>>        halt_poll_ns=10000 -- 15876.00 bits/s -- 172.2 %CPU
>>>        halt_poll_ns=20000 -- 15602.58 bits/s -- 185.4 %CPU
>>>        halt_poll_ns=30000 -- 15930.69 bits/s -- 194.4 %CPU
>>>        halt_poll_ns=40000 -- 16413.09 bits/s -- 195.3 %CPU
>>>        halt_poll_ns=50000 -- 16417.42 bits/s -- 196.3 %CPU
>>>
>>
>> Actually I'm not sure how much sense it makes to introduce this pv
>> stuff and the duplicate adaptive halt-polling logic as what has
>> already been done in kvm w/o obvious benefit for real workload like >
>> netperf. In addition, as you mentioned offline to me, enable both the
>> patchset and the adaptive halt-polling logic in kvm simultaneously can
>> result in more cpu power consumption. I remembered that David from
>
>
> If we use poll in KVM side, it will consume more cpu than in guest side. If
> use both two, then we can get the same performance as only enable guest side
> poll but it will cost more cpu because of poll KVM side. It means we should
> disable KVM side poll since it cannot give much improvement than  guest side
> except consume more cpu and large latency.

How can message passing workloads in windows guest survive?

Regards,
Wanpeng Li

>
>> Google mentioned that Windows Event Objects can get 2x latency
>> improvement in KVM FORUM, which means that the adaptive halt-polling
>> in kvm should be enabled by default. So if the windows guests and
>> linux guests are mixed on the same host, then this patchset will
>> result in more cpu power consumption if the customer enable the
>> polling in the linux guest. Anyway, if the patchset is finally
>> acceptable by maintainer, I will introduce the generic adaptive
>> halt-polling framework in kvm to avoid the duplicate logic.
>
>
> We will add more conditions than the current algorithm in future. But it's
> ok to use the one copy currently, we will do it in next version.
>
>
>
> --
> Yang
> Alibaba Cloud Computing

[toc] | [prev] | [next] | [standalone]

#1724806

FromYang Zhang <yang.zhang.wz@gmail.com>
Date2017-09-01 08:50 +0200
Message-ID<ukNWa-9F-7@gated-at.bofh.it>
In reply to#1722517
On 2017/8/29 22:02, Wanpeng Li wrote:
>> Here is the data we get when running benchmark netperf:
>>
>>     2. w/ patch:
>>        halt_poll_threshold=10000 -- 15803.89 bits/s -- 159.5 %CPU
>>        halt_poll_threshold=20000 -- 15899.04 bits/s -- 161.5 %CPU
>>        halt_poll_threshold=30000 -- 15642.38 bits/s -- 161.8 %CPU
>>        halt_poll_threshold=40000 -- 18040.76 bits/s -- 184.0 %CPU
>>        halt_poll_threshold=50000 -- 18877.61 bits/s -- 197.3 %CPU
>>
>>     3. kvm dynamic poll
>>        halt_poll_ns=10000 -- 15876.00 bits/s -- 172.2 %CPU
>>        halt_poll_ns=20000 -- 15602.58 bits/s -- 185.4 %CPU
>>        halt_poll_ns=30000 -- 15930.69 bits/s -- 194.4 %CPU
>>        halt_poll_ns=40000 -- 16413.09 bits/s -- 195.3 %CPU
>>        halt_poll_ns=50000 -- 16417.42 bits/s -- 196.3 %CPU
>>
> 
> Actually I'm not sure how much sense it makes to introduce this pv
> stuff and the duplicate adaptive halt-polling logic as what has
> already been done in kvm w/o obvious benefit for real workload like
> netperf. In addition, as you mentioned offline to me, enable both the
> patchset and the adaptive halt-polling logic in kvm simultaneously can
> result in more cpu power consumption. I remembered that David from

No.If we use poll in KVM side, it will consume more cpu than in guest 
side. If use both two, then we can get the performance as only enable 
guest side poll but it will cost more cpu because of poll in KVM side. 
It means we should disable KVM side poll since it cannot give much 
improvement than guest side except consume more cpu.

> Google mentioned that Windows Event Objects can get 2x latency
> improvement in KVM FORUM, which means that the adaptive halt-polling
> in kvm should be enabled by default. So if the windows guests and
> linux guests are mixed on the same host, then this patchset will
> result in more cpu power consumption if the customer enable the
> polling in the linux guest. Anyway, if the patchset is finally

I have said in last time, there already users using idle=poll in there 
VM, you *cannot* prevent them doing it. This patch provide a better 
solution than unconditional poll, we didn't introduce any worse stuff.

> acceptable by maintainer, I will introduce the generic adaptive
> halt-polling framework in kvm to avoid the duplicate logic.

We will add more conditions than the current algorithm in future. But 
it's ok to use one code currently, we will do it in next version.

> 
> Regards,
> Wanpeng Li
> 


-- 
Yang
Alibaba Cloud Computing

[toc] | [prev] | [next] | [standalone]

#1724814

FromWanpeng Li <kernellwp@gmail.com>
Date2017-09-01 09:00 +0200
Message-ID<ukO5P-dZ-11@gated-at.bofh.it>
In reply to#1724806
2017-09-01 14:44 GMT+08:00 Yang Zhang <yang.zhang.wz@gmail.com>:
> On 2017/8/29 22:02, Wanpeng Li wrote:
>>>
>>> Here is the data we get when running benchmark netperf:
>>>
>>>     2. w/ patch:
>>>        halt_poll_threshold=10000 -- 15803.89 bits/s -- 159.5 %CPU
>>>        halt_poll_threshold=20000 -- 15899.04 bits/s -- 161.5 %CPU
>>>        halt_poll_threshold=30000 -- 15642.38 bits/s -- 161.8 %CPU
>>>        halt_poll_threshold=40000 -- 18040.76 bits/s -- 184.0 %CPU
>>>        halt_poll_threshold=50000 -- 18877.61 bits/s -- 197.3 %CPU
>>>
>>>     3. kvm dynamic poll
>>>        halt_poll_ns=10000 -- 15876.00 bits/s -- 172.2 %CPU
>>>        halt_poll_ns=20000 -- 15602.58 bits/s -- 185.4 %CPU
>>>        halt_poll_ns=30000 -- 15930.69 bits/s -- 194.4 %CPU
>>>        halt_poll_ns=40000 -- 16413.09 bits/s -- 195.3 %CPU
>>>        halt_poll_ns=50000 -- 16417.42 bits/s -- 196.3 %CPU
>>>
>>
>> Actually I'm not sure how much sense it makes to introduce this pv
>> stuff and the duplicate adaptive halt-polling logic as what has
>> already been done in kvm w/o obvious benefit for real workload like
>> netperf. In addition, as you mentioned offline to me, enable both the
>> patchset and the adaptive halt-polling logic in kvm simultaneously can
>> result in more cpu power consumption. I remembered that David from
>
>
> No.If we use poll in KVM side, it will consume more cpu than in guest side.
> If use both two, then we can get the performance as only enable guest side
> poll but it will cost more cpu because of poll in KVM side. It means we
> should disable KVM side poll since it cannot give much improvement than
> guest side except consume more cpu.

The customers should have enough knowledge about what's the meaning of
the tunning which you exposed.

Regards,
Wanpeng Li

>
>> Google mentioned that Windows Event Objects can get 2x latency
>> improvement in KVM FORUM, which means that the adaptive halt-polling
>> in kvm should be enabled by default. So if the windows guests and
>> linux guests are mixed on the same host, then this patchset will
>> result in more cpu power consumption if the customer enable the
>> polling in the linux guest. Anyway, if the patchset is finally
>
>
> I have said in last time, there already users using idle=poll in there VM,
> you *cannot* prevent them doing it. This patch provide a better solution
> than unconditional poll, we didn't introduce any worse stuff.
>
>> acceptable by maintainer, I will introduce the generic adaptive
>> halt-polling framework in kvm to avoid the duplicate logic.
>
>
> We will add more conditions than the current algorithm in future. But it's
> ok to use one code currently, we will do it in next version.
>
>>
>> Regards,
>> Wanpeng Li
>>
>
>
> --
> Yang
> Alibaba Cloud Computing

[toc] | [prev] | [next] | [standalone]

#1724851

FromYang Zhang <yang.zhang.wz@gmail.com>
Date2017-09-01 10:00 +0200
Message-ID<ukP1V-TX-21@gated-at.bofh.it>
In reply to#1724814
On 2017/9/1 14:58, Wanpeng Li wrote:
> 2017-09-01 14:44 GMT+08:00 Yang Zhang <yang.zhang.wz@gmail.com>:
>> On 2017/8/29 22:02, Wanpeng Li wrote:
>>>>
>>>> Here is the data we get when running benchmark netperf:
>>>>
>>>>      2. w/ patch:
>>>>         halt_poll_threshold=10000 -- 15803.89 bits/s -- 159.5 %CPU
>>>>         halt_poll_threshold=20000 -- 15899.04 bits/s -- 161.5 %CPU
>>>>         halt_poll_threshold=30000 -- 15642.38 bits/s -- 161.8 %CPU
>>>>         halt_poll_threshold=40000 -- 18040.76 bits/s -- 184.0 %CPU
>>>>         halt_poll_threshold=50000 -- 18877.61 bits/s -- 197.3 %CPU
>>>>
>>>>      3. kvm dynamic poll
>>>>         halt_poll_ns=10000 -- 15876.00 bits/s -- 172.2 %CPU
>>>>         halt_poll_ns=20000 -- 15602.58 bits/s -- 185.4 %CPU
>>>>         halt_poll_ns=30000 -- 15930.69 bits/s -- 194.4 %CPU
>>>>         halt_poll_ns=40000 -- 16413.09 bits/s -- 195.3 %CPU
>>>>         halt_poll_ns=50000 -- 16417.42 bits/s -- 196.3 %CPU
>>>>
>>>
>>> Actually I'm not sure how much sense it makes to introduce this pv
>>> stuff and the duplicate adaptive halt-polling logic as what has
>>> already been done in kvm w/o obvious benefit for real workload like
>>> netperf. In addition, as you mentioned offline to me, enable both the
>>> patchset and the adaptive halt-polling logic in kvm simultaneously can
>>> result in more cpu power consumption. I remembered that David from
>>
>>
>> No.If we use poll in KVM side, it will consume more cpu than in guest side.
>> If use both two, then we can get the performance as only enable guest side
>> poll but it will cost more cpu because of poll in KVM side. It means we
>> should disable KVM side poll since it cannot give much improvement than
>> guest side except consume more cpu.
> 
> The customers should have enough knowledge about what's the meaning of
> the tunning which you exposed.

We have applied this patch to customize kernel for some real customers 
and we get positive feedback from them since the CPU never run at 100% 
even there is no task running. Also, this helps them to give more CPUs 
to other tasks and reduce the power consumption in their rack. Don't you 
think it is better?

> 
> Regards,
> Wanpeng Li
> 
>>
>>> Google mentioned that Windows Event Objects can get 2x latency
>>> improvement in KVM FORUM, which means that the adaptive halt-polling
>>> in kvm should be enabled by default. So if the windows guests and
>>> linux guests are mixed on the same host, then this patchset will
>>> result in more cpu power consumption if the customer enable the
>>> polling in the linux guest. Anyway, if the patchset is finally
>>
>>
>> I have said in last time, there already users using idle=poll in there VM,
>> you *cannot* prevent them doing it. This patch provide a better solution
>> than unconditional poll, we didn't introduce any worse stuff.
>>
>>> acceptable by maintainer, I will introduce the generic adaptive
>>> halt-polling framework in kvm to avoid the duplicate logic.
>>
>>
>> We will add more conditions than the current algorithm in future. But it's
>> ok to use one code currently, we will do it in next version.
>>
>>>
>>> Regards,
>>> Wanpeng Li
>>>
>>
>>
>> --
>> Yang
>> Alibaba Cloud Computing


-- 
Yang
Alibaba Cloud Computing

[toc] | [prev] | [next] | [standalone]

#1722553

From"Michael S. Tsirkin" <mst@redhat.com>
Date2017-08-29 17:00 +0200
Message-ID<ujQ9I-3UP-13@gated-at.bofh.it>
In reply to#1722395
On Tue, Aug 29, 2017 at 11:46:34AM +0000, Yang Zhang wrote:
> Some latency-intensive workload will see obviously performance 
> drop when running inside VM.

But are we trading a lot of CPU for a bit of lower latency?

> The main reason is that the overhead 
> is amplified when running inside VM. The most cost i have seen is 
> inside idle path. 
> 
> This patch introduces a new mechanism to poll for a while before 
> entering idle state. If schedule is needed during poll, then we 
> don't need to goes through the heavy overhead path. 

Isn't it the job of an idle driver to find the best way to
halt the CPU?

It looks like just by adding a cstate we can make it
halt at higher latencies only. And at lower latencies,
if it's doing a good job we can hopefully use mwait to
stop the CPU.

In fact I have been experimenting with exactly that.
Some initial results are encouraging but I could use help
with testing and especially tuning. If you can help
pls let me know!

Patch below is not intended for upstream - it's just
the fastest way I found to test things.
So it just uses command line arguments to configure the guest,
the right thing is through a combination of ACPI and CPUIDs
but let's decide whether we need this first.

RFC dontmerge PATCH intel_idle: add pv cstates when running on kvm

Usage:

kvm_pv_mwait - enables the feature. Note: you must have a recent
		host that allows guests to execute mwait without an exit,
		otherwise you will just get 100% CPU.

kvm_halt_target_residency - halt above this target residency.
		Should probably be a function of the cost of
		halt+wakeup.

kvm_halt_native - set to 0 if your VCPU does not match host.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>

---

diff --git a/drivers/idle/intel_idle.c b/drivers/idle/intel_idle.c
index c2ae819..6fa58ad 100644
--- a/drivers/idle/intel_idle.c
+++ b/drivers/idle/intel_idle.c
@@ -65,8 +65,10 @@
 #include <asm/intel-family.h>
 #include <asm/mwait.h>
 #include <asm/msr.h>
+#include <linux/kvm_para.h>
 
 #define INTEL_IDLE_VERSION "0.4.1"
+#define PREFIX "intel_idle: "
 
 static struct cpuidle_driver intel_idle_driver = {
 	.name = "intel_idle",
@@ -94,6 +96,7 @@ struct idle_cpu {
 };
 
 static const struct idle_cpu *icpu;
+static struct idle_cpu icpus;
 static struct cpuidle_device __percpu *intel_idle_cpuidle_devices;
 static int intel_idle(struct cpuidle_device *dev,
 			struct cpuidle_driver *drv, int index);
@@ -119,6 +122,49 @@ static struct cpuidle_state *cpuidle_state_table;
 #define flg2MWAIT(flags) (((flags) >> 24) & 0xFF)
 #define MWAIT2flg(eax) ((eax & 0xFF) << 24)
 
+static int intel_halt(struct cpuidle_device *dev,
+			struct cpuidle_driver *drv, int index)
+{
+	printk_once(KERN_ERR "safe_halt started\n");
+	safe_halt();
+	printk_once(KERN_ERR "safe_halt done\n");
+	return index;
+}
+
+static int kvm_halt_target_residency = 400; /* Halt above this target residency */
+module_param(kvm_halt_target_residency, int, 0444);
+static int kvm_halt_native = 1; /* Use native mwait substates */
+module_param(kvm_halt_native, int, 0444);
+static int kvm_pv_mwait = 0; /* Whether to do mwait within KVM */
+module_param(kvm_pv_mwait, int, 0444);
+
+static struct cpuidle_state kvm_halt_cstate = {
+	.name = "HALT-KVM",
+	.desc = "HALT",
+	.flags = MWAIT2flg(0x10),
+	.exit_latency = 0,
+	.target_residency = 0,
+	.enter = &intel_halt,
+};
+
+static struct cpuidle_state kvm_cstates[] = {
+	{
+		.name = "C1-NHM",
+		.desc = "MWAIT 0x00",
+		.flags = MWAIT2flg(0x00),
+		.exit_latency = 3,
+		.target_residency = 6,
+		.enter = &intel_idle,
+		.enter_freeze = intel_idle_freeze, },
+	{
+		.name = "HALT-KVM",
+		.desc = "HALT",
+		.flags = MWAIT2flg(0x10),
+		.exit_latency = 30,
+		.target_residency = 399,
+		.enter = &intel_halt, }
+};
+
 /*
  * States are indexed by the cstate number,
  * which is also the index into the MWAIT hint array.
@@ -927,8 +973,11 @@ static __cpuidle int intel_idle(struct cpuidle_device *dev,
 	if (!(lapic_timer_reliable_states & (1 << (cstate))))
 		tick_broadcast_enter();
 
+	printk_once(KERN_ERR "mwait_idle_with_hints started\n");
 	mwait_idle_with_hints(eax, ecx);
 
+	printk_once(KERN_ERR "mwait_idle_with_hints done\n");
+
 	if (!(lapic_timer_reliable_states & (1 << (cstate))))
 		tick_broadcast_exit();
 
@@ -989,6 +1038,10 @@ static const struct idle_cpu idle_cpu_tangier = {
 	.state_table = tangier_cstates,
 };
 
+static const struct idle_cpu idle_cpu_kvm = {
+	.state_table = kvm_cstates,
+};
+
 static const struct idle_cpu idle_cpu_lincroft = {
 	.state_table = atom_cstates,
 	.auto_demotion_disable_flags = ATM_LNC_C6_AUTO_DEMOTE,
@@ -1061,7 +1115,7 @@ static const struct idle_cpu idle_cpu_dnv = {
 };
 
 #define ICPU(model, cpu) \
-	{ X86_VENDOR_INTEL, 6, model, X86_FEATURE_MWAIT, (unsigned long)&cpu }
+	{ X86_VENDOR_INTEL, 6, model, X86_FEATURE_ANY, (unsigned long)&cpu }
 
 static const struct x86_cpu_id intel_idle_ids[] __initconst = {
 	ICPU(INTEL_FAM6_NEHALEM_EP,		idle_cpu_nehalem),
@@ -1125,19 +1180,39 @@ static int __init intel_idle_probe(void)
 		return -ENODEV;
 	}
 
-	if (boot_cpu_data.cpuid_level < CPUID_MWAIT_LEAF)
-		return -ENODEV;
+	icpus = *(struct idle_cpu *)id->driver_data;
+
+	if (kvm_pv_mwait) {
+
+		if (!kvm_halt_native)
+			icpus = idle_cpu_kvm;
+
+		pr_debug(PREFIX "MWAIT enabled by KVM\n");
+		mwait_substates = 0x1;
+		/*
+		 * these MSRs do not work on kvm maybe they should?
+		 * more likely we need to poke at CPUID before using MSRs
+		 */
+		icpus.auto_demotion_disable_flags = 0;
+		icpus.disable_promotion_to_c1e = 0;
+	} else {
+		if (!cpu_has(&boot_cpu_data, X86_FEATURE_MWAIT))
+			return -ENODEV;
+
+		if (boot_cpu_data.cpuid_level < CPUID_MWAIT_LEAF)
+			return -ENODEV;
 
-	cpuid(CPUID_MWAIT_LEAF, &eax, &ebx, &ecx, &mwait_substates);
+		cpuid(CPUID_MWAIT_LEAF, &eax, &ebx, &ecx, &mwait_substates);
 
-	if (!(ecx & CPUID5_ECX_EXTENSIONS_SUPPORTED) ||
-	    !(ecx & CPUID5_ECX_INTERRUPT_BREAK) ||
-	    !mwait_substates)
+		if (!(ecx & CPUID5_ECX_EXTENSIONS_SUPPORTED) ||
+		    !(ecx & CPUID5_ECX_INTERRUPT_BREAK) ||
+		    !mwait_substates)
 			return -ENODEV;
 
-	pr_debug("MWAIT substates: 0x%x\n", mwait_substates);
+		pr_debug(PREFIX "MWAIT substates: 0x%x\n", mwait_substates);
+	}
 
-	icpu = (const struct idle_cpu *)id->driver_data;
+	icpu = &icpus;
 	cpuidle_state_table = icpu->state_table;
 
 	pr_debug("v" INTEL_IDLE_VERSION " model 0x%X\n",
@@ -1340,6 +1415,11 @@ static void __init intel_idle_cpuidle_driver_init(void)
 		    (cpuidle_state_table[cstate].enter_freeze == NULL))
 			break;
 
+		if (kvm_pv_mwait &&
+		    cpuidle_state_table[cstate].target_residency >=
+		    kvm_halt_target_residency)
+			break;
+
 		if (cstate + 1 > max_cstate) {
 			pr_info("max_cstate %d reached\n", max_cstate);
 			break;
@@ -1353,7 +1433,7 @@ static void __init intel_idle_cpuidle_driver_init(void)
 					& MWAIT_SUBSTATE_MASK;
 
 		/* if NO sub-states for this state in CPUID, skip it */
-		if (num_substates == 0)
+		if (num_substates == 0 && !kvm_pv_mwait)
 			continue;
 
 		/* if state marked as disabled, skip it */
@@ -1375,6 +1455,20 @@ static void __init intel_idle_cpuidle_driver_init(void)
 		drv->state_count += 1;
 	}
 
+	if (kvm_halt_native && kvm_pv_mwait) {
+		drv->states[drv->state_count] =	/* structure copy */
+			kvm_halt_cstate;
+		drv->states[drv->state_count].exit_latency =
+			drv->state_count > 1 ?
+			drv->states[drv->state_count - 1].exit_latency + 1 : 1;
+		drv->states[drv->state_count].target_residency =
+			kvm_halt_target_residency;
+
+		drv->state_count += 1;
+	}
+
+	printk(KERN_ERR "detected states: %d\n\n",  drv->state_count);
+
 	if (icpu->byt_auto_demotion_disable_flag) {
 		wrmsrl(MSR_CC6_DEMOTION_POLICY_CONFIG, 0);
 		wrmsrl(MSR_MC6_DEMOTION_POLICY_CONFIG, 0);
@@ -1452,7 +1546,8 @@ static int __init intel_idle_init(void)
 		goto init_driver_fail;
 	}
 
-	if (boot_cpu_has(X86_FEATURE_ARAT))	/* Always Reliable APIC Timer */
+	if (boot_cpu_has(X86_FEATURE_ARAT) ||	/* Always Reliable APIC Timer */
+	    kvm_pv_mwait)
 		lapic_timer_reliable_states = LAPIC_TIMER_ALWAYS_RELIABLE;
 
 	retval = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "idle/intel:online",

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | linux.kernel

csiph-web