Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1736598 > unrolled thread

[patch 0/3] KVM KVM_HC_RT_PRIO hypercall support

Started byMarcelo Tosatti <mtosatti@redhat.com>
First post2017-09-21 13:50 +0200
Last post2017-09-22 08:30 +0200
Articles 4 — 2 participants

Back to article view | Back to linux.kernel

Contents

  [patch 0/3] KVM KVM_HC_RT_PRIO hypercall support Marcelo Tosatti <mtosatti@redhat.com> - 2017-09-21 13:50 +0200
    Re: [patch 0/3] KVM KVM_HC_RT_PRIO hypercall support Jan Kiszka <jan.kiszka@siemens.com> - 2017-09-21 19:50 +0200
      Re: [patch 0/3] KVM KVM_HC_RT_PRIO hypercall support Marcelo Tosatti <mtosatti@redhat.com> - 2017-09-22 03:50 +0200
        Re: [patch 0/3] KVM KVM_HC_RT_PRIO hypercall support Jan Kiszka <jan.kiszka@siemens.com> - 2017-09-22 08:30 +0200

#1736598 — [patch 0/3] KVM KVM_HC_RT_PRIO hypercall support

FromMarcelo Tosatti <mtosatti@redhat.com>
Date2017-09-21 13:50 +0200
Subject[patch 0/3] KVM KVM_HC_RT_PRIO hypercall support
Message-ID<us89r-rR-15@gated-at.bofh.it>
When executing guest vcpu-0 with FIFO:1 priority, which is necessary to
deal with the following situation:

VCPU-0 (housekeeping VCPU)              VCPU-1 (realtime VCPU)

raw_spin_lock(A)
interrupted, schedule task T-1          raw_spin_lock(A) (spin)

raw_spin_unlock(A)

Certain operations must interrupt guest vcpu-0 (see trace below).

To fix this issue, only change guest vcpu-0 to FIFO priority
on spinlock critical sections (see patch).

Hang trace
==========

Without FIFO priority:

qemu-kvm-6705  [002] ....1.. 767785.648964: kvm_exit: reason IO_INSTRUCTION rip 0xe8fe info 1f00039 0
qemu-kvm-6705  [002] ....1.. 767785.648965: kvm_exit: reason IO_INSTRUCTION rip 0xe911 info 3f60008 0
qemu-kvm-6705  [002] ....1.. 767785.648968: kvm_exit: reason IO_INSTRUCTION rip 0x8984 info 608000b 0
qemu-kvm-6705  [002] ....1.. 767785.648971: kvm_exit: reason IO_INSTRUCTION rip 0xb313 info 1f70008 0
qemu-kvm-6705  [002] ....1.. 767785.648974: kvm_exit: reason IO_INSTRUCTION rip 0xb514 info 3f60000 0
qemu-kvm-6705  [002] ....1.. 767785.648977: kvm_exit: reason PENDING_INTERRUPT rip 0x8052 info 0 0
qemu-kvm-6705  [002] ....1.. 767785.648980: kvm_exit: reason IO_INSTRUCTION rip 0xeee6 info 200040 0
qemu-kvm-6705  [002] ....1.. 767785.648999: kvm_exit: reason EPT_MISCONFIG rip 0x2120 info 0 0

With FIFO priority:

qemu-kvm-7636  [002] ....1.. 768218.205065: kvm_exit: reason IO_INSTRUCTION rip 0xb313 info 1f70008 0
qemu-kvm-7636  [002] ....1.. 768218.205068: kvm_exit: reason IO_INSTRUCTION rip 0x8984 info 608000b 0
qemu-kvm-7636  [002] ....1.. 768218.205071: kvm_exit: reason IO_INSTRUCTION rip 0xb313 info 1f70008 0
qemu-kvm-7636  [002] ....1.. 768218.205074: kvm_exit: reason IO_INSTRUCTION rip 0x8984 info 608000b 0
qemu-kvm-7636  [002] ....1.. 768218.205077: kvm_exit: reason IO_INSTRUCTION rip 0xb313 info 1f70008 0
..

Performance numbers (kernel compilation with make -j2)
======================================================

With hypercall: 4:40.  (make -j2)
Without hypercall: 3:38.  (make -j2)

Note for NFV workloads spinlock performance is not relevant
since DPDK should not enter the kernel (and housekeeping vcpu
performance is far from a key factor).

Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>

[toc] | [next] | [standalone]

#1736978

FromJan Kiszka <jan.kiszka@siemens.com>
Date2017-09-21 19:50 +0200
Message-ID<usdLP-41H-15@gated-at.bofh.it>
In reply to#1736598
On 2017-09-21 13:38, Marcelo Tosatti wrote:
> When executing guest vcpu-0 with FIFO:1 priority, which is necessary to
> deal with the following situation:
> 
> VCPU-0 (housekeeping VCPU)              VCPU-1 (realtime VCPU)
> 
> raw_spin_lock(A)
> interrupted, schedule task T-1          raw_spin_lock(A) (spin)
> 
> raw_spin_unlock(A)
> 
> Certain operations must interrupt guest vcpu-0 (see trace below).
> 
> To fix this issue, only change guest vcpu-0 to FIFO priority
> on spinlock critical sections (see patch).
> 
> Hang trace
> ==========
> 
> Without FIFO priority:
> 
> qemu-kvm-6705  [002] ....1.. 767785.648964: kvm_exit: reason IO_INSTRUCTION rip 0xe8fe info 1f00039 0
> qemu-kvm-6705  [002] ....1.. 767785.648965: kvm_exit: reason IO_INSTRUCTION rip 0xe911 info 3f60008 0
> qemu-kvm-6705  [002] ....1.. 767785.648968: kvm_exit: reason IO_INSTRUCTION rip 0x8984 info 608000b 0
> qemu-kvm-6705  [002] ....1.. 767785.648971: kvm_exit: reason IO_INSTRUCTION rip 0xb313 info 1f70008 0
> qemu-kvm-6705  [002] ....1.. 767785.648974: kvm_exit: reason IO_INSTRUCTION rip 0xb514 info 3f60000 0
> qemu-kvm-6705  [002] ....1.. 767785.648977: kvm_exit: reason PENDING_INTERRUPT rip 0x8052 info 0 0
> qemu-kvm-6705  [002] ....1.. 767785.648980: kvm_exit: reason IO_INSTRUCTION rip 0xeee6 info 200040 0
> qemu-kvm-6705  [002] ....1.. 767785.648999: kvm_exit: reason EPT_MISCONFIG rip 0x2120 info 0 0
> 
> With FIFO priority:
> 
> qemu-kvm-7636  [002] ....1.. 768218.205065: kvm_exit: reason IO_INSTRUCTION rip 0xb313 info 1f70008 0
> qemu-kvm-7636  [002] ....1.. 768218.205068: kvm_exit: reason IO_INSTRUCTION rip 0x8984 info 608000b 0
> qemu-kvm-7636  [002] ....1.. 768218.205071: kvm_exit: reason IO_INSTRUCTION rip 0xb313 info 1f70008 0
> qemu-kvm-7636  [002] ....1.. 768218.205074: kvm_exit: reason IO_INSTRUCTION rip 0x8984 info 608000b 0
> qemu-kvm-7636  [002] ....1.. 768218.205077: kvm_exit: reason IO_INSTRUCTION rip 0xb313 info 1f70008 0
> ..
> 
> Performance numbers (kernel compilation with make -j2)
> ======================================================
> 
> With hypercall: 4:40.  (make -j2)
> Without hypercall: 3:38.  (make -j2)
> 
> Note for NFV workloads spinlock performance is not relevant
> since DPDK should not enter the kernel (and housekeeping vcpu
> performance is far from a key factor).
> 
> Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
> 

That sounds familiar, though not yet the same: :)

http://git.kiszka.org/?p=linux-kvm.git;a=shortlog;h=refs/heads/queues/paravirt-sched
(paper: http://lwn.net/images/conf/rtlws11/papers/proc/p18.pdf)

I suppose your goal is not to enable the host to follow the guest
scheduler priority completely but only to have prio-ceiling for such
short critical sections. Maybe still useful to think ahead about future
extensions when actually introducing such an interface.

But shouldn't there be some limits on the maximum prio the guest can select?

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

[toc] | [prev] | [next] | [standalone]

#1737126

FromMarcelo Tosatti <mtosatti@redhat.com>
Date2017-09-22 03:50 +0200
Message-ID<uslgl-8oN-1@gated-at.bofh.it>
In reply to#1736978
On Thu, Sep 21, 2017 at 07:45:32PM +0200, Jan Kiszka wrote:
> On 2017-09-21 13:38, Marcelo Tosatti wrote:
> > When executing guest vcpu-0 with FIFO:1 priority, which is necessary to
> > deal with the following situation:
> > 
> > VCPU-0 (housekeeping VCPU)              VCPU-1 (realtime VCPU)
> > 
> > raw_spin_lock(A)
> > interrupted, schedule task T-1          raw_spin_lock(A) (spin)
> > 
> > raw_spin_unlock(A)
> > 
> > Certain operations must interrupt guest vcpu-0 (see trace below).
> > 
> > To fix this issue, only change guest vcpu-0 to FIFO priority
> > on spinlock critical sections (see patch).
> > 
> > Hang trace
> > ==========
> > 
> > Without FIFO priority:
> > 
> > qemu-kvm-6705  [002] ....1.. 767785.648964: kvm_exit: reason IO_INSTRUCTION rip 0xe8fe info 1f00039 0
> > qemu-kvm-6705  [002] ....1.. 767785.648965: kvm_exit: reason IO_INSTRUCTION rip 0xe911 info 3f60008 0
> > qemu-kvm-6705  [002] ....1.. 767785.648968: kvm_exit: reason IO_INSTRUCTION rip 0x8984 info 608000b 0
> > qemu-kvm-6705  [002] ....1.. 767785.648971: kvm_exit: reason IO_INSTRUCTION rip 0xb313 info 1f70008 0
> > qemu-kvm-6705  [002] ....1.. 767785.648974: kvm_exit: reason IO_INSTRUCTION rip 0xb514 info 3f60000 0
> > qemu-kvm-6705  [002] ....1.. 767785.648977: kvm_exit: reason PENDING_INTERRUPT rip 0x8052 info 0 0
> > qemu-kvm-6705  [002] ....1.. 767785.648980: kvm_exit: reason IO_INSTRUCTION rip 0xeee6 info 200040 0
> > qemu-kvm-6705  [002] ....1.. 767785.648999: kvm_exit: reason EPT_MISCONFIG rip 0x2120 info 0 0
> > 
> > With FIFO priority:
> > 
> > qemu-kvm-7636  [002] ....1.. 768218.205065: kvm_exit: reason IO_INSTRUCTION rip 0xb313 info 1f70008 0
> > qemu-kvm-7636  [002] ....1.. 768218.205068: kvm_exit: reason IO_INSTRUCTION rip 0x8984 info 608000b 0
> > qemu-kvm-7636  [002] ....1.. 768218.205071: kvm_exit: reason IO_INSTRUCTION rip 0xb313 info 1f70008 0
> > qemu-kvm-7636  [002] ....1.. 768218.205074: kvm_exit: reason IO_INSTRUCTION rip 0x8984 info 608000b 0
> > qemu-kvm-7636  [002] ....1.. 768218.205077: kvm_exit: reason IO_INSTRUCTION rip 0xb313 info 1f70008 0
> > ..
> > 
> > Performance numbers (kernel compilation with make -j2)
> > ======================================================
> > 
> > With hypercall: 4:40.  (make -j2)
> > Without hypercall: 3:38.  (make -j2)
> > 
> > Note for NFV workloads spinlock performance is not relevant
> > since DPDK should not enter the kernel (and housekeeping vcpu
> > performance is far from a key factor).
> > 
> > Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
> > 
> 
> That sounds familiar, though not yet the same: :)
> 
> http://git.kiszka.org/?p=linux-kvm.git;a=shortlog;h=refs/heads/queues/paravirt-sched
> (paper: http://lwn.net/images/conf/rtlws11/papers/proc/p18.pdf)
> 
> I suppose your goal is not to enable the host to follow the guest
> scheduler priority completely but only to have prio-ceiling for such
> short critical sections. Maybe still useful to think ahead about future
> extensions when actually introducing such an interface.

Hi Jan!

Hum... I'll take a look at your interface/paper and get back to you.

> But shouldn't there be some limits on the maximum prio the guest can select?

The SCHED_FIFO prio is fixed, selectable when QEMU starts. Do you
envision any other use case than a fixed priority value selectable
at QEMU initialization?

Thanks

[toc] | [prev] | [next] | [standalone]

#1737211

FromJan Kiszka <jan.kiszka@siemens.com>
Date2017-09-22 08:30 +0200
Message-ID<uspDk-2JB-17@gated-at.bofh.it>
In reply to#1737126
On 2017-09-22 03:19, Marcelo Tosatti wrote:
> On Thu, Sep 21, 2017 at 07:45:32PM +0200, Jan Kiszka wrote:
>> On 2017-09-21 13:38, Marcelo Tosatti wrote:
>>> When executing guest vcpu-0 with FIFO:1 priority, which is necessary to
>>> deal with the following situation:
>>>
>>> VCPU-0 (housekeeping VCPU)              VCPU-1 (realtime VCPU)
>>>
>>> raw_spin_lock(A)
>>> interrupted, schedule task T-1          raw_spin_lock(A) (spin)
>>>
>>> raw_spin_unlock(A)
>>>
>>> Certain operations must interrupt guest vcpu-0 (see trace below).
>>>
>>> To fix this issue, only change guest vcpu-0 to FIFO priority
>>> on spinlock critical sections (see patch).
>>>
>>> Hang trace
>>> ==========
>>>
>>> Without FIFO priority:
>>>
>>> qemu-kvm-6705  [002] ....1.. 767785.648964: kvm_exit: reason IO_INSTRUCTION rip 0xe8fe info 1f00039 0
>>> qemu-kvm-6705  [002] ....1.. 767785.648965: kvm_exit: reason IO_INSTRUCTION rip 0xe911 info 3f60008 0
>>> qemu-kvm-6705  [002] ....1.. 767785.648968: kvm_exit: reason IO_INSTRUCTION rip 0x8984 info 608000b 0
>>> qemu-kvm-6705  [002] ....1.. 767785.648971: kvm_exit: reason IO_INSTRUCTION rip 0xb313 info 1f70008 0
>>> qemu-kvm-6705  [002] ....1.. 767785.648974: kvm_exit: reason IO_INSTRUCTION rip 0xb514 info 3f60000 0
>>> qemu-kvm-6705  [002] ....1.. 767785.648977: kvm_exit: reason PENDING_INTERRUPT rip 0x8052 info 0 0
>>> qemu-kvm-6705  [002] ....1.. 767785.648980: kvm_exit: reason IO_INSTRUCTION rip 0xeee6 info 200040 0
>>> qemu-kvm-6705  [002] ....1.. 767785.648999: kvm_exit: reason EPT_MISCONFIG rip 0x2120 info 0 0
>>>
>>> With FIFO priority:
>>>
>>> qemu-kvm-7636  [002] ....1.. 768218.205065: kvm_exit: reason IO_INSTRUCTION rip 0xb313 info 1f70008 0
>>> qemu-kvm-7636  [002] ....1.. 768218.205068: kvm_exit: reason IO_INSTRUCTION rip 0x8984 info 608000b 0
>>> qemu-kvm-7636  [002] ....1.. 768218.205071: kvm_exit: reason IO_INSTRUCTION rip 0xb313 info 1f70008 0
>>> qemu-kvm-7636  [002] ....1.. 768218.205074: kvm_exit: reason IO_INSTRUCTION rip 0x8984 info 608000b 0
>>> qemu-kvm-7636  [002] ....1.. 768218.205077: kvm_exit: reason IO_INSTRUCTION rip 0xb313 info 1f70008 0
>>> ..
>>>
>>> Performance numbers (kernel compilation with make -j2)
>>> ======================================================
>>>
>>> With hypercall: 4:40.  (make -j2)
>>> Without hypercall: 3:38.  (make -j2)
>>>
>>> Note for NFV workloads spinlock performance is not relevant
>>> since DPDK should not enter the kernel (and housekeeping vcpu
>>> performance is far from a key factor).
>>>
>>> Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
>>>
>>
>> That sounds familiar, though not yet the same: :)
>>
>> http://git.kiszka.org/?p=linux-kvm.git;a=shortlog;h=refs/heads/queues/paravirt-sched
>> (paper: http://lwn.net/images/conf/rtlws11/papers/proc/p18.pdf)
>>
>> I suppose your goal is not to enable the host to follow the guest
>> scheduler priority completely but only to have prio-ceiling for such
>> short critical sections. Maybe still useful to think ahead about future
>> extensions when actually introducing such an interface.
> 
> Hi Jan!
> 
> Hum... I'll take a look at your interface/paper and get back to you.
> 
>> But shouldn't there be some limits on the maximum prio the guest can select?
> 
> The SCHED_FIFO prio is fixed, selectable when QEMU starts. Do you
> envision any other use case than a fixed priority value selectable
> at QEMU initialization?

Oh, indeed, this is a pure prio-ceiling variant with host-defined
ceiling value.

But it's very inefficient to use a hypercall for entering and leaving
each and every sections. I would strongly recommend using a lazy scheme
where the guest writes the desired state into a shared memory page, and
the host only evaluates that prior to taking a scheduling decision, or
at least only on real vmexits. We're using such scheme successfully to
accelerate the fast path of prio-ceiling for pthread mutexes in the
Xenomai real-time extension.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web