Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1740277 > unrolled thread

Re: [patch 3/3] x86: kvm guest side support for KVM_HC_RT_PRIO hypercall\

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [patch 3/3] x86: kvm guest side support for KVM_HC_RT_PRIO  hypercall\ Marcelo Tosatti <mtosatti@redhat.com> - 2017-09-27 01:30 +0200
    Re: [patch 3/3] x86: kvm guest side support for KVM_HC_RT_PRIO  hypercall\ Paolo Bonzini <pbonzini@redhat.com> - 2017-09-27 11:40 +0200
      Re: [patch 3/3] x86: kvm guest side support for KVM_HC_RT_PRIO  hypercall\ Marcelo Tosatti <mtosatti@redhat.com> - 2017-09-28 03:00 +0200
        Re: [patch 3/3] x86: kvm guest side support for KVM_HC_RT_PRIO  hypercall\ Paolo Bonzini <pbonzini@redhat.com> - 2017-09-28 09:30 +0200
          Re: [patch 3/3] x86: kvm guest side support for KVM_HC_RT_PRIO  hypercall\ Marcelo Tosatti <mtosatti@redhat.com> - 2017-09-29 01:10 +0200
            Re: [patch 3/3] x86: kvm guest side support for KVM_HC_RT_PRIO  hypercall\ Marcelo Tosatti <mtosatti@redhat.com> - 2017-09-29 01:10 +0200
            Re: [patch 3/3] x86: kvm guest side support for KVM_HC_RT_PRIO  hypercall\ Paolo Bonzini <pbonzini@redhat.com> - 2017-09-29 10:20 +0200
              Re: [patch 3/3] x86: kvm guest side support for KVM_HC_RT_PRIO  hypercall\ Marcelo Tosatti <mtosatti@redhat.com> - 2017-09-29 18:50 +0200
                Re: [patch 3/3] x86: kvm guest side support for KVM_HC_RT_PRIO  hypercall\ Paolo Bonzini <pbonzini@redhat.com> - 2017-09-29 19:10 +0200
                  Re: [patch 3/3] x86: kvm guest side support for KVM_HC_RT_PRIO  hypercall\ Marcelo Tosatti <mtosatti@redhat.com> - 2017-09-29 22:20 +0200
                    Re: [patch 3/3] x86: kvm guest side support for KVM_HC_RT_PRIO  hypercall\ Paolo Bonzini <pbonzini@redhat.com> - 2017-10-02 14:40 +0200
                      Re: [patch 3/3] x86: kvm guest side support for KVM_HC_RT_PRIO  hypercall\ Peter Zijlstra <peterz@infradead.org> - 2017-10-02 14:50 +0200

#1740277 — Re: [patch 3/3] x86: kvm guest side support for KVM_HC_RT_PRIO hypercall\

On Mon, Sep 25, 2017 at 05:12:42PM +0200, Paolo Bonzini wrote:
> On 25/09/2017 11:13, Peter Zijlstra wrote:
> > On Sun, Sep 24, 2017 at 11:57:53PM -0300, Marcelo Tosatti wrote:
> >> I think you are missing the following point:
> >>
> >> "vcpu0 can be interrupted when its not in a spinlock protected section, 
> >> otherwise it can't."
> 
> Who says that?  Certainly a driver can dedicate a single VCPU to
> periodic polling of the device, in such a way that the polling does not
> require a spinlock.

This sequence:


VCPU-0					VCPU-1 (running realtime workload)

takes spinlock A
scheduled out				
					spinlock(A) (busy spins until
						     VCPU-0 is scheduled
						     back in)
scheduled in
finishes execution of 
code under protected section
releases spinlock(A)			

					takes spinlock(A)

You get that point, right?

(*)

> >> So you _have_ to communicate to the host when the guest enters/leaves a
> >> critical section.
> >>
> >> So this point of "everything needs to be RT and the priorities must be
> >> designed carefully", is this: 
> >>
> >> 	WHEN in spinlock protected section (more specifically, when 
> >> 	spinlock protected section _shared with realtime vcpus_),
> >>
> >> 	priority of vcpu0 > priority of emulator thread
> >>
> >> 	OTHERWISE
> >>
> >> 	priority of vcpu0 < priority of emulator thread.
> 
> This is _not_ designed carefully, this is messy.

This is very precise to me. What is "messy" about it? (its clearly
defined).

> The emulator thread can interrupt the VCPU thread, so it has to be at
> higher RT priority (+ priority inheritance of mutexes).  

It can only do that _when_ the VCPU thread is not running a critical
section which a higher priority task depends on.

> Once you have
> done that we can decide on other approaches that e.g. let you get more
> sharing by placing housekeeping VCPUs at SCHED_NORMAL or SCHED_RR.

Well, if someone looks at (*) he sees that if the interruption delay 
(the length between "scheduled out" and "scheduled in" in that diagram)
exceeds a given threshold, that causes the realtime vcpu1 to also 
exceed processing of the realtime task for a given threshold. 

So when you say "The emulator thread can interrupt the VCPU thread", 
you're saying that it has to be modified to interrupt for a maximum
amount of time (say 15us).

Is that what you are suggesting?

> >> So emulator thread can interrupt and inject interrupts to vcpu0.
> > 
> > spinlock protected regions are not everything. What about lock-free
> > constructs where CPU's spin-wait on one another (there's plenty).
> > 
> > And I'm clearly ignorant of how this emulation thread works, but why
> > would it run for a long time? Either it is needed for forward progress
> > of the VCPU or its not. If its not, it shouldn't run.
> 
> The emulator thread 1) should not run for long period of times indeed,
> and 2) it is needed for forward progress of the VCPU.  So it has to be
> at higher RT priority.  I agree with Peter, sorry.  Spinlocks are a red
> herring here.
> 
> Paolo

Paolo, you don't control how many interruptions of the emulator thread
happen per second. So if you let the emulator thread interrupt the
emulator thread at all times, without some kind of bounding 
of these interruptions per time unit, you have a similar
problem as (*) (where the realtime task is scheduled).

Another approach to the problem was suggested to OpenStack.

[toc] | [next] | [standalone]

#1740567

On 27/09/2017 00:49, Marcelo Tosatti wrote:
> On Mon, Sep 25, 2017 at 05:12:42PM +0200, Paolo Bonzini wrote:
>> On 25/09/2017 11:13, Peter Zijlstra wrote:
>>> On Sun, Sep 24, 2017 at 11:57:53PM -0300, Marcelo Tosatti wrote:
>>>> I think you are missing the following point:
>>>>
>>>> "vcpu0 can be interrupted when its not in a spinlock protected section, 
>>>> otherwise it can't."
>>
>> Who says that?  Certainly a driver can dedicate a single VCPU to
>> periodic polling of the device, in such a way that the polling does not
>> require a spinlock.
> 
> This sequence:
> 
> 
> VCPU-0					VCPU-1 (running realtime workload)
> 
> takes spinlock A
> scheduled out				
> 					spinlock(A) (busy spins until
> 						     VCPU-0 is scheduled
> 						     back in)
> scheduled in
> finishes execution of 
> code under protected section
> releases spinlock(A)			
> 
> 					takes spinlock(A)

Yes, but then you have

                                        busy waits for flag to be set
  polls device
  scheduled out
  device receives data
                                        ...
  scheduled in
  set flag

or

  check for work to do
  scheduled out
                                        submit work
                                        busy waits for work to complete
  scheduled in
  do work

None of which have anything to do with spinlocks.  They're just
different ways to get priority inversion, and this...

>>>> So this point of "everything needs to be RT and the priorities must be
>>>> designed carefully", is this: 
>>>>
>>>> 	WHEN in spinlock protected section (more specifically, when 
>>>> 	spinlock protected section _shared with realtime vcpus_),
>>>>
>>>> 	priority of vcpu0 > priority of emulator thread
>>>>
>>>> 	OTHERWISE
>>>>
>>>> 	priority of vcpu0 < priority of emulator thread.
>>
>> This is _not_ designed carefully, this is messy.
> 
> This is very precise to me. What is "messy" about it? (its clearly
> defined).

... it's not how you design RT systems to avoid priority inversion.
It's just solving an instance of the issue.

>> The emulator thread can interrupt the VCPU thread, so it has to be at
>> higher RT priority (+ priority inheritance of mutexes).  
> 
> It can only do that _when_ the VCPU thread is not running a critical
> section which a higher priority task depends on.

All VCPUs must have the same priority.  There's no such thing as a
housekeeping VCPU.

There could be one sacrificial VCPU that you place on the same physical
CPU as the emulator thread, but that's it.  The whole system must run
smoothly even if you place those on the same physical CPU, without any
PV hacks.

> So when you say "The emulator thread can interrupt the VCPU thread", 
> you're saying that it has to be modified to interrupt for a maximum
> amount of time (say 15us).
> 
> Is that what you are suggesting?

This is correct.  But I think you are missing a fundamental point, that
is not specific to virt.  If a thread 1 can trigger an event in thread
2, and thread 2 runs at priority N, thread 1 must be placed at priority >N.

In this case, the emulator thread can signal I/O completion to the VCPU:
it doesn't matter if the VCPU is polling or using interrupts, the
emulator thread must be placed at higher priority than the VCPU.  This
is the root cause of the SeaBIOS issue that you were seeing.  Yes, it
was me who suggested moving VCPU0 and the emulator thread to
SCHED_NORMAL, but that was just a bandaid until the real fix was done
(which is to set up SCHED_FIFO priorities correctly and use PI mutexes).

In fact, this is not specific to the emulator thread.  It applies just
as well to vhost threads, to QEMU iothreads, even to the KVM PIT
emulation thread if the guest were using it.

> Paolo, you don't control how many interruptions of the emulator thread
> happen per second.

Indeed these non-VCPU threads should indeed run rarely and for a small
amount of time only to achieve bounded latencies in the VCPUs.  But if
they don't, those are simply bugs and we fix them.  In fact, sources of
frequent interruptions have all been fixed or moved outside the main
thread; for example, disks can use separate iothreads.  Configuration
problems are also bugs, just in a different place.

For a very basic VM that I've just tried (whatever QEMU does by default,
only tweak was not using the QEMU GUI), there is exactly one
interruption per second when the VM is idle.  That interruption in turn
is caused by udev periodically probing the guest CDROM (!).  So that's a
configuration problem: remove the CDROM, and it does one interruption
every 10-30 seconds, again all of them guest triggered.

Again: if you have many interruptions, it's not a flaw in KVM or QEMU's
design, it's just that someone is doing something stupid.  It could be
the guest (e.g. unnecessary devices or daemons as in the example above),
QEMU (e.g. the RTC emulation used to trigger QEMU timers twice a second
just to increment the clock), or the management (e.g. polling "is the VM
running" 50 times per second).  But it can and must be fixed.

The design should be the basic one: attribute the right priority to each
thread and things just work.

Paolo

> So if you let the emulator thread interrupt the
> emulator thread at all times, without some kind of bounding 
> of these interruptions per time unit, you have a similar
> problem as (*) (where the realtime task is scheduled).
> 

[toc] | [prev] | [next] | [standalone]

#1741120

On Wed, Sep 27, 2017 at 11:37:48AM +0200, Paolo Bonzini wrote:
> On 27/09/2017 00:49, Marcelo Tosatti wrote:
> > On Mon, Sep 25, 2017 at 05:12:42PM +0200, Paolo Bonzini wrote:
> >> On 25/09/2017 11:13, Peter Zijlstra wrote:
> >>> On Sun, Sep 24, 2017 at 11:57:53PM -0300, Marcelo Tosatti wrote:
> >>>> I think you are missing the following point:
> >>>>
> >>>> "vcpu0 can be interrupted when its not in a spinlock protected section, 
> >>>> otherwise it can't."
> >>
> >> Who says that?  Certainly a driver can dedicate a single VCPU to
> >> periodic polling of the device, in such a way that the polling does not
> >> require a spinlock.
> > 
> > This sequence:
> > 
> > 
> > VCPU-0					VCPU-1 (running realtime workload)
> > 
> > takes spinlock A
> > scheduled out				
> > 					spinlock(A) (busy spins until
> > 						     VCPU-0 is scheduled
> > 						     back in)
> > scheduled in
> > finishes execution of 
> > code under protected section
> > releases spinlock(A)			
> > 
> > 					takes spinlock(A)
> 
> Yes, but then you have
> 
>                                         busy waits for flag to be set
>   polls device
>   scheduled out
>   device receives data
>                                         ...
>   scheduled in
>   set flag
> 
> or
> 
>   check for work to do
>   scheduled out
>                                         submit work
>                                         busy waits for work to complete
>   scheduled in
>   do work
> 
> None of which have anything to do with spinlocks.  They're just
> different ways to get priority inversion, and this...
> 
> >>>> So this point of "everything needs to be RT and the priorities must be
> >>>> designed carefully", is this: 
> >>>>
> >>>> 	WHEN in spinlock protected section (more specifically, when 
> >>>> 	spinlock protected section _shared with realtime vcpus_),
> >>>>
> >>>> 	priority of vcpu0 > priority of emulator thread
> >>>>
> >>>> 	OTHERWISE
> >>>>
> >>>> 	priority of vcpu0 < priority of emulator thread.
> >>
> >> This is _not_ designed carefully, this is messy.
> > 
> > This is very precise to me. What is "messy" about it? (its clearly
> > defined).
> 
> ... it's not how you design RT systems to avoid priority inversion.
> It's just solving an instance of the issue.
> 
> >> The emulator thread can interrupt the VCPU thread, so it has to be at
> >> higher RT priority (+ priority inheritance of mutexes).  
> > 
> > It can only do that _when_ the VCPU thread is not running a critical
> > section which a higher priority task depends on.
> 
> All VCPUs must have the same priority.  There's no such thing as a
> housekeeping VCPU.
> 
> There could be one sacrificial VCPU that you place on the same physical
> CPU as the emulator thread, but that's it.  The whole system must run
> smoothly even if you place those on the same physical CPU, without any
> PV hacks.
> 
> > So when you say "The emulator thread can interrupt the VCPU thread", 
> > you're saying that it has to be modified to interrupt for a maximum
> > amount of time (say 15us).
> > 
> > Is that what you are suggesting?
> 
> This is correct.  But I think you are missing a fundamental point, that
> is not specific to virt.  If a thread 1 can trigger an event in thread
> 2, and thread 2 runs at priority N, thread 1 must be placed at priority >N.
> 
> In this case, the emulator thread can signal I/O completion to the VCPU:
> it doesn't matter if the VCPU is polling or using interrupts, the
> emulator thread must be placed at higher priority than the VCPU.  This
> is the root cause of the SeaBIOS issue that you were seeing.  Yes, it
> was me who suggested moving VCPU0 and the emulator thread to
> SCHED_NORMAL, but that was just a bandaid until the real fix was done
> (which is to set up SCHED_FIFO priorities correctly and use PI mutexes).
> 
> In fact, this is not specific to the emulator thread.  It applies just
> as well to vhost threads, to QEMU iothreads, even to the KVM PIT
> emulation thread if the guest were using it.
> 
> > Paolo, you don't control how many interruptions of the emulator thread
> > happen per second.
> 
> Indeed these non-VCPU threads should indeed run rarely and for a small
> amount of time only to achieve bounded latencies in the VCPUs.  But if
> they don't, those are simply bugs and we fix them.  In fact, sources of
> frequent interruptions have all been fixed or moved outside the main
> thread; for example, disks can use separate iothreads.  Configuration
> problems are also bugs, just in a different place.


> 
> For a very basic VM that I've just tried (whatever QEMU does by default,
> only tweak was not using the QEMU GUI), there is exactly one
> interruption per second when the VM is idle.  That interruption in turn
> is caused by udev periodically probing the guest CDROM (!).  So that's a
> configuration problem: remove the CDROM, and it does one interruption
> every 10-30 seconds, again all of them guest triggered.
> 
> Again: if you have many interruptions, it's not a flaw in KVM or QEMU's
> design, it's just that someone is doing something stupid.  It could be
> the guest (e.g. unnecessary devices or daemons as in the example above),
> QEMU (e.g. the RTC emulation used to trigger QEMU timers twice a second
> just to increment the clock), or the management (e.g. polling "is the VM
> running" 50 times per second).  But it can and must be fixed.

No, i mean you can run anything in VCPU-0 (it is valid to do that).
And that "anything" can generate 1 interrupt per second, 1000 or 10.000
interrupts per second. Which are all valid things to be done.

"I can't run a kernel compilation on VCPU-0 because that will impact
latency on the realtime VCPU-1" is not acceptable.

> The design should be the basic one: attribute the right priority to each
> thread and things just work.
> 
> Paolo
> 
> > So if you let the emulator thread interrupt the
> > emulator thread at all times, without some kind of bounding 
> > of these interruptions per time unit, you have a similar
> > problem as (*) (where the realtime task is scheduled).
> > 

[toc] | [prev] | [next] | [standalone]

#1741229

On 28/09/2017 02:44, Marcelo Tosatti wrote:
>> Again: if you have many interruptions, it's not a flaw in KVM or QEMU's
>> design, it's just that someone is doing something stupid.  It could be
>> the guest (e.g. unnecessary devices or daemons as in the example above),
>> QEMU (e.g. the RTC emulation used to trigger QEMU timers twice a second
>> just to increment the clock), or the management (e.g. polling "is the VM
>> running" 50 times per second).  But it can and must be fixed.
>
> No, i mean you can run anything in VCPU-0 (it is valid to do that).
> And that "anything" can generate 1 interrupt per second, 1000 or 10.000
> interrupts per second. Which are all valid things to be done.
> 
> "I can't run a kernel compilation on VCPU-0 because that will impact
> latency on the realtime VCPU-1" is not acceptable.

That shouldn't happen.  Sources of frequent interruptions have all been
fixed or moved outside the main thread.

If there are more left, report the bug and we'll see how to fix it in
userspace.

Paolo

[toc] | [prev] | [next] | [standalone]

#1741842

On Thu, Sep 28, 2017 at 09:22:02AM +0200, Paolo Bonzini wrote:
> On 28/09/2017 02:44, Marcelo Tosatti wrote:
> >> Again: if you have many interruptions, it's not a flaw in KVM or QEMU's
> >> design, it's just that someone is doing something stupid.  It could be
> >> the guest (e.g. unnecessary devices or daemons as in the example above),
> >> QEMU (e.g. the RTC emulation used to trigger QEMU timers twice a second
> >> just to increment the clock), or the management (e.g. polling "is the VM
> >> running" 50 times per second).  But it can and must be fixed.
> >
> > No, i mean you can run anything in VCPU-0 (it is valid to do that).
> > And that "anything" can generate 1 interrupt per second, 1000 or 10.000
> > interrupts per second. Which are all valid things to be done.
> > 
> > "I can't run a kernel compilation on VCPU-0 because that will impact
> > latency on the realtime VCPU-1" is not acceptable.
> 
> That shouldn't happen.  Sources of frequent interruptions have all been
> fixed or moved outside the main thread.
> 
> If there are more left, report the bug and we'll see how to fix it in
> userspace.
> 
> Paolo

What should not happen? The generation of 10.000 interrupts per second
(say disk IO completion) on a given workload ?

[toc] | [prev] | [next] | [standalone]

#1741843

On Thu, Sep 28, 2017 at 06:35:08PM -0300, Marcelo Tosatti wrote:
> On Thu, Sep 28, 2017 at 09:22:02AM +0200, Paolo Bonzini wrote:
> > On 28/09/2017 02:44, Marcelo Tosatti wrote:
> > >> Again: if you have many interruptions, it's not a flaw in KVM or QEMU's
> > >> design, it's just that someone is doing something stupid.  It could be
> > >> the guest (e.g. unnecessary devices or daemons as in the example above),
> > >> QEMU (e.g. the RTC emulation used to trigger QEMU timers twice a second
> > >> just to increment the clock), or the management (e.g. polling "is the VM
> > >> running" 50 times per second).  But it can and must be fixed.
> > >
> > > No, i mean you can run anything in VCPU-0 (it is valid to do that).
> > > And that "anything" can generate 1 interrupt per second, 1000 or 10.000
> > > interrupts per second. Which are all valid things to be done.
> > > 
> > > "I can't run a kernel compilation on VCPU-0 because that will impact
> > > latency on the realtime VCPU-1" is not acceptable.
> > 
> > That shouldn't happen.  Sources of frequent interruptions have all been
> > fixed or moved outside the main thread.
> > 
> > If there are more left, report the bug and we'll see how to fix it in
> > userspace.
> > 
> > Paolo
> 
> What should not happen? The generation of 10.000 interrupts per second
> (say disk IO completion) on a given workload ?

Are you suggesting that, workloads in vcpu-0 should be limited in the number
of interrupts (and durations of each interruption), so that the realtime vcpu-1's
latency requirement is met ?

I don't see how that suggestion can work because even if you make each
exit small, the frequency of them will cause a latency violation on
vcpu-1.

[toc] | [prev] | [next] | [standalone]

#1741963

On 28/09/2017 23:35, Marcelo Tosatti wrote:
> On Thu, Sep 28, 2017 at 09:22:02AM +0200, Paolo Bonzini wrote:
>> On 28/09/2017 02:44, Marcelo Tosatti wrote:
>>>> Again: if you have many interruptions, it's not a flaw in KVM or QEMU's
>>>> design, it's just that someone is doing something stupid.  It could be
>>>> the guest (e.g. unnecessary devices or daemons as in the example above),
>>>> QEMU (e.g. the RTC emulation used to trigger QEMU timers twice a second
>>>> just to increment the clock), or the management (e.g. polling "is the VM
>>>> running" 50 times per second).  But it can and must be fixed.
>>>
>>> No, i mean you can run anything in VCPU-0 (it is valid to do that).
>>> And that "anything" can generate 1 interrupt per second, 1000 or 10.000
>>> interrupts per second. Which are all valid things to be done.
>>>
>>> "I can't run a kernel compilation on VCPU-0 because that will impact
>>> latency on the realtime VCPU-1" is not acceptable.
>>
>> That shouldn't happen.  Sources of frequent interruptions have all been
>> fixed or moved outside the main thread.
>>
>> If there are more left, report the bug and we'll see how to fix it in
>> userspace.
> 
> What should not happen? The generation of 10.000 interrupts per second
> (say disk IO completion) on a given workload ?

If you know you have this kind disk workload, you must use virtio-blk or
virtio-scsi with iothreads and place the iothreads on their own physical
CPUs.

Among "run arbitrary workloads", "run real-time workloads", "pack stuff
into as few physical CPUs as possible", you can only pick two.

Paolo

[toc] | [prev] | [next] | [standalone]

#1742240

On Fri, Sep 29, 2017 at 10:18:25AM +0200, Paolo Bonzini wrote:
> On 28/09/2017 23:35, Marcelo Tosatti wrote:
> > On Thu, Sep 28, 2017 at 09:22:02AM +0200, Paolo Bonzini wrote:
> >> On 28/09/2017 02:44, Marcelo Tosatti wrote:
> >>>> Again: if you have many interruptions, it's not a flaw in KVM or QEMU's
> >>>> design, it's just that someone is doing something stupid.  It could be
> >>>> the guest (e.g. unnecessary devices or daemons as in the example above),
> >>>> QEMU (e.g. the RTC emulation used to trigger QEMU timers twice a second
> >>>> just to increment the clock), or the management (e.g. polling "is the VM
> >>>> running" 50 times per second).  But it can and must be fixed.
> >>>
> >>> No, i mean you can run anything in VCPU-0 (it is valid to do that).
> >>> And that "anything" can generate 1 interrupt per second, 1000 or 10.000
> >>> interrupts per second. Which are all valid things to be done.
> >>>
> >>> "I can't run a kernel compilation on VCPU-0 because that will impact
> >>> latency on the realtime VCPU-1" is not acceptable.
> >>
> >> That shouldn't happen.  Sources of frequent interruptions have all been
> >> fixed or moved outside the main thread.
> >>
> >> If there are more left, report the bug and we'll see how to fix it in
> >> userspace.
> > 
> > What should not happen? The generation of 10.000 interrupts per second
> > (say disk IO completion) on a given workload ?
> 
> If you know you have this kind disk workload, you must use virtio-blk or
> virtio-scsi with iothreads and place the iothreads on their own physical
> CPUs.
> 
> Among "run arbitrary workloads", "run real-time workloads", "pack stuff
> into as few physical CPUs as possible", you can only pick two.
> 
> Paolo

Thats not the state of things (userspace in vcpu-0 is not specially tailored
to not violate latencies in vcpu-1): that is not all user triggered
actions can be verified.

Think "updatedb", and so on...

[toc] | [prev] | [next] | [standalone]

#1742247

On 29/09/2017 18:40, Marcelo Tosatti wrote:
>> If you know you have this kind disk workload, you must use virtio-blk or
>> virtio-scsi with iothreads and place the iothreads on their own physical
>> CPUs.
>>
>> Among "run arbitrary workloads", "run real-time workloads", "pack stuff
>> into as few physical CPUs as possible", you can only pick two.
>
> Thats not the state of things (userspace in vcpu-0 is not specially tailored
> to not violate latencies in vcpu-1): that is not all user triggered
> actions can be verified.
> 
> Think "updatedb", and so on...

_Which_ spinlock is it that can cause unwanted latency while running
updatedb on VCPU0 and a real-time workload on VCPU1, and only so on virt
because of the emulator thread?  Is this still broken if you set up
priorities for the emulator thread correctly and use PI mutexes in QEMU?
 And if so, what is the cause of interruptions in the emulator thread
and how are these interruptions causing the jitter?

Priorities and priority inheritance (or lack of them) is a _known_
issue.  Jan was doing his KVM-RT things in 2009 and he was talking about
priorities[1] back then.  The effect of correct priorities is to _lower_
jitter, not to make it worse, and anyway certainly not worse than
SCHED_NORMAL I/O thread.  Once that's fixed, we can look at other problems.

Paolo

[1] http://static.lwn.net/images/conf/rtlws11/papers/proc/p18.pdf which
also mentions pv scheduling

[toc] | [prev] | [next] | [standalone]

#1742327

On Fri, Sep 29, 2017 at 07:05:41PM +0200, Paolo Bonzini wrote:
> On 29/09/2017 18:40, Marcelo Tosatti wrote:
> >> If you know you have this kind disk workload, you must use virtio-blk or
> >> virtio-scsi with iothreads and place the iothreads on their own physical
> >> CPUs.
> >>
> >> Among "run arbitrary workloads", "run real-time workloads", "pack stuff
> >> into as few physical CPUs as possible", you can only pick two.
> >
> > Thats not the state of things (userspace in vcpu-0 is not specially tailored
> > to not violate latencies in vcpu-1): that is not all user triggered
> > actions can be verified.
> > 
> > Think "updatedb", and so on...
> 
> _Which_ spinlock is it that can cause unwanted latency while running
> updatedb on VCPU0 and a real-time workload on VCPU1, and only so on virt
> because of the emulator thread?

Hundreds of them (the one being hit is in timer_interrupt), but i went 
to check and there are hundreds of raw spinlocks shared between the
kernel threads that run on isolated CPUs and vcpu-0.

>  Is this still broken if you set up
> priorities for the emulator thread correctly and use PI mutexes in QEMU?

I don't see why it would not, if you have to schedule the emulator
thread to process and inject I/O interrupts for example.

>  And if so, what is the cause of interruptions in the emulator thread
> and how are these interruptions causing the jitter?

Interrupt injections.

> Priorities and priority inheritance (or lack of them) is a _known_
> issue.  Jan was doing his KVM-RT things in 2009 and he was talking about
> priorities[1] back then.  The effect of correct priorities is to _lower_
> jitter, not to make it worse, and anyway certainly not worse than
> SCHED_NORMAL I/O thread.  Once that's fixed, we can look at other problems.
> 
> Paolo
> 
> [1] http://static.lwn.net/images/conf/rtlws11/papers/proc/p18.pdf which
> also mentions pv scheduling

[toc] | [prev] | [next] | [standalone]

#1743179

On 29/09/2017 22:17, Marcelo Tosatti wrote:
> On Fri, Sep 29, 2017 at 07:05:41PM +0200, Paolo Bonzini wrote:
>> On 29/09/2017 18:40, Marcelo Tosatti wrote:
>>> Thats not the state of things (userspace in vcpu-0 is not specially tailored
>>> to not violate latencies in vcpu-1): that is not all user triggered
>>> actions can be verified.
>>>
>>> Think "updatedb", and so on...
>>
>> _Which_ spinlock is it that can cause unwanted latency while running
>> updatedb on VCPU0 and a real-time workload on VCPU1, and only so on virt
>> because of the emulator thread?
> 
> Hundreds of them (the one being hit is in timer_interrupt), but i went 
> to check and there are hundreds of raw spinlocks shared between the
> kernel threads that run on isolated CPUs and vcpu-0.
> 
>>  Is this still broken if you set up
>> priorities for the emulator thread correctly and use PI mutexes in QEMU?
> 
> I don't see why it would not, if you have to schedule the emulator
> thread to process and inject I/O interrupts for example.

Yes, you're right if it's interrupt injections.  If it's unexpected disk
accesses, you can just add a QEMU I/O thread on a different physical
CPU.  The same physical CPU can host I/O threads for different guests if
you expect them to do little.

I don't understand why is it correct to delay interrupt injection just
because VCPU0 is running in a spinlock-protected region?  I just cannot
see the reason why it's safe and not a recipe for priority inversions.

Paolo

>>  And if so, what is the cause of interruptions in the emulator thread
>> and how are these interruptions causing the jitter?
> 
> Interrupt injections.
> 
>> Priorities and priority inheritance (or lack of them) is a _known_
>> issue.  Jan was doing his KVM-RT things in 2009 and he was talking about
>> priorities[1] back then.  The effect of correct priorities is to _lower_
>> jitter, not to make it worse, and anyway certainly not worse than
>> SCHED_NORMAL I/O thread.  Once that's fixed, we can look at other problems.
>>
>> Paolo
>>
>> [1] http://static.lwn.net/images/conf/rtlws11/papers/proc/p18.pdf which
>> also mentions pv scheduling

[toc] | [prev] | [next] | [standalone]

#1743194

On Mon, Oct 02, 2017 at 02:30:33PM +0200, Paolo Bonzini wrote:
> I don't understand why is it correct to delay interrupt injection just
> because VCPU0 is running in a spinlock-protected region?  I just cannot
> see the reason why it's safe and not a recipe for priority inversions.

It is indeed not right. Something like:

	raw_spin_lock(&some_lock);

	/* do crud */

	raw_spin_unlock(&some_lock);

Should not hold off the interrupt that tells you your finger is in
imminent danger of becoming detached. Only when we do
local_irq_disable() (ie. raw_spin_lock_irq*() and the like) should we
avoid interrupt delivery.

This whole fixation on spinlock regions is misguided and must stop, its
wrong on all levels.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web