Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1225973 > unrolled thread

Re: [PATCH] KVM: add halt_attempted_poll to VCPU stats

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH] KVM: add halt_attempted_poll to VCPU stats Christian Borntraeger <borntraeger@de.ibm.com> - 2015-09-16 12:30 +0200
    Re: [PATCH] KVM: add halt_attempted_poll to VCPU stats Wanpeng Li <wanpeng.li@hotmail.com> - 2015-09-16 14:20 +0200
    Re: [PATCH] KVM: add halt_attempted_poll to VCPU stats David Hildenbrand <dahi@linux.vnet.ibm.com> - 2015-09-17 12:30 +0200

#1225973 — Re: [PATCH] KVM: add halt_attempted_poll to VCPU stats

Am 15.09.2015 um 18:27 schrieb Paolo Bonzini:
> This new statistic can help diagnosing VCPUs that, for any reason,
> trigger bad behavior of halt_poll_ns autotuning.
> 
> For example, say halt_poll_ns = 480000, and wakeups are spaced exactly
> like 479us, 481us, 479us, 481us. Then KVM always fails polling and wastes
> 10+20+40+80+160+320+480 = 1110 microseconds out of every
> 479+481+479+481+479+481+479 = 3359 microseconds. The VCPU then
> is consuming about 30% more CPU than it would use without
> polling.  This would show as an abnormally high number of
> attempted polling compared to the successful polls.
> 
> Cc: Christian Borntraeger <borntraeger@de.ibm.com<
> Cc: David Matlack <dmatlack@google.com>
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

Acked-by: Christian Borntraeger <borntraeger@de.ibm.com>

yes, this will help to detect some bad cases, but not all.

PS: 
upstream maintenance keeps me really busy at the moment :-)
I am looking into a case right now, where auto polling goes 
completely nuts on my system:

guest1: 8vcpus		guest2: 1 vcpu
iperf with 25 process (-P25) from guest1 to guest2.

I/O interrupts on s390 are floating (pending on all CPUs) so on 
ALL VCPUs that go to sleep, polling will consider any pending
network interrupt as successful poll. So with auto polling the
guest consumes up to 5 host CPUs without auto polling only 1.
Reducing  halt_poll_ns to 100000 seems to work (goes back to 
1 cpu).

The proper way might be to feedback the result of the
interrupt dequeue into the heuristics. Don't know yet how
to handle that properly.

Christian

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1226041

On 9/16/15 6:12 PM, Christian Borntraeger wrote:
> Am 15.09.2015 um 18:27 schrieb Paolo Bonzini:
>> This new statistic can help diagnosing VCPUs that, for any reason,
>> trigger bad behavior of halt_poll_ns autotuning.
>>
>> For example, say halt_poll_ns = 480000, and wakeups are spaced exactly
>> like 479us, 481us, 479us, 481us. Then KVM always fails polling and wastes
>> 10+20+40+80+160+320+480 = 1110 microseconds out of every
>> 479+481+479+481+479+481+479 = 3359 microseconds. The VCPU then

For the first 481 us, block_ns should be 481us, block_ns > 
halt_poll_ns(480us) and long halt is detected, the vcpu->halt_poll_ns 
will be shrinked.

>> is consuming about 30% more CPU than it would use without
>> polling.  This would show as an abnormally high number of
>> attempted polling compared to the successful polls.
>>
>> Cc: Christian Borntraeger <borntraeger@de.ibm.com<
>> Cc: David Matlack <dmatlack@google.com>
>> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> Acked-by: Christian Borntraeger <borntraeger@de.ibm.com>
>
> yes, this will help to detect some bad cases, but not all.
>
> PS:
> upstream maintenance keeps me really busy at the moment :-)
> I am looking into a case right now, where auto polling goes
> completely nuts on my system:
>
> guest1: 8vcpus		guest2: 1 vcpu
> iperf with 25 process (-P25) from guest1 to guest2.
>
> I/O interrupts on s390 are floating (pending on all CPUs) so on
> ALL VCPUs that go to sleep, polling will consider any pending
> network interrupt as successful poll. So with auto polling the
> guest consumes up to 5 host CPUs without auto polling only 1.
> Reducing  halt_poll_ns to 100000 seems to work (goes back to
> 1 cpu).
>
> The proper way might be to feedback the result of the
> interrupt dequeue into the heuristics. Don't know yet how
> to handle that properly.

If this can be reproduced on x86 platform?

Regards,
Wanpeng Li
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1226883

> Am 15.09.2015 um 18:27 schrieb Paolo Bonzini:
> > This new statistic can help diagnosing VCPUs that, for any reason,
> > trigger bad behavior of halt_poll_ns autotuning.
> > 
> > For example, say halt_poll_ns = 480000, and wakeups are spaced exactly
> > like 479us, 481us, 479us, 481us. Then KVM always fails polling and wastes
> > 10+20+40+80+160+320+480 = 1110 microseconds out of every
> > 479+481+479+481+479+481+479 = 3359 microseconds. The VCPU then
> > is consuming about 30% more CPU than it would use without
> > polling.  This would show as an abnormally high number of
> > attempted polling compared to the successful polls.
> > 
> > Cc: Christian Borntraeger <borntraeger@de.ibm.com<
> > Cc: David Matlack <dmatlack@google.com>
> > Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> 
> Acked-by: Christian Borntraeger <borntraeger@de.ibm.com>
> 
> yes, this will help to detect some bad cases, but not all.
> 
> PS: 
> upstream maintenance keeps me really busy at the moment :-)
> I am looking into a case right now, where auto polling goes 
> completely nuts on my system:
> 
> guest1: 8vcpus		guest2: 1 vcpu
> iperf with 25 process (-P25) from guest1 to guest2.
> 
> I/O interrupts on s390 are floating (pending on all CPUs) so on 
> ALL VCPUs that go to sleep, polling will consider any pending
> network interrupt as successful poll. So with auto polling the
> guest consumes up to 5 host CPUs without auto polling only 1.
> Reducing  halt_poll_ns to 100000 seems to work (goes back to 
> 1 cpu).
> 
> The proper way might be to feedback the result of the
> interrupt dequeue into the heuristics. Don't know yet how
> to handle that properly.
> 
> Christian

I think the main problem is that we have two different kinds of wakeups, and
they can't be properly reported for now. "runnability" says nothing about the
"reason".

a) "forced wakeup" - "local workload"
- "local" interrupt/timer pending
- signal
- (request bits, nested irqs ... for some archs)
-> Impacts halt_poll_ns

b) "trial wakeup" - "floating" workload
-> floating interrupts
Another vcpu might be faster and dequeue the floating interrupt.
However, if we have a high I/O load, we want all VCPUs to reduce their
halt_poll_ns value. Special cases would also be:
- Only one VCPU in the system
- Only one VCPU running
- Only one VCPU that is enabled for this type of interrupt
-> Impacts halt_poll_ns only partially

So kvm_arch_vcpu_runnable() returns true for multiple
VCPUs, although only one might get the interrupt. If we could change that
internally (one VCPU reserving that interrupt), we might get this working out of
the box. As kvm_arch_vcpu_runnable is on the hot path and also called from other
VCPUs, this isn't that trivial. Will play with it. Until then, I'll prepare a
patch to disable it for s390x, just as Paolo suggested.

David

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web