Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1270480 > unrolled thread

Re: [PATCH] KVM: x86: Add lowest-priority support for vt-d posted-interrupts

Started byRadim Krčmář <rkrcmar@redhat.com>
First post2015-11-16 20:10 +0100
Last post2015-11-26 15:10 +0100
Articles 16 — 4 participants

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH] KVM: x86: Add lowest-priority support for vt-d  posted-interrupts Radim Krčmář <rkrcmar@redhat.com> - 2015-11-16 20:10 +0100
    Re: [PATCH] KVM: x86: Add lowest-priority support for vt-d  posted-interrupts Paolo Bonzini <pbonzini@redhat.com> - 2015-11-17 10:50 +0100
      RE: [PATCH] KVM: x86: Add lowest-priority support for vt-d  posted-interrupts "Wu, Feng" <feng.wu@intel.com> - 2015-11-24 02:30 +0100
        Re: [PATCH] KVM: x86: Add lowest-priority support for vt-d  posted-interrupts Radim Krcmár <rkrcmar@redhat.com> - 2015-11-24 15:40 +0100
          Re: [PATCH] KVM: x86: Add lowest-priority support for vt-d  posted-interrupts Paolo Bonzini <pbonzini@redhat.com> - 2015-11-24 15:40 +0100
            RE: [PATCH] KVM: x86: Add lowest-priority support for vt-d  posted-interrupts "Wu, Feng" <feng.wu@intel.com> - 2015-11-25 03:00 +0100
              Re: [PATCH] KVM: x86: Add lowest-priority support for vt-d  posted-interrupts Paolo Bonzini <pbonzini@redhat.com> - 2015-11-25 12:40 +0100
    RE: [PATCH] KVM: x86: Add lowest-priority support for vt-d  posted-interrupts "Wu, Feng" <feng.wu@intel.com> - 2015-11-24 02:30 +0100
      Re: [PATCH] KVM: x86: Add lowest-priority support for vt-d  posted-interrupts Radim Krčmář <rkrcmar@redhat.com> - 2015-11-24 15:40 +0100
        Re: [PATCH] KVM: x86: Add lowest-priority support for vt-d  posted-interrupts Radim Krčmář <rkrcmar@redhat.com> - 2015-11-24 15:50 +0100
        RE: [PATCH] KVM: x86: Add lowest-priority support for vt-d  posted-interrupts "Wu, Feng" <feng.wu@intel.com> - 2015-11-25 04:30 +0100
          Re: [PATCH] KVM: x86: Add lowest-priority support for vt-d  posted-interrupts Radim Krcmár <rkrcmar@redhat.com> - 2015-11-25 15:20 +0100
            Re: [PATCH] KVM: x86: Add lowest-priority support for vt-d  posted-interrupts Paolo Bonzini <pbonzini@redhat.com> - 2015-11-25 15:40 +0100
              Re: [PATCH] KVM: x86: Add lowest-priority support for vt-d  posted-interrupts Radim Krčmář <rkrcmar@redhat.com> - 2015-11-25 16:50 +0100
                RE: [PATCH] KVM: x86: Add lowest-priority support for vt-d  posted-interrupts "Wu, Feng" <feng.wu@intel.com> - 2015-11-26 07:30 +0100
                  Re: [PATCH] KVM: x86: Add lowest-priority support for vt-d  posted-interrupts Radim Krcmár <rkrcmar@redhat.com> - 2015-11-26 15:10 +0100

#1270480 — Re: [PATCH] KVM: x86: Add lowest-priority support for vt-d posted-interrupts

FromRadim Krčmář <rkrcmar@redhat.com>
Date2015-11-16 20:10 +0100
SubjectRe: [PATCH] KVM: x86: Add lowest-priority support for vt-d posted-interrupts
Message-ID<qvxk6-2PK-13@gated-at.bofh.it>
2015-11-09 10:46+0800, Feng Wu:
> Use vector-hashing to handle lowest-priority interrupts for
> posted-interrupts. As an example, modern Intel CPUs use this
> method to handle lowest-priority interrupts.

(I don't think it's a good idea that the algorithm differs from non-PI
 lowest priority delivery.  I'd make them both vector-hashing, which
 would be "fun" to explain to people expecting round robin ...)

> Signed-off-by: Feng Wu <feng.wu@intel.com>
> ---
> diff --git a/arch/x86/kvm/irq_comm.c b/arch/x86/kvm/irq_comm.c
> +/*
> + * This routine handles lowest-priority interrupts using vector-hashing
> + * mechanism. As an example, modern Intel CPUs use this method to handle
> + * lowest-priority interrupts.
> + *
> + * Here is the details about the vector-hashing mechanism:
> + * 1. For lowest-priority interrupts, store all the possible destination
> + *    vCPUs in an array.
> + * 2. Use "guest vector % max number of destination vCPUs" to find the right
> + *    destination vCPU in the array for the lowest-priority interrupt.
> + */

(Is Skylake i7-6700 a modern Intel CPU?
 I didn't manage to get hashing ... all interrupts always went to the
 lowest APIC ID in the set :/
 Is there a simple way to verify the algorithm?)

> +struct kvm_vcpu *kvm_intr_vector_hashing_dest(struct kvm *kvm,
> +					      struct kvm_lapic_irq *irq)
> +
> +{
> +	unsigned long dest_vcpu_bitmap[BITS_TO_LONGS(KVM_MAX_VCPUS)];
> +	unsigned int dest_vcpus = 0;
> +	struct kvm_vcpu *vcpu;
> +	unsigned int i, mod, idx = 0;
> +
> +	vcpu = kvm_intr_vector_hashing_dest_fast(kvm, irq);
> +	if (vcpu)
> +		return vcpu;

I think the rest of this function shouldn't be implemented:
 - Shorthands are only for IPIs and hence don't need to be handled,
 - Lowest priority physical broadcast is not supported,
 - Lowest priority cluster logical broadcast is not supported,
 - No point in optimizing mixed xAPIC and x2APIC mode,
 - The rest is handled by kvm_intr_vector_hashing_dest_fast().
   (Even lowest priority flat logical "broadcast".)
 - We do the work twice when vcpu == NULL means that there is no
   matching destination.

Is there a valid case that can be resolved by going through all vcpus?

> +
> +	memset(dest_vcpu_bitmap, 0, sizeof(dest_vcpu_bitmap));
> +
> +	kvm_for_each_vcpu(i, vcpu, kvm) {
> +		if (!kvm_apic_present(vcpu))
> +			continue;
> +
> +		if (!kvm_apic_match_dest(vcpu, NULL, irq->shorthand,
> +					irq->dest_id, irq->dest_mode))
> +			continue;
> +
> +		__set_bit(vcpu->vcpu_id, dest_vcpu_bitmap);
> +		dest_vcpus++;
> +	}
> +
> +	if (dest_vcpus == 0)
> +		return NULL;
> +
> +	mod = irq->vector % dest_vcpus;
> +
> +	for (i = 0; i <= mod; i++) {
> +		idx = find_next_bit(dest_vcpu_bitmap, KVM_MAX_VCPUS, idx) + 1;
> +		BUG_ON(idx >= KVM_MAX_VCPUS);
> +	}
> +
> +	return kvm_get_vcpu(kvm, idx - 1);
> +}
> +EXPORT_SYMBOL_GPL(kvm_intr_vector_hashing_dest);
> +
> diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c
> @@ -816,6 +816,63 @@ out:
> +struct kvm_vcpu *kvm_intr_vector_hashing_dest_fast(struct kvm *kvm,
> +						   struct kvm_lapic_irq *irq)

We now have three very similar functions :(

  kvm_irq_delivery_to_apic_fast
  kvm_intr_is_single_vcpu_fast
  kvm_intr_vector_hashing_dest_fast

By utilizing the gcc optimizer, they can be merged without introducing
many instructions to the hot path, kvm_irq_delivery_to_apic_fast.
(I would eventually do it, so you can save time by ignoring this.)

Thanks.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1271017

FromPaolo Bonzini <pbonzini@redhat.com>
Date2015-11-17 10:50 +0100
Message-ID<qvL3I-3gw-21@gated-at.bofh.it>
In reply to#1270480

On 16/11/2015 20:03, Radim Krčmář wrote:
> 2015-11-09 10:46+0800, Feng Wu:
>> Use vector-hashing to handle lowest-priority interrupts for
>> posted-interrupts. As an example, modern Intel CPUs use this
>> method to handle lowest-priority interrupts.
> 
> (I don't think it's a good idea that the algorithm differs from non-PI
>  lowest priority delivery.  I'd make them both vector-hashing, which
>  would be "fun" to explain to people expecting round robin ...)

Yup, I would make it a module option.  Thanks very much Radim for
helping with the review.

Paolo

>> Signed-off-by: Feng Wu <feng.wu@intel.com>
>> ---
>> diff --git a/arch/x86/kvm/irq_comm.c b/arch/x86/kvm/irq_comm.c
>> +/*
>> + * This routine handles lowest-priority interrupts using vector-hashing
>> + * mechanism. As an example, modern Intel CPUs use this method to handle
>> + * lowest-priority interrupts.
>> + *
>> + * Here is the details about the vector-hashing mechanism:
>> + * 1. For lowest-priority interrupts, store all the possible destination
>> + *    vCPUs in an array.
>> + * 2. Use "guest vector % max number of destination vCPUs" to find the right
>> + *    destination vCPU in the array for the lowest-priority interrupt.
>> + */
> 
> (Is Skylake i7-6700 a modern Intel CPU?
>  I didn't manage to get hashing ... all interrupts always went to the
>  lowest APIC ID in the set :/
>  Is there a simple way to verify the algorithm?)
> 
>> +struct kvm_vcpu *kvm_intr_vector_hashing_dest(struct kvm *kvm,
>> +					      struct kvm_lapic_irq *irq)
>> +
>> +{
>> +	unsigned long dest_vcpu_bitmap[BITS_TO_LONGS(KVM_MAX_VCPUS)];
>> +	unsigned int dest_vcpus = 0;
>> +	struct kvm_vcpu *vcpu;
>> +	unsigned int i, mod, idx = 0;
>> +
>> +	vcpu = kvm_intr_vector_hashing_dest_fast(kvm, irq);
>> +	if (vcpu)
>> +		return vcpu;
> 
> I think the rest of this function shouldn't be implemented:
>  - Shorthands are only for IPIs and hence don't need to be handled,
>  - Lowest priority physical broadcast is not supported,
>  - Lowest priority cluster logical broadcast is not supported,
>  - No point in optimizing mixed xAPIC and x2APIC mode,
>  - The rest is handled by kvm_intr_vector_hashing_dest_fast().
>    (Even lowest priority flat logical "broadcast".)
>  - We do the work twice when vcpu == NULL means that there is no
>    matching destination.
> 
> Is there a valid case that can be resolved by going through all vcpus?
> 
>> +
>> +	memset(dest_vcpu_bitmap, 0, sizeof(dest_vcpu_bitmap));
>> +
>> +	kvm_for_each_vcpu(i, vcpu, kvm) {
>> +		if (!kvm_apic_present(vcpu))
>> +			continue;
>> +
>> +		if (!kvm_apic_match_dest(vcpu, NULL, irq->shorthand,
>> +					irq->dest_id, irq->dest_mode))
>> +			continue;
>> +
>> +		__set_bit(vcpu->vcpu_id, dest_vcpu_bitmap);
>> +		dest_vcpus++;
>> +	}
>> +
>> +	if (dest_vcpus == 0)
>> +		return NULL;
>> +
>> +	mod = irq->vector % dest_vcpus;
>> +
>> +	for (i = 0; i <= mod; i++) {
>> +		idx = find_next_bit(dest_vcpu_bitmap, KVM_MAX_VCPUS, idx) + 1;
>> +		BUG_ON(idx >= KVM_MAX_VCPUS);
>> +	}
>> +
>> +	return kvm_get_vcpu(kvm, idx - 1);
>> +}
>> +EXPORT_SYMBOL_GPL(kvm_intr_vector_hashing_dest);
>> +
>> diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c
>> @@ -816,6 +816,63 @@ out:
>> +struct kvm_vcpu *kvm_intr_vector_hashing_dest_fast(struct kvm *kvm,
>> +						   struct kvm_lapic_irq *irq)
> 
> We now have three very similar functions :(
> 
>   kvm_irq_delivery_to_apic_fast
>   kvm_intr_is_single_vcpu_fast
>   kvm_intr_vector_hashing_dest_fast
> 
> By utilizing the gcc optimizer, they can be merged without introducing
> many instructions to the hot path, kvm_irq_delivery_to_apic_fast.
> (I would eventually do it, so you can save time by ignoring this.)
> 
> Thanks.
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1276003

From"Wu, Feng" <feng.wu@intel.com>
Date2015-11-24 02:30 +0100
Message-ID<qyaAG-1iQ-1@gated-at.bofh.it>
In reply to#1271017
DQoNCj4gLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gRnJvbTogUGFvbG8gQm9uemluaSBb
bWFpbHRvOnBib256aW5pQHJlZGhhdC5jb21dDQo+IFNlbnQ6IFR1ZXNkYXksIE5vdmVtYmVyIDE3
LCAyMDE1IDU6NDEgUE0NCj4gVG86IFJhZGltIEtyxI1tw6HFmSA8cmtyY21hckByZWRoYXQuY29t
PjsgV3UsIEZlbmcgPGZlbmcud3VAaW50ZWwuY29tPg0KPiBDYzoga3ZtQHZnZXIua2VybmVsLm9y
ZzsgbGludXgta2VybmVsQHZnZXIua2VybmVsLm9yZw0KPiBTdWJqZWN0OiBSZTogW1BBVENIXSBL
Vk06IHg4NjogQWRkIGxvd2VzdC1wcmlvcml0eSBzdXBwb3J0IGZvciB2dC1kIHBvc3RlZC0NCj4g
aW50ZXJydXB0cw0KPiANCj4gDQo+IA0KPiBPbiAxNi8xMS8yMDE1IDIwOjAzLCBSYWRpbSBLcsSN
bcOhxZkgd3JvdGU6DQo+ID4gMjAxNS0xMS0wOSAxMDo0NiswODAwLCBGZW5nIFd1Og0KPiA+PiBV
c2UgdmVjdG9yLWhhc2hpbmcgdG8gaGFuZGxlIGxvd2VzdC1wcmlvcml0eSBpbnRlcnJ1cHRzIGZv
cg0KPiA+PiBwb3N0ZWQtaW50ZXJydXB0cy4gQXMgYW4gZXhhbXBsZSwgbW9kZXJuIEludGVsIENQ
VXMgdXNlIHRoaXMNCj4gPj4gbWV0aG9kIHRvIGhhbmRsZSBsb3dlc3QtcHJpb3JpdHkgaW50ZXJy
dXB0cy4NCj4gPg0KPiA+IChJIGRvbid0IHRoaW5rIGl0J3MgYSBnb29kIGlkZWEgdGhhdCB0aGUg
YWxnb3JpdGhtIGRpZmZlcnMgZnJvbSBub24tUEkNCj4gPiAgbG93ZXN0IHByaW9yaXR5IGRlbGl2
ZXJ5LiAgSSdkIG1ha2UgdGhlbSBib3RoIHZlY3Rvci1oYXNoaW5nLCB3aGljaA0KPiA+ICB3b3Vs
ZCBiZSAiZnVuIiB0byBleHBsYWluIHRvIHBlb3BsZSBleHBlY3Rpbmcgcm91bmQgcm9iaW4gLi4u
KQ0KPiANCj4gWXVwLCBJIHdvdWxkIG1ha2UgaXQgYSBtb2R1bGUgb3B0aW9uLiAgVGhhbmtzIHZl
cnkgbXVjaCBSYWRpbSBmb3INCj4gaGVscGluZyB3aXRoIHRoZSByZXZpZXcuDQoNClRoYW5rcyBm
b3IgeW91ciBndXlzJyByZXZpZXcuIFllcywgd2UgY2FuIGludHJvZHVjZSBhIG1vZHVsZSBvcHRp
b24NCmZvciBpdC4gQWNjb3JkaW5nIHRvIFJhZGltJ3MgY29tbWVudHMgYWJvdmUsIHdlIG5lZWQg
dXNlIHRoZQ0Kc2FtZSBwb2xpY3kgZm9yIFBJIGFuZCBub24tUEkgbG93ZXN0LXByaW9yaXR5IGlu
dGVycnVwdHMsIHNvIGhlcmUgaXMgdGhlDQpxdWVzdGlvbjogZm9yIHZlY3RvciBoYXNoaW5nLCBp
dCBpcyBlYXN5IHRvIGFwcGx5IGl0IGZvciBib3RoIG5vbi1QSSBhbmQgUEkNCmNhc2UsIGhvd2V2
ZXIsIGZvciBSb3VuZC1Sb2JpbiwgaW4gbm9uLVBJIGNhc2UsIHRoZSByb3VuZCByb2JpbiBjb3Vu
dGVyDQppcyB1c2VkIGFuZCB1cGRhdGVkIHdoZW4gdGhlIGludGVycnVwdCBpcyBpbmplY3RlZCB0
byBndWVzdCwgYnV0IGZvcg0KUEkgY2FzZSwgdGhlIGludGVycnVwdCBpcyBpbmplY3RlZCB0byBn
dWVzdCB0b3RhbGx5IGJ5IGhhcmR3YXJlLCBzb2Z0d2FyZQ0KY2Fubm90IGNvbnRyb2wgaXQgd2hp
bGUgaW50ZXJydXB0IGRlbGl2ZXJ5LCB3ZSBjYW4gb25seSBkZWNpZGUgdGhlDQpkZXN0aW5hdGlv
biB2Q1BVIGZvciB0aGUgUEkgaW50ZXJydXB0IGluIHRoZSBpbml0aWFsIGNvbmZpZ3VyYXRpb24N
CnRpbWUgKGd1ZXN0IHVwZGF0ZSB2TVNJIC0+IFFFTVUgLT4gS1ZNKS4gRG8geW91IGd1eXMgaGF2
ZSBhbnkgZ29vZA0Kc3VnZ2VzdGlvbiB0byBkbyByb3VuZCByb2JpbiBmb3IgUEkgbG93ZXN0LXBy
aW9yaXR5PyBTZWVtcyBSb3VuZCByb2Jpbg0KaXMgbm90IGEgZ29vZCB3YXkgZm9yIFBJIGxvd2Vz
dC1wcmlvcml0eSBpbnRlcnJ1cHRzLiBBbnkgY29tbWVudHMNCmFyZSBhcHByZWNpYXRlZCENCg0K
VGhhbmtzLA0KRmVuZw0KDQoNCj4gDQo+IFBhb2xvDQo+IA0KPiA+PiBTaWduZWQtb2ZmLWJ5OiBG
ZW5nIFd1IDxmZW5nLnd1QGludGVsLmNvbT4NCj4gPj4gLS0tDQo+ID4+IGRpZmYgLS1naXQgYS9h
cmNoL3g4Ni9rdm0vaXJxX2NvbW0uYyBiL2FyY2gveDg2L2t2bS9pcnFfY29tbS5jDQo+ID4+ICsv
Kg0KPiA+PiArICogVGhpcyByb3V0aW5lIGhhbmRsZXMgbG93ZXN0LXByaW9yaXR5IGludGVycnVw
dHMgdXNpbmcgdmVjdG9yLWhhc2hpbmcNCj4gPj4gKyAqIG1lY2hhbmlzbS4gQXMgYW4gZXhhbXBs
ZSwgbW9kZXJuIEludGVsIENQVXMgdXNlIHRoaXMgbWV0aG9kIHRvDQo+IGhhbmRsZQ0KPiA+PiAr
ICogbG93ZXN0LXByaW9yaXR5IGludGVycnVwdHMuDQo+ID4+ICsgKg0KPiA+PiArICogSGVyZSBp
cyB0aGUgZGV0YWlscyBhYm91dCB0aGUgdmVjdG9yLWhhc2hpbmcgbWVjaGFuaXNtOg0KPiA+PiAr
ICogMS4gRm9yIGxvd2VzdC1wcmlvcml0eSBpbnRlcnJ1cHRzLCBzdG9yZSBhbGwgdGhlIHBvc3Np
YmxlIGRlc3RpbmF0aW9uDQo+ID4+ICsgKiAgICB2Q1BVcyBpbiBhbiBhcnJheS4NCj4gPj4gKyAq
IDIuIFVzZSAiZ3Vlc3QgdmVjdG9yICUgbWF4IG51bWJlciBvZiBkZXN0aW5hdGlvbiB2Q1BVcyIg
dG8gZmluZCB0aGUNCj4gcmlnaHQNCj4gPj4gKyAqICAgIGRlc3RpbmF0aW9uIHZDUFUgaW4gdGhl
IGFycmF5IGZvciB0aGUgbG93ZXN0LXByaW9yaXR5IGludGVycnVwdC4NCj4gPj4gKyAqLw0KPiA+
DQo+ID4gKElzIFNreWxha2UgaTctNjcwMCBhIG1vZGVybiBJbnRlbCBDUFU/DQo+ID4gIEkgZGlk
bid0IG1hbmFnZSB0byBnZXQgaGFzaGluZyAuLi4gYWxsIGludGVycnVwdHMgYWx3YXlzIHdlbnQg
dG8gdGhlDQo+ID4gIGxvd2VzdCBBUElDIElEIGluIHRoZSBzZXQgOi8NCj4gPiAgSXMgdGhlcmUg
YSBzaW1wbGUgd2F5IHRvIHZlcmlmeSB0aGUgYWxnb3JpdGhtPykNCj4gPg0KPiA+PiArc3RydWN0
IGt2bV92Y3B1ICprdm1faW50cl92ZWN0b3JfaGFzaGluZ19kZXN0KHN0cnVjdCBrdm0gKmt2bSwN
Cj4gPj4gKwkJCQkJICAgICAgc3RydWN0IGt2bV9sYXBpY19pcnEgKmlycSkNCj4gPj4gKw0KPiA+
PiArew0KPiA+PiArCXVuc2lnbmVkIGxvbmcNCj4gZGVzdF92Y3B1X2JpdG1hcFtCSVRTX1RPX0xP
TkdTKEtWTV9NQVhfVkNQVVMpXTsNCj4gPj4gKwl1bnNpZ25lZCBpbnQgZGVzdF92Y3B1cyA9IDA7
DQo+ID4+ICsJc3RydWN0IGt2bV92Y3B1ICp2Y3B1Ow0KPiA+PiArCXVuc2lnbmVkIGludCBpLCBt
b2QsIGlkeCA9IDA7DQo+ID4+ICsNCj4gPj4gKwl2Y3B1ID0ga3ZtX2ludHJfdmVjdG9yX2hhc2hp
bmdfZGVzdF9mYXN0KGt2bSwgaXJxKTsNCj4gPj4gKwlpZiAodmNwdSkNCj4gPj4gKwkJcmV0dXJu
IHZjcHU7DQo+ID4NCj4gPiBJIHRoaW5rIHRoZSByZXN0IG9mIHRoaXMgZnVuY3Rpb24gc2hvdWxk
bid0IGJlIGltcGxlbWVudGVkOg0KPiA+ICAtIFNob3J0aGFuZHMgYXJlIG9ubHkgZm9yIElQSXMg
YW5kIGhlbmNlIGRvbid0IG5lZWQgdG8gYmUgaGFuZGxlZCwNCj4gPiAgLSBMb3dlc3QgcHJpb3Jp
dHkgcGh5c2ljYWwgYnJvYWRjYXN0IGlzIG5vdCBzdXBwb3J0ZWQsDQo+ID4gIC0gTG93ZXN0IHBy
aW9yaXR5IGNsdXN0ZXIgbG9naWNhbCBicm9hZGNhc3QgaXMgbm90IHN1cHBvcnRlZCwNCj4gPiAg
LSBObyBwb2ludCBpbiBvcHRpbWl6aW5nIG1peGVkIHhBUElDIGFuZCB4MkFQSUMgbW9kZSwNCj4g
PiAgLSBUaGUgcmVzdCBpcyBoYW5kbGVkIGJ5IGt2bV9pbnRyX3ZlY3Rvcl9oYXNoaW5nX2Rlc3Rf
ZmFzdCgpLg0KPiA+ICAgIChFdmVuIGxvd2VzdCBwcmlvcml0eSBmbGF0IGxvZ2ljYWwgImJyb2Fk
Y2FzdCIuKQ0KPiA+ICAtIFdlIGRvIHRoZSB3b3JrIHR3aWNlIHdoZW4gdmNwdSA9PSBOVUxMIG1l
YW5zIHRoYXQgdGhlcmUgaXMgbm8NCj4gPiAgICBtYXRjaGluZyBkZXN0aW5hdGlvbi4NCj4gPg0K
PiA+IElzIHRoZXJlIGEgdmFsaWQgY2FzZSB0aGF0IGNhbiBiZSByZXNvbHZlZCBieSBnb2luZyB0
aHJvdWdoIGFsbCB2Y3B1cz8NCj4gPg0KPiA+PiArDQo+ID4+ICsJbWVtc2V0KGRlc3RfdmNwdV9i
aXRtYXAsIDAsIHNpemVvZihkZXN0X3ZjcHVfYml0bWFwKSk7DQo+ID4+ICsNCj4gPj4gKwlrdm1f
Zm9yX2VhY2hfdmNwdShpLCB2Y3B1LCBrdm0pIHsNCj4gPj4gKwkJaWYgKCFrdm1fYXBpY19wcmVz
ZW50KHZjcHUpKQ0KPiA+PiArCQkJY29udGludWU7DQo+ID4+ICsNCj4gPj4gKwkJaWYgKCFrdm1f
YXBpY19tYXRjaF9kZXN0KHZjcHUsIE5VTEwsIGlycS0+c2hvcnRoYW5kLA0KPiA+PiArCQkJCQlp
cnEtPmRlc3RfaWQsIGlycS0+ZGVzdF9tb2RlKSkNCj4gPj4gKwkJCWNvbnRpbnVlOw0KPiA+PiAr
DQo+ID4+ICsJCV9fc2V0X2JpdCh2Y3B1LT52Y3B1X2lkLCBkZXN0X3ZjcHVfYml0bWFwKTsNCj4g
Pj4gKwkJZGVzdF92Y3B1cysrOw0KPiA+PiArCX0NCj4gPj4gKw0KPiA+PiArCWlmIChkZXN0X3Zj
cHVzID09IDApDQo+ID4+ICsJCXJldHVybiBOVUxMOw0KPiA+PiArDQo+ID4+ICsJbW9kID0gaXJx
LT52ZWN0b3IgJSBkZXN0X3ZjcHVzOw0KPiA+PiArDQo+ID4+ICsJZm9yIChpID0gMDsgaSA8PSBt
b2Q7IGkrKykgew0KPiA+PiArCQlpZHggPSBmaW5kX25leHRfYml0KGRlc3RfdmNwdV9iaXRtYXAs
IEtWTV9NQVhfVkNQVVMsDQo+IGlkeCkgKyAxOw0KPiA+PiArCQlCVUdfT04oaWR4ID49IEtWTV9N
QVhfVkNQVVMpOw0KPiA+PiArCX0NCj4gPj4gKw0KPiA+PiArCXJldHVybiBrdm1fZ2V0X3ZjcHUo
a3ZtLCBpZHggLSAxKTsNCj4gPj4gK30NCj4gPj4gK0VYUE9SVF9TWU1CT0xfR1BMKGt2bV9pbnRy
X3ZlY3Rvcl9oYXNoaW5nX2Rlc3QpOw0KPiA+PiArDQo+ID4+IGRpZmYgLS1naXQgYS9hcmNoL3g4
Ni9rdm0vbGFwaWMuYyBiL2FyY2gveDg2L2t2bS9sYXBpYy5jDQo+ID4+IEBAIC04MTYsNiArODE2
LDYzIEBAIG91dDoNCj4gPj4gK3N0cnVjdCBrdm1fdmNwdSAqa3ZtX2ludHJfdmVjdG9yX2hhc2hp
bmdfZGVzdF9mYXN0KHN0cnVjdCBrdm0gKmt2bSwNCj4gPj4gKwkJCQkJCSAgIHN0cnVjdCBrdm1f
bGFwaWNfaXJxICppcnEpDQo+ID4NCj4gPiBXZSBub3cgaGF2ZSB0aHJlZSB2ZXJ5IHNpbWlsYXIg
ZnVuY3Rpb25zIDooDQo+ID4NCj4gPiAgIGt2bV9pcnFfZGVsaXZlcnlfdG9fYXBpY19mYXN0DQo+
ID4gICBrdm1faW50cl9pc19zaW5nbGVfdmNwdV9mYXN0DQo+ID4gICBrdm1faW50cl92ZWN0b3Jf
aGFzaGluZ19kZXN0X2Zhc3QNCj4gPg0KPiA+IEJ5IHV0aWxpemluZyB0aGUgZ2NjIG9wdGltaXpl
ciwgdGhleSBjYW4gYmUgbWVyZ2VkIHdpdGhvdXQgaW50cm9kdWNpbmcNCj4gPiBtYW55IGluc3Ry
dWN0aW9ucyB0byB0aGUgaG90IHBhdGgsIGt2bV9pcnFfZGVsaXZlcnlfdG9fYXBpY19mYXN0Lg0K
PiA+IChJIHdvdWxkIGV2ZW50dWFsbHkgZG8gaXQsIHNvIHlvdSBjYW4gc2F2ZSB0aW1lIGJ5IGln
bm9yaW5nIHRoaXMuKQ0KPiA+DQo+ID4gVGhhbmtzLg0KPiA+DQo=
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1276518

FromRadim Krcmár <rkrcmar@redhat.com>
Date2015-11-24 15:40 +0100
Message-ID<qymVb-Ro-13@gated-at.bofh.it>
In reply to#1276003
2015-11-24 01:26+0000, Wu, Feng:
>> From: Paolo Bonzini [mailto:pbonzini@redhat.com]
>> On 16/11/2015 20:03, Radim Krčmář wrote:
>> > 2015-11-09 10:46+0800, Feng Wu:
>> >> Use vector-hashing to handle lowest-priority interrupts for
>> >> posted-interrupts. As an example, modern Intel CPUs use this
>> >> method to handle lowest-priority interrupts.
>> >
>> > (I don't think it's a good idea that the algorithm differs from non-PI
>> >  lowest priority delivery.  I'd make them both vector-hashing, which
>> >  would be "fun" to explain to people expecting round robin ...)
>> 
>> Yup, I would make it a module option.  Thanks very much Radim for
>> helping with the review.
> 
> Thanks for your guys' review. Yes, we can introduce a module option
> for it. According to Radim's comments above, we need use the
> same policy for PI and non-PI lowest-priority interrupts, so here is the
> question: for vector hashing, it is easy to apply it for both non-PI and PI
> case, however, for Round-Robin, in non-PI case, the round robin counter
> is used and updated when the interrupt is injected to guest, but for
> PI case, the interrupt is injected to guest totally by hardware, software
> cannot control it while interrupt delivery, we can only decide the
> destination vCPU for the PI interrupt in the initial configuration
> time (guest update vMSI -> QEMU -> KVM). Do you guys have any good
> suggestion to do round robin for PI lowest-priority? Seems Round robin
> is not a good way for PI lowest-priority interrupts. Any comments
> are appreciated!

It's meaningless to try dynamic algorithms with PI so if we allow both
lowest priority algorithms, I'd let PI handle any lowest priority only
with vector hashing.  (It's an ugly compromise.)
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1276523

FromPaolo Bonzini <pbonzini@redhat.com>
Date2015-11-24 15:40 +0100
Message-ID<qymVc-Ro-33@gated-at.bofh.it>
In reply to#1276518

On 24/11/2015 15:35, Radim Krcmár wrote:
> > Thanks for your guys' review. Yes, we can introduce a module option
> > for it. According to Radim's comments above, we need use the
> > same policy for PI and non-PI lowest-priority interrupts, so here is the
> > question: for vector hashing, it is easy to apply it for both non-PI and PI
> > case, however, for Round-Robin, in non-PI case, the round robin counter
> > is used and updated when the interrupt is injected to guest, but for
> > PI case, the interrupt is injected to guest totally by hardware, software
> > cannot control it while interrupt delivery, we can only decide the
> > destination vCPU for the PI interrupt in the initial configuration
> > time (guest update vMSI -> QEMU -> KVM). Do you guys have any good
> > suggestion to do round robin for PI lowest-priority? Seems Round robin
> > is not a good way for PI lowest-priority interrupts. Any comments
> > are appreciated!
>
> It's meaningless to try dynamic algorithms with PI so if we allow both
> lowest priority algorithms, I'd let PI handle any lowest priority only
> with vector hashing.  (It's an ugly compromise.)

For now, I would just keep the 4.4 behavior, i.e. disable PI unless
there is a single destination || vector hashing is enabled.  We can flip
the switch later.

Paolo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1276964

From"Wu, Feng" <feng.wu@intel.com>
Date2015-11-25 03:00 +0100
Message-ID<qyxxg-7M5-9@gated-at.bofh.it>
In reply to#1276523
DQoNCj4gLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gRnJvbTogUGFvbG8gQm9uemluaSBb
bWFpbHRvOnBib256aW5pQHJlZGhhdC5jb21dDQo+IFNlbnQ6IFR1ZXNkYXksIE5vdmVtYmVyIDI0
LCAyMDE1IDEwOjM4IFBNDQo+IFRvOiBSYWRpbSBLcmNtw6FyIDxya3JjbWFyQHJlZGhhdC5jb20+
OyBXdSwgRmVuZyA8ZmVuZy53dUBpbnRlbC5jb20+DQo+IENjOiBrdm1Admdlci5rZXJuZWwub3Jn
OyBsaW51eC1rZXJuZWxAdmdlci5rZXJuZWwub3JnDQo+IFN1YmplY3Q6IFJlOiBbUEFUQ0hdIEtW
TTogeDg2OiBBZGQgbG93ZXN0LXByaW9yaXR5IHN1cHBvcnQgZm9yIHZ0LWQgcG9zdGVkLQ0KPiBp
bnRlcnJ1cHRzDQo+IA0KPiANCj4gDQo+IE9uIDI0LzExLzIwMTUgMTU6MzUsIFJhZGltIEtyY23D
oXIgd3JvdGU6DQo+ID4gPiBUaGFua3MgZm9yIHlvdXIgZ3V5cycgcmV2aWV3LiBZZXMsIHdlIGNh
biBpbnRyb2R1Y2UgYSBtb2R1bGUgb3B0aW9uDQo+ID4gPiBmb3IgaXQuIEFjY29yZGluZyB0byBS
YWRpbSdzIGNvbW1lbnRzIGFib3ZlLCB3ZSBuZWVkIHVzZSB0aGUNCj4gPiA+IHNhbWUgcG9saWN5
IGZvciBQSSBhbmQgbm9uLVBJIGxvd2VzdC1wcmlvcml0eSBpbnRlcnJ1cHRzLCBzbyBoZXJlIGlz
IHRoZQ0KPiA+ID4gcXVlc3Rpb246IGZvciB2ZWN0b3IgaGFzaGluZywgaXQgaXMgZWFzeSB0byBh
cHBseSBpdCBmb3IgYm90aCBub24tUEkgYW5kIFBJDQo+ID4gPiBjYXNlLCBob3dldmVyLCBmb3Ig
Um91bmQtUm9iaW4sIGluIG5vbi1QSSBjYXNlLCB0aGUgcm91bmQgcm9iaW4gY291bnRlcg0KPiA+
ID4gaXMgdXNlZCBhbmQgdXBkYXRlZCB3aGVuIHRoZSBpbnRlcnJ1cHQgaXMgaW5qZWN0ZWQgdG8g
Z3Vlc3QsIGJ1dCBmb3INCj4gPiA+IFBJIGNhc2UsIHRoZSBpbnRlcnJ1cHQgaXMgaW5qZWN0ZWQg
dG8gZ3Vlc3QgdG90YWxseSBieSBoYXJkd2FyZSwgc29mdHdhcmUNCj4gPiA+IGNhbm5vdCBjb250
cm9sIGl0IHdoaWxlIGludGVycnVwdCBkZWxpdmVyeSwgd2UgY2FuIG9ubHkgZGVjaWRlIHRoZQ0K
PiA+ID4gZGVzdGluYXRpb24gdkNQVSBmb3IgdGhlIFBJIGludGVycnVwdCBpbiB0aGUgaW5pdGlh
bCBjb25maWd1cmF0aW9uDQo+ID4gPiB0aW1lIChndWVzdCB1cGRhdGUgdk1TSSAtPiBRRU1VIC0+
IEtWTSkuIERvIHlvdSBndXlzIGhhdmUgYW55IGdvb2QNCj4gPiA+IHN1Z2dlc3Rpb24gdG8gZG8g
cm91bmQgcm9iaW4gZm9yIFBJIGxvd2VzdC1wcmlvcml0eT8gU2VlbXMgUm91bmQgcm9iaW4NCj4g
PiA+IGlzIG5vdCBhIGdvb2Qgd2F5IGZvciBQSSBsb3dlc3QtcHJpb3JpdHkgaW50ZXJydXB0cy4g
QW55IGNvbW1lbnRzDQo+ID4gPiBhcmUgYXBwcmVjaWF0ZWQhDQo+ID4NCj4gPiBJdCdzIG1lYW5p
bmdsZXNzIHRvIHRyeSBkeW5hbWljIGFsZ29yaXRobXMgd2l0aCBQSSBzbyBpZiB3ZSBhbGxvdyBi
b3RoDQo+ID4gbG93ZXN0IHByaW9yaXR5IGFsZ29yaXRobXMsIEknZCBsZXQgUEkgaGFuZGxlIGFu
eSBsb3dlc3QgcHJpb3JpdHkgb25seQ0KPiA+IHdpdGggdmVjdG9yIGhhc2hpbmcuICAoSXQncyBh
biB1Z2x5IGNvbXByb21pc2UuKQ0KPiANCj4gRm9yIG5vdywgSSB3b3VsZCBqdXN0IGtlZXAgdGhl
IDQuNCBiZWhhdmlvciwgaS5lLiBkaXNhYmxlIFBJIHVubGVzcw0KPiB0aGVyZSBpcyBhIHNpbmds
ZSBkZXN0aW5hdGlvbiB8fCB2ZWN0b3IgaGFzaGluZyBpcyBlbmFibGVkLiAgV2UgY2FuIGZsaXAN
Cj4gdGhlIHN3aXRjaCBsYXRlci4NCg0KT2theSwgbGV0IG1lIHRyeSB0byB1bmRlcnN0YW5kIHRo
aXMgY2xlYXJseToNCi0gV2Ugd2lsbCBoYXZlIGEgbmV3IEtWTSBjb21tYW5kIGxpbmUgcGFyYW1l
dGVyIHRvIGluZGljYXRlIHdoZXRoZXINCiAgdmVjdG9yIGhhc2hpbmcgaXMgZW5hYmxlZC4NCi0g
SWYgaXQgaXMgbm90IGVuYWJsZWQsIGZvciBQSSwgd2UgY2FuIG9ubHkgc3VwcG9ydCBzaW5nbGUg
ZGVzdGluYXRpb24gbG93ZXN0DQogIHByaW9yaXR5IGludGVycnVwdHMsIGZvciBub24tUEksIHdl
IGNvbnRpbnVlIHRvIHVzZSBSUi4NCi0gSWYgaXQgaXMgZW5hYmxlZCwgZm9yIFBJIGFuZCBub24t
UEkgd2UgdXNlIHZlY3RvciBoYXNoaW5nIGZvciBib3RoIG9mIHRoZW0uDQoNCklzIHRoaXMgdGhl
IGNhc2UgeW91IGhhdmUgaW4gbWluZD8gVGhhbmtzIGEgbG90IQ0KDQpUaGFua3MsDQpGZW5nDQoN
Cj4gDQo+IFBhb2xvDQo=
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1277265

FromPaolo Bonzini <pbonzini@redhat.com>
Date2015-11-25 12:40 +0100
Message-ID<qyGAz-5Fi-37@gated-at.bofh.it>
In reply to#1276964

On 25/11/2015 02:58, Wu, Feng wrote:
> Okay, let me try to understand this clearly:
> - We will have a new KVM command line parameter to indicate whether
>   vector hashing is enabled.
> - If it is not enabled, for PI, we can only support single destination lowest
>   priority interrupts, for non-PI, we continue to use RR.
> - If it is enabled, for PI and non-PI we use vector hashing for both of them.
> 
> Is this the case you have in mind? Thanks a lot!

Yes, thanks!

Paolo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1276004

From"Wu, Feng" <feng.wu@intel.com>
Date2015-11-24 02:30 +0100
Message-ID<qyaAG-1iQ-3@gated-at.bofh.it>
In reply to#1270480

> -----Original Message-----
> From: Radim Krčmář [mailto:rkrcmar@redhat.com]
> Sent: Tuesday, November 17, 2015 3:03 AM
> To: Wu, Feng <feng.wu@intel.com>
> Cc: pbonzini@redhat.com; kvm@vger.kernel.org; linux-
> kernel@vger.kernel.org
> Subject: Re: [PATCH] KVM: x86: Add lowest-priority support for vt-d posted-
> interrupts
> 
> 2015-11-09 10:46+0800, Feng Wu:
> > Use vector-hashing to handle lowest-priority interrupts for
> > posted-interrupts. As an example, modern Intel CPUs use this
> > method to handle lowest-priority interrupts.
> 
> (I don't think it's a good idea that the algorithm differs from non-PI
>  lowest priority delivery.  I'd make them both vector-hashing, which
>  would be "fun" to explain to people expecting round robin ...)
> 
> > Signed-off-by: Feng Wu <feng.wu@intel.com>
> > ---
> > diff --git a/arch/x86/kvm/irq_comm.c b/arch/x86/kvm/irq_comm.c
> > +/*
> > + * This routine handles lowest-priority interrupts using vector-hashing
> > + * mechanism. As an example, modern Intel CPUs use this method to
> handle
> > + * lowest-priority interrupts.
> > + *
> > + * Here is the details about the vector-hashing mechanism:
> > + * 1. For lowest-priority interrupts, store all the possible destination
> > + *    vCPUs in an array.
> > + * 2. Use "guest vector % max number of destination vCPUs" to find the
> right
> > + *    destination vCPU in the array for the lowest-priority interrupt.
> > + */
> 
> (Is Skylake i7-6700 a modern Intel CPU?
>  I didn't manage to get hashing ... all interrupts always went to the
>  lowest APIC ID in the set :/
>  Is there a simple way to verify the algorithm?)

Sorry for the late response, I try to get more information about vector
hashing before getting back to you. Here is the response from our
hardware architect:

"I don't think we do any vector hashing on our client parts.  This may be why the customer is not able to detect this on Skylake client silicon.
The vector hashing is micro-architectural and something we had done on server parts.

If you look at the haswell server CPU spec (https://www-ssl.intel.com/content/dam/www/public/us/en/documents/datasheets/xeon-e5-v3-datasheet-vol-2.pdf)
In section 4.1.2, you will see an IntControl register (this is a register controlled/configured by BIOS) - see below.

If you look at bits 6:4 in that register, you see the option we offer in hardware for what kind of redirection is applied to lowest priority interrupts.
There are three options:
1.	Fixed priority  
2.	Redirect last 
3.	Hash Vector

If picking vector hash, then bits 10:8 specifies the APIC-ID bits used for the hashing."

Thanks,
Feng


> 
> > +struct kvm_vcpu *kvm_intr_vector_hashing_dest(struct kvm *kvm,
> > +					      struct kvm_lapic_irq *irq)
> > +
> > +{
> > +	unsigned long
> dest_vcpu_bitmap[BITS_TO_LONGS(KVM_MAX_VCPUS)];
> > +	unsigned int dest_vcpus = 0;
> > +	struct kvm_vcpu *vcpu;
> > +	unsigned int i, mod, idx = 0;
> > +
> > +	vcpu = kvm_intr_vector_hashing_dest_fast(kvm, irq);
> > +	if (vcpu)
> > +		return vcpu;
> 
> I think the rest of this function shouldn't be implemented:
>  - Shorthands are only for IPIs and hence don't need to be handled,
>  - Lowest priority physical broadcast is not supported,
>  - Lowest priority cluster logical broadcast is not supported,
>  - No point in optimizing mixed xAPIC and x2APIC mode,
>  - The rest is handled by kvm_intr_vector_hashing_dest_fast().
>    (Even lowest priority flat logical "broadcast".)
>  - We do the work twice when vcpu == NULL means that there is no
>    matching destination.
> 
> Is there a valid case that can be resolved by going through all vcpus?
> 
> > +
> > +	memset(dest_vcpu_bitmap, 0, sizeof(dest_vcpu_bitmap));
> > +
> > +	kvm_for_each_vcpu(i, vcpu, kvm) {
> > +		if (!kvm_apic_present(vcpu))
> > +			continue;
> > +
> > +		if (!kvm_apic_match_dest(vcpu, NULL, irq->shorthand,
> > +					irq->dest_id, irq->dest_mode))
> > +			continue;
> > +
> > +		__set_bit(vcpu->vcpu_id, dest_vcpu_bitmap);
> > +		dest_vcpus++;
> > +	}
> > +
> > +	if (dest_vcpus == 0)
> > +		return NULL;
> > +
> > +	mod = irq->vector % dest_vcpus;
> > +
> > +	for (i = 0; i <= mod; i++) {
> > +		idx = find_next_bit(dest_vcpu_bitmap, KVM_MAX_VCPUS,
> idx) + 1;
> > +		BUG_ON(idx >= KVM_MAX_VCPUS);
> > +	}
> > +
> > +	return kvm_get_vcpu(kvm, idx - 1);
> > +}
> > +EXPORT_SYMBOL_GPL(kvm_intr_vector_hashing_dest);
> > +
> > diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c
> > @@ -816,6 +816,63 @@ out:
> > +struct kvm_vcpu *kvm_intr_vector_hashing_dest_fast(struct kvm *kvm,
> > +						   struct kvm_lapic_irq *irq)
> 
> We now have three very similar functions :(
> 
>   kvm_irq_delivery_to_apic_fast
>   kvm_intr_is_single_vcpu_fast
>   kvm_intr_vector_hashing_dest_fast
> 
> By utilizing the gcc optimizer, they can be merged without introducing
> many instructions to the hot path, kvm_irq_delivery_to_apic_fast.
> (I would eventually do it, so you can save time by ignoring this.)
> 
> Thanks.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1276514

FromRadim Krčmář <rkrcmar@redhat.com>
Date2015-11-24 15:40 +0100
Message-ID<qymVb-Ro-3@gated-at.bofh.it>
In reply to#1276004
2015-11-24 01:26+0000, Wu, Feng:
> "I don't think we do any vector hashing on our client parts.  This may be why the customer is not able to detect this on Skylake client silicon.
> The vector hashing is micro-architectural and something we had done on server parts.
> 
> If you look at the haswell server CPU spec (https://www-ssl.intel.com/content/dam/www/public/us/en/documents/datasheets/xeon-e5-v3-datasheet-vol-2.pdf)
> In section 4.1.2, you will see an IntControl register (this is a register controlled/configured by BIOS) - see below.

Thank you!

> If you look at bits 6:4 in that register, you see the option we offer in hardware for what kind of redirection is applied to lowest priority interrupts.
> There are three options:
> 1.	Fixed priority  
> 2.	Redirect last 
> 3.	Hash Vector
> 
> If picking vector hash, then bits 10:8 specifies the APIC-ID bits used for the hashing."

The hash function just interprets a subset of vector's bits as a number
and uses that as a starting offset in a search for an enabled APIC
within the destination set?

For example:
The x2APIC destination is 0x00000055 (= first four even APICs in cluster
0), the vector is 0b11100000, and bits 10:8 of IntControl are 000.

000 means that bits 7:4 of vector are selected, thus the vector hash is
0b1110 = 14, so the round-robin effectively does 14 % 4 (because we only
have 4 destinations) and delivers to the 3rd possible APIC (= ID 6)?
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1276528

FromRadim Krčmář <rkrcmar@redhat.com>
Date2015-11-24 15:50 +0100
Message-ID<qyn4R-Xh-1@gated-at.bofh.it>
In reply to#1276514
2015-11-24 15:31+0100, Radim Krčmář:
> 000 means that bits 7:4 of vector are selected, thus the vector hash is
> 0b1110 = 14, so the round-robin effectively does 14 % 4 (because we only
> have 4 destinations) and delivers to the 3rd possible APIC (= ID 6)?

Ah, 3rd APIC in the set has ID 4, of course :)
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1276992

From"Wu, Feng" <feng.wu@intel.com>
Date2015-11-25 04:30 +0100
Message-ID<qyyWl-v3-1@gated-at.bofh.it>
In reply to#1276514

> -----Original Message-----
> From: Radim Krčmář [mailto:rkrcmar@redhat.com]
> Sent: Tuesday, November 24, 2015 10:32 PM
> To: Wu, Feng <feng.wu@intel.com>
> Cc: pbonzini@redhat.com; kvm@vger.kernel.org; linux-
> kernel@vger.kernel.org
> Subject: Re: [PATCH] KVM: x86: Add lowest-priority support for vt-d posted-
> interrupts
> 
> 2015-11-24 01:26+0000, Wu, Feng:
> > "I don't think we do any vector hashing on our client parts.  This may be
> why the customer is not able to detect this on Skylake client silicon.
> > The vector hashing is micro-architectural and something we had done on
> server parts.
> >
> > If you look at the haswell server CPU spec (https://www-
> ssl.intel.com/content/dam/www/public/us/en/documents/datasheets/xeon-
> e5-v3-datasheet-vol-2.pdf)
> > In section 4.1.2, you will see an IntControl register (this is a register
> controlled/configured by BIOS) - see below.
> 
> Thank you!
> 
> > If you look at bits 6:4 in that register, you see the option we offer in
> hardware for what kind of redirection is applied to lowest priority interrupts.
> > There are three options:
> > 1.	Fixed priority
> > 2.	Redirect last
> > 3.	Hash Vector
> >
> > If picking vector hash, then bits 10:8 specifies the APIC-ID bits used for the
> hashing."
> 
> The hash function just interprets a subset of vector's bits as a number
> and uses that as a starting offset in a search for an enabled APIC
> within the destination set?
> 
> For example:
> The x2APIC destination is 0x00000055 (= first four even APICs in cluster
> 0), the vector is 0b11100000, and bits 10:8 of IntControl are 000.
> 
> 000 means that bits 7:4 of vector are selected, thus the vector hash is
> 0b1110 = 14, so the round-robin effectively does 14 % 4 (because we only
> have 4 destinations) and delivers to the 3rd possible APIC (= ID 6)?

In my current implementation, I don't select a subset of vector's bits as
the number, instead, I use the whole vector number. For software emulation
p. o. v, do we really need to select a subset of the vector's bits as the base
number? What is your opinion? Thanks a lot!

Thank,
Feng
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1277419

FromRadim Krcmár <rkrcmar@redhat.com>
Date2015-11-25 15:20 +0100
Message-ID<qyJ5o-7oe-17@gated-at.bofh.it>
In reply to#1276992
2015-11-25 03:21+0000, Wu, Feng:
> From: Radim Krčmář [mailto:rkrcmar@redhat.com]
>> The hash function just interprets a subset of vector's bits as a number
>> and uses that as a starting offset in a search for an enabled APIC
>> within the destination set?
>> 
>> For example:
>> The x2APIC destination is 0x00000055 (= first four even APICs in cluster
>> 0), the vector is 0b11100000, and bits 10:8 of IntControl are 000.
>> 
>> 000 means that bits 7:4 of vector are selected, thus the vector hash is
>> 0b1110 = 14, so the round-robin effectively does 14 % 4 (because we only
>> have 4 destinations) and delivers to the 3rd possible APIC (= ID 6)?
> 
> In my current implementation, I don't select a subset of vector's bits as
> the number, instead, I use the whole vector number. For software emulation
> p. o. v, do we really need to select a subset of the vector's bits as the base
> number? What is your opinion? Thanks a lot!

I think it's ok to pick any algorithm we like.  It's unlikely that
software would recognize and take advantage of the hardware algorithm
without adding a special treatment for KVM.
(I'd vote for the simple pick-first-APIC lowest priority algorithm ...
 I don't see much point in complicating lowest priority when it doesn't
 deliver to lowest priority CPU anyway.)

I mainly wanted to know what real hardware really does, because there is
a lot of alternatives that still fit into the Xeon documentation.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1277432

FromPaolo Bonzini <pbonzini@redhat.com>
Date2015-11-25 15:40 +0100
Message-ID<qyJoJ-7uW-7@gated-at.bofh.it>
In reply to#1277419

On 25/11/2015 15:12, Radim Krcmár wrote:
> I think it's ok to pick any algorithm we like.  It's unlikely that
> software would recognize and take advantage of the hardware algorithm
> without adding a special treatment for KVM.
> (I'd vote for the simple pick-first-APIC lowest priority algorithm ...
>  I don't see much point in complicating lowest priority when it doesn't
>  deliver to lowest priority CPU anyway.)

Vector hashing is an improvement for the common case where all vectors
are set to all CPUs.  Sure you can get an unlucky assignment, but it's
still better than pick-first-APIC.

Paolo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1277509

FromRadim Krčmář <rkrcmar@redhat.com>
Date2015-11-25 16:50 +0100
Message-ID<qyKuv-8c2-49@gated-at.bofh.it>
In reply to#1277432
2015-11-25 15:38+0100, Paolo Bonzini:
> On 25/11/2015 15:12, Radim Krcmár wrote:
>> I think it's ok to pick any algorithm we like.  It's unlikely that
>> software would recognize and take advantage of the hardware algorithm
>> without adding a special treatment for KVM.
>> (I'd vote for the simple pick-first-APIC lowest priority algorithm ...
>>  I don't see much point in complicating lowest priority when it doesn't
>>  deliver to lowest priority CPU anyway.)
> 
> Vector hashing is an improvement for the common case where all vectors
> are set to all CPUs.  Sure you can get an unlucky assignment, but it's
> still better than pick-first-APIC.

Yeah, hashing has a valid use case, but a subtle weighting of drawbacks
led me to prefer pick-first-APIC ...

(I'd prefer to have simple code in KVM and depend on static IRQ balancing
 in a guest to handle the distribution.
 The guest could get the unlucky assignment anyway, so it should be
 prepared;  and hashing just made KVM worse in that case.  Guests might
 also configure physical x(2)APIC, where is no lowest priority.
 And if the guest doesn't do anything with IRQs, then it might not even
 care about the impact that our choice has.)
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1277991

From"Wu, Feng" <feng.wu@intel.com>
Date2015-11-26 07:30 +0100
Message-ID<qyYe6-ZD-13@gated-at.bofh.it>
In reply to#1277509
DQoNCj4gLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gRnJvbTogUmFkaW0gS3LEjW3DocWZ
IFttYWlsdG86cmtyY21hckByZWRoYXQuY29tXQ0KPiBTZW50OiBXZWRuZXNkYXksIE5vdmVtYmVy
IDI1LCAyMDE1IDExOjQzIFBNDQo+IFRvOiBQYW9sbyBCb256aW5pIDxwYm9uemluaUByZWRoYXQu
Y29tPg0KPiBDYzogV3UsIEZlbmcgPGZlbmcud3VAaW50ZWwuY29tPjsga3ZtQHZnZXIua2VybmVs
Lm9yZzsgbGludXgtDQo+IGtlcm5lbEB2Z2VyLmtlcm5lbC5vcmcNCj4gU3ViamVjdDogUmU6IFtQ
QVRDSF0gS1ZNOiB4ODY6IEFkZCBsb3dlc3QtcHJpb3JpdHkgc3VwcG9ydCBmb3IgdnQtZCBwb3N0
ZWQtDQo+IGludGVycnVwdHMNCj4gDQo+IDIwMTUtMTEtMjUgMTU6MzgrMDEwMCwgUGFvbG8gQm9u
emluaToNCj4gPiBPbiAyNS8xMS8yMDE1IDE1OjEyLCBSYWRpbSBLcmNtw6FyIHdyb3RlOg0KPiA+
PiBJIHRoaW5rIGl0J3Mgb2sgdG8gcGljayBhbnkgYWxnb3JpdGhtIHdlIGxpa2UuICBJdCdzIHVu
bGlrZWx5IHRoYXQNCj4gPj4gc29mdHdhcmUgd291bGQgcmVjb2duaXplIGFuZCB0YWtlIGFkdmFu
dGFnZSBvZiB0aGUgaGFyZHdhcmUgYWxnb3JpdGhtDQo+ID4+IHdpdGhvdXQgYWRkaW5nIGEgc3Bl
Y2lhbCB0cmVhdG1lbnQgZm9yIEtWTS4NCj4gPj4gKEknZCB2b3RlIGZvciB0aGUgc2ltcGxlIHBp
Y2stZmlyc3QtQVBJQyBsb3dlc3QgcHJpb3JpdHkgYWxnb3JpdGhtIC4uLg0KPiA+PiAgSSBkb24n
dCBzZWUgbXVjaCBwb2ludCBpbiBjb21wbGljYXRpbmcgbG93ZXN0IHByaW9yaXR5IHdoZW4gaXQg
ZG9lc24ndA0KPiA+PiAgZGVsaXZlciB0byBsb3dlc3QgcHJpb3JpdHkgQ1BVIGFueXdheS4pDQo+
ID4NCj4gPiBWZWN0b3IgaGFzaGluZyBpcyBhbiBpbXByb3ZlbWVudCBmb3IgdGhlIGNvbW1vbiBj
YXNlIHdoZXJlIGFsbCB2ZWN0b3JzDQo+ID4gYXJlIHNldCB0byBhbGwgQ1BVcy4gIFN1cmUgeW91
IGNhbiBnZXQgYW4gdW5sdWNreSBhc3NpZ25tZW50LCBidXQgaXQncw0KPiA+IHN0aWxsIGJldHRl
ciB0aGFuIHBpY2stZmlyc3QtQVBJQy4NCj4gDQo+IFllYWgsIGhhc2hpbmcgaGFzIGEgdmFsaWQg
dXNlIGNhc2UsIGJ1dCBhIHN1YnRsZSB3ZWlnaHRpbmcgb2YgZHJhd2JhY2tzDQo+IGxlZCBtZSB0
byBwcmVmZXIgcGljay1maXJzdC1BUElDIC4uLg0KDQpJcyBpdCBwb3NzaWJsZSB0aGF0IHBpY2st
Zmlyc3QtQVBJQyBwb2xpY3kgbWFrZSBjZXJ0YWluIHZDUFUncyBpcnEgd29ya2xvYWQgdG9vDQpo
ZWF2eT8NCg0KPiANCj4gKEknZCBwcmVmZXIgdG8gaGF2ZSBzaW1wbGUgY29kZSBpbiBLVk0gYW5k
IGRlcGVuZCBvbiBzdGF0aWMgSVJRIGJhbGFuY2luZw0KPiAgaW4gYSBndWVzdCB0byBoYW5kbGUg
dGhlIGRpc3RyaWJ1dGlvbi4NCj4gIFRoZSBndWVzdCBjb3VsZCBnZXQgdGhlIHVubHVja3kgYXNz
aWdubWVudCBhbnl3YXksIHNvIGl0IHNob3VsZCBiZQ0KPiAgcHJlcGFyZWQ7ICBhbmQgaGFzaGlu
ZyBqdXN0IG1hZGUgS1ZNIHdvcnNlIGluIHRoYXQgY2FzZS4gIEd1ZXN0cyBtaWdodA0KPiAgYWxz
byBjb25maWd1cmUgcGh5c2ljYWwgeCgyKUFQSUMsIHdoZXJlIGlzIG5vIGxvd2VzdCBwcmlvcml0
eS4NCj4gIEFuZCBpZiB0aGUgZ3Vlc3QgZG9lc24ndCBkbyBhbnl0aGluZyB3aXRoIElSUXMsIHRo
ZW4gaXQgbWlnaHQgbm90IGV2ZW4NCj4gIGNhcmUgYWJvdXQgdGhlIGltcGFjdCB0aGF0IG91ciBj
aG9pY2UgaGFzLikNCg0KRG8gZG8geW91IGd1eXMgaGF2ZSBhbiBhZ3JlZW1lbnQgb24gaG93IHRv
IGhhbmRsZSB0aGlzPyBPciB3ZSBjYW4gaW1wbGVtZW50DQp0aGUgdmVjdG9yIGhhc2hpbmcgYXQg
dGhlIGN1cnJlbnQgc3RhZ2UuIHRoZW4gd2UgY2FuIGltcHJvdmUgaXQgbGlrZSBSYWRpbSBtZW50
aW9uZWQNCmFib3ZlIGlmIGl0IGlzIHJlYWxseSBuZWVkZWQ/IA0KDQpUaGFua3MsDQpGZW5nDQo=
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1278247

FromRadim Krcmár <rkrcmar@redhat.com>
Date2015-11-26 15:10 +0100
Message-ID<qz5pg-6dE-3@gated-at.bofh.it>
In reply to#1277991
2015-11-26 06:24+0000, Wu, Feng:
>> From: Radim Krčmář [mailto:rkrcmar@redhat.com]
>> 2015-11-25 15:38+0100, Paolo Bonzini:
>>> On 25/11/2015 15:12, Radim Krcmár wrote:
>>>> I think it's ok to pick any algorithm we like.  It's unlikely that
>>>> software would recognize and take advantage of the hardware algorithm
>>>> without adding a special treatment for KVM.
>>>> (I'd vote for the simple pick-first-APIC lowest priority algorithm ...
>>>>  I don't see much point in complicating lowest priority when it doesn't
>>>>  deliver to lowest priority CPU anyway.)
>>>
>>> Vector hashing is an improvement for the common case where all vectors
>>> are set to all CPUs.  Sure you can get an unlucky assignment, but it's
>>> still better than pick-first-APIC.
>> 
>> Yeah, hashing has a valid use case, but a subtle weighting of drawbacks
>> led me to prefer pick-first-APIC ...
> 
> Is it possible that pick-first-APIC policy make certain vCPU's irq workload too
> heavy?

It is, but vector hashing doesn't eliminate that possibility, just makes
it significantly less likely.
irqbalanced takes care of proper distribution in Linux guests.  I'm not
sure what other OS do, but they should have something like that as well.

>> (I'd prefer to have simple code in KVM and depend on static IRQ balancing
>>  in a guest to handle the distribution.
>>  The guest could get the unlucky assignment anyway, so it should be
>>  prepared;  and hashing just made KVM worse in that case.  Guests might
>>  also configure physical x(2)APIC, where is no lowest priority.
>>  And if the guest doesn't do anything with IRQs, then it might not even
>>  care about the impact that our choice has.)
> 
> Do do you guys have an agreement on how to handle this? Or we can implement
> the vector hashing at the current stage. then we can improve it like Radim mentioned
> above if it is really needed? 

Vector hashing is definitely an improvement over the current situation;
I'll agree with any algorithm if it is reasonably implemented.

(v1 fails delivery if chosen APIC is disabled, misuses KVM_MAX_VCPUS as
 bitmap size, and counting number of bits is better done with hweight16()
 -- these bugs would hopefully be fixed by having a common functinon :])
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web