Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1267787 > unrolled thread
| Started by | Jason Wang <jasowang@redhat.com> |
|---|---|
| First post | 2015-11-12 11:20 +0100 |
| Last post | 2015-11-13 10:30 +0100 |
| Articles | 4 — 2 participants |
Back to article view | Back to linux.kernel
[PATCH net-next RFC V3 0/3] basic busy polling support for vhost_net Jason Wang <jasowang@redhat.com> - 2015-11-12 11:20 +0100
Re: [PATCH net-next RFC V3 0/3] basic busy polling support for vhost_net Jason Wang <jasowang@redhat.com> - 2015-11-12 11:30 +0100
Re: [PATCH net-next RFC V3 0/3] basic busy polling support for vhost_net Felipe Franciosi <felipe@nutanix.com> - 2015-11-12 13:10 +0100
Re: [PATCH net-next RFC V3 0/3] basic busy polling support for vhost_net Jason Wang <jasowang@redhat.com> - 2015-11-13 10:30 +0100
| From | Jason Wang <jasowang@redhat.com> |
|---|---|
| Date | 2015-11-12 11:20 +0100 |
| Subject | [PATCH net-next RFC V3 0/3] basic busy polling support for vhost_net |
| Message-ID | <qtX90-7C4-3@gated-at.bofh.it> |
Hi all:
This series tries to add basic busy polling for vhost net. The idea is
simple: at the end of tx/rx processing, busy polling for new tx added
descriptor and rx receive socket for a while. The maximum number of
time (in us) could be spent on busy polling was specified ioctl.
Test were done through:
- 50 us as busy loop timeout
- Netperf 2.6
- Two machines with back to back connected ixgbe
- Guest with 1 vcpu and 1 queue
Results:
- For stream workload, ioexits were reduced dramatically in medium
size (1024-2048) of tx (at most -39%) and almost all rx (at most
-79%) as a result of polling. This compensate for the possible
wasted cpu cycles more or less. That porbably why we can still see
some increasing in the normalized throughput in some cases.
- Throughput of tx were increased (at most 105%) expect for the huge
write (16384). And we can send more packets in the case (+tpkts were
increased).
- Very minor rx regression in some cases.
- Improvemnt on TCP_RR (at most 16%).
size/session/+thu%/+normalize%/+tpkts%/+rpkts%/+ioexits%/
64/ 1/ +9%/ -17%/ +5%/ +10%/ -2%
64/ 2/ +8%/ -18%/ +6%/ +10%/ -1%
64/ 4/ +4%/ -21%/ +6%/ +10%/ -1%
64/ 8/ +9%/ -17%/ +6%/ +9%/ -2%
256/ 1/ +20%/ -1%/ +15%/ +11%/ -9%
256/ 2/ +15%/ -6%/ +15%/ +8%/ -8%
256/ 4/ +17%/ -4%/ +16%/ +8%/ -8%
256/ 8/ -61%/ -69%/ +16%/ +10%/ -10%
512/ 1/ +15%/ -3%/ +19%/ +18%/ -11%
512/ 2/ +19%/ 0%/ +19%/ +13%/ -10%
512/ 4/ +18%/ -2%/ +18%/ +15%/ -10%
512/ 8/ +17%/ -1%/ +18%/ +15%/ -11%
1024/ 1/ +25%/ +4%/ +27%/ +16%/ -21%
1024/ 2/ +28%/ +8%/ +25%/ +15%/ -22%
1024/ 4/ +25%/ +5%/ +25%/ +14%/ -21%
1024/ 8/ +27%/ +7%/ +25%/ +16%/ -21%
2048/ 1/ +32%/ +12%/ +31%/ +22%/ -38%
2048/ 2/ +33%/ +12%/ +30%/ +23%/ -36%
2048/ 4/ +31%/ +10%/ +31%/ +24%/ -37%
2048/ 8/ +105%/ +75%/ +33%/ +23%/ -39%
16384/ 1/ 0%/ -14%/ +2%/ 0%/ +19%
16384/ 2/ 0%/ -13%/ +19%/ -13%/ +17%
16384/ 4/ 0%/ -12%/ +3%/ 0%/ +2%
16384/ 8/ 0%/ -11%/ -2%/ +1%/ +1%
size/session/+thu%/+normalize%/+tpkts%/+rpkts%/+ioexits%/
64/ 1/ -7%/ -23%/ +4%/ +6%/ -74%
64/ 2/ -2%/ -12%/ +2%/ +2%/ -55%
64/ 4/ +2%/ -5%/ +10%/ -2%/ -43%
64/ 8/ -5%/ -5%/ +11%/ -34%/ -59%
256/ 1/ -6%/ -16%/ +9%/ +11%/ -60%
256/ 2/ +3%/ -4%/ +6%/ -3%/ -28%
256/ 4/ 0%/ -5%/ -9%/ -9%/ -10%
256/ 8/ -3%/ -6%/ -12%/ -9%/ -40%
512/ 1/ -4%/ -17%/ -10%/ +21%/ -34%
512/ 2/ 0%/ -9%/ -14%/ -3%/ -30%
512/ 4/ 0%/ -4%/ -18%/ -12%/ -4%
512/ 8/ -1%/ -4%/ -1%/ -5%/ +4%
1024/ 1/ 0%/ -16%/ +12%/ +11%/ -10%
1024/ 2/ 0%/ -11%/ 0%/ +5%/ -31%
1024/ 4/ 0%/ -4%/ -7%/ +1%/ -22%
1024/ 8/ -5%/ -6%/ -17%/ -29%/ -79%
2048/ 1/ 0%/ -16%/ +1%/ +9%/ -10%
2048/ 2/ 0%/ -12%/ +7%/ +9%/ -26%
2048/ 4/ 0%/ -7%/ -4%/ +3%/ -64%
2048/ 8/ -1%/ -5%/ -6%/ +4%/ -20%
16384/ 1/ 0%/ -12%/ +11%/ +7%/ -20%
16384/ 2/ 0%/ -7%/ +1%/ +5%/ -26%
16384/ 4/ 0%/ -5%/ +12%/ +22%/ -23%
16384/ 8/ 0%/ -1%/ -8%/ +5%/ -3%
size/session/+thu%/+normalize%/+tpkts%/+rpkts%/+ioexits%/
1/ 1/ +9%/ -29%/ +9%/ +9%/ +9%
1/ 25/ +6%/ -18%/ +6%/ +6%/ -1%
1/ 50/ +6%/ -19%/ +5%/ +5%/ -2%
1/ 100/ +5%/ -19%/ +4%/ +4%/ -3%
64/ 1/ +10%/ -28%/ +10%/ +10%/ +10%
64/ 25/ +8%/ -18%/ +7%/ +7%/ -2%
64/ 50/ +8%/ -17%/ +8%/ +8%/ -1%
64/ 100/ +8%/ -17%/ +8%/ +8%/ -1%
256/ 1/ +10%/ -28%/ +10%/ +10%/ +10%
256/ 25/ +15%/ -13%/ +15%/ +15%/ 0%
256/ 50/ +16%/ -14%/ +18%/ +18%/ +2%
256/ 100/ +15%/ -13%/ +12%/ +12%/ -2%
Changes from V2:
- poll also at the end of rx handling
- factor out the polling logic and optimize the code a little bit
- add two ioctls to get and set the busy poll timeout
- test on ixgbe (which can give more stable and reproducable numbers)
instead of mlx4.
Changes from V1:
- Add a comment for vhost_has_work() to explain why it could be
lockless
- Add param description for busyloop_timeout
- Split out the busy polling logic into a new helper
- Check and exit the loop when there's a pending signal
- Disable preemption during busy looping to make sure lock_clock() was
correctly used.
Jason Wang (3):
vhost: introduce vhost_has_work()
vhost: introduce vhost_vq_more_avail()
vhost_net: basic polling support
drivers/vhost/net.c | 77 +++++++++++++++++++++++++++++++++++++++++++---
drivers/vhost/vhost.c | 48 +++++++++++++++++++++++------
drivers/vhost/vhost.h | 3 ++
include/uapi/linux/vhost.h | 11 +++++++
4 files changed, 125 insertions(+), 14 deletions(-)
--
2.1.4
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [next] | [standalone]
| From | Jason Wang <jasowang@redhat.com> |
|---|---|
| Date | 2015-11-12 11:30 +0100 |
| Subject | Re: [PATCH net-next RFC V3 0/3] basic busy polling support for vhost_net |
| Message-ID | <qtXiG-7Fc-9@gated-at.bofh.it> |
| In reply to | #1267787 |
On 11/12/2015 06:16 PM, Jason Wang wrote: > Hi all: > > This series tries to add basic busy polling for vhost net. The idea is > simple: at the end of tx/rx processing, busy polling for new tx added > descriptor and rx receive socket for a while. The maximum number of > time (in us) could be spent on busy polling was specified ioctl. > > Test were done through: > > - 50 us as busy loop timeout > - Netperf 2.6 > - Two machines with back to back connected ixgbe > - Guest with 1 vcpu and 1 queue > > Results: > - For stream workload, ioexits were reduced dramatically in medium > size (1024-2048) of tx (at most -39%) and almost all rx (at most > -79%) as a result of polling. This compensate for the possible > wasted cpu cycles more or less. That porbably why we can still see > some increasing in the normalized throughput in some cases. > - Throughput of tx were increased (at most 105%) expect for the huge > write (16384). And we can send more packets in the case (+tpkts were > increased). > - Very minor rx regression in some cases. > - Improvemnt on TCP_RR (at most 16%). Forget to mention, the following test results by order are: 1) Guest TX 2) Guest RX 3) TCP_RR > size/session/+thu%/+normalize%/+tpkts%/+rpkts%/+ioexits%/ > 64/ 1/ +9%/ -17%/ +5%/ +10%/ -2% > 64/ 2/ +8%/ -18%/ +6%/ +10%/ -1% > 64/ 4/ +4%/ -21%/ +6%/ +10%/ -1% > 64/ 8/ +9%/ -17%/ +6%/ +9%/ -2% > 256/ 1/ +20%/ -1%/ +15%/ +11%/ -9% > 256/ 2/ +15%/ -6%/ +15%/ +8%/ -8% > 256/ 4/ +17%/ -4%/ +16%/ +8%/ -8% > 256/ 8/ -61%/ -69%/ +16%/ +10%/ -10% > 512/ 1/ +15%/ -3%/ +19%/ +18%/ -11% > 512/ 2/ +19%/ 0%/ +19%/ +13%/ -10% > 512/ 4/ +18%/ -2%/ +18%/ +15%/ -10% > 512/ 8/ +17%/ -1%/ +18%/ +15%/ -11% > 1024/ 1/ +25%/ +4%/ +27%/ +16%/ -21% > 1024/ 2/ +28%/ +8%/ +25%/ +15%/ -22% > 1024/ 4/ +25%/ +5%/ +25%/ +14%/ -21% > 1024/ 8/ +27%/ +7%/ +25%/ +16%/ -21% > 2048/ 1/ +32%/ +12%/ +31%/ +22%/ -38% > 2048/ 2/ +33%/ +12%/ +30%/ +23%/ -36% > 2048/ 4/ +31%/ +10%/ +31%/ +24%/ -37% > 2048/ 8/ +105%/ +75%/ +33%/ +23%/ -39% > 16384/ 1/ 0%/ -14%/ +2%/ 0%/ +19% > 16384/ 2/ 0%/ -13%/ +19%/ -13%/ +17% > 16384/ 4/ 0%/ -12%/ +3%/ 0%/ +2% > 16384/ 8/ 0%/ -11%/ -2%/ +1%/ +1% > size/session/+thu%/+normalize%/+tpkts%/+rpkts%/+ioexits%/ > 64/ 1/ -7%/ -23%/ +4%/ +6%/ -74% > 64/ 2/ -2%/ -12%/ +2%/ +2%/ -55% > 64/ 4/ +2%/ -5%/ +10%/ -2%/ -43% > 64/ 8/ -5%/ -5%/ +11%/ -34%/ -59% > 256/ 1/ -6%/ -16%/ +9%/ +11%/ -60% > 256/ 2/ +3%/ -4%/ +6%/ -3%/ -28% > 256/ 4/ 0%/ -5%/ -9%/ -9%/ -10% > 256/ 8/ -3%/ -6%/ -12%/ -9%/ -40% > 512/ 1/ -4%/ -17%/ -10%/ +21%/ -34% > 512/ 2/ 0%/ -9%/ -14%/ -3%/ -30% > 512/ 4/ 0%/ -4%/ -18%/ -12%/ -4% > 512/ 8/ -1%/ -4%/ -1%/ -5%/ +4% > 1024/ 1/ 0%/ -16%/ +12%/ +11%/ -10% > 1024/ 2/ 0%/ -11%/ 0%/ +5%/ -31% > 1024/ 4/ 0%/ -4%/ -7%/ +1%/ -22% > 1024/ 8/ -5%/ -6%/ -17%/ -29%/ -79% > 2048/ 1/ 0%/ -16%/ +1%/ +9%/ -10% > 2048/ 2/ 0%/ -12%/ +7%/ +9%/ -26% > 2048/ 4/ 0%/ -7%/ -4%/ +3%/ -64% > 2048/ 8/ -1%/ -5%/ -6%/ +4%/ -20% > 16384/ 1/ 0%/ -12%/ +11%/ +7%/ -20% > 16384/ 2/ 0%/ -7%/ +1%/ +5%/ -26% > 16384/ 4/ 0%/ -5%/ +12%/ +22%/ -23% > 16384/ 8/ 0%/ -1%/ -8%/ +5%/ -3% > size/session/+thu%/+normalize%/+tpkts%/+rpkts%/+ioexits%/ > 1/ 1/ +9%/ -29%/ +9%/ +9%/ +9% > 1/ 25/ +6%/ -18%/ +6%/ +6%/ -1% > 1/ 50/ +6%/ -19%/ +5%/ +5%/ -2% > 1/ 100/ +5%/ -19%/ +4%/ +4%/ -3% > 64/ 1/ +10%/ -28%/ +10%/ +10%/ +10% > 64/ 25/ +8%/ -18%/ +7%/ +7%/ -2% > 64/ 50/ +8%/ -17%/ +8%/ +8%/ -1% > 64/ 100/ +8%/ -17%/ +8%/ +8%/ -1% > 256/ 1/ +10%/ -28%/ +10%/ +10%/ +10% > 256/ 25/ +15%/ -13%/ +15%/ +15%/ 0% > 256/ 50/ +16%/ -14%/ +18%/ +18%/ +2% > 256/ 100/ +15%/ -13%/ +12%/ +12%/ -2% > > Changes from V2: > - poll also at the end of rx handling > - factor out the polling logic and optimize the code a little bit > - add two ioctls to get and set the busy poll timeout > - test on ixgbe (which can give more stable and reproducable numbers) > instead of mlx4. > > Changes from V1: > - Add a comment for vhost_has_work() to explain why it could be > lockless > - Add param description for busyloop_timeout > - Split out the busy polling logic into a new helper > - Check and exit the loop when there's a pending signal > - Disable preemption during busy looping to make sure lock_clock() was > correctly used. > > Jason Wang (3): > vhost: introduce vhost_has_work() > vhost: introduce vhost_vq_more_avail() > vhost_net: basic polling support > > drivers/vhost/net.c | 77 +++++++++++++++++++++++++++++++++++++++++++--- > drivers/vhost/vhost.c | 48 +++++++++++++++++++++++------ > drivers/vhost/vhost.h | 3 ++ > include/uapi/linux/vhost.h | 11 +++++++ > 4 files changed, 125 insertions(+), 14 deletions(-) > -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Felipe Franciosi <felipe@nutanix.com> |
|---|---|
| Date | 2015-11-12 13:10 +0100 |
| Subject | Re: [PATCH net-next RFC V3 0/3] basic busy polling support for vhost_net |
| Message-ID | <qtYRs-gc-21@gated-at.bofh.it> |
| In reply to | #1267791 |
SGkgSmFzb24sDQoNCkkgdW5kZXJzdGFuZCB5b3VyIGJ1c3kgbG9vcCB0aW1lb3V0IGlzIHF1aXRl IGNvbnNlcnZhdGl2ZSBhdCA1MHVzLiBEaWQgeW91IHRyeSBhbnkgb3RoZXIgdmFsdWVzPw0KDQpB bHNvLCBkaWQgeW91IG1lYXN1cmUgaG93IHBvbGxpbmcgYWZmZWN0cyBtYW55IFZNcyB0YWxraW5n IHRvIGVhY2ggb3RoZXIgKGUuZy4gMjAgVk1zIG9uIGVhY2ggaG9zdCwgcGVyaGFwcyB3aXRoIHNl dmVyYWwgdk5JQ3MgZWFjaCwgdHJhbnNtaXR0aW5nIHRvIGEgY29ycmVzcG9uZGluZyBWTS92TklD IHBhaXIgb24gYW5vdGhlciBob3N0KT8NCg0KDQpPbiBhIGNvbXBsZXRlIHNlcGFyYXRlIGV4cGVy aW1lbnQgKGJ1c3kgd2FpdGluZyBvbiBzdG9yYWdlIEkvTyByaW5ncyBvbiBYZW4pLCBJIGhhdmUg b2JzZXJ2ZWQgdGhhdCBiaWdnZXIgdGltZW91dHMgZ2F2ZSBiaWdnZXIgYmVuZWZpdHMuIE9uIHRo ZSBvdGhlciBoYW5kLCBhbGwgY2FzZXMgdGhhdCBjb250ZW5kZWQgZm9yIENQVSB3ZXJlIGJhZGx5 IGh1cnQgd2l0aCBhbnkgc29ydCBvZiBwb2xsaW5nLg0KDQpUaGUgY2FzZXMgdGhhdCBjb250ZW5k ZWQgZm9yIENQVSBjb25zaXN0ZWQgb2YgbWFueSBWTXMgZ2VuZXJhdGluZyB3b3JrbG9hZCBvdmVy IHZlcnkgZmFzdCBJL08gZGV2aWNlcyAoaW4gdGhhdCBjYXNlLCBzZXZlcmFsIE5WTWUgZGV2aWNl cyBvbiBhIHNpbmdsZSBob3N0KS4gQW5kIHRoZSBtZXRyaWMgdGhhdCBnb3QgYWZmZWN0ZWQgd2Fz IGFnZ3JlZ2F0ZSB0aHJvdWdocHV0IGZyb20gYWxsIFZNcy4NCg0KVGhlIHNvbHV0aW9uIHdhcyB0 byBkZXRlcm1pbmUgd2hldGhlciB0byBwb2xsIGRlcGVuZGluZyBvbiB0aGUgaG9zdCdzIG92ZXJh bGwgQ1BVIHV0aWxpc2F0aW9uIGF0IHRoYXQgbW9tZW50LiBUaGF0IGdhdmUgbWUgdGhlIGJlc3Qg b2YgYm90aCB3b3JsZHMgYXMgcG9sbGluZyBtYWRlIGV2ZXJ5dGhpbmcgZmFzdGVyIHdpdGhvdXQg c2xvd2luZyBkb3duIGFueSBvdGhlciBtZXRyaWMuDQoNClRoYW5rcywNCkZlbGlwZQ0KDQoNCg0K T24gMTIvMTEvMjAxNSAxMDoyMCwgImt2bS1vd25lckB2Z2VyLmtlcm5lbC5vcmcgb24gYmVoYWxm IG9mIEphc29uIFdhbmciIDxrdm0tb3duZXJAdmdlci5rZXJuZWwub3JnIG9uIGJlaGFsZiBvZiBq YXNvd2FuZ0ByZWRoYXQuY29tPiB3cm90ZToNCg0KPg0KPg0KPk9uIDExLzEyLzIwMTUgMDY6MTYg UE0sIEphc29uIFdhbmcgd3JvdGU6DQo+PiBIaSBhbGw6DQo+Pg0KPj4gVGhpcyBzZXJpZXMgdHJp ZXMgdG8gYWRkIGJhc2ljIGJ1c3kgcG9sbGluZyBmb3Igdmhvc3QgbmV0LiBUaGUgaWRlYSBpcw0K Pj4gc2ltcGxlOiBhdCB0aGUgZW5kIG9mIHR4L3J4IHByb2Nlc3NpbmcsIGJ1c3kgcG9sbGluZyBm b3IgbmV3IHR4IGFkZGVkDQo+PiBkZXNjcmlwdG9yIGFuZCByeCByZWNlaXZlIHNvY2tldCBmb3Ig YSB3aGlsZS4gVGhlIG1heGltdW0gbnVtYmVyIG9mDQo+PiB0aW1lIChpbiB1cykgY291bGQgYmUg c3BlbnQgb24gYnVzeSBwb2xsaW5nIHdhcyBzcGVjaWZpZWQgaW9jdGwuDQo+Pg0KPj4gVGVzdCB3 ZXJlIGRvbmUgdGhyb3VnaDoNCj4+DQo+PiAtIDUwIHVzIGFzIGJ1c3kgbG9vcCB0aW1lb3V0DQo+ PiAtIE5ldHBlcmYgMi42DQo+PiAtIFR3byBtYWNoaW5lcyB3aXRoIGJhY2sgdG8gYmFjayBjb25u ZWN0ZWQgaXhnYmUNCj4+IC0gR3Vlc3Qgd2l0aCAxIHZjcHUgYW5kIDEgcXVldWUNCj4+DQo+PiBS ZXN1bHRzOg0KPj4gLSBGb3Igc3RyZWFtIHdvcmtsb2FkLCBpb2V4aXRzIHdlcmUgcmVkdWNlZCBk cmFtYXRpY2FsbHkgaW4gbWVkaXVtDQo+PiAgIHNpemUgKDEwMjQtMjA0OCkgb2YgdHggKGF0IG1v c3QgLTM5JSkgYW5kIGFsbW9zdCBhbGwgcnggKGF0IG1vc3QNCj4+ICAgLTc5JSkgYXMgYSByZXN1 bHQgb2YgcG9sbGluZy4gVGhpcyBjb21wZW5zYXRlIGZvciB0aGUgcG9zc2libGUNCj4+ICAgd2Fz dGVkIGNwdSBjeWNsZXMgbW9yZSBvciBsZXNzLiBUaGF0IHBvcmJhYmx5IHdoeSB3ZSBjYW4gc3Rp bGwgc2VlDQo+PiAgIHNvbWUgaW5jcmVhc2luZyBpbiB0aGUgbm9ybWFsaXplZCB0aHJvdWdocHV0 IGluIHNvbWUgY2FzZXMuDQo+PiAtIFRocm91Z2hwdXQgb2YgdHggd2VyZSBpbmNyZWFzZWQgKGF0 IG1vc3QgMTA1JSkgZXhwZWN0IGZvciB0aGUgaHVnZQ0KPj4gICB3cml0ZSAoMTYzODQpLiBBbmQg d2UgY2FuIHNlbmQgbW9yZSBwYWNrZXRzIGluIHRoZSBjYXNlICgrdHBrdHMgd2VyZQ0KPj4gICBp bmNyZWFzZWQpLg0KPj4gLSBWZXJ5IG1pbm9yIHJ4IHJlZ3Jlc3Npb24gaW4gc29tZSBjYXNlcy4N Cj4+IC0gSW1wcm92ZW1udCBvbiBUQ1BfUlIgKGF0IG1vc3QgMTYlKS4NCj4NCj5Gb3JnZXQgdG8g bWVudGlvbiwgdGhlIGZvbGxvd2luZyB0ZXN0IHJlc3VsdHMgYnkgb3JkZXIgYXJlOg0KPg0KPjEp IEd1ZXN0IFRYDQo+MikgR3Vlc3QgUlgNCj4zKSBUQ1BfUlINCj4NCj4+IHNpemUvc2Vzc2lvbi8r dGh1JS8rbm9ybWFsaXplJS8rdHBrdHMlLytycGt0cyUvK2lvZXhpdHMlLw0KPj4gICAgNjQvICAg ICAxLyAgICs5JS8gIC0xNyUvICAgKzUlLyAgKzEwJS8gICAtMiUNCj4+ICAgIDY0LyAgICAgMi8g ICArOCUvICAtMTglLyAgICs2JS8gICsxMCUvICAgLTElDQo+PiAgICA2NC8gICAgIDQvICAgKzQl LyAgLTIxJS8gICArNiUvICArMTAlLyAgIC0xJQ0KPj4gICAgNjQvICAgICA4LyAgICs5JS8gIC0x NyUvICAgKzYlLyAgICs5JS8gICAtMiUNCj4+ICAgMjU2LyAgICAgMS8gICsyMCUvICAgLTElLyAg KzE1JS8gICsxMSUvICAgLTklDQo+PiAgIDI1Ni8gICAgIDIvICArMTUlLyAgIC02JS8gICsxNSUv ICAgKzglLyAgIC04JQ0KPj4gICAyNTYvICAgICA0LyAgKzE3JS8gICAtNCUvICArMTYlLyAgICs4 JS8gICAtOCUNCj4+ICAgMjU2LyAgICAgOC8gIC02MSUvICAtNjklLyAgKzE2JS8gICsxMCUvICAt MTAlDQo+PiAgIDUxMi8gICAgIDEvICArMTUlLyAgIC0zJS8gICsxOSUvICArMTglLyAgLTExJQ0K Pj4gICA1MTIvICAgICAyLyAgKzE5JS8gICAgMCUvICArMTklLyAgKzEzJS8gIC0xMCUNCj4+ICAg NTEyLyAgICAgNC8gICsxOCUvICAgLTIlLyAgKzE4JS8gICsxNSUvICAtMTAlDQo+PiAgIDUxMi8g ICAgIDgvICArMTclLyAgIC0xJS8gICsxOCUvICArMTUlLyAgLTExJQ0KPj4gIDEwMjQvICAgICAx LyAgKzI1JS8gICArNCUvICArMjclLyAgKzE2JS8gIC0yMSUNCj4+ICAxMDI0LyAgICAgMi8gICsy OCUvICAgKzglLyAgKzI1JS8gICsxNSUvICAtMjIlDQo+PiAgMTAyNC8gICAgIDQvICArMjUlLyAg ICs1JS8gICsyNSUvICArMTQlLyAgLTIxJQ0KPj4gIDEwMjQvICAgICA4LyAgKzI3JS8gICArNyUv ICArMjUlLyAgKzE2JS8gIC0yMSUNCj4+ICAyMDQ4LyAgICAgMS8gICszMiUvICArMTIlLyAgKzMx JS8gICsyMiUvICAtMzglDQo+PiAgMjA0OC8gICAgIDIvICArMzMlLyAgKzEyJS8gICszMCUvICAr MjMlLyAgLTM2JQ0KPj4gIDIwNDgvICAgICA0LyAgKzMxJS8gICsxMCUvICArMzElLyAgKzI0JS8g IC0zNyUNCj4+ICAyMDQ4LyAgICAgOC8gKzEwNSUvICArNzUlLyAgKzMzJS8gICsyMyUvICAtMzkl DQo+PiAxNjM4NC8gICAgIDEvICAgIDAlLyAgLTE0JS8gICArMiUvICAgIDAlLyAgKzE5JQ0KPj4g MTYzODQvICAgICAyLyAgICAwJS8gIC0xMyUvICArMTklLyAgLTEzJS8gICsxNyUNCj4+IDE2Mzg0 LyAgICAgNC8gICAgMCUvICAtMTIlLyAgICszJS8gICAgMCUvICAgKzIlDQo+PiAxNjM4NC8gICAg IDgvICAgIDAlLyAgLTExJS8gICAtMiUvICAgKzElLyAgICsxJQ0KPj4gc2l6ZS9zZXNzaW9uLyt0 aHUlLytub3JtYWxpemUlLyt0cGt0cyUvK3Jwa3RzJS8raW9leGl0cyUvDQo+PiAgICA2NC8gICAg IDEvICAgLTclLyAgLTIzJS8gICArNCUvICAgKzYlLyAgLTc0JQ0KPj4gICAgNjQvICAgICAyLyAg IC0yJS8gIC0xMiUvICAgKzIlLyAgICsyJS8gIC01NSUNCj4+ICAgIDY0LyAgICAgNC8gICArMiUv ICAgLTUlLyAgKzEwJS8gICAtMiUvICAtNDMlDQo+PiAgICA2NC8gICAgIDgvICAgLTUlLyAgIC01 JS8gICsxMSUvICAtMzQlLyAgLTU5JQ0KPj4gICAyNTYvICAgICAxLyAgIC02JS8gIC0xNiUvICAg KzklLyAgKzExJS8gIC02MCUNCj4+ICAgMjU2LyAgICAgMi8gICArMyUvICAgLTQlLyAgICs2JS8g ICAtMyUvICAtMjglDQo+PiAgIDI1Ni8gICAgIDQvICAgIDAlLyAgIC01JS8gICAtOSUvICAgLTkl LyAgLTEwJQ0KPj4gICAyNTYvICAgICA4LyAgIC0zJS8gICAtNiUvICAtMTIlLyAgIC05JS8gIC00 MCUNCj4+ICAgNTEyLyAgICAgMS8gICAtNCUvICAtMTclLyAgLTEwJS8gICsyMSUvICAtMzQlDQo+ PiAgIDUxMi8gICAgIDIvICAgIDAlLyAgIC05JS8gIC0xNCUvICAgLTMlLyAgLTMwJQ0KPj4gICA1 MTIvICAgICA0LyAgICAwJS8gICAtNCUvICAtMTglLyAgLTEyJS8gICAtNCUNCj4+ICAgNTEyLyAg ICAgOC8gICAtMSUvICAgLTQlLyAgIC0xJS8gICAtNSUvICAgKzQlDQo+PiAgMTAyNC8gICAgIDEv ICAgIDAlLyAgLTE2JS8gICsxMiUvICArMTElLyAgLTEwJQ0KPj4gIDEwMjQvICAgICAyLyAgICAw JS8gIC0xMSUvICAgIDAlLyAgICs1JS8gIC0zMSUNCj4+ICAxMDI0LyAgICAgNC8gICAgMCUvICAg LTQlLyAgIC03JS8gICArMSUvICAtMjIlDQo+PiAgMTAyNC8gICAgIDgvICAgLTUlLyAgIC02JS8g IC0xNyUvICAtMjklLyAgLTc5JQ0KPj4gIDIwNDgvICAgICAxLyAgICAwJS8gIC0xNiUvICAgKzEl LyAgICs5JS8gIC0xMCUNCj4+ICAyMDQ4LyAgICAgMi8gICAgMCUvICAtMTIlLyAgICs3JS8gICAr OSUvICAtMjYlDQo+PiAgMjA0OC8gICAgIDQvICAgIDAlLyAgIC03JS8gICAtNCUvICAgKzMlLyAg LTY0JQ0KPj4gIDIwNDgvICAgICA4LyAgIC0xJS8gICAtNSUvICAgLTYlLyAgICs0JS8gIC0yMCUN Cj4+IDE2Mzg0LyAgICAgMS8gICAgMCUvICAtMTIlLyAgKzExJS8gICArNyUvICAtMjAlDQo+PiAx NjM4NC8gICAgIDIvICAgIDAlLyAgIC03JS8gICArMSUvICAgKzUlLyAgLTI2JQ0KPj4gMTYzODQv ICAgICA0LyAgICAwJS8gICAtNSUvICArMTIlLyAgKzIyJS8gIC0yMyUNCj4+IDE2Mzg0LyAgICAg OC8gICAgMCUvICAgLTElLyAgIC04JS8gICArNSUvICAgLTMlDQo+PiBzaXplL3Nlc3Npb24vK3Ro dSUvK25vcm1hbGl6ZSUvK3Rwa3RzJS8rcnBrdHMlLytpb2V4aXRzJS8NCj4+ICAgICAxLyAgICAg MS8gICArOSUvICAtMjklLyAgICs5JS8gICArOSUvICAgKzklDQo+PiAgICAgMS8gICAgMjUvICAg KzYlLyAgLTE4JS8gICArNiUvICAgKzYlLyAgIC0xJQ0KPj4gICAgIDEvICAgIDUwLyAgICs2JS8g IC0xOSUvICAgKzUlLyAgICs1JS8gICAtMiUNCj4+ICAgICAxLyAgIDEwMC8gICArNSUvICAtMTkl LyAgICs0JS8gICArNCUvICAgLTMlDQo+PiAgICA2NC8gICAgIDEvICArMTAlLyAgLTI4JS8gICsx MCUvICArMTAlLyAgKzEwJQ0KPj4gICAgNjQvICAgIDI1LyAgICs4JS8gIC0xOCUvICAgKzclLyAg ICs3JS8gICAtMiUNCj4+ICAgIDY0LyAgICA1MC8gICArOCUvICAtMTclLyAgICs4JS8gICArOCUv ICAgLTElDQo+PiAgICA2NC8gICAxMDAvICAgKzglLyAgLTE3JS8gICArOCUvICAgKzglLyAgIC0x JQ0KPj4gICAyNTYvICAgICAxLyAgKzEwJS8gIC0yOCUvICArMTAlLyAgKzEwJS8gICsxMCUNCj4+ ICAgMjU2LyAgICAyNS8gICsxNSUvICAtMTMlLyAgKzE1JS8gICsxNSUvICAgIDAlDQo+PiAgIDI1 Ni8gICAgNTAvICArMTYlLyAgLTE0JS8gICsxOCUvICArMTglLyAgICsyJQ0KPj4gICAyNTYvICAg MTAwLyAgKzE1JS8gIC0xMyUvICArMTIlLyAgKzEyJS8gICAtMiUNCj4+DQo+PiBDaGFuZ2VzIGZy b20gVjI6DQo+PiAtIHBvbGwgYWxzbyBhdCB0aGUgZW5kIG9mIHJ4IGhhbmRsaW5nDQo+PiAtIGZh Y3RvciBvdXQgdGhlIHBvbGxpbmcgbG9naWMgYW5kIG9wdGltaXplIHRoZSBjb2RlIGEgbGl0dGxl IGJpdA0KPj4gLSBhZGQgdHdvIGlvY3RscyB0byBnZXQgYW5kIHNldCB0aGUgYnVzeSBwb2xsIHRp bWVvdXQNCj4+IC0gdGVzdCBvbiBpeGdiZSAod2hpY2ggY2FuIGdpdmUgbW9yZSBzdGFibGUgYW5k IHJlcHJvZHVjYWJsZSBudW1iZXJzKQ0KPj4gICBpbnN0ZWFkIG9mIG1seDQuDQo+Pg0KPj4gQ2hh bmdlcyBmcm9tIFYxOg0KPj4gLSBBZGQgYSBjb21tZW50IGZvciB2aG9zdF9oYXNfd29yaygpIHRv IGV4cGxhaW4gd2h5IGl0IGNvdWxkIGJlDQo+PiAgIGxvY2tsZXNzDQo+PiAtIEFkZCBwYXJhbSBk ZXNjcmlwdGlvbiBmb3IgYnVzeWxvb3BfdGltZW91dA0KPj4gLSBTcGxpdCBvdXQgdGhlIGJ1c3kg cG9sbGluZyBsb2dpYyBpbnRvIGEgbmV3IGhlbHBlcg0KPj4gLSBDaGVjayBhbmQgZXhpdCB0aGUg bG9vcCB3aGVuIHRoZXJlJ3MgYSBwZW5kaW5nIHNpZ25hbA0KPj4gLSBEaXNhYmxlIHByZWVtcHRp b24gZHVyaW5nIGJ1c3kgbG9vcGluZyB0byBtYWtlIHN1cmUgbG9ja19jbG9jaygpIHdhcw0KPj4g ICBjb3JyZWN0bHkgdXNlZC4NCj4+DQo+PiBKYXNvbiBXYW5nICgzKToNCj4+ICAgdmhvc3Q6IGlu dHJvZHVjZSB2aG9zdF9oYXNfd29yaygpDQo+PiAgIHZob3N0OiBpbnRyb2R1Y2Ugdmhvc3RfdnFf bW9yZV9hdmFpbCgpDQo+PiAgIHZob3N0X25ldDogYmFzaWMgcG9sbGluZyBzdXBwb3J0DQo+Pg0K Pj4gIGRyaXZlcnMvdmhvc3QvbmV0LmMgICAgICAgIHwgNzcgKysrKysrKysrKysrKysrKysrKysr KysrKysrKysrKysrKysrKysrKysrKy0tLQ0KPj4gIGRyaXZlcnMvdmhvc3Qvdmhvc3QuYyAgICAg IHwgNDggKysrKysrKysrKysrKysrKysrKysrKystLS0tLS0NCj4+ICBkcml2ZXJzL3Zob3N0L3Zo b3N0LmggICAgICB8ICAzICsrDQo+PiAgaW5jbHVkZS91YXBpL2xpbnV4L3Zob3N0LmggfCAxMSAr KysrKysrDQo+PiAgNCBmaWxlcyBjaGFuZ2VkLCAxMjUgaW5zZXJ0aW9ucygrKSwgMTQgZGVsZXRp b25zKC0pDQo+Pg0KPg0KPi0tDQo+VG8gdW5zdWJzY3JpYmUgZnJvbSB0aGlzIGxpc3Q6IHNlbmQg dGhlIGxpbmUgInVuc3Vic2NyaWJlIGt2bSIgaW4NCj50aGUgYm9keSBvZiBhIG1lc3NhZ2UgdG8g bWFqb3Jkb21vQHZnZXIua2VybmVsLm9yZw0KPk1vcmUgbWFqb3Jkb21vIGluZm8gYXQgIGh0dHA6 Ly92Z2VyLmtlcm5lbC5vcmcvbWFqb3Jkb21vLWluZm8uaHRtbA0K -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Jason Wang <jasowang@redhat.com> |
|---|---|
| Date | 2015-11-13 10:30 +0100 |
| Subject | Re: [PATCH net-next RFC V3 0/3] basic busy polling support for vhost_net |
| Message-ID | <quiQa-4yT-21@gated-at.bofh.it> |
| In reply to | #1267864 |
On 11/12/2015 08:02 PM, Felipe Franciosi wrote:
> Hi Jason,
>
> I understand your busy loop timeout is quite conservative at 50us. Did you try any other values?
I've also tried 20us. And results shows 50us was better in:
- very small packet tx (e.g 64bytes at most 46% improvement)
- TCP_RR (at most 11% improvement)
But I will test bigger values. In fact, for net itself, we can be even
more aggressive: make vhost poll forever but I haven't tired this.
>
> Also, did you measure how polling affects many VMs talking to each other (e.g. 20 VMs on each host, perhaps with several vNICs each, transmitting to a corresponding VM/vNIC pair on another host)?
Not yet, in my todo list.
>
>
> On a complete separate experiment (busy waiting on storage I/O rings on Xen), I have observed that bigger timeouts gave bigger benefits. On the other hand, all cases that contended for CPU were badly hurt with any sort of polling.
>
> The cases that contended for CPU consisted of many VMs generating workload over very fast I/O devices (in that case, several NVMe devices on a single host). And the metric that got affected was aggregate throughput from all VMs.
>
> The solution was to determine whether to poll depending on the host's overall CPU utilisation at that moment. That gave me the best of both worlds as polling made everything faster without slowing down any other metric.
You mean a threshold and exit polling when it exceeds this? I use a
simpler method: just exit the busy loop when there's more than one
processes is in running state. I test this method in the past for socket
busy read (http://www.gossamer-threads.com/lists/linux/kernel/1997531)
which seems can solve the issue. But haven't tested this for vhost
polling. Will run some simple test (e.g pin two vhost threads in one
host cpu), and see how well it perform.
Thanks
>
> Thanks,
> Felipe
>
>
>
> On 12/11/2015 10:20, "kvm-owner@vger.kernel.org on behalf of Jason Wang" <kvm-owner@vger.kernel.org on behalf of jasowang@redhat.com> wrote:
>
>>
>> On 11/12/2015 06:16 PM, Jason Wang wrote:
>>> Hi all:
>>>
>>> This series tries to add basic busy polling for vhost net. The idea is
>>> simple: at the end of tx/rx processing, busy polling for new tx added
>>> descriptor and rx receive socket for a while. The maximum number of
>>> time (in us) could be spent on busy polling was specified ioctl.
>>>
>>> Test were done through:
>>>
>>> - 50 us as busy loop timeout
>>> - Netperf 2.6
>>> - Two machines with back to back connected ixgbe
>>> - Guest with 1 vcpu and 1 queue
>>>
>>> Results:
>>> - For stream workload, ioexits were reduced dramatically in medium
>>> size (1024-2048) of tx (at most -39%) and almost all rx (at most
>>> -79%) as a result of polling. This compensate for the possible
>>> wasted cpu cycles more or less. That porbably why we can still see
>>> some increasing in the normalized throughput in some cases.
>>> - Throughput of tx were increased (at most 105%) expect for the huge
>>> write (16384). And we can send more packets in the case (+tpkts were
>>> increased).
>>> - Very minor rx regression in some cases.
>>> - Improvemnt on TCP_RR (at most 16%).
>> Forget to mention, the following test results by order are:
>>
>> 1) Guest TX
>> 2) Guest RX
>> 3) TCP_RR
>>
>>> size/session/+thu%/+normalize%/+tpkts%/+rpkts%/+ioexits%/
>>> 64/ 1/ +9%/ -17%/ +5%/ +10%/ -2%
>>> 64/ 2/ +8%/ -18%/ +6%/ +10%/ -1%
>>> 64/ 4/ +4%/ -21%/ +6%/ +10%/ -1%
>>> 64/ 8/ +9%/ -17%/ +6%/ +9%/ -2%
>>> 256/ 1/ +20%/ -1%/ +15%/ +11%/ -9%
>>> 256/ 2/ +15%/ -6%/ +15%/ +8%/ -8%
>>> 256/ 4/ +17%/ -4%/ +16%/ +8%/ -8%
>>> 256/ 8/ -61%/ -69%/ +16%/ +10%/ -10%
>>> 512/ 1/ +15%/ -3%/ +19%/ +18%/ -11%
>>> 512/ 2/ +19%/ 0%/ +19%/ +13%/ -10%
>>> 512/ 4/ +18%/ -2%/ +18%/ +15%/ -10%
>>> 512/ 8/ +17%/ -1%/ +18%/ +15%/ -11%
>>> 1024/ 1/ +25%/ +4%/ +27%/ +16%/ -21%
>>> 1024/ 2/ +28%/ +8%/ +25%/ +15%/ -22%
>>> 1024/ 4/ +25%/ +5%/ +25%/ +14%/ -21%
>>> 1024/ 8/ +27%/ +7%/ +25%/ +16%/ -21%
>>> 2048/ 1/ +32%/ +12%/ +31%/ +22%/ -38%
>>> 2048/ 2/ +33%/ +12%/ +30%/ +23%/ -36%
>>> 2048/ 4/ +31%/ +10%/ +31%/ +24%/ -37%
>>> 2048/ 8/ +105%/ +75%/ +33%/ +23%/ -39%
>>> 16384/ 1/ 0%/ -14%/ +2%/ 0%/ +19%
>>> 16384/ 2/ 0%/ -13%/ +19%/ -13%/ +17%
>>> 16384/ 4/ 0%/ -12%/ +3%/ 0%/ +2%
>>> 16384/ 8/ 0%/ -11%/ -2%/ +1%/ +1%
>>> size/session/+thu%/+normalize%/+tpkts%/+rpkts%/+ioexits%/
>>> 64/ 1/ -7%/ -23%/ +4%/ +6%/ -74%
>>> 64/ 2/ -2%/ -12%/ +2%/ +2%/ -55%
>>> 64/ 4/ +2%/ -5%/ +10%/ -2%/ -43%
>>> 64/ 8/ -5%/ -5%/ +11%/ -34%/ -59%
>>> 256/ 1/ -6%/ -16%/ +9%/ +11%/ -60%
>>> 256/ 2/ +3%/ -4%/ +6%/ -3%/ -28%
>>> 256/ 4/ 0%/ -5%/ -9%/ -9%/ -10%
>>> 256/ 8/ -3%/ -6%/ -12%/ -9%/ -40%
>>> 512/ 1/ -4%/ -17%/ -10%/ +21%/ -34%
>>> 512/ 2/ 0%/ -9%/ -14%/ -3%/ -30%
>>> 512/ 4/ 0%/ -4%/ -18%/ -12%/ -4%
>>> 512/ 8/ -1%/ -4%/ -1%/ -5%/ +4%
>>> 1024/ 1/ 0%/ -16%/ +12%/ +11%/ -10%
>>> 1024/ 2/ 0%/ -11%/ 0%/ +5%/ -31%
>>> 1024/ 4/ 0%/ -4%/ -7%/ +1%/ -22%
>>> 1024/ 8/ -5%/ -6%/ -17%/ -29%/ -79%
>>> 2048/ 1/ 0%/ -16%/ +1%/ +9%/ -10%
>>> 2048/ 2/ 0%/ -12%/ +7%/ +9%/ -26%
>>> 2048/ 4/ 0%/ -7%/ -4%/ +3%/ -64%
>>> 2048/ 8/ -1%/ -5%/ -6%/ +4%/ -20%
>>> 16384/ 1/ 0%/ -12%/ +11%/ +7%/ -20%
>>> 16384/ 2/ 0%/ -7%/ +1%/ +5%/ -26%
>>> 16384/ 4/ 0%/ -5%/ +12%/ +22%/ -23%
>>> 16384/ 8/ 0%/ -1%/ -8%/ +5%/ -3%
>>> size/session/+thu%/+normalize%/+tpkts%/+rpkts%/+ioexits%/
>>> 1/ 1/ +9%/ -29%/ +9%/ +9%/ +9%
>>> 1/ 25/ +6%/ -18%/ +6%/ +6%/ -1%
>>> 1/ 50/ +6%/ -19%/ +5%/ +5%/ -2%
>>> 1/ 100/ +5%/ -19%/ +4%/ +4%/ -3%
>>> 64/ 1/ +10%/ -28%/ +10%/ +10%/ +10%
>>> 64/ 25/ +8%/ -18%/ +7%/ +7%/ -2%
>>> 64/ 50/ +8%/ -17%/ +8%/ +8%/ -1%
>>> 64/ 100/ +8%/ -17%/ +8%/ +8%/ -1%
>>> 256/ 1/ +10%/ -28%/ +10%/ +10%/ +10%
>>> 256/ 25/ +15%/ -13%/ +15%/ +15%/ 0%
>>> 256/ 50/ +16%/ -14%/ +18%/ +18%/ +2%
>>> 256/ 100/ +15%/ -13%/ +12%/ +12%/ -2%
>>>
>>> Changes from V2:
>>> - poll also at the end of rx handling
>>> - factor out the polling logic and optimize the code a little bit
>>> - add two ioctls to get and set the busy poll timeout
>>> - test on ixgbe (which can give more stable and reproducable numbers)
>>> instead of mlx4.
>>>
>>> Changes from V1:
>>> - Add a comment for vhost_has_work() to explain why it could be
>>> lockless
>>> - Add param description for busyloop_timeout
>>> - Split out the busy polling logic into a new helper
>>> - Check and exit the loop when there's a pending signal
>>> - Disable preemption during busy looping to make sure lock_clock() was
>>> correctly used.
>>>
>>> Jason Wang (3):
>>> vhost: introduce vhost_has_work()
>>> vhost: introduce vhost_vq_more_avail()
>>> vhost_net: basic polling support
>>>
>>> drivers/vhost/net.c | 77 +++++++++++++++++++++++++++++++++++++++++++---
>>> drivers/vhost/vhost.c | 48 +++++++++++++++++++++++------
>>> drivers/vhost/vhost.h | 3 ++
>>> include/uapi/linux/vhost.h | 11 +++++++
>>> 4 files changed, 125 insertions(+), 14 deletions(-)
>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
> N�����r��y���b�X��ǧv�^�){.n�+����{����zX����ܨ}���Ơz�&j:+v�������zZ+��+zf���h���~����i���z� �w���?����&�)ߢf��^jǫy�m��@A�a��� 0��h��i
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [standalone]
Back to top | Article view | linux.kernel
csiph-web