Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1264460 > unrolled thread

linux-next network throughput performance regression

Started bySimon Xiao <sixiao@microsoft.com>
First post2015-11-06 21:40 +0100
Last post2015-11-09 06:40 +0100
Articles 11 — 6 participants

Back to article view | Back to linux.kernel

Contents

  linux-next network throughput performance regression Simon Xiao <sixiao@microsoft.com> - 2015-11-06 21:40 +0100
    Re: linux-next network throughput performance regression David Ahern <dsa@cumulusnetworks.com> - 2015-11-06 21:50 +0100
      Re: linux-next network throughput performance regression David Ahern <dsa@cumulusnetworks.com> - 2015-11-06 22:40 +0100
        Re: linux-next network throughput performance regression Eric Dumazet <eric.dumazet@gmail.com> - 2015-11-07 20:40 +0100
          Re: linux-next network throughput performance regression Eric Dumazet <eric.dumazet@gmail.com> - 2015-11-07 21:00 +0100
            RE: linux-next network throughput performance regression Simon Xiao <sixiao@microsoft.com> - 2015-11-09 21:30 +0100
              Re: linux-next network throughput performance regression Eric Dumazet <eric.dumazet@gmail.com> - 2015-11-10 00:10 +0100
          Re: linux-next network throughput performance regression David Miller <davem@davemloft.net> - 2015-11-09 04:00 +0100
            Re: linux-next network throughput performance regression David Miller <davem@davemloft.net> - 2015-11-09 04:30 +0100
              Re: linux-next network throughput performance regression Dave Airlie <airlied@gmail.com> - 2015-11-09 04:40 +0100
              Re: linux-next network throughput performance regression Tom Herbert <tom@herbertland.com> - 2015-11-09 06:40 +0100

#1264460 — linux-next network throughput performance regression

FromSimon Xiao <sixiao@microsoft.com>
Date2015-11-06 21:40 +0100
Subjectlinux-next network throughput performance regression
Message-ID<qrVXI-V4-5@gated-at.bofh.it>
I compared the network throughput performance on SLES12 bare metal servers, between SLES12 default kernel and latest linux-next (2015-11-05) kernel, based on the test results, I suspect there is a network regression exists on Linux-Next over the 40G Ethernet network:
a) iperf3 reports 50% performance drop with single TCP stream on latest linux-next;
b) iperf3 reports 10% ~ 30% performance drop with 2 to 128 TCP streams on latest linux-next;
Another throughput benchmarking tool (ntttcp-for-linux) test result is also listed at the end of the email for reference.


Server configuration:
------------------------------
Two servers (one client and one server, cross linked by 40G Ethernet), which have:
a) CPU: Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz, 2 sockets, 16 CPUs, cache size : 20480 KB
b) Memory: 64 GB
c) Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro], 40G Ethernet, default driver


Test with iperf3:
------------------------------
iperf3: https://github.com/esnet/iperf

a) SLES12 default kernel, network throughput tested by iperf3:
Test Connections	1	2	4	8	16	32	64	128	
Throughput (G bps)	36.7	37.3	37.6	37.7	37.7	37.7	37.7	25.7		

b) SLES12 + Linux-Next 20151105, network throughput tested by iperf3:
Test Connections	1	2	4	8	16	32	64	128
Throughput (G bps)	18.2	32.2	34.6	32.8	27.6	32.0	27.0	21.3
Percentage dropped	-50%	-14%	-8%	-13%	-27%	-15%	-28%	-17%


Test with ntttcp-for-linux:
------------------------------
ntttcp-for-linux: https://github.com/Microsoft/ntttcp-for-linux 

a) SLES12 default kernel, network throughput tested by ntttcp-for-linux:
Test Connections	1	2	4	8	16	32	64	128	256	512
Throughput (Gbps)	36.19	37.29	37.67	37.68	37.7	37.72	37.74	37.76	37.81	37.9	

b) SLES12 + Linux-Next 20151105, network throughput tested by ntttcp-for-linux:
Test Connections	1	2	4	8	16	32	64	128	256	512	
Throughput (Gbps)	28.12	34.01	37.6	36.53	32.94	33.07	33.63	33.44	33.83	34.42	
Percentage dropped	-22%	-9%	0%	-3%	-13%	-12%	-11%	-11%	-11%	-9%
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1264514

FromDavid Ahern <dsa@cumulusnetworks.com>
Date2015-11-06 21:50 +0100
Message-ID<qrW7q-YG-45@gated-at.bofh.it>
In reply to#1264460
On 11/6/15 1:31 PM, Simon Xiao wrote:
> I compared the network throughput performance on SLES12 bare metal servers, between SLES12 default kernel and latest linux-next (2015-11-05) kernel, based on the test results, I suspect there is a network regression exists on Linux-Next over the 40G Ethernet network:
> a) iperf3 reports 50% performance drop with single TCP stream on latest linux-next;
> b) iperf3 reports 10% ~ 30% performance drop with 2 to 128 TCP streams on latest linux-next;
> Another throughput benchmarking tool (ntttcp-for-linux) test result is also listed at the end of the email for reference.
>

Can you post your kernel config file?
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1264579

FromDavid Ahern <dsa@cumulusnetworks.com>
Date2015-11-06 22:40 +0100
Message-ID<qrWTM-1C2-15@gated-at.bofh.it>
In reply to#1264514
On 11/6/15 2:18 PM, Simon Xiao wrote:
> The .config file used to build linux-next kernel is attached to this mail.

Thanks.

Failed to notice this on the first response; my brain filled in. Why 
linux-next tree? Can you try net-next which is more relevant for this 
mailing list, post the top commit id and config file used?


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1264920

FromEric Dumazet <eric.dumazet@gmail.com>
Date2015-11-07 20:40 +0100
Message-ID<qshvb-6vE-1@gated-at.bofh.it>
In reply to#1264579
On Fri, 2015-11-06 at 14:30 -0700, David Ahern wrote:
> On 11/6/15 2:18 PM, Simon Xiao wrote:
> > The .config file used to build linux-next kernel is attached to this mail.
> 
> Thanks.
> 
> Failed to notice this on the first response; my brain filled in. Why 
> linux-next tree? Can you try net-next which is more relevant for this 
> mailing list, post the top commit id and config file used?

Throughput on a single TCP flow for a 40G NIC can be tricky to tune.

Make sure IRQ are properly setup/balanced, as I know that IRQ names were
changed recently and your scripts might have not noticed...

Also "ethtool -c eth0" might show very different interrupt coalescing
params ?

I too have a Mellanox 40Gb in my lab and saw no difference in
performance with recent kernels.

Of course, a simple "perf record -a -g sleep 4 ; perf report" might
point to some obvious issue. Like unexpected segmentation in case of
forwarding...



--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1264921

FromEric Dumazet <eric.dumazet@gmail.com>
Date2015-11-07 21:00 +0100
Message-ID<qshOy-6Ci-5@gated-at.bofh.it>
In reply to#1264920
On Sat, 2015-11-07 at 11:35 -0800, Eric Dumazet wrote:
> On Fri, 2015-11-06 at 14:30 -0700, David Ahern wrote:
> > On 11/6/15 2:18 PM, Simon Xiao wrote:
> > > The .config file used to build linux-next kernel is attached to this mail.
> > 
> > Thanks.
> > 
> > Failed to notice this on the first response; my brain filled in. Why 
> > linux-next tree? Can you try net-next which is more relevant for this 
> > mailing list, post the top commit id and config file used?
> 
> Throughput on a single TCP flow for a 40G NIC can be tricky to tune.
> 
> Make sure IRQ are properly setup/balanced, as I know that IRQ names were
> changed recently and your scripts might have not noticed...
> 
> Also "ethtool -c eth0" might show very different interrupt coalescing
> params ?
> 
> I too have a Mellanox 40Gb in my lab and saw no difference in
> performance with recent kernels.
> 
> Of course, a simple "perf record -a -g sleep 4 ; perf report" might
> point to some obvious issue. Like unexpected segmentation in case of
> forwarding...
> 
> 

I did a test with current net tree on both sender and receiver

lpaa23:~# ./netperf -H 10.246.7.152
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
10.246.7.152 () port 0 AF_INET
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

 87380  16384  16384    10.00    26864.98   
lpaa23:~# ethtool -c eth1
Coalesce parameters for eth1:
Adaptive RX: on  TX: off
stats-block-usecs: 0
sample-interval: 0
pkt-rate-low: 400000
pkt-rate-high: 450000

rx-usecs: 16
rx-frames: 44
rx-usecs-irq: 0
rx-frames-irq: 0

tx-usecs: 16
tx-frames: 16
tx-usecs-irq: 0
tx-frames-irq: 256

rx-usecs-low: 0
rx-frame-low: 0
tx-usecs-low: 0
tx-frame-low: 0

rx-usecs-high: 128
rx-frame-high: 0
tx-usecs-high: 0
tx-frame-high: 0

lpaa23:~# ethtool -C eth1 tx-usecs 4 tx-frames 4
lpaa23:~# ./netperf -H 10.246.7.152
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
10.246.7.152 () port 0 AF_INET
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

 87380  16384  16384    10.00    30206.27   


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1265990

FromSimon Xiao <sixiao@microsoft.com>
Date2015-11-09 21:30 +0100
Message-ID<qt1eH-2O0-9@gated-at.bofh.it>
In reply to#1264921
VGhhbmtzIEVyaWMgdG8gcHJvdmlkZSB0aGUgZGF0YS4gSSBhbSBsb29waW5nIFRvbSAoYXMgSSBh
bSBsb29raW5nIGludG8gaGlzIHJlY2VudCBwYXRjaGVzKSBhbmQgT2xhZiAoZnJvbSBTdXNlKS4N
Cg0KU28sIGlmIEkgdW5kZXJzdGFuZCBpdCBjb3JyZWN0bHksIHlvdSBhcmUgcnVubmluZyBuZXRw
ZXJmIHdpdGggc2luZ2xlIFRDUCBjb25uZWN0aW9uLCBhbmQgeW91IGdvdCB+MjZHYnBzIGluaXRp
YWxseSBhbmQgZ290IH4zMEdicHMgYWZ0ZXIgdHVybmluZyB0aGUgdHgtdXNlY3MgYW5kIHR4LWZy
YW1lcy4NCg0KRG8geW91IGhhdmUgYSBiYXNlbGluZSBvbiB5b3VyIGVudmlyb25tZW50IGZvciB0
aGUgYmVzdC9tYXgvb3IgcGVhayB0aHJvdWdocHV0Pw0KQWdhaW4sIGluIG15IGVudmlyb25tZW50
IChTTEVTIGJhcmUgbWV0YWwpLCBpZiB1c2UgU0xFUzEyIGRlZmF1bHQga2VybmVsIGFzIGEgYmFz
ZWxpbmUsIHdlIGNhbiBzZWUgc2lnbmlmaWNhbnQgcGVyZm9ybWFuY2UgZHJvcCAoMTAlIH4gNTAl
KSBvbiBsYXRlc3QgbGludXgtbmV4dCBrZXJuZWwuIA0KQWJzb2x1dGVseSBJIHdpbGwgdHJ5IHRo
ZSBzYW1lIHRlc3Qgb24gbmV0LW5leHQgc29vbiBhbmQgdXBkYXRlIHRoZSByZXN1bHRzIHRvIGhl
cmUgbGF0ZXIuDQoNClRoYW5rcywNClNpbW9uDQoNCg0KPiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2Ut
LS0tLQ0KPiBGcm9tOiBFcmljIER1bWF6ZXQgW21haWx0bzplcmljLmR1bWF6ZXRAZ21haWwuY29t
XQ0KPiBTZW50OiBTYXR1cmRheSwgTm92ZW1iZXIgNywgMjAxNSAxMTo1MCBBTQ0KPiBUbzogRGF2
aWQgQWhlcm4gPGRzYUBjdW11bHVzbmV0d29ya3MuY29tPg0KPiBDYzogU2ltb24gWGlhbyA8c2l4
aWFvQG1pY3Jvc29mdC5jb20+OyBkZXZlbEBsaW51eGRyaXZlcnByb2plY3Qub3JnOw0KPiBuZXRk
ZXZAdmdlci5rZXJuZWwub3JnOyBsaW51eC1rZXJuZWxAdmdlci5rZXJuZWwub3JnOyBEYXZpZCBN
aWxsZXINCj4gPGRhdmVtQGRhdmVtbG9mdC5uZXQ+OyBLWSBTcmluaXZhc2FuIDxreXNAbWljcm9z
b2Z0LmNvbT47IEhhaXlhbmcNCj4gWmhhbmcgPGhhaXlhbmd6QG1pY3Jvc29mdC5jb20+DQo+IFN1
YmplY3Q6IFJlOiBsaW51eC1uZXh0IG5ldHdvcmsgdGhyb3VnaHB1dCBwZXJmb3JtYW5jZSByZWdy
ZXNzaW9uDQo+IA0KPiBPbiBTYXQsIDIwMTUtMTEtMDcgYXQgMTE6MzUgLTA4MDAsIEVyaWMgRHVt
YXpldCB3cm90ZToNCj4gPiBPbiBGcmksIDIwMTUtMTEtMDYgYXQgMTQ6MzAgLTA3MDAsIERhdmlk
IEFoZXJuIHdyb3RlOg0KPiA+ID4gT24gMTEvNi8xNSAyOjE4IFBNLCBTaW1vbiBYaWFvIHdyb3Rl
Og0KPiA+ID4gPiBUaGUgLmNvbmZpZyBmaWxlIHVzZWQgdG8gYnVpbGQgbGludXgtbmV4dCBrZXJu
ZWwgaXMgYXR0YWNoZWQgdG8gdGhpcyBtYWlsLg0KPiA+ID4NCj4gPiA+IFRoYW5rcy4NCj4gPiA+
DQo+ID4gPiBGYWlsZWQgdG8gbm90aWNlIHRoaXMgb24gdGhlIGZpcnN0IHJlc3BvbnNlOyBteSBi
cmFpbiBmaWxsZWQgaW4uIFdoeQ0KPiA+ID4gbGludXgtbmV4dCB0cmVlPyBDYW4geW91IHRyeSBu
ZXQtbmV4dCB3aGljaCBpcyBtb3JlIHJlbGV2YW50IGZvcg0KPiA+ID4gdGhpcyBtYWlsaW5nIGxp
c3QsIHBvc3QgdGhlIHRvcCBjb21taXQgaWQgYW5kIGNvbmZpZyBmaWxlIHVzZWQ/DQo+ID4NCj4g
PiBUaHJvdWdocHV0IG9uIGEgc2luZ2xlIFRDUCBmbG93IGZvciBhIDQwRyBOSUMgY2FuIGJlIHRy
aWNreSB0byB0dW5lLg0KPiA+DQo+ID4gTWFrZSBzdXJlIElSUSBhcmUgcHJvcGVybHkgc2V0dXAv
YmFsYW5jZWQsIGFzIEkga25vdyB0aGF0IElSUSBuYW1lcw0KPiA+IHdlcmUgY2hhbmdlZCByZWNl
bnRseSBhbmQgeW91ciBzY3JpcHRzIG1pZ2h0IGhhdmUgbm90IG5vdGljZWQuLi4NCj4gPg0KPiA+
IEFsc28gImV0aHRvb2wgLWMgZXRoMCIgbWlnaHQgc2hvdyB2ZXJ5IGRpZmZlcmVudCBpbnRlcnJ1
cHQgY29hbGVzY2luZw0KPiA+IHBhcmFtcyA/DQo+ID4NCj4gPiBJIHRvbyBoYXZlIGEgTWVsbGFu
b3ggNDBHYiBpbiBteSBsYWIgYW5kIHNhdyBubyBkaWZmZXJlbmNlIGluDQo+ID4gcGVyZm9ybWFu
Y2Ugd2l0aCByZWNlbnQga2VybmVscy4NCj4gPg0KPiA+IE9mIGNvdXJzZSwgYSBzaW1wbGUgInBl
cmYgcmVjb3JkIC1hIC1nIHNsZWVwIDQgOyBwZXJmIHJlcG9ydCIgbWlnaHQNCj4gPiBwb2ludCB0
byBzb21lIG9idmlvdXMgaXNzdWUuIExpa2UgdW5leHBlY3RlZCBzZWdtZW50YXRpb24gaW4gY2Fz
ZSBvZg0KPiA+IGZvcndhcmRpbmcuLi4NCj4gPg0KPiA+DQo+IA0KPiBJIGRpZCBhIHRlc3Qgd2l0
aCBjdXJyZW50IG5ldCB0cmVlIG9uIGJvdGggc2VuZGVyIGFuZCByZWNlaXZlcg0KPiANCj4gbHBh
YTIzOn4jIC4vbmV0cGVyZiAtSCAxMC4yNDYuNy4xNTINCj4gTUlHUkFURUQgVENQIFNUUkVBTSBU
RVNUIGZyb20gMC4wLjAuMCAoMC4wLjAuMCkgcG9ydCAwIEFGX0lORVQgdG8NCj4gMTAuMjQ2Ljcu
MTUyICgpIHBvcnQgMCBBRl9JTkVUDQo+IFJlY3YgICBTZW5kICAgIFNlbmQNCj4gU29ja2V0IFNv
Y2tldCAgTWVzc2FnZSAgRWxhcHNlZA0KPiBTaXplICAgU2l6ZSAgICBTaXplICAgICBUaW1lICAg
ICBUaHJvdWdocHV0DQo+IGJ5dGVzICBieXRlcyAgIGJ5dGVzICAgIHNlY3MuICAgIDEwXjZiaXRz
L3NlYw0KPiANCj4gIDg3MzgwICAxNjM4NCAgMTYzODQgICAgMTAuMDAgICAgMjY4NjQuOTgNCj4g
bHBhYTIzOn4jIGV0aHRvb2wgLWMgZXRoMQ0KPiBDb2FsZXNjZSBwYXJhbWV0ZXJzIGZvciBldGgx
Og0KPiBBZGFwdGl2ZSBSWDogb24gIFRYOiBvZmYNCj4gc3RhdHMtYmxvY2stdXNlY3M6IDANCj4g
c2FtcGxlLWludGVydmFsOiAwDQo+IHBrdC1yYXRlLWxvdzogNDAwMDAwDQo+IHBrdC1yYXRlLWhp
Z2g6IDQ1MDAwMA0KPiANCj4gcngtdXNlY3M6IDE2DQo+IHJ4LWZyYW1lczogNDQNCj4gcngtdXNl
Y3MtaXJxOiAwDQo+IHJ4LWZyYW1lcy1pcnE6IDANCj4gDQo+IHR4LXVzZWNzOiAxNg0KPiB0eC1m
cmFtZXM6IDE2DQo+IHR4LXVzZWNzLWlycTogMA0KPiB0eC1mcmFtZXMtaXJxOiAyNTYNCj4gDQo+
IHJ4LXVzZWNzLWxvdzogMA0KPiByeC1mcmFtZS1sb3c6IDANCj4gdHgtdXNlY3MtbG93OiAwDQo+
IHR4LWZyYW1lLWxvdzogMA0KPiANCj4gcngtdXNlY3MtaGlnaDogMTI4DQo+IHJ4LWZyYW1lLWhp
Z2g6IDANCj4gdHgtdXNlY3MtaGlnaDogMA0KPiB0eC1mcmFtZS1oaWdoOiAwDQo+IA0KPiBscGFh
MjM6fiMgZXRodG9vbCAtQyBldGgxIHR4LXVzZWNzIDQgdHgtZnJhbWVzIDQNCg0KDQo+IGxwYWEy
Mzp+IyAuL25ldHBlcmYgLUgNCj4gMTAuMjQ2LjcuMTUyIE1JR1JBVEVEIFRDUCBTVFJFQU0gVEVT
VCBmcm9tIDAuMC4wLjAgKDAuMC4wLjApIHBvcnQgMA0KPiBBRl9JTkVUIHRvDQo+IDEwLjI0Ni43
LjE1MiAoKSBwb3J0IDAgQUZfSU5FVA0KPiBSZWN2ICAgU2VuZCAgICBTZW5kDQo+IFNvY2tldCBT
b2NrZXQgIE1lc3NhZ2UgIEVsYXBzZWQNCj4gU2l6ZSAgIFNpemUgICAgU2l6ZSAgICAgVGltZSAg
ICAgVGhyb3VnaHB1dA0KPiBieXRlcyAgYnl0ZXMgICBieXRlcyAgICBzZWNzLiAgICAxMF42Yml0
cy9zZWMNCj4gDQo+ICA4NzM4MCAgMTYzODQgIDE2Mzg0ICAgIDEwLjAwICAgIDMwMjA2LjI3DQo+
IA0KDQo=
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1266070

FromEric Dumazet <eric.dumazet@gmail.com>
Date2015-11-10 00:10 +0100
Message-ID<qt3Jv-5gt-13@gated-at.bofh.it>
In reply to#1265990
On Mon, 2015-11-09 at 20:23 +0000, Simon Xiao wrote:
> Thanks Eric to provide the data. I am looping Tom (as I am looking into his recent patches) and Olaf (from Suse).
> 
> So, if I understand it correctly, you are running netperf with single
> TCP connection, and you got ~26Gbps initially and got ~30Gbps after
> turning the tx-usecs and tx-frames.
> 
> Do you have a baseline on your environment for the best/max/or peak
> throughput?

The peak on my lab pair is about 34Gbits, usually I get this if I pin
the receiving thread on a cpu, otherwise process scheduler can really
hurt too much.

lpaa23:~# DUMP_TCP_INFO=1 ./netperf -H lpaa24 -l 20 -Cc -T ,1
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to lpaa24.prod.google.com () port 0 AF_INET : cpu bind
tcpi_rto 201000 tcpi_ato 0 tcpi_pmtu 1500 tcpi_rcv_ssthresh 29200
tcpi_rtt 101 tcpi_rttvar 15 tcpi_snd_ssthresh 289 tpci_snd_cwnd 289
tcpi_reordering 3 tcpi_total_retrans 453
Recv   Send    Send                          Utilization       Service Demand
Socket Socket  Message  Elapsed              Send     Recv     Send    Recv
Size   Size    Size     Time     Throughput  local    remote   local   remote
bytes  bytes   bytes    secs.    10^6bits/s  % S      % S      us/KB   us/KB

 87380  16384  16384    20.00      33975.99   1.27     3.36     0.147   0.389  

Not too bad, I don't recall reaching more than that ever.


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1265308

FromDavid Miller <davem@davemloft.net>
Date2015-11-09 04:00 +0100
Message-ID<qsKQx-hN-1@gated-at.bofh.it>
In reply to#1264920
From: Dexuan Cui <decui@microsoft.com>
Date: Mon, 9 Nov 2015 02:39:24 +0000

>> Throughput on a single TCP flow for a 40G NIC can be tricky to tune.
> Why is a single TCP flow trickier than multiple TCP flows?
> IMO it should be easier to analyze the issue of a single TCP flow?

Because a single TCP flow can only use one of the many TX queues
that such modern NICs have.

The single TX queue becomes the bottleneck.

Whereas if you have several TCP flows, all of them can use independant
TX queues on the NIC in parallel to fill the link with traffic.

That's why.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1265319

FromDavid Miller <davem@davemloft.net>
Date2015-11-09 04:30 +0100
Message-ID<qsLjz-OT-3@gated-at.bofh.it>
In reply to#1265308
From: Dexuan Cui <decui@microsoft.com>
Date: Mon, 9 Nov 2015 03:11:35 +0000

>> -----Original Message-----
>> From: David Miller [mailto:davem@davemloft.net]
>> Sent: Monday, November 9, 2015 10:53
>> To: Dexuan Cui <decui@microsoft.com>
>> Cc: eric.dumazet@gmail.com; dsa@cumulusnetworks.com; Simon Xiao
>> <sixiao@microsoft.com>; netdev@vger.kernel.org; Haiyang Zhang
>> <haiyangz@microsoft.com>; linux-kernel@vger.kernel.org;
>> devel@linuxdriverproject.org
>> Subject: Re: linux-next network throughput performance regression
>> 
>> From: Dexuan Cui <decui@microsoft.com>
>> Date: Mon, 9 Nov 2015 02:39:24 +0000
>> 
>> >> Throughput on a single TCP flow for a 40G NIC can be tricky to tune.
>> > Why is a single TCP flow trickier than multiple TCP flows?
>> > IMO it should be easier to analyze the issue of a single TCP flow?
>> 
>> Because a single TCP flow can only use one of the many TX queues
>> that such modern NICs have.
>> 
>> The single TX queue becomes the bottleneck.
>> 
>> Whereas if you have several TCP flows, all of them can use independant
>> TX queues on the NIC in parallel to fill the link with traffic.
>> 
>> That's why.
> 
> Thanks, David!
> I understand 1 TX queue is the bottleneck (however in Simon's
> test, TX=1 => 36.7Gb/s, TX=8 => 37.7 Gb/s, so it looks the TX=1 bottleneck
> is not so obvious).
> I'm just wondering how the bottleneck became much narrower with
> recent linux-next in Simon's result (36.7 Gb/s vs. 18.2 Gb/s). IMO there
> must be some latency somewhere.

I think the whole thing here is that you misinterpreted what Eric said.

He is not arguing that some regression did, or did not, happen.

He instead was making the basic statement about the fact that due to
the lack of paralellness a single stream TCP case is harder to
optimize for high speed NICs.

That is all.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1265320

FromDave Airlie <airlied@gmail.com>
Date2015-11-09 04:40 +0100
Message-ID<qsLtf-RR-1@gated-at.bofh.it>
In reply to#1265319
On 9 November 2015 at 13:23, David Miller <davem@davemloft.net> wrote:
> From: Dexuan Cui <decui@microsoft.com>
> Date: Mon, 9 Nov 2015 03:11:35 +0000
>
>>> -----Original Message-----
>>> From: David Miller [mailto:davem@davemloft.net]
>>> Sent: Monday, November 9, 2015 10:53
>>> To: Dexuan Cui <decui@microsoft.com>
>>> Cc: eric.dumazet@gmail.com; dsa@cumulusnetworks.com; Simon Xiao
>>> <sixiao@microsoft.com>; netdev@vger.kernel.org; Haiyang Zhang
>>> <haiyangz@microsoft.com>; linux-kernel@vger.kernel.org;
>>> devel@linuxdriverproject.org
>>> Subject: Re: linux-next network throughput performance regression
>>>
>>> From: Dexuan Cui <decui@microsoft.com>
>>> Date: Mon, 9 Nov 2015 02:39:24 +0000
>>>
>>> >> Throughput on a single TCP flow for a 40G NIC can be tricky to tune.
>>> > Why is a single TCP flow trickier than multiple TCP flows?
>>> > IMO it should be easier to analyze the issue of a single TCP flow?
>>>
>>> Because a single TCP flow can only use one of the many TX queues
>>> that such modern NICs have.
>>>
>>> The single TX queue becomes the bottleneck.
>>>
>>> Whereas if you have several TCP flows, all of them can use independant
>>> TX queues on the NIC in parallel to fill the link with traffic.
>>>
>>> That's why.
>>
>> Thanks, David!
>> I understand 1 TX queue is the bottleneck (however in Simon's
>> test, TX=1 => 36.7Gb/s, TX=8 => 37.7 Gb/s, so it looks the TX=1 bottleneck
>> is not so obvious).
>> I'm just wondering how the bottleneck became much narrower with
>> recent linux-next in Simon's result (36.7 Gb/s vs. 18.2 Gb/s). IMO there
>> must be some latency somewhere.
>
> I think the whole thing here is that you misinterpreted what Eric said.
>
> He is not arguing that some regression did, or did not, happen.
>
> He instead was making the basic statement about the fact that due to
> the lack of paralellness a single stream TCP case is harder to
> optimize for high speed NICs.
>
> That is all.

We recently had a regression tracked down in a similiar area that was
because of link order.

Dave.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1265363

FromTom Herbert <tom@herbertland.com>
Date2015-11-09 06:40 +0100
Message-ID<qsNlo-2cc-5@gated-at.bofh.it>
In reply to#1265319
On Sun, Nov 8, 2015 at 7:31 PM, Dexuan Cui <decui@microsoft.com> wrote:
>> From: David Miller [mailto:davem@davemloft.net]
>> Sent: Monday, November 9, 2015 11:24
>> ...
>> > Thanks, David!
>> > I understand 1 TX queue is the bottleneck (however in Simon's
>> > test, TX=1 => 36.7Gb/s, TX=8 => 37.7 Gb/s, so it looks the TX=1 bottleneck
>> > is not so obvious).
>> > I'm just wondering how the bottleneck became much narrower with
>> > recent linux-next in Simon's result (36.7 Gb/s vs. 18.2 Gb/s). IMO there
>> > must be some latency somewhere.
>>
>> I think the whole thing here is that you misinterpreted what Eric said.
>>
>> He is not arguing that some regression did, or did not, happen.
>>
>> He instead was making the basic statement about the fact that due to
>> the lack of paralellness a single stream TCP case is harder to
>> optimize for high speed NICs.
>>
>> That is all.
> Thanks, I got it.
> I'm actually new to network performance tuning, trying to understand
> all the related details. :-)
>

You might want to look at
https://www.kernel.org/doc/Documentation/networking/scaling.txt as an
introduction to the scaling capabilities of the stack.

Tom

> Thanks,
> -- Dexuan
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web