Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.debian.user > #268128 > unrolled thread
| Started by | Stefan K <Shadow_7@gmx.net> |
|---|---|
| First post | 2024-03-07 10:20 +0100 |
| Last post | 2024-03-09 20:20 +0100 |
| Articles | 14 — 6 participants |
Back to article view | Back to linux.debian.user
very poor nfs performance Stefan K <Shadow_7@gmx.net> - 2024-03-07 10:20 +0100
Re: very poor nfs performance Ralph Aichinger <ra@h5.or.at> - 2024-03-07 11:40 +0100
Aw: Re: very poor nfs performance Stefan K <Shadow_7@gmx.net> - 2024-03-07 12:40 +0100
Re: very poor nfs performance Mike Kupfer <kupfer@rawbw.com> - 2024-03-08 07:10 +0100
Aw: Re: very poor nfs performance Stefan K <Shadow_7@gmx.net> - 2024-03-08 17:10 +0100
Re: very poor nfs performance debian-user@howorth.org.uk - 2024-03-08 17:20 +0100
Aw: Re: very poor nfs performance Stefan K <Shadow_7@gmx.net> - 2024-03-08 17:40 +0100
Re: very poor nfs performance debian-user@howorth.org.uk - 2024-03-08 17:50 +0100
Aw: Re: Re: very poor nfs performance Stefan K <Shadow_7@gmx.net> - 2024-03-08 17:40 +0100
Re: Aw: Re: Re: very poor nfs performance Mike Kupfer <kupfer@rawbw.com> - 2024-03-08 18:20 +0100
Re: very poor nfs performance Dan Ritter <dsr@randomstring.org> - 2024-03-08 19:00 +0100
Re: very poor nfs performance Michel Verdier <mv524@free.fr> - 2024-03-08 12:00 +0100
Aw: Re: very poor nfs performance Stefan K <Shadow_7@gmx.net> - 2024-03-08 17:00 +0100
Re: very poor nfs performance Ralph Aichinger <ra@h5.or.at> - 2024-03-09 20:20 +0100
| From | Stefan K <Shadow_7@gmx.net> |
|---|---|
| Date | 2024-03-07 10:20 +0100 |
| Subject | very poor nfs performance |
| Message-ID | <Ifi4V-eZNA-1@gated-at.bofh.it> |
Hello guys, I hope someone can help me with my problem. Our NFS performance ist very bad, like ~20MB/s, mountoption looks like that: rw,relatime,sync,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,local_lock=none The NFS server (debian 12) is a ZFS Fileserver with 40G network connection, also the read/write performance is good: fio --rw=readwrite --rwmixread=70 --name=test --size=50G --direct=1 --bsrange=4k-1M --numjobs=30 --group_reporting --filename=/zfs_storage/test/asdfg --runtime=600 --time_based READ: bw=11.1GiB/s (11.9GB/s), 11.1GiB/s-11.1GiB/s (11.9GB/s-11.9GB/s), io=6665GiB (7156GB), run=600004-600004msec WRITE: bw=4875MiB/s (5112MB/s), 4875MiB/s-4875MiB/s (5112MB/s-5112MB/s), io=2856GiB (3067GB), run=600004-600004msec Only one nfs client(debian 12) is connected via 10G, since we host also database files on the nfs share, 'sync'-mountoption is important (more or less), but it should still be much faster than 20MB/s so can somebody tell me whats wrong or what should I change to speed that up? thanks in advance. best regards Stefan
[toc] | [next] | [standalone]
| From | Ralph Aichinger <ra@h5.or.at> |
|---|---|
| Date | 2024-03-07 11:40 +0100 |
| Message-ID | <Ifjkm-f0rZ-13@gated-at.bofh.it> |
| In reply to | #268128 |
On Thu, 2024-03-07 at 10:13 +0100, Stefan K wrote: > Hello guys, > > I hope someone can help me with my problem. > Our NFS performance ist very bad, like ~20MB/s, mountoption looks > like that: Are both sides agreeing on MTU (using Jumbo frames or not)? Have you tested the network with iperf (or simiar), does this happen only with NFS or also with other network traffic? /ralph
[toc] | [prev] | [next] | [standalone]
| From | Stefan K <Shadow_7@gmx.net> |
|---|---|
| Date | 2024-03-07 12:40 +0100 |
| Subject | Aw: Re: very poor nfs performance |
| Message-ID | <Ifkgp-f116-1@gated-at.bofh.it> |
| In reply to | #268132 |
Hi Ralph, I just tested it with scp and I got 262MB/s So it's not a network issue, just a NFS issue, somehow. best regards Stefan > Gesendet: Donnerstag, 07. März 2024 um 11:22 Uhr > Von: "Ralph Aichinger" <ra@h5.or.at> > An: debian-user@lists.debian.org > Betreff: Re: very poor nfs performance > > On Thu, 2024-03-07 at 10:13 +0100, Stefan K wrote: > > Hello guys, > > > > I hope someone can help me with my problem. > > Our NFS performance ist very bad, like ~20MB/s, mountoption looks > > like that: > > Are both sides agreeing on MTU (using Jumbo frames or not)? > > Have you tested the network with iperf (or simiar), does this happen > only with NFS or also with other network traffic? > > /ralph > >
[toc] | [prev] | [next] | [standalone]
| From | Mike Kupfer <kupfer@rawbw.com> |
|---|---|
| Date | 2024-03-08 07:10 +0100 |
| Message-ID | <IfBAB-fbW2-1@gated-at.bofh.it> |
| In reply to | #268128 |
Stefan K wrote: > 'sync'-mountoption is important (more or less), but it should still be > much faster than 20MB/s I don't know if "sync" could be entirely responsible for such a slowdown, but it's likely at least contributing, particularly if the application is doing small I/Os at the system call level. You could try removing the "sync" option, just as an experiment, to see how much it is contributing to the slowdown. mike
[toc] | [prev] | [next] | [standalone]
| From | Stefan K <Shadow_7@gmx.net> |
|---|---|
| Date | 2024-03-08 17:10 +0100 |
| Subject | Aw: Re: very poor nfs performance |
| Message-ID | <IfKXf-fhDy-1@gated-at.bofh.it> |
| In reply to | #268168 |
> You could try removing the "sync" option, just as an experiment, to see > how much it is contributing to the slowdown. If I don't use sync I got around 300MB/s (tested with 600MB-file) .. that's ok (far from great), but since there are database files on the nfs it can cause file/database corruption, so we would like to use sync option best regards Stefan
[toc] | [prev] | [next] | [standalone]
| From | debian-user@howorth.org.uk |
|---|---|
| Date | 2024-03-08 17:20 +0100 |
| Message-ID | <IfL6V-fhGP-1@gated-at.bofh.it> |
| In reply to | #268176 |
Stefan K <Shadow_7@gmx.net> wrote: > > You could try removing the "sync" option, just as an experiment, to > > see how much it is contributing to the slowdown. > > If I don't use sync I got around 300MB/s (tested with 600MB-file) .. > that's ok (far from great), but since there are database files on the > nfs it can cause file/database corruption, so we would like to use > sync option Run the database on the machine that stores the files and perform database access remotely over the net instead. ? > best regards > Stefan
[toc] | [prev] | [next] | [standalone]
| From | Stefan K <Shadow_7@gmx.net> |
|---|---|
| Date | 2024-03-08 17:40 +0100 |
| Subject | Aw: Re: very poor nfs performance |
| Message-ID | <IfLqh-fhMY-9@gated-at.bofh.it> |
| In reply to | #268178 |
> Run the database on the machine that stores the files and perform > database access remotely over the net instead. ? yes, but this doesn't resolve the performance issue with nfs
[toc] | [prev] | [next] | [standalone]
| From | debian-user@howorth.org.uk |
|---|---|
| Date | 2024-03-08 17:50 +0100 |
| Message-ID | <IfLzX-fhQf-9@gated-at.bofh.it> |
| In reply to | #268181 |
Stefan K <Shadow_7@gmx.net> wrote: > > Run the database on the machine that stores the files and perform > > database access remotely over the net instead. ? > > yes, but this doesn't resolve the performance issue with nfs But it removes your issue that forces you to use the sync option.
[toc] | [prev] | [next] | [standalone]
| From | Stefan K <Shadow_7@gmx.net> |
|---|---|
| Date | 2024-03-08 17:40 +0100 |
| Subject | Aw: Re: Re: very poor nfs performance |
| Message-ID | <IfLqh-fhMY-1@gated-at.bofh.it> |
| In reply to | #268168 |
> Can you partition the files into 2 different shares? Put the database > files in one share and access them using "sync", and put the rest of the > files in a different share, with no "sync"? this could be a solution, but I want to understand why is it so slow and fix that
[toc] | [prev] | [next] | [standalone]
| From | Mike Kupfer <kupfer@rawbw.com> |
|---|---|
| Date | 2024-03-08 18:20 +0100 |
| Subject | Re: Aw: Re: Re: very poor nfs performance |
| Message-ID | <IfM30-fif7-1@gated-at.bofh.it> |
| In reply to | #268180 |
Stefan K wrote: > > Can you partition the files into 2 different shares? Put the database > > files in one share and access them using "sync", and put the rest of the > > files in a different share, with no "sync"? > this could be a solution, but I want to understand why is it so slow and fix that It's inherent in how sync works. Over-the-wire calls are expensive. NFS implementations try to get acceptable performance by extensive caching, using asynchronous operations when possible, and by issuing a smaller number of large RPCs (rather than a larger number of small RPCs). The sync option defeats all of those mechanisms. mike
[toc] | [prev] | [next] | [standalone]
| From | Dan Ritter <dsr@randomstring.org> |
|---|---|
| Date | 2024-03-08 19:00 +0100 |
| Message-ID | <IfMFI-fisn-3@gated-at.bofh.it> |
| In reply to | #268183 |
Mike Kupfer wrote: > Stefan K wrote: > > > > Can you partition the files into 2 different shares? Put the database > > > files in one share and access them using "sync", and put the rest of the > > > files in a different share, with no "sync"? > > this could be a solution, but I want to understand why is it so slow and fix that > > It's inherent in how sync works. Over-the-wire calls are expensive. > NFS implementations try to get acceptable performance by extensive > caching, using asynchronous operations when possible, and by issuing a > smaller number of large RPCs (rather than a larger number of small > RPCs). The sync option defeats all of those mechanisms. It is also the case that databases absolutely need sync to work properly, so running them over NFS is a bad idea. At most, a sqlite DB can be OK -- because sqlite is single user. -dsr-
[toc] | [prev] | [next] | [standalone]
| From | Michel Verdier <mv524@free.fr> |
|---|---|
| Date | 2024-03-08 12:00 +0100 |
| Message-ID | <IfG7f-feAi-5@gated-at.bofh.it> |
| In reply to | #268128 |
On 2024-03-07, Stefan K wrote: > I hope someone can help me with my problem. > Our NFS performance ist very bad, like ~20MB/s, mountoption looks like that: > rw,relatime,sync,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,local_lock=none What are the mount options for your zfs filesystem ? You could test with noatime if you don't need access times. And perhaps with lazytime instead of relatime. Could you provide us nfsstat -v
[toc] | [prev] | [next] | [standalone]
| From | Stefan K <Shadow_7@gmx.net> |
|---|---|
| Date | 2024-03-08 17:00 +0100 |
| Subject | Aw: Re: very poor nfs performance |
| Message-ID | <IfKNz-fhl6-1@gated-at.bofh.it> |
| In reply to | #268170 |
> You could test with noatime if you don't need access times. > And perhaps with lazytime instead of relatime. Mountoptions are: type zfs (rw,xattr,noacl) I get you point, but when you look at my fio output, the performance is quiet good > Could you provide us > nfsstat -v server: nfsstat -v Server packet stats: packets udp tcp tcpconn 509979521 0 510004972 2 Server rpc stats: calls badcalls badfmt badauth badclnt 509971853 0 0 0 0 Server reply cache: hits misses nocache 0 0 509980028 Server io stats: read write 1587531840 3079615002 Server read ahead cache: size 0-10% 10-20% 20-30% 30-40% 40-50% 50-60% 60-70% 70-80% 80-90% 90-100% notfound 0 0 0 0 0 0 0 0 0 0 0 0 Server file handle cache: lookup anon ncachedir ncachenondir stale 0 0 0 0 0 Server nfs v4: null compound 2 0% 509976662 99% Server nfs v4 operations: op0-unused op1-unused op2-future access close 0 0% 0 0% 0 0% 5015903 0% 3091693 0% commit create delegpurge delegreturn getattr 314634 0% 149836 0% 0 0% 1615740 0% 390748077 20% getfh link lock lockt locku 2573550 0% 0 0% 17 0% 0 0% 15 0% lookup lookup_root nverify open openattr 3931149 0% 0 0% 0 0% 3131045 0% 0 0% open_conf open_dgrd putfh putpubfh putrootfh 0 0% 3 0% 510522216 26% 0 0% 4 0% read readdir readlink remove rename 59976532 3% 421791 0% 0 0% 429965 0% 244564 0% renew restorefh savefh secinfo setattr 0 0% 0 0% 542231 0% 0 0% 845324 0% setcltid setcltidconf verify write rellockowner 0 0% 0 0% 0 0% 404569758 21% 0 0% bc_ctl bind_conn exchange_id create_ses destroy_ses 0 0% 0 0% 4 0% 2 0% 1 0% free_stateid getdirdeleg getdevinfo getdevlist layoutcommit 15 0% 0 0% 0 0% 0 0% 0 0% layoutget layoutreturn secinfononam sequence set_ssv 0 0% 0 0% 2 0% 509980018 26% 0 0% test_stateid want_deleg destroy_clid reclaim_comp allocate 10 0% 0 0% 1 0% 2 0% 164 0% copy copy_notify deallocate ioadvise layouterror 297667 0% 0 0% 0 0% 0 0% 0 0% layoutstats offloadcancel offloadstatus readplus seek 0 0% 0 0% 0 0% 0 0% 0 0% write_same 0 0% client: nfsstat -v Client packet stats: packets udp tcp tcpconn 0 0 0 0 Client rpc stats: calls retrans authrefrsh 37415730 0 37425651 Client nfs v4: null read write commit open 1 0% 4107833 10% 30388717 81% 2516 0% 55493 0% open_conf open_noat open_dgrd close setattr 0 0% 194252 0% 0 0% 247380 0% 75890 0% fsinfo renew setclntid confirm lock 459 0% 0 0% 0 0% 0 0% 4 0% lockt locku access getattr lookup 0 0% 2 0% 131533 0% 1497029 4% 318056 0% lookup_root remove rename link symlink 1 0% 31656 0% 15877 0% 0 0% 0 0% create pathconf statfs readlink readdir 7019 0% 458 0% 170522 0% 0 0% 30007 0% server_caps delegreturn getacl setacl fs_locations 917 0% 118109 0% 0 0% 0 0% 0 0% rel_lkowner secinfo fsid_present exchange_id create_session 0 0% 0 0% 0 0% 2 0% 1 0% destroy_session sequence get_lease_time reclaim_comp layoutget 0 0% 0 0% 0 0% 1 0% 0 0% getdevinfo layoutcommit layoutreturn secinfo_no test_stateid 0 0% 0 0% 0 0% 1 0% 0 0% free_stateid getdevicelist bind_conn_to_ses destroy_clientid seek 2 0% 0 0% 0 0% 0 0% 0 0% allocate deallocate layoutstats clone 10 0% 0 0% 0 0% 0 0% thanks for your help best regards Stefan
[toc] | [prev] | [next] | [standalone]
| From | Ralph Aichinger <ra@h5.or.at> |
|---|---|
| Date | 2024-03-09 20:20 +0100 |
| Message-ID | <IgaoF-fwVz-9@gated-at.bofh.it> |
| In reply to | #268128 |
On Sat, 2024-03-09 at 13:54 +0100, hw wrote: > > NFS can be hard on network card drivers > IPv6 may be faster than IPv4 > the network cable might suck > the switch might suck or block stuff As iperf and other network protocols were confirmed to be fast by the OP it is very unlikely that it is a straight network problem. Yes, these effects do exist occasionally (weird interactions of higher level protocols and the low level stuff), but it is very rare. The cable that is so specifically broken to slow down NFS but not scp might exist, but it is very unlikely. /ralph
[toc] | [prev] | [standalone]
Back to top | Article view | linux.debian.user
csiph-web