Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #199225 > unrolled thread

Slow XFS write

Back to article view | Back to linux.debian.user

Contents

  Slow XFS write Martin LEUSCH <sce.tech@imereos.fr> - 2018-08-22 12:00 +0200
    Re: Slow XFS write Michael Stone <mstone@debian.org> - 2018-08-22 13:20 +0200
      Re: Slow XFS write Martin LEUSCH <sce.tech@imereos.fr> - 2018-08-22 14:30 +0200
        Re: Slow XFS write Michael Stone <mstone@debian.org> - 2018-08-22 15:20 +0200
          Re: Slow XFS write Martin LEUSCH <sce.tech@imereos.fr> - 2018-08-22 16:10 +0200
            Re: Slow XFS write Michael Stone <mstone@debian.org> - 2018-08-22 18:00 +0200
              Re: Slow XFS write Michael Stone <mstone@debian.org> - 2018-08-22 18:00 +0200
    Re: Slow XFS write Martin LEUSCH <sce.tech@imereos.fr> - 2018-08-22 16:20 +0200
      Re: Slow XFS write Reco <recoverym4n@gmail.com> - 2018-08-22 19:40 +0200
    Re: Slow XFS write Stefan K <shadow_7@gmx.net> - 2018-08-23 09:10 +0200
      Re: Slow XFS write Martin LEUSCH <sce.tech@imereos.fr> - 2018-08-23 11:20 +0200

#199225 — Slow XFS write

Hi,


I have a NFS server with a hardware RAID5 on 3 HDD of 6 TB. I have a 
system partition with ext4 and a data partition with XFS.

I get only 10 MB/s in write speed on the XFS data partition and 80 MB/s 
on the system partition.


XFS mount option:

/dev/sda4 on /var/srv/nfs type xfs (rw,relatime,attr2,inode64,noquota)

Is there a way to have better performance? Is XFS a good choice in this 
situation?

[toc] | [next] | [standalone]

#199230

On Wed, Aug 22, 2018 at 11:41:08AM +0200, Martin LEUSCH wrote:
>I have a NFS server with a hardware RAID5 on 3 HDD of 6 TB. I have a 
>system partition with ext4 and a data partition with XFS.
>
>I get only 10 MB/s in write speed on the XFS data partition and 80 
>MB/s on the system partition.

how are you testing this?

>Is there a way to have better performance? Is XFS a good choice in 
>this situation?

it's unlikely that the filesystem is the issue, both xfs and ext4 are 
more than capable of saturating a 3 disk raid5.

Mike Stone

[toc] | [prev] | [next] | [standalone]

#199231

[Multipart message — attachments visible in raw view] — view raw

Le 22/08/2018 à 13:15, Michael Stone a écrit :
> On Wed, Aug 22, 2018 at 11:41:08AM +0200, Martin LEUSCH wrote:
>> I have a NFS server with a hardware RAID5 on 3 HDD of 6 TB. I have a 
>> system partition with ext4 and a data partition with XFS.
>>
>> I get only 10 MB/s in write speed on the XFS data partition and 80 
>> MB/s on the system partition.
>
> how are you testing this? 
I tested write speed with dd command like "dd if=/dev/zero 
of=/var/srv/nfs/testfile bs=1G count=1 oflag=direct".
The 10 MB/s for the data partition also correspond to the behavior I get 
in real situation, when I copy a big file on NFS, 12GB are copied 
quickly then hang for 15 or 20 minutes. When I copy big file with scp, 
It copy quickly the at the beginning then decrease to 10 MB/s till the end.

[toc] | [prev] | [next] | [standalone]

#199232

On Wed, Aug 22, 2018 at 02:25:51PM +0200, Martin LEUSCH wrote:
>I tested write speed with dd command like "dd if=/dev/zero of=/var/srv/nfs/
>testfile bs=1G count=1 oflag=direct".
>The 10 MB/s for the data partition also correspond to the behavior I get in
>real situation, when I copy a big file on NFS, 12GB are copied quickly then
>hang for 15 or 20 minutes. When I copy big file with scp, It copy quickly the
>at the beginning then decrease to 10 MB/s till the end.

what happens with
dd if=/dev/zero of=/var/srv/nfs/testfile bs=64k count=16k conv=fsync

And this is direct on the server, not via NFS, right?

[toc] | [prev] | [next] | [standalone]

#199235

Le 22/08/2018 à 15:17, Michael Stone a écrit :
> On Wed, Aug 22, 2018 at 02:25:51PM +0200, Martin LEUSCH wrote:
>> I tested write speed with dd command like "dd if=/dev/zero 
>> of=/var/srv/nfs/
>> testfile bs=1G count=1 oflag=direct".
>> The 10 MB/s for the data partition also correspond to the behavior I 
>> get in
>> real situation, when I copy a big file on NFS, 12GB are copied 
>> quickly then
>> hang for 15 or 20 minutes. When I copy big file with scp, It copy 
>> quickly the
>> at the beginning then decrease to 10 MB/s till the end.
>
> what happens with
> dd if=/dev/zero of=/var/srv/nfs/testfile bs=64k count=16k conv=fsync
>
> And this is direct on the server, not via NFS, right?
>
I've got same result as previous test.
tests with dd command are executed directly on the server.

[toc] | [prev] | [next] | [standalone]

#199240

On Wed, Aug 22, 2018 at 04:02:42PM +0200, Martin LEUSCH wrote:
>>what happens with
>>dd if=/dev/zero of=/var/srv/nfs/testfile bs=64k count=16k conv=fsync
>>
>>And this is direct on the server, not via NFS, right?
>>
>I've got same result as previous test.
>tests with dd command are executed directly on the server.

Can you rebuild the partition? If so, unmount it then perform the dd 
directly to the device. (of=/dev/sda2 or whatever) What's the hardware 
raid vendor? Honestly, the performance of ext4 partition is also 
horrible, just not as bad as the xfs partition. What's the hardware raid 
model? Does it have a battery backed cache? (I will guess not, because 
the test is only 1G and that should fit into most current caches?) If 
the test directly to the partition shows good performance, then it might 
be an alignment issue; try recreating the fs with `mkfs.xfs -d 
su=64k,sw=2 ...`. If the performance is the same to the raw partition 
(and still much worse than the root partition) then the partition 
alignment itself might be off--which means recreating the partitions. 
(fsck and mkfs.xfs normally handle this automatically, but that depends 
on the raid controller+driver passing the information up, and per your 
xfsinfo output, that didn't happen.)

Mike Stone

[toc] | [prev] | [next] | [standalone]

#199241

On Wed, Aug 22, 2018 at 11:53:25AM -0400, Michael Stone wrote:
>Can you rebuild the partition? If so, unmount it then perform the dd 
>directly to the device.

To be clear, ^^^ this will destroy the filesystem.

Mike Stone

[toc] | [prev] | [next] | [standalone]

#199236

To complete the description there is infos about the XFS partition:

meta-data=/dev/sda4              isize=256    agcount=11, agsize=268435455 blks
          =                       sectsz=512   attr=2, projid32bit=1
          =                       crc=0        finobt=0
data     =                       bsize=4096   blocks=2920268544, imaxpct=5
          =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
log      =internal               bsize=4096   blocks=521728, version=2
          =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

And infos about the RAID5 volume:

Virtual Drive: 0 (Target Id: 0)
Name                :
RAID Level          : Primary-5, Secondary-0, RAID Level Qualifier-3
Size                : 10.915 TB
Sector Size         : 512
Parity Size         : 5.457 TB
State               : Optimal
Strip Size          : 64 KB
Number Of Drives    : 3
Span Depth          : 1
Default Cache Policy: WriteThrough, ReadAheadNone, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteThrough, ReadAheadNone, Direct, No Write Cache if Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy   : Disk's Default
Encryption Type     : None
Default Power Savings Policy: Controller Defined
Current Power Savings Policy: None
Can spin up in 1 minute: Yes
LD has drives that support T10 power conditions: No
LD's IO profile supports MAX power savings with cached writes: No
Bad Blocks Exist: No
Is VD Cached: No

Le 22/08/2018 à 11:41, Martin LEUSCH a écrit :
> Hi,
>
>
> I have a NFS server with a hardware RAID5 on 3 HDD of 6 TB. I have a 
> system partition with ext4 and a data partition with XFS.
>
> I get only 10 MB/s in write speed on the XFS data partition and 80 
> MB/s on the system partition.
>
>
> XFS mount option:
>
> /dev/sda4 on /var/srv/nfs type xfs (rw,relatime,attr2,inode64,noquota)
>
> Is there a way to have better performance? Is XFS a good choice in 
> this situation?
>
>

[toc] | [prev] | [next] | [standalone]

#199247

	Hi.

On Wed, Aug 22, 2018 at 04:15:30PM +0200, Martin LEUSCH wrote:
> To complete the description there is infos about the XFS partition:
> 
> meta-data=/dev/sda4              isize=256    agcount=11, agsize=268435455 blks
>          =                       sectsz=512   attr=2, projid32bit=1
>          =                       crc=0        finobt=0
> data     =                       bsize=4096   blocks=2920268544, imaxpct=5
>          =                       sunit=0      swidth=0 blks
> naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
> log      =internal               bsize=4096   blocks=521728, version=2
>          =                       sectsz=512   sunit=0 blks, lazy-count=1
> realtime =none                   extsz=4096   blocks=0, rtextents=0

Nothing unusual IMO. Certainly nothing that could explain such a
slowdown.

Can you provide a result of 'perf top'? Just to be sure it's xfs that's
to be blamed.

> And infos about the RAID5 volume:

Looks suspiciously similar to LSI MegaRAID.
Is controller firmware current? Is it possible to upgrade it?
Since you seem to have BBU, have you considered enabling WriteBack mode?

Reco

[toc] | [prev] | [next] | [standalone]

#199271

Hello,

did you really need the hardware controller? I suggest to use Software/MD raid, ob btrfs with raid1 or zfs with raidz1



On Wednesday, August 22, 2018 11:41:08 AM CEST Martin LEUSCH wrote:
> Hi,
> 
> 
> I have a NFS server with a hardware RAID5 on 3 HDD of 6 TB. I have a 
> system partition with ext4 and a data partition with XFS.
> 
> I get only 10 MB/s in write speed on the XFS data partition and 80 MB/s 
> on the system partition.
> 
> 
> XFS mount option:
> 
> /dev/sda4 on /var/srv/nfs type xfs (rw,relatime,attr2,inode64,noquota)
> 
> Is there a way to have better performance? Is XFS a good choice in this 
> situation?
> 
> 
> 

[toc] | [prev] | [next] | [standalone]

#199274

Hi all,

After some research on my RAID controller, PERC H310 Mini on a Dell 
server, I found a lot performance issue. It does not have WriteBack 
cache or BBU. I do not tested to write directly on the partition because 
I have some backup to do before but I think reinstall the server on a 
software RAID is the best solution here.

Thanks to all

Le 23/08/2018 à 09:07, Stefan K a écrit :
> Hello,
>
> did you really need the hardware controller? I suggest to use Software/MD raid, ob btrfs with raid1 or zfs with raidz1

> Looks suspiciously similar to LSI MegaRAID.
> Is controller firmware current? Is it possible to upgrade it?
> Since you seem to have BBU, have you considered enabling WriteBack mode?
>
> Reco

> Can you rebuild the partition? If so, unmount it then perform the dd 
> directly to the device. (of=/dev/sda2 or whatever) What's the hardware 
> raid vendor? Honestly, the performance of ext4 partition is also 
> horrible, just not as bad as the xfs partition. What's the hardware 
> raid model? Does it have a battery backed cache? (I will guess not, 
> because the test is only 1G and that should fit into most current 
> caches?) If the test directly to the partition shows good performance, 
> then it might be an alignment issue; try recreating the fs with 
> `mkfs.xfs -d su=64k,sw=2 ...`. If the performance is the same to the 
> raw partition (and still much worse than the root partition) then the 
> partition alignment itself might be off--which means recreating the 
> partitions. (fsck and mkfs.xfs normally handle this automatically, but 
> that depends on the raid controller+driver passing the information up, 
> and per your xfsinfo output, that didn't happen.)
>
> Mike Stone 

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web