Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #161869 > unrolled thread

Intel 82576 Gigabit on Debian 7 slow speed.

Back to article view | Back to linux.debian.user

Contents

  Intel 82576 Gigabit on Debian 7 slow speed. Mimiko <vbvbrj@gmail.com> - 2015-12-18 12:40 +0100
    Re: Intel 82576 Gigabit on Debian 7 slow speed. Michael Beck <eliteknipser@gmail.com> - 2015-12-18 15:40 +0100
      Re: Intel 82576 Gigabit on Debian 7 slow speed. Mimiko <vbvbrj@gmail.com> - 2015-12-18 18:50 +0100
        Re: Intel 82576 Gigabit on Debian 7 slow speed. Dan Ritter <dsr@randomstring.org> - 2015-12-18 23:50 +0100
          Re: Intel 82576 Gigabit on Debian 7 slow speed. Mimiko <vbvbrj@gmail.com> - 2015-12-19 16:50 +0100
            Re: Intel 82576 Gigabit on Debian 7 slow speed. Sven Hartge <sven@svenhartge.de> - 2015-12-19 17:30 +0100
              Re: Intel 82576 Gigabit on Debian 7 slow speed. Frank Pikelner <frank.pikelner@gmail.com> - 2015-12-20 01:40 +0100
            Re: Intel 82576 Gigabit on Debian 7 slow speed. David Christensen <dpchrist@holgerdanske.com> - 2015-12-20 04:30 +0100
              Re: Intel 82576 Gigabit on Debian 7 slow speed. Mimiko <vbvbrj@gmail.com> - 2015-12-20 17:40 +0100
                Re: Intel 82576 Gigabit on Debian 7 slow speed. David Christensen <dpchrist@holgerdanske.com> - 2015-12-21 09:00 +0100
                  Re: Intel 82576 Gigabit on Debian 7 slow speed. Mimiko <vbvbrj@gmail.com> - 2015-12-22 08:30 +0100
                    Re: Intel 82576 Gigabit on Debian 7 slow speed. David Christensen <dpchrist@holgerdanske.com> - 2015-12-22 11:00 +0100
                Re: Intel 82576 Gigabit on Debian 7 slow speed. David Christensen <dpchrist@holgerdanske.com> - 2015-12-22 23:00 +0100
              Re: Intel 82576 Gigabit on Debian 7 slow speed. Mimiko <vbvbrj@gmail.com> - 2015-12-20 18:20 +0100
                Re: Intel 82576 Gigabit on Debian 7 slow speed. David Christensen <dpchrist@holgerdanske.com> - 2015-12-21 09:30 +0100
                  Re: Intel 82576 Gigabit on Debian 7 slow speed. David Christensen <dpchrist@holgerdanske.com> - 2015-12-21 10:00 +0100
        Re: Intel 82576 Gigabit on Debian 7 slow speed. David Christensen <dpchrist@holgerdanske.com> - 2015-12-19 06:20 +0100
    Re: Intel 82576 Gigabit on Debian 7 slow speed. David Christensen <dpchrist@holgerdanske.com> - 2015-12-19 03:20 +0100

#161869 — Intel 82576 Gigabit on Debian 7 slow speed.

Hello.

I've bonded two onboard Intel 82576 Gigabit networks on an supermicro 
server for load balancing (round-robin). It is working, but transfer 
rate is about 10-20MB/s, while on same type of server the same 
configuration in windows I get around 100MB/s.

cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: load balancing (round-robin)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 200
Down Delay (ms): 200

Slave Interface: eth0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Slave queue ID: 0

Slave Interface: eth1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Slave queue ID: 0

I've tried disable bond and leave only eth0, same result. I've tried to 
update to backported newer kernel with no success. Copying with via ssh 
is same speed. If lots of users starts to use the samba shares, even ssh 
lags.

What could be the problem?

Thank you.

[toc] | [next] | [standalone]

#161875

> Am 18.12.2015 um 12:35 schrieb Mimiko <vbvbrj@gmail.com>:
> 
> Hello.
> 
> I've bonded two onboard Intel 82576 Gigabit networks on an supermicro server for load balancing (round-robin). It is working, but transfer rate is about 10-20MB/s, while on same type of server the same configuration in windows I get around 100MB/s.
> 
> cat /proc/net/bonding/bond0
> Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
> 
> Bonding Mode: load balancing (round-robin)
> MII Status: up
> MII Polling Interval (ms): 100
> Up Delay (ms): 200
> Down Delay (ms): 200
> 
> Slave Interface: eth0
> MII Status: up
> Speed: 1000 Mbps
> Duplex: full
> Link Failure Count: 0
> Slave queue ID: 0
> 
> Slave Interface: eth1
> MII Status: up
> Speed: 1000 Mbps
> Duplex: full
> Link Failure Count: 0
> Slave queue ID: 0
> 
> I've tried disable bond and leave only eth0, same result. I've tried to update to backported newer kernel with no success. Copying with via ssh is same speed. If lots of users starts to use the samba shares, even ssh lags.
> 
> What could be the problem?
> 
> Thank you.
> 

Hi,

Any lost packets? 
Real real full-duplex (ethtool)?
What does the (managed?) switch say? 
Are the cables okay (cat6?)?
Which kind of transfer?nfs/cifs/iscsi or raw iperf-test?

Gruß,

Michael Beck

[toc] | [prev] | [next] | [standalone]

#161887

On 18.12.2015 16:32, Michael Beck wrote:
 > Any lost packets?

ifconfig
bond0     Link encap:Ethernet  HWaddr
           UP BROADCAST RUNNING MASTER MULTICAST  MTU:9000  Metric:1
           RX packets:654308767 errors:0 dropped:5238 overruns:0 frame:0
           TX packets:761897714 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:343017135502 (319.4 GiB)  TX bytes:877734258061 
(817.4 GiB)

eth0      Link encap:Ethernet  HWaddr
           UP BROADCAST RUNNING SLAVE MULTICAST  MTU:9000  Metric:1
           RX packets:426388262 errors:0 dropped:0 overruns:0 frame:0
           TX packets:504528230 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000
           RX bytes:201613380511 (187.7 GiB)  TX bytes:593461966181 
(552.7 GiB)
           Memory:fafe0000-fb000000

eth1      Link encap:Ethernet  HWaddr
           UP BROADCAST RUNNING SLAVE MULTICAST  MTU:9000  Metric:1
           RX packets:227920505 errors:0 dropped:0 overruns:0 frame:0
           TX packets:257369484 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000
           RX bytes:141403754991 (131.6 GiB)  TX bytes:284272291880 
(264.7 GiB)
           Memory:faf60000-faf80000

I think 5238 dropped packets for several days compared to received 
packets is negligible.

 > Real real full-duplex (ethtool)?

ethtool -i bond0
driver: bonding
version: 3.7.1
firmware-version: 2
bus-info:
supports-statistics: no
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

ethtool eth0
Settings for eth0:
         Supported ports: [ TP ]
         Supported link modes:   10baseT/Half 10baseT/Full
                                 100baseT/Half 100baseT/Full
                                 1000baseT/Full
         Supported pause frame use: Symmetric
         Supports auto-negotiation: Yes
         Advertised link modes:  10baseT/Half 10baseT/Full
                                 100baseT/Half 100baseT/Full
                                 1000baseT/Full
         Advertised pause frame use: Symmetric
         Advertised auto-negotiation: Yes
         Speed: 1000Mb/s
         Duplex: Full
         Port: Twisted Pair
         PHYAD: 1
         Transceiver: internal
         Auto-negotiation: on
         MDI-X: Unknown
         Supports Wake-on: pumbg
         Wake-on: g
         Current message level: 0x00000007 (7)
                                drv probe link
         Link detected: yes

ethtool eth1
Settings for eth1:
         Supported ports: [ TP ]
         Supported link modes:   10baseT/Half 10baseT/Full
                                 100baseT/Half 100baseT/Full
                                 1000baseT/Full
         Supported pause frame use: Symmetric
         Supports auto-negotiation: Yes
         Advertised link modes:  10baseT/Half 10baseT/Full
                                 100baseT/Half 100baseT/Full
                                 1000baseT/Full
         Advertised pause frame use: Symmetric
         Advertised auto-negotiation: Yes
         Speed: 1000Mb/s
         Duplex: Full
         Port: Twisted Pair
         PHYAD: 1
         Transceiver: internal
         Auto-negotiation: on
         MDI-X: Unknown
         Supports Wake-on: pumbg
         Wake-on: g
         Current message level: 0x00000007 (7)
                                drv probe link
         Link detected: yes

 > What does the (managed?) switch say?

It shows 1Gbit.

 > Are the cables okay (cat6?)?

Yes. Cables are new cat6e.

 > Which kind of transfer?nfs/cifs/iscsi or raw iperf-test?

I've tried send/receive cifs (samba). Also I've tried sendng file with 
pscp using Kitty. Both give same low speed.
NFS or iSCSI isn't used.

iperf -c ip
------------------------------------------------------------
Client connecting to ip, TCP port 5001
TCP window size: 23.5 KByte (default)
------------------------------------------------------------
[  3] local ip port 36389 connected with ip port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   835 MBytes   700 Mbits/sec

70% of 1Gbit. Is this seems a problem with samba?

iperf -uc 10.10.0.254 -b 1000m
------------------------------------------------------------
Client connecting to ip, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size:  224 KByte (default)
------------------------------------------------------------
[  3] local ip port 33348 connected with ip port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   966 MBytes   810 Mbits/sec
[  3] Sent 688757 datagrams
[  3] WARNING: did not receive ack of last datagram after 10 tries.

dd if=/dev/zero of=test.bin bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 24.5841 s, 427 MB/s

dd if=test.bin of=/dev/null bs=1M
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 2.24662 s, 4.7 GB/s

Disk is a zfs raid:
zpool create -f -m none -o ashift=12 zfspool raidz2 .... (total 8 x 1TB 
+ 8 x 2TB disk in SATA2 supermicro backplane)
zfs set atime=off zfspool
zfs set dedup=off zfspool
zfs create -V 4T zfspool/backup
zfs set compression=lz4 zfspool/backup
mkfs.ext4 -b 4096 -q /dev/zvol/zfspool/backup

Is this samba problem or zfs problem?

[toc] | [prev] | [next] | [standalone]

#161899

On Fri, Dec 18, 2015 at 07:44:58PM +0200, Mimiko wrote:
> iperf -c ip
> ------------------------------------------------------------
> Client connecting to ip, TCP port 5001
> TCP window size: 23.5 KByte (default)
> ------------------------------------------------------------
> [  3] local ip port 36389 connected with ip port 5001
> [ ID] Interval       Transfer     Bandwidth
> [  3]  0.0-10.0 sec   835 MBytes   700 Mbits/sec
> 
> 70% of 1Gbit. Is this seems a problem with samba?

Do you have jumbo packet support on your switch and on all
possible clients? 5000 - 9000 byte ethernet packets can improve
raw throughput, at a (usually minor) cost in latency.

> dd if=/dev/zero of=test.bin bs=1M count=10000
> 10000+0 records in
> 10000+0 records out
> 10485760000 bytes (10 GB) copied, 24.5841 s, 427 MB/s

Looks normal.

> dd if=test.bin of=/dev/null bs=1M
> 10000+0 records in
> 10000+0 records out
> 10485760000 bytes (10 GB) copied, 2.24662 s, 4.7 GB/s

Pulled from cache.

> Disk is a zfs raid:
> zpool create -f -m none -o ashift=12 zfspool raidz2 .... (total 8 x
> 1TB + 8 x 2TB disk in SATA2 supermicro backplane)
> zfs set atime=off zfspool
> zfs set dedup=off zfspool
> zfs create -V 4T zfspool/backup
> zfs set compression=lz4 zfspool/backup
> mkfs.ext4 -b 4096 -q /dev/zvol/zfspool/backup
> 
> Is this samba problem or zfs problem?

RAIDZ2 is not a high-speed solution, it's a medium-safety
solution. Layering ext4 on top of ZFS blocks doesn't make
it faster, it just extends the reliability of ZFS to the 
ext4 fs.

Do you have a requirement for putting ext4 on top of ZFS blocks?
Can you use ZFS instead?


-dsr-

[toc] | [prev] | [next] | [standalone]

#161911

Hello.

After reviewing the results of test, I've modified smb.conf. I've added 
max protocol = SMB2 and removed SO_RCVBUF=8192 SO_SNDBUF=8192 from 
socket options. The read speed from this server increased to 40MB/s, the 
write speed to this server increased to 30MB/s.

On 19.12.2015 00:43, Dan Ritter wrote:
> Do you have jumbo packet support on your switch and on all
> possible clients? 5000 - 9000 byte ethernet packets can improve
> raw throughput, at a (usually minor) cost in latency.

Yes, main switches are set for jumbo frames, but half of computers and 
switches are old and can't handle jumbo frames. That's why servers are 
not set for jumbo frames, only the switches are ready.

> RAIDZ2 is not a high-speed solution, it's a medium-safety
> solution. Layering ext4 on top of ZFS blocks doesn't make
> it faster, it just extends the reliability of ZFS to the
> ext4 fs.
>
> Do you have a requirement for putting ext4 on top of ZFS blocks?
> Can you use ZFS instead?

Yes, raidz2 is a compromise for us for performance-safety. I wanted to 
allow up two 2 hdd to fail and be able to sustain the data. I've choosed 
ext4 on zfs, because this is not only a backup server, but also a file 
server, which stores files with ACL support in a Microsoft Active 
Directory file access policy. Also I want to set up drdb to sync 
partitions to another file server. If ZFS file system will allow this - 
I would switch to it.

BUT, this server boots from an md mirror on two separate SSD's. Reading 
and writing to this disks give same low speed.


On 19.12.2015 04:14, David Christensen wrote:
 > What motherboard?  What CPU?  What and how much RAM?  What disk/RAID?

Its a supermicro server based on SuperMicro X8DTi-F motherboard with 2 x 
Intel Xeon E5620 CPU @ 2,4GHz with 18 GB ram.

 > What version of Debian?  Which kernel?  Any other customizations?

Message title states Debian 7 which is Wheezy, Linux 3.2.0-4-amd64 #1 
SMP Debian 3.2.68-1+deb7u6 x86_64 GNU/Linux. All updates installed. It's 
a standard install with ZFS for software raid and file server.

 > But, if the cables ran at 100 MB/s under Windows you're probably okay.
 > What is your switch?

Same cable types, same server type on windows gives faster file 
transfer. All servers is on gigabit switch.

 > iperf should be testing the hardware, device drivers, and kernel on both
 > ends.  What make/ model Ethernet chips do you have on the two hosts?
 >
 >      # lspci | grep -i ethernet

iperf between two Debian 7 servers gives a good performance almost. The 
problem server has Intel 82576 dual gigabit.
On the other server it is Broadcom Corporation NetXtreme BCM5705 
Gigabit. The othe server gives 80MB/s on read and 60MB/s on write. Still 
not 100MB/s as on windows, but it is 1 cpu older server.

 > Is there anything else between the two computers?

Only the switch.

 > I assume you ran that on your Samba server (?).

Ofcourse.

 > A stream of zeros is very easy to compress and can give misleading
 > results.  It's better to use random numbers.  (A hardware RNG is 
useful for this.)

Yes, zeros are easily compressed. I've tried to use /dev/random as 
source, but this device give me less then 1KB/s. When I used 
/dev/urandom, I've got 8,6MB/s and dd used 100% of one core.

 > I assume you're running the latest ZFS on Linux?  (ZFS-fuse is
 > considerably slower and is behind on features.)

Yes, ZFS on Linux and all latest updates.

 > Why did you have to -f (force) creation?

Don't know why I did used this option back then. I think it would be 
working withou -f.

 > LUKS?  On the raw drives, or on partitions?  How many CPU's and cores?
 > HT?  AES-NI?

I used raw drives 8 x 2TB and 8 x 1TB. 2 x CPU gives me 16 cores. This 
is BIOS settings:
Ratio CMOS Setting: 18
C1E Support: Disabled
Hardware Prefetcher: Enabled
Adjacent Cache Line Prefetch: Enabled
DCU Prefetcher: Enabled
Data Reuse Optimisation: Enabled
MPS and ACPI MADT ordering: Modern ordering
Max CPUID Value Limit: Disabled
Intel(R) Virtualization Tech: Enabled
Execute-Disable Bit Capability: Enabled
Intel AES-NI: Disabled
Simultaneous Multi-Threading: Enabled
Active Processor Cores: All
Intel(R) EIST tech: Disabled
Intel(R) C-STATE tech: Disabled
Clock Spread Spectrum: Disabled

 > Different-sized drives makes things more interesting.  What arrangements
 > did you try?   What did you settle on, and why?

I've read a lot about zfs before setting up this server and used mostly 
recommended ashift=12 and other settings to suit my needs.

 > Why raidz2?

To allow up two 2 disk to fail.

 > Turning off atime is okay if none of the applications that use the file
 > systems need it.

Thats why it is off.

 > General de-duplication can be risky, but for many backup solutions it is
 > a killer feature (along with replication).

18GB is not enough for good deduplication, even I would like to have it.

 > A ZFS volume with an ext4 file system?  Why?

ACL access list and future use of drdb.

 > For backups?  I keep mine on dedicated disks in another machine.

This is a backup server with file server.



So hope to cope with windows speed in file transfer.

[toc] | [prev] | [next] | [standalone]

#161915

Mimiko <vbvbrj@gmail.com> wrote:

> After reviewing the results of test, I've modified smb.conf. I've added 
> max protocol = SMB2 and removed SO_RCVBUF=8192 SO_SNDBUF=8192 from 
> socket options. The read speed from this server increased to 40MB/s, the 
> write speed to this server increased to 30MB/s.

The SO_*BUF options have been deprecated a long long time ago. They made
sense with a 10MBit/s network but reduce the speed with anything faster,
as you have seen.

Unfortunately many HOWTOs still contain them and advise to use them
without understanding what they do. A kind of cargo cult.

Grüße,
Sven.

-- 
Sigmentation fault. Core dumped.

[toc] | [prev] | [next] | [standalone]

#161923

There appear to be driver issues discussed in other threads with
respect to Intel driver and slow throughput due to interrupts and CPU
offloading. May want to review driver parameters and look at trying a
few changes.

Frank

On Sat, Dec 19, 2015 at 11:24 AM, Sven Hartge <sven@svenhartge.de> wrote:
> Mimiko <vbvbrj@gmail.com> wrote:
>
>> After reviewing the results of test, I've modified smb.conf. I've added
>> max protocol = SMB2 and removed SO_RCVBUF=8192 SO_SNDBUF=8192 from
>> socket options. The read speed from this server increased to 40MB/s, the
>> write speed to this server increased to 30MB/s.
>
> The SO_*BUF options have been deprecated a long long time ago. They made
> sense with a 10MBit/s network but reduce the speed with anything faster,
> as you have seen.
>
> Unfortunately many HOWTOs still contain them and advise to use them
> without understanding what they do. A kind of cargo cult.
>
> Grüße,
> Sven.
>
> --
> Sigmentation fault. Core dumped.
>

[toc] | [prev] | [next] | [standalone]

#161925

On 12/19/2015 07:48 AM, Mimiko wrote:
> After reviewing the results of test, I've modified smb.conf. I've added
> max protocol = SMB2 and removed SO_RCVBUF=8192 SO_SNDBUF=8192 from
> socket options. The read speed from this server increased to 40MB/s, the
> write speed to this server increased to 30MB/s.

Improving Samba is good, but I would optimize the file system first. 
You've got a small mountain of hardware, and your disk throughput 
numbers should be an order of magnitude better.


>Also I want to set up drdb to sync partitions to another file server.

Do you mean drbd?

     http://drbd.linbit.com/


> BUT, this server boots from an md mirror on two separate SSD's. Reading
> and writing to this disks give same low speed.

LUKS?  Above or below md?


> Its a supermicro server based on SuperMicro X8DTi-F motherboard

     http://www.supermicro.com/products/motherboard/QPI/5500/X8DTi-F.cfm

The motherboard web page says 6 @ SATA2 ports.  Where do you get the 
other 12?  SATA2 or SATA3?


What interface(s) do the HDD's and SSD's have?


 > with 2 x Intel Xeon E5620 CPU @ 2,4GHz

 
http://ark.intel.com/products/47925/Intel-Xeon-Processor-E5620-12M-Cache-2_40-GHz-5_86-GTs-Intel-QPI?q=Intel%C2%AE%20Xeon%C2%AE%20Processor%20E5620%20%2812M%20Cache,%202.40%20GHz,%205.86%20GT/s%20Intel%C2%AE%20QPI%29

Each CPU has 4 cores, HT (so, 8 threads), and AES-NI.


 > with 18 GB ram.

The motherboard supports 192 GB of RAM.


Do you know if your amount of RAM is too much, too little, or just right?


> Same cable types, same server type on windows gives faster file
> transfer. All servers is on gigabit switch.

Then it's software and/or configuration.


> iperf between two Debian 7 servers gives a good performance almost. The
> problem server has Intel 82576 dual gigabit.
> On the other server it is Broadcom Corporation NetXtreme BCM5705
> Gigabit. The othe server gives 80MB/s on read and 60MB/s on write. Still
> not 100MB/s as on windows, but it is 1 cpu older server.

I ran iperf earlier between two of my machines with Intel desktop 
Gigabit Ethernet chips, a few meters of Category 5e cable, and a 
consumer Gigabit switchin between, and got ~950 MB/s.  You might want to 
find another machine with Intel Ethernet chips and test again.


> Yes, zeros are easily compressed. I've tried to use /dev/random as
> source, but this device give me less then 1KB/s.

/dev/random is meant for small amounts of high-quality random numbers 
(e.g. seeds and keys).  It draws directly from the the kernel entropy 
pool, and blocks when the pool is low.


> When I used /dev/urandom, I've got 8,6MB/s and dd used 100% of one core.

/dev/urandom uses the entropy pool sparingly and generates the rest with 
a software PRNG.


>> LUKS?  On the raw drives, or on partitions?
> I used raw drives 8 x 2TB and 8 x 1TB.

So, 16 LUKS containers below ZFS?  If you have fewer than 16 ZFS 
volumes, you could put LUKS above ZFS and below ext4.


> This is BIOS settings:
> Ratio CMOS Setting: 18
> C1E Support: Disabled
> Hardware Prefetcher: Enabled
> Adjacent Cache Line Prefetch: Enabled
> DCU Prefetcher: Enabled
> Data Reuse Optimisation: Enabled
> MPS and ACPI MADT ordering: Modern ordering
> Max CPUID Value Limit: Disabled
> Intel(R) Virtualization Tech: Enabled
> Execute-Disable Bit Capability: Enabled
> Intel AES-NI: Disabled
                 ^^^^^^^^
If you are running LUKS, that would explain low HDD sequential 
throughput (latency causes low random throughput) and low SSD overall 
throughput.  Enable AES-NI in the BIOS settings and test again.


> Simultaneous Multi-Threading: Enabled
> Active Processor Cores: All
> Intel(R) EIST tech: Disabled
> Intel(R) C-STATE tech: Disabled
> Clock Spread Spectrum: Disabled

Are those the defaults?  I typically reset to the BIOS default settings, 
and then change as few as possible after careful consideration and 
followed by testing.


>> Different-sized drives makes things more interesting.  What arrangements
>> did you try?   What did you settle on, and why?
>
> I've read a lot about zfs before setting up this server and used mostly
> recommended ashift=12 and other settings to suit my needs.
>
>> Why raidz2?
>
> To allow up two 2 disk to fail.

Without seeing your exact 'zpool create ...' invocation, it is difficult 
to comment.


>> A ZFS volume with an ext4 file system?  Why?
>
> ACL access list and future use of drdb.

Perhaps you should file bug reports/ feature requests.


> So hope to cope with windows speed in file transfer.

There are many complex decisions to make; it's easy to make a mistake. 
Once you figure it out, either OS should give good results.


David

[toc] | [prev] | [next] | [standalone]

#161944

On 20.12.2015 02:36, Frank Pikelner wrote:
 > There appear to be driver issues discussed in other threads with
 > respect to Intel driver and slow throughput due to interrupts and CPU
 > offloading. May want to review driver parameters and look at trying a
 > few changes.

Yes, I've read about that. It is written that new kernel resolve problem 
with Intel drivers. I've upgraded to a backported kernel and results 
were the same. So I reversed.

How to view driver parameters and which device's driver?


On 20.12.2015 05:24, David Christensen wrote:
 > Do you mean drbd?

Yes. :)

 > LUKS? Above or below md?

If LUKS you mean Linux Unified Key Setup, then I don't use id.

 > The motherboard web page says 6 @ SATA2 ports. Where do you get the
 > other 12? SATA2 or SATA3?
 >
 >
 > What interface(s) do the HDD's and SSD's have?

The HDD's are connected thru SuperMicro SAS RAID AOC-USASLP-L8i (PCI-E). 
Sorry to not mention. All disks are in an enclosure. This card works at 
SATA2 speed, although disks can handle SATA3.

Also SSD's are new Intel disk which have SATA3, but motherboard supports 
only SATA2. SSD's a connected to motheboard directly.

 > The motherboard supports 192 GB of RAM.

There is only 18RAM available as of budget.

 > If you are running LUKS, that would explain low HDD sequential
 > throughput (latency causes low random throughput) and low SSD overall
 > throughput. Enable AES-NI in the BIOS settings and test again.

Enabled this settings - and no effect.

 >> Simultaneous Multi-Threading: Enabled
 >> Active Processor Cores: All
 >> Intel(R) EIST tech: Disabled
 >> Intel(R) C-STATE tech: Disabled
 >> Clock Spread Spectrum: Disabled
 >
 > Are those the defaults? I typically reset to the BIOS default settings,
 > and then change as few as possible after careful consideration and
 > followed by testing.

Some are default, some are changed based on my readings in Internet.

 > Without seeing your exact 'zpool create ...' invocation, it is difficult
 > to comment.

zpool create -f -m none -o ashift=12 zfspool raidz2 
wwn-0x50000xxxxxxxxxxx ..... (all disks).

 >>> A ZFS volume with an ext4 file system? Why?
 >>
 >> ACL access list and future use of drdb.
 >
 > Perhaps you should file bug reports/ feature requests.

It will take a longer time than I will start to use drbd. If no one 
fired a request, or it wasn't implemented yet, then ZFS on Linux thinks 
it is not useful, I think.

 > Improving Samba is good, but I would optimize the file system first.
 > You've got a small mountain of hardware, and your disk throughput
 > numbers should be an order of magnitude better.

I'm open to suggestions to optimize file system and keep the needs.

[toc] | [prev] | [next] | [standalone]

#161957

On 12/20/2015 08:36 AM, Mimiko wrote:
> If LUKS you mean Linux Unified Key Setup, then I don't use [it].

Okay.


> The HDD's are connected thru SuperMicro SAS RAID AOC-USASLP-L8i
> (PCI-E).

  http://www.supermicro.com.tr/AOC-USASLP-L8i.cfm.htm

So, one RAID card, eight SAS channels, and one 2 TB and one 1 TB drive
per channel (?).


Are you using any hardware RAID or JBOD capabilities under Wheezy?


> All disks are in an enclosure.

External enclosure?  That card is for drives within in the same
enclosure as the card.


Where do the drives get power?


> This card works at SATA2 speed, although disks can handle SATA3.

What is the make/ model of the drives?


> Also SSD's are new Intel disk which have SATA3, but motherboard
> supports only SATA2. SSD's a connected to motheboard directly.

Try disconnecting the RAID card and drives, and benchmarking just the SSD's.


> There is only 18RAM available as of budget.

Do you know if your amount of RAM is too much, too little, or just right?


> Enabled [AES-NI in CMOS setup] - and no effect.

Since you are not running encrypted drives, AES-NI won't help you there. 
  But, it should help with SSL.


> Some [CMOS settings[ are default, some are changed based on my
> readings in Internet.

Okay, so long as you understand each.


> zpool create -f -m none -o ashift=12 zfspool raidz2
> wwn-0x50000xxxxxxxxxxx ..... (all disks).

What does ZFS report for the size of the pool?


> [bug reports/ feature requests] will take a longer time than I will
> start to use drbd. If no one fired a request, or it wasn't
> implemented yet, then ZFS on Linux thinks it is not useful, I think.

Okay.


> I'm open to suggestions to optimize file system and keep the needs.

When you ran Windows:

1.  Which version of Windows?

2.  Were the hardware and and CMOS settings identical to what they are
with Wheezy?

3.  Did you use the AOC-USASLP-L8i RAID hardware capabilities?  If so,
how did you configure it?

4.  Did you use any software RAID, Windows dynamic disks, etc.?  If so,
what software and how did you configure it?


David

[toc] | [prev] | [next] | [standalone]

#161983

On 21.12.2015 09:58, David Christensen wrote:
> On 12/20/2015 08:36 AM, Mimiko wrote:
>> The HDD's are connected thru SuperMicro SAS RAID AOC-USASLP-L8i
>> (PCI-E).
>
>   http://www.supermicro.com.tr/AOC-USASLP-L8i.cfm.htm
>
> So, one RAID card, eight SAS channels, and one 2 TB and one 1 TB drive
> per channel (?).

No. One channel is connected to 8 x 2TB the other channel to 8 x 1TB.

> Are you using any hardware RAID or JBOD capabilities under Wheezy?

JBOD for software raid. The windows server uses HW RAID6.

>> All disks are in an enclosure.
>
> External enclosure?  That card is for drives within in the same
> enclosure as the card.

Well, it is more the expansion. In same server 2U.

> Where do the drives get power?

 From same power as MB.

>> This card works at SATA2 speed, although disks can handle SATA3.
>
> What is the make/ model of the drives?

Toshiba 2TB 3.5" MG03ACA200 SATA and Toshiba 1TB 3.5" MG03ACA100 SATA.

> Try disconnecting the RAID card and drives, and benchmarking just the
> SSD's.

Its impossible for now.

>> There is only 18RAM available as of budget.
>
> Do you know if your amount of RAM is too much, too little, or just right?

I don't understand the meaning of this. More RAM is better, but this is 
what we have on this server.

>> zpool create -f -m none -o ashift=12 zfspool raidz2
>> wwn-0x50000xxxxxxxxxxx ..... (all disks).
>
> What does ZFS report for the size of the pool?

zfs list
NAME                   USED  AVAIL  REFER  MOUNTPOINT
zfspool               11.5T  39.7G   335K  none

> When you ran Windows:
>
> 1.  Which version of Windows?

Windows Server 2008 R2 x64

> 2.  Were the hardware and and CMOS settings identical to what they are
> with Wheezy?

Almost identical. Windows server has 48GB RAM and 8 x 2TB HDD and no 
separate SSD's.

> 3.  Did you use the AOC-USASLP-L8i RAID hardware capabilities?  If so,
> how did you configure it?

Windows is on HW RAID6.

> 4.  Did you use any software RAID, Windows dynamic disks, etc.?  If so,
> what software and how did you configure it?

No.

 > Do you have a laptop or desktop (or two) with Gigabit that you can use
 > for testing?

Only one laptop. May be one desktop too. Its easier to setup another 
server with same MB but lower disk count and ram. So keep in touch. As 
this is disappointing for Debian to loose to windows in file share.

 > You have 24 GB of drives for RAID.  How many file systems(s), and what
 > size each, do you need?

There are now around 8 ext4 filesystems with different sizes.

 > Are you able to move services off the Wheezy Samba server so you can
 > take drastic measures?

Unfortunately no. Its a lot of data. This is only server with lot of 
disk space.


700Mbit/s is about 80MB/s. So I think its not intel network card the 
issue. Something is with samba or zfs.

Thank you.

[toc] | [prev] | [next] | [standalone]

#161989

On 12/21/2015 11:20 PM, Mimiko wrote:
> One [RAID] channel is connected to 8 x 2TB the other channel to 8 x 1TB.

What is the part number of your cables?


> JBOD for software raid [on Wheezy].

I thought the 'zpool create ...' invocation showed all 16 drives in one 
raidz2 pool (?).  This is not the same as JBOD.


> The windows server uses HW RAID6.

Hardware RAID is generally considered to be faster than software RAID, 
but the latter offers more opportunities for tuning.


> [Drives are enclosed] in same server 2U.
> [Drives get power] from same power as MB.
> Toshiba 2TB 3.5" MG03ACA200 SATA and Toshiba 1TB 3.5" MG03ACA100 SATA.

Okay.

 
http://toshiba.semicon-storage.com/us/product/storage-products/enterprise-hdd.html


> [Disconnect RAID and benchmark system SSD's] is impossible for now.

Schedule a maintenance window.


 > I don't understand the meaning of ["do you know if your amount of RAM
 > is too much, too little, or just right?"].  More RAM is better, but
 > this is what we have on this server.

What I'm suggesting is that you find some metric(s) to tell you how much 
memory Linux, ZFS, and Samba are using:

- Too little memory is bad, for reasons we all already know.

- Too much memory is a waste -- the extra memory could be put into 
another machine where it is needed.


> zfs list
> NAME                   USED  AVAIL  REFER  MOUNTPOINT
> zfspool               11.5T  39.7G   335K  none

Ouch.  You need to free up space soon, or you're going to have a disaster.


Slow throughput is a symptom of full drives.


I suspect that your 'zpool create ...' invocation with differing drive 
sizes resulted in ZFS treating all the drives as the same size as the 
smallest drive -- e.g. 16 @ 1 TB.  So, it appears that raidz2 cost you 4 
TB, or 25% of your raw space.


> Windows Server 2008 R2 x64

Okay.


> Windows server has 48GB RAM and 8 x 2TB HDD and no separate SSD's.
 > Windows is on HW RAID6.
 > No [software RAID or dynamic disks].

You are making an apples to oranges comparison.

     48 GB vs. 18 GB -- Linux has 1/3 the RAM

     8 drives on 8 channels vs. 16 drives on 8 channels -- Linux has 
twice the buss contention.

     hardware RAID vs. software RAID -- I don't think the Linux software 
RAID is set up optimally


That said, any computer with drives that are 99.6% full is not going to 
benchmark well; it's going to crash.


> Only one laptop. May be one desktop too. Its easier to setup another
> server with same MB but lower disk count and ram.

It's good to have spare hardware for running tests, doing maintenance, 
and for emergencies (like now).


> So keep in touch. As this is disappointing for Debian to loose to windows in file share.

I think either can perform well, when properly tuned.


> There are now around 8 ext4 filesystems with different sizes.

Okay.


 > Unfortunately [I am unable to move services off the machine]. Its a
 > lot of data. This is only server with lot of disk space.

Been there, done that, and lost data, time, money, credibility, 
business, etc..



> 700Mbit/s is about 80MB/s. So I think its not intel network card the
> issue. Something is with samba or zfs.

I agree that Gigabit interfaces should run at close to capacity, but the 
iperf numbers you posted previously show otherwise.


David

[toc] | [prev] | [next] | [standalone]

#162004

On 12/20/2015 08:36 AM, Mimiko wrote:
> zpool create -f -m none -o ashift=12 zfspool raidz2
> wwn-0x50000xxxxxxxxxxx ..... (all disks).

On 12/21/2015 11:20 PM, Mimiko wrote:
 > One channel is connected to 8 x 2TB the other channel to 8 x 1TB.
...
 > zfs list
 > NAME                   USED  AVAIL  REFER  MOUNTPOINT
 > zfspool               11.5T  39.7G   335K  none

RTFM zpool:

     http://linux.die.net/man/8/zpool

     Virtual Devices (vdevs)
     ...
     raidz
     ...
         A raidz group can have single-, double- , or triple parity,
         meaning that the raidz group can sustain one, two, or three
         failures, respectively, without losing any data. ... [The]
         raidz2 vdev type specifies a double-parity raidz group;
         ...
         A raidz group with N disks of size X with P parity disks can
         hold approximately (N-P)*X bytes and can withstand P device(s)
         failing before data integrity is compromised. The minimum
         number of devices in a raidz group is one more than the number
         of parity disks. The recommended number is between 3 and 9 to
         help increase performance.


Doing the math for your 'zpool create' invocation:

     N = 16	(total number of drives)
     X = 1 TB	(size of smallest drive)
     P = 2	(raidz2)

     size = (16 - 2) * 1 TB = 14 TB


But your 'zfs list' implies a pool size of 12 TB.  I don't understand...


RTFM RAID card:

     http://www.supermicro.com.tr/AOC-USASLP-L8i.pdf

     Chapter 2 Connectors, Jumpers and Indicators
     ...
     2-1     Front Connectors and Pin Definitions
     ...
     Internal SAS Connectors
     ...
     Use a single port SAS "ipass" cable (SuperMicro order number
     CBL-0108L-02)


Are you using Supermicro cable CBL-0108L-02?

     http://www.supermicro.com/support/resources/sas/


Are you using Supermicro chassis 836E2-R800V or 836E2-R800B?

     http://www.supermicro.com/products/chassis/3U/836/SC836E2-R800.cfm


David

[toc] | [prev] | [next] | [standalone]

#161945

I've tested using iperf the file server to the windows server and I've 
got 400Mbits. The other linux server got 600Mbits. Also I've tested 
iperf between windows server and another same type supermicro with 
hyper-v and got around 650Mbits.

I've tested samba speed from windows to other linux server. It gives 
90MB/s at read from that server and 70MB/s of write. 2 times of the file 
server with problem.

Using smbclient:
 From file server to other liux box - 48MB/s
from othe linux box to file server - 65MB/s

This is what I could test more.

[toc] | [prev] | [next] | [standalone]

#161958

On 12/19/2015 07:24 PM, David Christensen wrote:
 > I ran iperf ... ~950 MB/s.
                         ^^^^
Oops -- make that 950 Mbps.


On 12/20/2015 09:13 AM, Mimiko wrote:
> I've tested using iperf the file server to the windows server and I've
> got 400Mbits.

Something is wrong.


> The other linux server got 600Mbits.

Something is wrong, too.


> Also I've tested
> iperf between windows server and another same type supermicro with
> hyper-v and got around 650Mbits.

The hypervisor adds another layer, but I would still expect it to be 
900+ Mbps.  Something is wrong, three.


Do you have a laptop or desktop (or two) with Gigabit that you can use 
for testing?


Download a live CD image, burn two discs/ USB flash drives, and try again:

     https://www.debian.org/CD/live/

     debian-live-8.2.0-amd64-standard.iso


> I've tested samba speed from windows to other linux server. It gives
> 90MB/s at read from that server and 70MB/s of write. 2 times of the file
> server with problem.
>
> Using smbclient:
>  From file server to other liux box - 48MB/s
> from othe linux box to file server - 65MB/s
>
> This is what I could test more.

Both Ethernet and RAID need to be running well before Samba can run well.


You have 24 GB of drives for RAID.  How many file systems(s), and what 
size each, do you need?


Are you able to move services off the Wheezy Samba server so you can 
take drastic measures?


David

[toc] | [prev] | [next] | [standalone]

#161959

On 12/21/2015 12:24 AM, David Christensen wrote:
> Download a live CD image, burn two discs/ USB flash drives, and try again:
>
> https://www.debian.org/CD/live/
>
>      debian-live-8.2.0-amd64-standard.iso

I just realized that the live disc might not have iperf...


Perhaps Knoppix:

     http://knopper.net/knoppix/index-en.html


But the best solution I've found is installing and running Debian on USB 
flash drives in computers that are "new enough".  USB flash drives are 
cheap, they work well enough for light use, and they are very useful for 
troubleshooting.  Get two, set one up with a fresh install of Jessie, 
install iperf and whatever other tools you like, save the raw image, 
clone it to the second drive, and use those for testing.


David

[toc] | [prev] | [next] | [standalone]

#161904

On 12/18/2015 09:44 AM, Mimiko wrote:
> ifconfig
> bond0     Link encap:Ethernet  HWaddr
>            UP BROADCAST RUNNING MASTER MULTICAST  MTU:9000  Metric:1
>            RX packets:654308767 errors:0 dropped:5238 overruns:0 frame:0
>            TX packets:761897714 errors:0 dropped:0 overruns:0 carrier:0
>            collisions:0 txqueuelen:0
>            RX bytes:343017135502 (319.4 GiB)  TX bytes:877734258061
> (817.4 GiB)
>
> eth0      Link encap:Ethernet  HWaddr
>            UP BROADCAST RUNNING SLAVE MULTICAST  MTU:9000  Metric:1
>            RX packets:426388262 errors:0 dropped:0 overruns:0 frame:0
>            TX packets:504528230 errors:0 dropped:0 overruns:0 carrier:0
>            collisions:0 txqueuelen:1000
>            RX bytes:201613380511 (187.7 GiB)  TX bytes:593461966181
> (552.7 GiB)
>            Memory:fafe0000-fb000000
>
> eth1      Link encap:Ethernet  HWaddr
>            UP BROADCAST RUNNING SLAVE MULTICAST  MTU:9000  Metric:1
>            RX packets:227920505 errors:0 dropped:0 overruns:0 frame:0
>            TX packets:257369484 errors:0 dropped:0 overruns:0 carrier:0
>            collisions:0 txqueuelen:1000
>            RX bytes:141403754991 (131.6 GiB)  TX bytes:284272291880
> (264.7 GiB)
>            Memory:faf60000-faf80000

What version of Debian?  Which kernel?  Any other customizations?


> I think 5238 dropped packets for several days compared to received
> packets is negligible.

Likely so.


> ethtool -i bond0
> driver: bonding
> version: 3.7.1
> firmware-version: 2
> bus-info:
> supports-statistics: no
> supports-test: no
> supports-eeprom-access: no
> supports-register-dump: no
> supports-priv-flags: no
>
> ethtool eth0
> Settings for eth0:
>          Supported ports: [ TP ]
>          Supported link modes:   10baseT/Half 10baseT/Full
>                                  100baseT/Half 100baseT/Full
>                                  1000baseT/Full
>          Supported pause frame use: Symmetric
>          Supports auto-negotiation: Yes
>          Advertised link modes:  10baseT/Half 10baseT/Full
>                                  100baseT/Half 100baseT/Full
>                                  1000baseT/Full
>          Advertised pause frame use: Symmetric
>          Advertised auto-negotiation: Yes
>          Speed: 1000Mb/s
>          Duplex: Full
>          Port: Twisted Pair
>          PHYAD: 1
>          Transceiver: internal
>          Auto-negotiation: on
>          MDI-X: Unknown
>          Supports Wake-on: pumbg
>          Wake-on: g
>          Current message level: 0x00000007 (7)
>                                 drv probe link
>          Link detected: yes
>
> ethtool eth1
> Settings for eth1:
>          Supported ports: [ TP ]
>          Supported link modes:   10baseT/Half 10baseT/Full
>                                  100baseT/Half 100baseT/Full
>                                  1000baseT/Full
>          Supported pause frame use: Symmetric
>          Supports auto-negotiation: Yes
>          Advertised link modes:  10baseT/Half 10baseT/Full
>                                  100baseT/Half 100baseT/Full
>                                  1000baseT/Full
>          Advertised pause frame use: Symmetric
>          Advertised auto-negotiation: Yes
>          Speed: 1000Mb/s
>          Duplex: Full
>          Port: Twisted Pair
>          PHYAD: 1
>          Transceiver: internal
>          Auto-negotiation: on
>          MDI-X: Unknown
>          Supports Wake-on: pumbg
>          Wake-on: g
>          Current message level: 0x00000007 (7)
>                                 drv probe link
>          Link detected: yes

I don't see any obvious problems.


>  > What does the (managed?) switch say?
>
> It shows 1Gbit.
>
>  > Are the cables okay (cat6?)?
>
> Yes. Cables are new cat6e.

Category 6E is an oddball:

     https://en.wikipedia.org/wiki/Category_6_cable#Category_6e

But, if the cables ran at 100 MB/s under Windows you're probably okay.


>  > Which kind of transfer?nfs/cifs/iscsi or raw iperf-test?
>
> I've tried send/receive cifs (samba). Also I've tried sendng file with
> pscp using Kitty. Both give same low speed.
> NFS or iSCSI isn't used.

Okay.


> iperf -c ip
> ------------------------------------------------------------
> Client connecting to ip, TCP port 5001
> TCP window size: 23.5 KByte (default)
> ------------------------------------------------------------
> [  3] local ip port 36389 connected with ip port 5001
> [ ID] Interval       Transfer     Bandwidth
> [  3]  0.0-10.0 sec   835 MBytes   700 Mbits/sec
>
> 70% of 1Gbit. Is this seems a problem with samba?

iperf should be testing the hardware, device drivers, and kernel on both 
ends.  What make/ model Ethernet chips do you have on the two hosts?

     # lspci | grep -i ethernet


What is your switch?


Is there anything else between the two computers?


> dd if=/dev/zero of=test.bin bs=1M count=10000
> 10000+0 records in
> 10000+0 records out
> 10485760000 bytes (10 GB) copied, 24.5841 s, 427 MB/s

I assume you ran that on your Samba server (?).


A stream of zeros is very easy to compress and can give misleading 
results.  It's better to use random numbers.  (A hardware RNG is useful 
for this.)


> dd if=test.bin of=/dev/null bs=1M
> 10000+0 records in
> 10000+0 records out
> 10485760000 bytes (10 GB) copied, 2.24662 s, 4.7 GB/s

The 4.7 GB/s came from the kernel page cache.  You want to sync(1) and 
drop all caches strategically when benchmarking:

     http://www.linuxinsight.com/proc_sys_vm_drop_caches.html


> Disk is a zfs raid:
> zpool create -f -m none -o ashift=12 zfspool raidz2 .... (total 8 x 1TB
> + 8 x 2TB disk in SATA2 supermicro backplane)
> zfs set atime=off zfspool
> zfs set dedup=off zfspool
> zfs create -V 4T zfspool/backup
> zfs set compression=lz4 zfspool/backup
> mkfs.ext4 -b 4096 -q /dev/zvol/zfspool/backup
>
> Is this samba problem or zfs problem?

I assume you're running the latest ZFS on Linux?  (ZFS-fuse is 
considerably slower and is behind on features.)


Why did you have to -f (force) creation?


LUKS?  On the raw drives, or on partitions?  How many CPU's and cores? 
HT?  AES-NI?


Different-sized drives makes things more interesting.  What arrangements 
did you try?   What did you settle on, and why?


Why raidz2?


Turning off atime is okay if none of the applications that use the file 
systems need it.


General de-duplication can be risky, but for many backup solutions it is 
a killer feature (along with replication).


A ZFS volume with an ext4 file system?  Why?


For backups?  I keep mine on dedicated disks in another machine.


Compression is generally regarded as good.


Lots of RAM is recommended in all cases.  How much RAM does you server 
have?  Also note that some people strongly recommend ECC RAM.  Is your 
RAM ECC?


David

[toc] | [prev] | [next] | [standalone]

#161903

On 12/18/2015 03:35 AM, Mimiko wrote:
> I've bonded two onboard Intel 82576 Gigabit networks on an supermicro
> server for load balancing (round-robin). It is working, but transfer
> rate is about 10-20MB/s, while on same type of server the same
> configuration in windows I get around 100MB/s.

How did you measure the transfer rate of 10-20 MB/s?


What kind and how long of cables?  Have you tested the cables with an 
Ethernet cable tester? I bought the Ideal LinkMaster Tester years ago, 
and it has saved my sanity more than once:

     http://www.idealind.com/prodDetail.do?prodId=62-200


What make/ model device are the cables connected to?


> I've tried disable bond and leave only eth0, same result. I've tried to
> update to backported newer kernel with no success. Copying with via ssh
> is same speed. If lots of users starts to use the samba shares, even ssh
> lags.

What motherboard?  What CPU?  What and how much RAM?  What disk/RAID?


David

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web