Groups > linux.debian.user > #208446 > unrolled thread

Speed Problem Copying Files

Started by	Lothar Schilling <ls@proasyl.de>
First post	2019-05-09 11:00 +0200
Last post	2019-05-14 22:10 +0200
Articles	20 on this page of 32 — 13 participants

Back to article view | Back to linux.debian.user

  Speed Problem Copying Files Lothar Schilling <ls@proasyl.de> - 2019-05-09 11:00 +0200
    Re: Speed Problem Copying Files Jonas Smedegaard <jonas@jones.dk> - 2019-05-09 11:20 +0200
      Re: Speed Problem Copying Files Lothar Schilling <ls@proasyl.de> - 2019-05-09 11:50 +0200
        Re: Speed Problem Copying Files Kevin DAGNEAUX <kevin.dagneaux@fiitelcom.fr> - 2019-05-09 12:00 +0200
          Re: Speed Problem Copying Files Lothar Schilling <ls@proasyl.de> - 2019-05-09 12:30 +0200
            Re: Speed Problem Copying Files Martin <keine-eile@online.de> - 2019-05-09 13:30 +0200
              Re: Speed Problem Copying Files Lothar Schilling <ls@proasyl.de> - 2019-05-09 14:50 +0200
                Re: Speed Problem Copying Files Martin <keine-eile@online.de> - 2019-05-09 16:00 +0200
                Re: Speed Problem Copying Files "Thomas Schmitt" <scdbackup@gmx.net> - 2019-05-09 17:20 +0200
        Re: Speed Problem Copying Files Jonas Smedegaard <jonas@jones.dk> - 2019-05-09 12:30 +0200
          Re: Speed Problem Copying Files Jonas Smedegaard <jonas@jones.dk> - 2019-05-09 13:00 +0200
            Re: Speed Problem Copying Files Lothar Schilling <ls@proasyl.de> - 2019-05-09 13:20 +0200
              Re: Speed Problem Copying Files Keith Christian <keith1christian@gmail.com> - 2019-05-09 13:30 +0200
                Re: Speed Problem Copying Files Lothar Schilling <ls@proasyl.de> - 2019-05-09 14:50 +0200
    Re: Speed Problem Copying Files Gene Heskett <gheskett@shentel.net> - 2019-05-09 16:20 +0200
      Re: Speed Problem Copying Files "Jeremy Nicoll" <jn.ml.dbn.25@letterboxes.org> - 2019-05-09 16:30 +0200
        Re: Speed Problem Copying Files Dan Ritter <dsr@randomstring.org> - 2019-05-09 17:00 +0200
        Re: Speed Problem Copying Files Gene Heskett <gheskett@shentel.net> - 2019-05-09 17:00 +0200
    Re: Speed Problem Copying Files David Christensen <dpchrist@holgerdanske.com> - 2019-05-10 07:50 +0200
      Re: Speed Problem Copying Files Lothar Schilling <ls@proasyl.de> - 2019-05-10 11:00 +0200
        Re: Speed Problem Copying Files Lothar Schilling <ls@proasyl.de> - 2019-05-13 10:40 +0200
          Re: Speed Problem Copying Files Lothar Schilling <ls@proasyl.de> - 2019-05-13 11:00 +0200
            Re: Speed Problem Copying Files Tixy <tixy@yxit.co.uk> - 2019-05-13 11:20 +0200
            Re: Speed Problem Copying Files Curt <curty@free.fr> - 2019-05-13 11:40 +0200
            Re: Speed Problem Copying Files Michael Stone <mstone@debian.org> - 2019-05-14 13:40 +0200
          Re: Speed Problem Copying Files Tixy <tixy@yxit.co.uk> - 2019-05-13 11:00 +0200
          Re: Speed Problem Copying Files David Christensen <dpchrist@holgerdanske.com> - 2019-05-14 02:30 +0200
            Re: Speed Problem Copying Files Lothar Schilling <ls@proasyl.de> - 2019-05-14 09:40 +0200
              Re: Speed Problem Copying Files David Christensen <dpchrist@holgerdanske.com> - 2019-05-14 18:30 +0200
      Re: Speed Problem Copying Files Michael Stone <mstone@debian.org> - 2019-05-14 13:30 +0200
        Re: Speed Problem Copying Files David Christensen <dpchrist@holgerdanske.com> - 2019-05-14 21:20 +0200
          Re: Speed Problem Copying Files Michael Stone <mstone@debian.org> - 2019-05-14 22:10 +0200

Page 1 of 2 [1] 2 Next page →

#208446 — Speed Problem Copying Files

From	Lothar Schilling <ls@proasyl.de>
Date	2019-05-09 11:00 +0200
Subject	Speed Problem Copying Files
Message-ID	<xVN4f-3Rc-15@gated-at.bofh.it>

[Multipart message — attachments visible in raw view] — view raw

Hi everybody,

for years I have used CentOS for our server landscape. Now I decided to
give Debian a try. I just set up a Stretch 9.8 system supposed to become
our main backup server. So I set up a backup job wih rsync. But the
going is really very very slooooow. Trying to figure out what's happening:

  * iperf -c [host] => bandwith almost 1000 Mbit, that's fine.
  * dd if=/dev/zero of=/daten/testfile bs=1G count=10 oflag=direct =>
    10737418240 Bytes (11 GB, 10 GiB) kopiert, 40,4992 s, 265 MB/s, so
    that's fine as well

But whenever I try to rsync, cp or scp - whether copy files on the local
hard disk only or over the network - speed goes down to 500 kB/s.

Filesystem is ext4.

I don't have any clue about what's going on. Any kind of help would be
appreciated, thank you!

Lothar

[toc] | [next] | [standalone]

#208447

From	Jonas Smedegaard <jonas@jones.dk>
Date	2019-05-09 11:20 +0200
Message-ID	<xVNnA-4dU-17@gated-at.bofh.it>
In reply to	#208446

[Multipart message — attachments visible in raw view] — view raw

Quoting Lothar Schilling (2019-05-09 10:49:32)
> for years I have used CentOS for our server landscape. Now I decided 
> to give Debian a try.

Welcome to Debian!

I sincerely hope you will appreciate Debian.


> I just set up a Stretch 9.8 system supposed to become our main backup 
> server. So I set up a backup job wih rsync. But the going is really 
> very very slooooow. Trying to figure out what's happening:
> 
>   * iperf -c [host] => bandwith almost 1000 Mbit, that's fine.
>   * dd if=/dev/zero of=/daten/testfile bs=1G count=10 oflag=direct =>
>     10737418240 Bytes (11 GB, 10 GiB) kopiert, 40,4992 s, 265 MB/s, so
>     that's fine as well
> 
> But whenever I try to rsync, cp or scp - whether copy files on the local
> hard disk only or over the network - speed goes down to 500 kB/s.
> 
> Filesystem is ext4.
> 
> I don't have any clue about what's going on. Any kind of help would be
> appreciated, thank you!

Is _only_ transfer speed affected?

I am no expert in this, but imagine that if you rsync massive amounts 
involving hardlinks then memory becomes a problem too.

Perhaps run atop to monitor bottlenecks live


 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/

 [x] quote me freely  [ ] ask before reusing  [ ] keep private

[toc] | [prev] | [next] | [standalone]

#208448

From	Lothar Schilling <ls@proasyl.de>
Date	2019-05-09 11:50 +0200
Message-ID	<xVNQB-4nq-5@gated-at.bofh.it>
In reply to	#208447

Am 09.05.2019 um 11:14 schrieb Jonas Smedegaard:
> Quoting Lothar Schilling (2019-05-09 10:49:32)
>> for years I have used CentOS for our server landscape. Now I decided 
>> to give Debian a try.
> Welcome to Debian!
>
> I sincerely hope you will appreciate Debian.
>
>
>> I just set up a Stretch 9.8 system supposed to become our main backup 
>> server. So I set up a backup job wih rsync. But the going is really 
>> very very slooooow. Trying to figure out what's happening:
>>
>>   * iperf -c [host] => bandwith almost 1000 Mbit, that's fine.
>>   * dd if=/dev/zero of=/daten/testfile bs=1G count=10 oflag=direct =>
>>     10737418240 Bytes (11 GB, 10 GiB) kopiert, 40,4992 s, 265 MB/s, so
>>     that's fine as well
>>
>> But whenever I try to rsync, cp or scp - whether copy files on the local
>> hard disk only or over the network - speed goes down to 500 kB/s.
>>
>> Filesystem is ext4.
>>
>> I don't have any clue about what's going on. Any kind of help would be
>> appreciated, thank you!
> Is _only_ transfer speed affected?
>
> I am no expert in this, but imagine that if you rsync massive amounts 
> involving hardlinks then memory becomes a problem too.
>
> Perhaps run atop to monitor bottlenecks live
>
>
>  - Jonas
>
It is most definitely not a memory or cpu problem. It's a HP Proliant
with 32 GB RAM and a Intel Xeon CPU 2.40GHz with 4 cores. Also the
problem stays the same if I just copy one large file.

[toc] | [prev] | [next] | [standalone]

#208449

From	Kevin DAGNEAUX <kevin.dagneaux@fiitelcom.fr>
Date	2019-05-09 12:00 +0200
Message-ID	<xVO0i-4qM-3@gated-at.bofh.it>
In reply to	#208448

[Multipart message — attachments visible in raw view] — view raw

Le 09/05/2019 à 11:46, Lothar Schilling a écrit :
> Am 09.05.2019 um 11:14 schrieb Jonas Smedegaard:
>> Quoting Lothar Schilling (2019-05-09 10:49:32)
>>> for years I have used CentOS for our server landscape. Now I decided
>>> to give Debian a try.
>> Welcome to Debian!
>>
>> I sincerely hope you will appreciate Debian.
>>
>>
>>> I just set up a Stretch 9.8 system supposed to become our main backup
>>> server. So I set up a backup job wih rsync. But the going is really
>>> very very slooooow. Trying to figure out what's happening:
>>>
>>>    * iperf -c [host] => bandwith almost 1000 Mbit, that's fine.
>>>    * dd if=/dev/zero of=/daten/testfile bs=1G count=10 oflag=direct =>
>>>      10737418240 Bytes (11 GB, 10 GiB) kopiert, 40,4992 s, 265 MB/s, so
>>>      that's fine as well
>>>
>>> But whenever I try to rsync, cp or scp - whether copy files on the local
>>> hard disk only or over the network - speed goes down to 500 kB/s.
>>>
>>> Filesystem is ext4.
>>>
>>> I don't have any clue about what's going on. Any kind of help would be
>>> appreciated, thank you!
>> Is _only_ transfer speed affected?
>>
>> I am no expert in this, but imagine that if you rsync massive amounts
>> involving hardlinks then memory becomes a problem too.
>>
>> Perhaps run atop to monitor bottlenecks live
>>
>>
>>   - Jonas
>>
> It is most definitely not a memory or cpu problem. It's a HP Proliant
> with 32 GB RAM and a Intel Xeon CPU 2.40GHz with 4 cores. Also the
> problem stays the same if I just copy one large file.

Check if it's related to the disk speed (hdparm and/or iotop).

Kevin

[toc] | [prev] | [next] | [standalone]

#208451

From	Lothar Schilling <ls@proasyl.de>
Date	2019-05-09 12:30 +0200
Message-ID	<xVOtk-4Q8-17@gated-at.bofh.it>
In reply to	#208449

Am 09.05.2019 um 11:51 schrieb Kevin DAGNEAUX:
> Le 09/05/2019 à 11:46, Lothar Schilling a écrit :
>> Am 09.05.2019 um 11:14 schrieb Jonas Smedegaard:
>>> Quoting Lothar Schilling (2019-05-09 10:49:32)
>>>> for years I have used CentOS for our server landscape. Now I decided
>>>> to give Debian a try.
>>> Welcome to Debian!
>>>
>>> I sincerely hope you will appreciate Debian.
>>>
>>>
>>>> I just set up a Stretch 9.8 system supposed to become our main backup
>>>> server. So I set up a backup job wih rsync. But the going is really
>>>> very very slooooow. Trying to figure out what's happening:
>>>>
>>>>    * iperf -c [host] => bandwith almost 1000 Mbit, that's fine.
>>>>    * dd if=/dev/zero of=/daten/testfile bs=1G count=10 oflag=direct =>
>>>>      10737418240 Bytes (11 GB, 10 GiB) kopiert, 40,4992 s, 265
>>>> MB/s, so
>>>>      that's fine as well
>>>>
>>>> But whenever I try to rsync, cp or scp - whether copy files on the
>>>> local
>>>> hard disk only or over the network - speed goes down to 500 kB/s.
>>>>
>>>> Filesystem is ext4.
>>>>
>>>> I don't have any clue about what's going on. Any kind of help would be
>>>> appreciated, thank you!
>>> Is _only_ transfer speed affected?
>>>
>>> I am no expert in this, but imagine that if you rsync massive amounts
>>> involving hardlinks then memory becomes a problem too.
>>>
>>> Perhaps run atop to monitor bottlenecks live
>>>
>>>
>>>   - Jonas
>>>
>> It is most definitely not a memory or cpu problem. It's a HP Proliant
>> with 32 GB RAM and a Intel Xeon CPU 2.40GHz with 4 cores. Also the
>> problem stays the same if I just copy one large file.
>
> Check if it's related to the disk speed (hdparm and/or iotop).
>
> Kevin
>

hdparm -tT /dev/sda
/dev/sda:
 Timing cached reads:   13348 MB in  2.00 seconds = 6683.42 MB/sec
 Timing buffered disk reads: 1014 MB in  3.00 seconds = 337.72 MB/sec

iotop -o (for rsync and cp)
Total DISK READ :       0.00 B/s | Total DISK WRITE :     476.15 K/s
Actual DISK READ:       0.00 B/s | Actual DISK WRITE:     487.86 K/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
19531 be/4 root        0.00 B/s  476.15 K/s  0.00 % 99.24 % rsync
--info=progress2 /daten/testfile /daten/testfile2

iotop -o (for dd)
Total DISK READ :       0.00 B/s | Total DISK WRITE :     297.68 M/s
Actual DISK READ:       0.00 B/s | Actual DISK WRITE:     297.68 M/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
19557 be/4 root        0.00 B/s  297.68 M/s  0.00 % 99.99 % dd
if=/dev/zero of=/daten/testfile bs=1G count=10 oflag=direct

[toc] | [prev] | [next] | [standalone]

#208455

From	Martin <keine-eile@online.de>
Date	2019-05-09 13:30 +0200
Message-ID	<xVPpn-5oW-1@gated-at.bofh.it>
In reply to	#208451

[..]
> hdparm -tT /dev/sda
> /dev/sda:
>  Timing cached reads:   13348 MB in  2.00 seconds = 6683.42 MB/sec
>  Timing buffered disk reads: 1014 MB in  3.00 seconds = 337.72 MB/sec
> 
> iotop -o (for rsync and cp)
> Total DISK READ :       0.00 B/s | Total DISK WRITE :     476.15 K/s
> Actual DISK READ:       0.00 B/s | Actual DISK WRITE:     487.86 K/s
>   TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
> 19531 be/4 root        0.00 B/s  476.15 K/s  0.00 % 99.24 % rsync
> --info=progress2 /daten/testfile /daten/testfile2
> 
> iotop -o (for dd)
> Total DISK READ :       0.00 B/s | Total DISK WRITE :     297.68 M/s
> Actual DISK READ:       0.00 B/s | Actual DISK WRITE:     297.68 M/s
>   TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
> 19557 be/4 root        0.00 B/s  297.68 M/s  0.00 % 99.99 % dd
> if=/dev/zero of=/daten/testfile bs=1G count=10 oflag=direct

Show us the 'dd if=/daten/testfile bs=1G oflag=direct of=/dev/null', please.
If this is as slow as this ~480k/s above, check your disk's health status. Like with smartmontools or some disk-utility software.

Martin

[toc] | [prev] | [next] | [standalone]

#208462

From	Lothar Schilling <ls@proasyl.de>
Date	2019-05-09 14:50 +0200
Message-ID	<xVQEO-65h-3@gated-at.bofh.it>
In reply to	#208455

Am 09.05.2019 um 13:27 schrieb Martin:
> [..]
>> hdparm -tT /dev/sda
>> /dev/sda:
>>  Timing cached reads:   13348 MB in  2.00 seconds = 6683.42 MB/sec
>>  Timing buffered disk reads: 1014 MB in  3.00 seconds = 337.72 MB/sec
>>
>> iotop -o (for rsync and cp)
>> Total DISK READ :       0.00 B/s | Total DISK WRITE :     476.15 K/s
>> Actual DISK READ:       0.00 B/s | Actual DISK WRITE:     487.86 K/s
>>   TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
>> 19531 be/4 root        0.00 B/s  476.15 K/s  0.00 % 99.24 % rsync
>> --info=progress2 /daten/testfile /daten/testfile2
>>
>> iotop -o (for dd)
>> Total DISK READ :       0.00 B/s | Total DISK WRITE :     297.68 M/s
>> Actual DISK READ:       0.00 B/s | Actual DISK WRITE:     297.68 M/s
>>   TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
>> 19557 be/4 root        0.00 B/s  297.68 M/s  0.00 % 99.99 % dd
>> if=/dev/zero of=/daten/testfile bs=1G count=10 oflag=direct
> Show us the 'dd if=/daten/testfile bs=1G oflag=direct of=/dev/null', please.
> If this is as slow as this ~480k/s above, check your disk's health status. Like with smartmontools or some disk-utility software.
>
> Martin
>
Fast enough...

dd if=/daten/testfile bs=1G oflag=direct of=/daten/testfile2
10+0 Datensätze ein
10+0 Datensätze aus
10737418240 Bytes (11 GB, 10 GiB) kopiert, 72,7297 s, 148 MB/s

dd if=/daten/testfile of=/dev/null
20971520+0 Datensätze ein
20971520+0 Datensätze aus
10737418240 Bytes (11 GB, 10 GiB) kopiert, 36,6887 s, 293 MB/s

[toc] | [prev] | [next] | [standalone]

#208467

From	Martin <keine-eile@online.de>
Date	2019-05-09 16:00 +0200
Message-ID	<xVRKy-6Iz-9@gated-at.bofh.it>
In reply to	#208462

Am 09.05.19 um 14:43 schrieb Lothar Schilling:
> Am 09.05.2019 um 13:27 schrieb Martin:
>> [..]
>>> hdparm -tT /dev/sda
>>> /dev/sda:
>>>  Timing cached reads:   13348 MB in  2.00 seconds = 6683.42 MB/sec
>>>  Timing buffered disk reads: 1014 MB in  3.00 seconds = 337.72 MB/sec
>>>
>>> iotop -o (for rsync and cp)
>>> Total DISK READ :       0.00 B/s | Total DISK WRITE :     476.15 K/s
>>> Actual DISK READ:       0.00 B/s | Actual DISK WRITE:     487.86 K/s
>>>   TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
>>> 19531 be/4 root        0.00 B/s  476.15 K/s  0.00 % 99.24 % rsync
>>> --info=progress2 /daten/testfile /daten/testfile2
>>>
>>> iotop -o (for dd)
>>> Total DISK READ :       0.00 B/s | Total DISK WRITE :     297.68 M/s
>>> Actual DISK READ:       0.00 B/s | Actual DISK WRITE:     297.68 M/s
>>>   TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
>>> 19557 be/4 root        0.00 B/s  297.68 M/s  0.00 % 99.99 % dd
>>> if=/dev/zero of=/daten/testfile bs=1G count=10 oflag=direct
>> Show us the 'dd if=/daten/testfile bs=1G oflag=direct of=/dev/null', please.
>> If this is as slow as this ~480k/s above, check your disk's health status. Like with smartmontools or some disk-utility software.
>>
>> Martin
>>
> Fast enough...
> 
> dd if=/daten/testfile bs=1G oflag=direct of=/daten/testfile2
> 10+0 Datensätze ein
> 10+0 Datensätze aus
> 10737418240 Bytes (11 GB, 10 GiB) kopiert, 72,7297 s, 148 MB/s
> 
> dd if=/daten/testfile of=/dev/null
> 20971520+0 Datensätze ein
> 20971520+0 Datensätze aus
> 10737418240 Bytes (11 GB, 10 GiB) kopiert, 36,6887 s, 293 MB/s
> 

Well, this looks quite consistent.
How does your system load look, like in top? Do you see any high 'wait' or 'system' numbers?

Then, I have seen a thing called 'systemd.resource-control'. Which I have close to zero knowledge about. Can systemd throttle a copy command? May be some one from this list can tell.

[toc] | [prev] | [next] | [standalone]

#208474

From	"Thomas Schmitt" <scdbackup@gmx.net>
Date	2019-05-09 17:20 +0200
Message-ID	<xVSZX-7DF-1@gated-at.bofh.it>
In reply to	#208462

Hi,

Lothar Schilling wrote:
> Fast enough...
> dd if=/daten/testfile bs=1G oflag=direct of=/daten/testfile2
> 10737418240 Bytes (11 GB, 10 GiB) kopiert, 72,7297 s, 148 MB/s

So this is sufficiently fast, but

  cp /daten/testfile /daten/testfile2

lasts 2000 seconds ?

> dd if=/daten/testfile of=/dev/null
> 10737418240 Bytes (11 GB, 10 GiB) kopiert, 36,6887 s, 293 MB/s

It is not about the number of read/write operations.
(How fast is bs=1G on the second try ?)

------------------------------------------------------------------------

It is also not about the filesystem and its health status.
So where could be any difference ?

dd is an i/o dinosaur. It performs read/write without much mind.

cp uses a centralized copy facility by a function named "copy ()".
This facility makes much ado about sparse files:
  https://sources.debian.org/src/coreutils/8.30-3/src/copy.c/#L387
  https://sources.debian.org/src/coreutils/8.30-3/src/copy.c/#L224

man cp says:

  By default, sparse SOURCE files are detected by a crude  heuristic  and
  the corresponding DEST file is made sparse as well.  That is the behav‐
  ior selected by --sparse=auto.  Specify  --sparse=always  to  create  a
  sparse  DEST  file  whenever  the  SOURCE  file  contains a long enough
  sequence of zero bytes.  Use  --sparse=never  to  inhibit  creation  of
  sparse files.

So i'd try cp with --sparse=never in order to see whether it then resembles
the behavior of dd.

Have a nice day :)

Thomas

[toc] | [prev] | [next] | [standalone]

#208452

From	Jonas Smedegaard <jonas@jones.dk>
Date	2019-05-09 12:30 +0200
Message-ID	<xVOtk-4Q8-21@gated-at.bofh.it>
In reply to	#208448

[Multipart message — attachments visible in raw view] — view raw

Quoting Lothar Schilling (2019-05-09 11:46:00)
> Am 09.05.2019 um 11:14 schrieb Jonas Smedegaard:
> > Quoting Lothar Schilling (2019-05-09 10:49:32)
> >> I just set up a Stretch 9.8 system supposed to become our main 
> >> backup server. So I set up a backup job wih rsync. But the going is 
> >> really very very slooooow. Trying to figure out what's happening:
> >>
> >>   * iperf -c [host] => bandwith almost 1000 Mbit, that's fine.
> >>   * dd if=/dev/zero of=/daten/testfile bs=1G count=10 oflag=direct 
> >>     => 10737418240 Bytes (11 GB, 10 GiB) kopiert, 40,4992 s, 265 
> >>     MB/s, so that's fine as well
> >>
> >> But whenever I try to rsync, cp or scp - whether copy files on the 
> >> local hard disk only or over the network - speed goes down to 500 
> >> kB/s.
> >>
> >> Filesystem is ext4.
> >>
> >> I don't have any clue about what's going on. Any kind of help would 
> >> be appreciated, thank you!
> > Is _only_ transfer speed affected?
> >
> > I am no expert in this, but imagine that if you rsync massive 
> > amounts involving hardlinks then memory becomes a problem too.
> >
> > Perhaps run atop to monitor bottlenecks live
> >
> >
> >  - Jonas
> >
> It is most definitely not a memory or cpu problem. It's a HP Proliant 
> with 32 GB RAM and a Intel Xeon CPU 2.40GHz with 4 cores. Also the 
> problem stays the same if I just copy one large file.

Fair enough :-)

Another shot in the dark: Did you install appropriate firmware packages?

 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/

 [x] quote me freely  [ ] ask before reusing  [ ] keep private

[toc] | [prev] | [next] | [standalone]

#208453

From	Jonas Smedegaard <jonas@jones.dk>
Date	2019-05-09 13:00 +0200
Message-ID	<xVOWl-4ZM-1@gated-at.bofh.it>
In reply to	#208452

[Multipart message — attachments visible in raw view] — view raw

[ replying to list, not discretely ]

Quoting Lothar Schilling (2019-05-09 12:36:55)
> Am 09.05.2019 um 12:26 schrieb Jonas Smedegaard:
> > Quoting Lothar Schilling (2019-05-09 11:46:00)
> >> Am 09.05.2019 um 11:14 schrieb Jonas Smedegaard:
> >>> Quoting Lothar Schilling (2019-05-09 10:49:32)
> >>>> I just set up a Stretch 9.8 system supposed to become our main 
> >>>> backup server. So I set up a backup job wih rsync. But the going is 
> >>>> really very very slooooow. Trying to figure out what's happening:
> >>>>
> >>>>   * iperf -c [host] => bandwith almost 1000 Mbit, that's fine.
> >>>>   * dd if=/dev/zero of=/daten/testfile bs=1G count=10 oflag=direct 
> >>>>     => 10737418240 Bytes (11 GB, 10 GiB) kopiert, 40,4992 s, 265 
> >>>>     MB/s, so that's fine as well
> >>>>
> >>>> But whenever I try to rsync, cp or scp - whether copy files on the 
> >>>> local hard disk only or over the network - speed goes down to 500 
> >>>> kB/s.
> >>>>
> >>>> Filesystem is ext4.
> >>>>
> >>>> I don't have any clue about what's going on. Any kind of help would 
> >>>> be appreciated, thank you!
> >>> Is _only_ transfer speed affected?
> >>>
> >>> I am no expert in this, but imagine that if you rsync massive 
> >>> amounts involving hardlinks then memory becomes a problem too.
> >>>
> >>> Perhaps run atop to monitor bottlenecks live
> >>>
> >>>
> >>>  - Jonas
> >>>
> >> It is most definitely not a memory or cpu problem. It's a HP Proliant 
> >> with 32 GB RAM and a Intel Xeon CPU 2.40GHz with 4 cores. Also the 
> >> problem stays the same if I just copy one large file.
> > Fair enough :-)
> >
> > Another shot in the dark: Did you install appropriate firmware packages?
> >
> >  - Jonas
> >
> No, actually I did not. But on that very same machine I had installed
> Centos 7 a few days earlier. There were no problems.

I am not familiar with Centos but imagine they build kernels differently 
than Debian - in particular handle firmware differently.  That's why I 
bring up that detail.


NB! Debian mailinglist posting style is to post only to list: 
https://www.debian.org/MailingLists/#codeofconduct

(I guess your posting to me discretely was simply an error)


 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/

 [x] quote me freely  [ ] ask before reusing  [ ] keep private

[toc] | [prev] | [next] | [standalone]

#208454

From	Lothar Schilling <ls@proasyl.de>
Date	2019-05-09 13:20 +0200
Message-ID	<xVPfH-5lN-7@gated-at.bofh.it>
In reply to	#208453

Am 09.05.2019 um 12:50 schrieb Jonas Smedegaard:
> [ replying to list, not discretely ]
>
> Quoting Lothar Schilling (2019-05-09 12:36:55)
>> Am 09.05.2019 um 12:26 schrieb Jonas Smedegaard:
>>> Quoting Lothar Schilling (2019-05-09 11:46:00)
>>>> Am 09.05.2019 um 11:14 schrieb Jonas Smedegaard:
>>>>> Quoting Lothar Schilling (2019-05-09 10:49:32)
>>>>>> I just set up a Stretch 9.8 system supposed to become our main 
>>>>>> backup server. So I set up a backup job wih rsync. But the going is 
>>>>>> really very very slooooow. Trying to figure out what's happening:
>>>>>>
>>>>>>   * iperf -c [host] => bandwith almost 1000 Mbit, that's fine.
>>>>>>   * dd if=/dev/zero of=/daten/testfile bs=1G count=10 oflag=direct 
>>>>>>     => 10737418240 Bytes (11 GB, 10 GiB) kopiert, 40,4992 s, 265 
>>>>>>     MB/s, so that's fine as well
>>>>>>
>>>>>> But whenever I try to rsync, cp or scp - whether copy files on the 
>>>>>> local hard disk only or over the network - speed goes down to 500 
>>>>>> kB/s.
>>>>>>
>>>>>> Filesystem is ext4.
>>>>>>
>>>>>> I don't have any clue about what's going on. Any kind of help would 
>>>>>> be appreciated, thank you!
>>>>> Is _only_ transfer speed affected?
>>>>>
>>>>> I am no expert in this, but imagine that if you rsync massive 
>>>>> amounts involving hardlinks then memory becomes a problem too.
>>>>>
>>>>> Perhaps run atop to monitor bottlenecks live
>>>>>
>>>>>
>>>>>  - Jonas
>>>>>
>>>> It is most definitely not a memory or cpu problem. It's a HP Proliant 
>>>> with 32 GB RAM and a Intel Xeon CPU 2.40GHz with 4 cores. Also the 
>>>> problem stays the same if I just copy one large file.
>>> Fair enough :-)
>>>
>>> Another shot in the dark: Did you install appropriate firmware packages?
>>>
>>>  - Jonas
>>>
>> No, actually I did not. But on that very same machine I had installed
>> Centos 7 a few days earlier. There were no problems.
> I am not familiar with Centos but imagine they build kernels differently 
> than Debian - in particular handle firmware differently.  That's why I 
> bring up that detail.
>
> NB! Debian mailinglist posting style is to post only to list: 
> https://www.debian.org/MailingLists/#codeofconduct
>
> (I guess your posting to me discretely was simply an error)
>
>
>  - Jonas

Yes, posting to you  instead of the list was by accident. Sorry. I still
do not think, though, that this is firmware-related. But I'll ask HP
Support. In the meantime I hope there is going to more suggestions for a
solution to my weird problem...

[toc] | [prev] | [next] | [standalone]

#208456

From	Keith Christian <keith1christian@gmail.com>
Date	2019-05-09 13:30 +0200
Message-ID	<xVPpn-5oW-5@gated-at.bofh.it>
In reply to	#208454

[Multipart message — attachments visible in raw view] — view raw

What is the rsync command line, could there be a —bwlimit option in it?

[toc] | [prev] | [next] | [standalone]

#208463

From	Lothar Schilling <ls@proasyl.de>
Date	2019-05-09 14:50 +0200
Message-ID	<xVQEO-65h-5@gated-at.bofh.it>
In reply to	#208456

Am 09.05.2019 um 13:23 schrieb Keith Christian:
> What is the rsync command line, could there be a —bwlimit option in it?

No, there isn't. Anyway, cp has the same problem.

[toc] | [prev] | [next] | [standalone]

#208469

From	Gene Heskett <gheskett@shentel.net>
Date	2019-05-09 16:20 +0200
Message-ID	<xVS3T-74l-1@gated-at.bofh.it>
In reply to	#208446

On Thursday 09 May 2019 04:49:32 am Lothar Schilling wrote:

> Hi everybody,
>
> for years I have used CentOS for our server landscape. Now I decided
> to give Debian a try. I just set up a Stretch 9.8 system supposed to
> become our main backup server. So I set up a backup job wih rsync. But
> the going is really very very slooooow. Trying to figure out what's
> happening:
>
>   * iperf -c [host] => bandwith almost 1000 Mbit, that's fine.
>   * dd if=/dev/zero of=/daten/testfile bs=1G count=10 oflag=direct =>
>     10737418240 Bytes (11 GB, 10 GiB) kopiert, 40,4992 s, 265 MB/s, so
>     that's fine as well
>
> But whenever I try to rsync, cp or scp - whether copy files on the
> local hard disk only or over the network - speed goes down to 500
> kB/s.
>
> Filesystem is ext4.
>
> I don't have any clue about what's going on. Any kind of help would be
> appreciated, thank you!
>
> Lothar

This is an interesting question. I am useing mc to copy data from old 
wheezy drive to new stretch drive, both with speeds well above 
100M/second performance, but when dealing with small files, theres a 
penalty for obtaining the space to write to, and for the cleanup after, 
and its substantial. So I see speeds below 5 megs when dealing with 
spinning rust and my maildir format email corpus. Drive seek speeds and 
rotational latency will all figure into that, and its one of the reasons 
an SSD seems so much faster because they seek in a microsecond, and they 
don't have to wait until the sector passes under the heads of the drive.

Anything that will speed up that aspect of moving data will surely be 
appreciated, so I'll read carefully the responses here.

Thank you.

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>

[toc] | [prev] | [next] | [standalone]

#208471

From	"Jeremy Nicoll" <jn.ml.dbn.25@letterboxes.org>
Date	2019-05-09 16:30 +0200
Message-ID	<xVSdA-77C-9@gated-at.bofh.it>
In reply to	#208469

On Thu, 9 May 2019, at 15:17, Gene Heskett wrote:
> ... and its one of the reasons 
> an SSD seems so much faster because they seek in a microsecond

In what sense does an SSD have "seek time"?     Seek time is
a tightly defined thing.

There must be delays in SSD firmware's processing, I suppose, but
it isn't seek time.

Also, I wonder how you can say that that delay is a microsecond long? 

Which manuafacturer, which product, what level of firmware?

-- 
Jeremy Nicoll - my opinions are my own.

[toc] | [prev] | [next] | [standalone]

#208472

From	Dan Ritter <dsr@randomstring.org>
Date	2019-05-09 17:00 +0200
Message-ID	<xVSGB-7hG-1@gated-at.bofh.it>
In reply to	#208471

Jeremy Nicoll wrote: 
> On Thu, 9 May 2019, at 15:17, Gene Heskett wrote:
> > ... and its one of the reasons 
> > an SSD seems so much faster because they seek in a microsecond
> 
> In what sense does an SSD have "seek time"?     Seek time is
> a tightly defined thing.
> 
> There must be delays in SSD firmware's processing, I suppose, but
> it isn't seek time.

In this case, the latency between the OS issuing a read/write request
and the drive beginning to fulfill it. 

IOPs is usually a more useful way of putting it: a spinning disk
offers between 80 and 150 I/O operations (seeks) per second; a
typical SSD offers between 15000 and 600000 IOPs.

-dsr-

[toc] | [prev] | [next] | [standalone]

#208473

From	Gene Heskett <gheskett@shentel.net>
Date	2019-05-09 17:00 +0200
Message-ID	<xVSGB-7hG-5@gated-at.bofh.it>
In reply to	#208471

On Thursday 09 May 2019 10:28:10 am Jeremy Nicoll wrote:

> On Thu, 9 May 2019, at 15:17, Gene Heskett wrote:
> > ... and its one of the reasons
> > an SSD seems so much faster because they seek in a microsecond
>
> In what sense does an SSD have "seek time"?     Seek time is
> a tightly defined thing.
>
> There must be delays in SSD firmware's processing, I suppose, but
> it isn't seek time.
>
> Also, I wonder how you can say that that delay is a microsecond long?
>
IDK for sure, but its just a rewrite of it data pointer, si I use 
microsecond to connote the instant factor.

> Which manuafacturer, which product, what level of firmware?

I have a mixture of kinston and adata SSD's here, all in the basic 40 to 
60 Gb sizes, and working lots faster than spinning rust. For most of 
what I do with those machines, a 20 Gb drive is more than enough.

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>

[toc] | [prev] | [next] | [standalone]

#208494

From	David Christensen <dpchrist@holgerdanske.com>
Date	2019-05-10 07:50 +0200
Message-ID	<xW6zT-7v1-1@gated-at.bofh.it>
In reply to	#208446

On 5/9/19 1:49 AM, Lothar Schilling wrote:
> Hi everybody,
> 
> for years I have used CentOS for our server landscape. Now I decided to
> give Debian a try. I just set up a Stretch 9.8 system supposed to become
> our main backup server. So I set up a backup job wih rsync. But the
> going is really very very slooooow. Trying to figure out what's happening:
> 
>    * iperf -c [host] => bandwith almost 1000 Mbit, that's fine.
>    * dd if=/dev/zero of=/daten/testfile bs=1G count=10 oflag=direct =>
>      10737418240 Bytes (11 GB, 10 GiB) kopiert, 40,4992 s, 265 MB/s, so
>      that's fine as well
> 
> But whenever I try to rsync, cp or scp - whether copy files on the local
> hard disk only or over the network - speed goes down to 500 kB/s.
> 
> Filesystem is ext4.
> 
> I don't have any clue about what's going on. Any kind of help would be
> appreciated, thank you!
> 
> Lothar


On 5/9/19 2:46 AM, Lothar Schilling wrote:> Am 09.05.2019 um 11:14 
schrieb Jonas Smedegaard:
 >> Is _only_ transfer speed affected?
 >>
 >> I am no expert in this, but imagine that if you rsync massive amounts
 >> involving hardlinks then memory becomes a problem too.
 >>
 >> Perhaps run atop to monitor bottlenecks live
 >>
 >>
 >>   - Jonas
 >>
 > It is most definitely not a memory or cpu problem. It's a HP Proliant
 > with 32 GB RAM and a Intel Xeon CPU 2.40GHz with 4 cores. Also the
 > problem stays the same if I just copy one large file.


On 5/9/19 3:26 AM, Lothar Schilling wrote:> Am 09.05.2019 um 11:51 
schrieb Kevin DAGNEAUX:
 >> Check if it's related to the disk speed (hdparm and/or iotop).
 >>
 >> Kevin
 >>
 >
 > hdparm -tT /dev/sda
 > /dev/sda:
 >   Timing cached reads:   13348 MB in  2.00 seconds = 6683.42 MB/sec
 >   Timing buffered disk reads: 1014 MB in  3.00 seconds = 337.72 MB/sec
 >
 > iotop -o (for rsync and cp)
 > Total DISK READ :       0.00 B/s | Total DISK WRITE :     476.15 K/s
 > Actual DISK READ:       0.00 B/s | Actual DISK WRITE:     487.86 K/s
 >    TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
 > 19531 be/4 root        0.00 B/s  476.15 K/s  0.00 % 99.24 % rsync
 > --info=progress2 /daten/testfile /daten/testfile2
 >
 > iotop -o (for dd)
 > Total DISK READ :       0.00 B/s | Total DISK WRITE :     297.68 M/s
 > Actual DISK READ:       0.00 B/s | Actual DISK WRITE:     297.68 M/s
 >    TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
 > 19557 be/4 root        0.00 B/s  297.68 M/s  0.00 % 99.99 % dd
 > if=/dev/zero of=/daten/testfile bs=1G count=10 oflag=direct


On 5/9/19 5:43 AM, Lothar Schilling wrote:> Am 09.05.2019 um 13:27 
schrieb Martin:
 >> [..]
 >>> hdparm -tT /dev/sda
 >>> /dev/sda:
 >>>   Timing cached reads:   13348 MB in  2.00 seconds = 6683.42 MB/sec
 >>>   Timing buffered disk reads: 1014 MB in  3.00 seconds = 337.72 MB/sec
 >>>
 >>> iotop -o (for rsync and cp)
 >>> Total DISK READ :       0.00 B/s | Total DISK WRITE :     476.15 K/s
 >>> Actual DISK READ:       0.00 B/s | Actual DISK WRITE:     487.86 K/s
 >>>    TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
 >>> 19531 be/4 root        0.00 B/s  476.15 K/s  0.00 % 99.24 % rsync
 >>> --info=progress2 /daten/testfile /daten/testfile2
 >>>
 >>> iotop -o (for dd)
 >>> Total DISK READ :       0.00 B/s | Total DISK WRITE :     297.68 M/s
 >>> Actual DISK READ:       0.00 B/s | Actual DISK WRITE:     297.68 M/s
 >>>    TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
 >>> 19557 be/4 root        0.00 B/s  297.68 M/s  0.00 % 99.99 % dd
 >>> if=/dev/zero of=/daten/testfile bs=1G count=10 oflag=direct
 >> Show us the 'dd if=/daten/testfile bs=1G oflag=direct of=/dev/null', 
please.
 >> If this is as slow as this ~480k/s above, check your disk's health 
status. Like with smartmontools or some disk-utility software.
 >>
 >> Martin
 >>
 > Fast enough...
 >
 > dd if=/daten/testfile bs=1G oflag=direct of=/daten/testfile2
 > 10+0 Datensätze ein
 > 10+0 Datensätze aus
 > 10737418240 Bytes (11 GB, 10 GiB) kopiert, 72,7297 s, 148 MB/s
 >
 > dd if=/daten/testfile of=/dev/null
 > 20971520+0 Datensätze ein
 > 20971520+0 Datensätze aus
 > 10737418240 Bytes (11 GB, 10 GiB) kopiert, 36,6887 s, 293 MB/s


rsync(1) can be fickle, especially when host operating systems and/or 
rsync versions mismatch.  But, you also seem to have file system issues.


The above information could be more useful if it were presented in 
context -- e.g. complete console sessions -- meaningful prompts (date, 
time, username, hostname, working directory), exact commands input, and 
exact output printed.


Please use the Bash shell for the root account and the following line to 
/root/.bashrc:

     export PS1='\n\D{%Y-%m-%d %H:%M:%S} '${USER}'@\h \w\n\$ '


Then start a new shell, su(1) to root, change to a directory within the 
file system that is having issues, and run the following commands. 
Adjust the commands for your hosts and network, as required ('po' is my 
workstation, 'dpchrist' is my user account, 'samba' is my file server, 
and '/var/local/samba/dpchrist' is my storage directory on my file server):

2019-05-09 21:59:55 root@po /mnt/scratch
# cat /etc/debian_version
9.9

2019-05-09 22:00:00 root@po /mnt/scratch
# uname -a
Linux po 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1 (2019-04-12) x86_64 GNU/Linux

2019-05-09 22:00:05 root@po /mnt/scratch
# lsblk
NAME           MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda              8:0    0  55.9G  0 disk
|-sda1           8:1    0   953M  0 part  /boot
|-sda2           8:2    0   1.9G  0 part
| `-sda2_crypt 253:1    0   1.9G  0 crypt [SWAP]
|-sda3           8:3    0   9.3G  0 part
| `-sda3_crypt 253:0    0   9.3G  0 crypt /
`-sda4           8:4    0  38.2G  0 part
   `-sda4_crypt 253:2    0  38.2G  0 crypt /mnt/scratch
sr0             11:0    1  1024M  0 rom

2019-05-09 22:00:10 root@po /mnt/scratch
# mount | egrep 'sd[a-z]'
/dev/mapper/sda3_crypt on / type btrfs 
(rw,relatime,ssd,space_cache,subvolid=5,subvol=/)
/dev/sda1 on /boot type btrfs 
(rw,relatime,ssd,space_cache,subvolid=5,subvol=/)
/dev/mapper/sda4_crypt on /mnt/scratch type btrfs 
(rw,relatime,ssd,space_cache,subvolid=5,subvol=/)

2019-05-09 22:00:19 root@po /mnt/scratch
# df | egrep 'sd[a-z]'
/dev/mapper/sda3_crypt   9763840 5540800   4160128  58% /
/dev/sda1                 975872   80892    800244  10% /boot
/dev/mapper/sda4_crypt  40056832 1066752  38732224   3% /mnt/scratch

2019-05-09 22:00:27 root@po /mnt/scratch
# time dd if=/dev/urandom of=foo bs=1M count=1K conv=fsync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 15.987 s, 67.2 MB/s

real	0m15.988s
user	0m0.000s
sys	0m3.976s

2019-05-09 22:02:13 root@po /mnt/scratch
# echo 3 > /proc/sys/vm/drop_caches && time dd if=foo of=/dev/null bs=1M 
count=1K
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.49821 s, 430 MB/s

real	0m2.518s
user	0m0.000s
sys	0m0.252s

2019-05-09 22:27:32 root@po /mnt/scratch
# echo 3 > /proc/sys/vm/drop_caches && f=foo && g=bar && 
uh=dpchrist@samba && p=/var/local/samba/dpchrist/test && ssh $uh rm -f 
$p/$g && rsync -a --stats $f $uh:$p/$g

Number of files: 1 (reg: 1)
Number of created files: 1 (reg: 1)
Number of deleted files: 0
Number of regular files transferred: 1
Total file size: 1,073,741,824 bytes
Total transferred file size: 1,073,741,824 bytes
Literal data: 1,073,741,824 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 1,074,004,051
Total bytes received: 35

sent 1,074,004,051 bytes  received 35 bytes  113,053,061.68 bytes/sec
total size is 1,073,741,824  speedup is 1.00

2019-05-09 22:27:52 root@po /mnt/scratch
# echo 3 > /proc/sys/vm/drop_caches && f=foo && g=bar && 
uh=dpchrist@samba && p=/var/local/samba/dpchrist/test && rm -f $g && 
rsync -a --stats $uh:$p/$g $g

Number of files: 1 (reg: 1)
Number of created files: 1 (reg: 1)
Number of deleted files: 0
Number of regular files transferred: 1
Total file size: 1,073,741,824 bytes
Total transferred file size: 1,073,741,824 bytes
Literal data: 1,073,741,824 bytes
Matched data: 0 bytes
File list size: 43
File list generation time: 0.022 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 43
Total bytes received: 1,074,004,061

sent 43 bytes  received 1,074,004,061 bytes  102,286,105.14 bytes/sec
total size is 1,073,741,824  speedup is 1.00


Add in other commands as you see fit, such as whatever generated the 
output in your previous posts.


Please reply with your *complete* console session (you can remove all of 
the above in your reply).


David

[toc] | [prev] | [next] | [standalone]

#208498

From	Lothar Schilling <ls@proasyl.de>
Date	2019-05-10 11:00 +0200
Message-ID	<xW9xL-M5-3@gated-at.bofh.it>
In reply to	#208494

Am 10.05.2019 um 07:48 schrieb David Christensen:
> On 5/9/19 1:49 AM, Lothar Schilling wrote:
>> Hi everybody,
>>
>> for years I have used CentOS for our server landscape. Now I decided to
>> give Debian a try. I just set up a Stretch 9.8 system supposed to become
>> our main backup server. So I set up a backup job wih rsync. But the
>> going is really very very slooooow. Trying to figure out what's
>> happening:
>>
>>    * iperf -c [host] => bandwith almost 1000 Mbit, that's fine.
>>    * dd if=/dev/zero of=/daten/testfile bs=1G count=10 oflag=direct =>
>>      10737418240 Bytes (11 GB, 10 GiB) kopiert, 40,4992 s, 265 MB/s, so
>>      that's fine as well
>>
>> But whenever I try to rsync, cp or scp - whether copy files on the local
>> hard disk only or over the network - speed goes down to 500 kB/s.
>>
>> Filesystem is ext4.
>>
>> I don't have any clue about what's going on. Any kind of help would be
>> appreciated, thank you!
>>
>> Lothar
>
>
> On 5/9/19 2:46 AM, Lothar Schilling wrote:> Am 09.05.2019 um 11:14
> schrieb Jonas Smedegaard:
> >> Is _only_ transfer speed affected?
> >>
> >> I am no expert in this, but imagine that if you rsync massive amounts
> >> involving hardlinks then memory becomes a problem too.
> >>
> >> Perhaps run atop to monitor bottlenecks live
> >>
> >>
> >>   - Jonas
> >>
> > It is most definitely not a memory or cpu problem. It's a HP Proliant
> > with 32 GB RAM and a Intel Xeon CPU 2.40GHz with 4 cores. Also the
> > problem stays the same if I just copy one large file.
>
>
> On 5/9/19 3:26 AM, Lothar Schilling wrote:> Am 09.05.2019 um 11:51
> schrieb Kevin DAGNEAUX:
> >> Check if it's related to the disk speed (hdparm and/or iotop).
> >>
> >> Kevin
> >>
> >
> > hdparm -tT /dev/sda
> > /dev/sda:
> >   Timing cached reads:   13348 MB in  2.00 seconds = 6683.42 MB/sec
> >   Timing buffered disk reads: 1014 MB in  3.00 seconds = 337.72 MB/sec
> >
> > iotop -o (for rsync and cp)
> > Total DISK READ :       0.00 B/s | Total DISK WRITE :     476.15 K/s
> > Actual DISK READ:       0.00 B/s | Actual DISK WRITE:     487.86 K/s
> >    TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
> > 19531 be/4 root        0.00 B/s  476.15 K/s  0.00 % 99.24 % rsync
> > --info=progress2 /daten/testfile /daten/testfile2
> >
> > iotop -o (for dd)
> > Total DISK READ :       0.00 B/s | Total DISK WRITE :     297.68 M/s
> > Actual DISK READ:       0.00 B/s | Actual DISK WRITE:     297.68 M/s
> >    TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
> > 19557 be/4 root        0.00 B/s  297.68 M/s  0.00 % 99.99 % dd
> > if=/dev/zero of=/daten/testfile bs=1G count=10 oflag=direct
>
>
> On 5/9/19 5:43 AM, Lothar Schilling wrote:> Am 09.05.2019 um 13:27
> schrieb Martin:
> >> [..]
> >>> hdparm -tT /dev/sda
> >>> /dev/sda:
> >>>   Timing cached reads:   13348 MB in  2.00 seconds = 6683.42 MB/sec
> >>>   Timing buffered disk reads: 1014 MB in  3.00 seconds = 337.72
> MB/sec
> >>>
> >>> iotop -o (for rsync and cp)
> >>> Total DISK READ :       0.00 B/s | Total DISK WRITE :     476.15 K/s
> >>> Actual DISK READ:       0.00 B/s | Actual DISK WRITE:     487.86 K/s
> >>>    TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>   
> COMMAND
> >>> 19531 be/4 root        0.00 B/s  476.15 K/s  0.00 % 99.24 % rsync
> >>> --info=progress2 /daten/testfile /daten/testfile2
> >>>
> >>> iotop -o (for dd)
> >>> Total DISK READ :       0.00 B/s | Total DISK WRITE :     297.68 M/s
> >>> Actual DISK READ:       0.00 B/s | Actual DISK WRITE:     297.68 M/s
> >>>    TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>   
> COMMAND
> >>> 19557 be/4 root        0.00 B/s  297.68 M/s  0.00 % 99.99 % dd
> >>> if=/dev/zero of=/daten/testfile bs=1G count=10 oflag=direct
> >> Show us the 'dd if=/daten/testfile bs=1G oflag=direct
> of=/dev/null', please.
> >> If this is as slow as this ~480k/s above, check your disk's health
> status. Like with smartmontools or some disk-utility software.
> >>
> >> Martin
> >>
> > Fast enough...
> >
> > dd if=/daten/testfile bs=1G oflag=direct of=/daten/testfile2
> > 10+0 Datensätze ein
> > 10+0 Datensätze aus
> > 10737418240 Bytes (11 GB, 10 GiB) kopiert, 72,7297 s, 148 MB/s
> >
> > dd if=/daten/testfile of=/dev/null
> > 20971520+0 Datensätze ein
> > 20971520+0 Datensätze aus
> > 10737418240 Bytes (11 GB, 10 GiB) kopiert, 36,6887 s, 293 MB/s
>
>
> rsync(1) can be fickle, especially when host operating systems and/or
> rsync versions mismatch.  But, you also seem to have file system issues.
>
>
> The above information could be more useful if it were presented in
> context -- e.g. complete console sessions -- meaningful prompts (date,
> time, username, hostname, working directory), exact commands input,
> and exact output printed.
>
>
> Please use the Bash shell for the root account and the following line
> to /root/.bashrc:
>
>     export PS1='\n\D{%Y-%m-%d %H:%M:%S} '${USER}'@\h \w\n\$ '
>
>
> Then start a new shell, su(1) to root, change to a directory within
> the file system that is having issues, and run the following commands.
> Adjust the commands for your hosts and network, as required ('po' is
> my workstation, 'dpchrist' is my user account, 'samba' is my file
> server, and '/var/local/samba/dpchrist' is my storage directory on my
> file server):
>
> 2019-05-09 21:59:55 root@po /mnt/scratch
> # cat /etc/debian_version
> 9.9
>
> 2019-05-09 22:00:00 root@po /mnt/scratch
> # uname -a
> Linux po 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1 (2019-04-12) x86_64
> GNU/Linux
>
> 2019-05-09 22:00:05 root@po /mnt/scratch
> # lsblk
> NAME           MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
> sda              8:0    0  55.9G  0 disk
> |-sda1           8:1    0   953M  0 part  /boot
> |-sda2           8:2    0   1.9G  0 part
> | `-sda2_crypt 253:1    0   1.9G  0 crypt [SWAP]
> |-sda3           8:3    0   9.3G  0 part
> | `-sda3_crypt 253:0    0   9.3G  0 crypt /
> `-sda4           8:4    0  38.2G  0 part
>   `-sda4_crypt 253:2    0  38.2G  0 crypt /mnt/scratch
> sr0             11:0    1  1024M  0 rom
>
> 2019-05-09 22:00:10 root@po /mnt/scratch
> # mount | egrep 'sd[a-z]'
> /dev/mapper/sda3_crypt on / type btrfs
> (rw,relatime,ssd,space_cache,subvolid=5,subvol=/)
> /dev/sda1 on /boot type btrfs
> (rw,relatime,ssd,space_cache,subvolid=5,subvol=/)
> /dev/mapper/sda4_crypt on /mnt/scratch type btrfs
> (rw,relatime,ssd,space_cache,subvolid=5,subvol=/)
>
> 2019-05-09 22:00:19 root@po /mnt/scratch
> # df | egrep 'sd[a-z]'
> /dev/mapper/sda3_crypt   9763840 5540800   4160128  58% /
> /dev/sda1                 975872   80892    800244  10% /boot
> /dev/mapper/sda4_crypt  40056832 1066752  38732224   3% /mnt/scratch
>
> 2019-05-09 22:00:27 root@po /mnt/scratch
> # time dd if=/dev/urandom of=foo bs=1M count=1K conv=fsync
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 15.987 s, 67.2 MB/s
>
> real    0m15.988s
> user    0m0.000s
> sys    0m3.976s
>
> 2019-05-09 22:02:13 root@po /mnt/scratch
> # echo 3 > /proc/sys/vm/drop_caches && time dd if=foo of=/dev/null
> bs=1M count=1K
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.49821 s, 430 MB/s
>
> real    0m2.518s
> user    0m0.000s
> sys    0m0.252s
>
> 2019-05-09 22:27:32 root@po /mnt/scratch
> # echo 3 > /proc/sys/vm/drop_caches && f=foo && g=bar &&
> uh=dpchrist@samba && p=/var/local/samba/dpchrist/test && ssh $uh rm -f
> $p/$g && rsync -a --stats $f $uh:$p/$g
>
> Number of files: 1 (reg: 1)
> Number of created files: 1 (reg: 1)
> Number of deleted files: 0
> Number of regular files transferred: 1
> Total file size: 1,073,741,824 bytes
> Total transferred file size: 1,073,741,824 bytes
> Literal data: 1,073,741,824 bytes
> Matched data: 0 bytes
> File list size: 0
> File list generation time: 0.001 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 1,074,004,051
> Total bytes received: 35
>
> sent 1,074,004,051 bytes  received 35 bytes  113,053,061.68 bytes/sec
> total size is 1,073,741,824  speedup is 1.00
>
> 2019-05-09 22:27:52 root@po /mnt/scratch
> # echo 3 > /proc/sys/vm/drop_caches && f=foo && g=bar &&
> uh=dpchrist@samba && p=/var/local/samba/dpchrist/test && rm -f $g &&
> rsync -a --stats $uh:$p/$g $g
>
> Number of files: 1 (reg: 1)
> Number of created files: 1 (reg: 1)
> Number of deleted files: 0
> Number of regular files transferred: 1
> Total file size: 1,073,741,824 bytes
> Total transferred file size: 1,073,741,824 bytes
> Literal data: 1,073,741,824 bytes
> Matched data: 0 bytes
> File list size: 43
> File list generation time: 0.022 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 43
> Total bytes received: 1,074,004,061
>
> sent 43 bytes  received 1,074,004,061 bytes  102,286,105.14 bytes/sec
> total size is 1,073,741,824  speedup is 1.00
>
>
> Add in other commands as you see fit, such as whatever generated the
> output in your previous posts.
>
>
> Please reply with your *complete* console session (you can remove all
> of the above in your reply).
>
>
> David
>
Wow, thank you! Getting into that on Monday hoping I can deliver...

[toc] | [prev] | [next] | [standalone]

Page 1 of 2 [1] 2 Next page →

csiph-web

Speed Problem Copying Files

Contents

#208446 — Speed Problem Copying Files

#208447

#208448

#208449

#208451

#208455

#208462

#208467

#208474

#208452

#208453

#208454

#208456

#208463

#208469

#208471

#208472

#208473

#208494

#208498