Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #208446 > unrolled thread

Speed Problem Copying Files

Started byLothar Schilling <ls@proasyl.de>
First post2019-05-09 11:00 +0200
Last post2019-05-14 22:10 +0200
Articles 12 on this page of 32 — 13 participants

Back to article view | Back to linux.debian.user

Contents

  Speed Problem Copying Files Lothar Schilling <ls@proasyl.de> - 2019-05-09 11:00 +0200
    Re: Speed Problem Copying Files Jonas Smedegaard <jonas@jones.dk> - 2019-05-09 11:20 +0200
      Re: Speed Problem Copying Files Lothar Schilling <ls@proasyl.de> - 2019-05-09 11:50 +0200
        Re: Speed Problem Copying Files Kevin DAGNEAUX <kevin.dagneaux@fiitelcom.fr> - 2019-05-09 12:00 +0200
          Re: Speed Problem Copying Files Lothar Schilling <ls@proasyl.de> - 2019-05-09 12:30 +0200
            Re: Speed Problem Copying Files Martin <keine-eile@online.de> - 2019-05-09 13:30 +0200
              Re: Speed Problem Copying Files Lothar Schilling <ls@proasyl.de> - 2019-05-09 14:50 +0200
                Re: Speed Problem Copying Files Martin <keine-eile@online.de> - 2019-05-09 16:00 +0200
                Re: Speed Problem Copying Files "Thomas Schmitt" <scdbackup@gmx.net> - 2019-05-09 17:20 +0200
        Re: Speed Problem Copying Files Jonas Smedegaard <jonas@jones.dk> - 2019-05-09 12:30 +0200
          Re: Speed Problem Copying Files Jonas Smedegaard <jonas@jones.dk> - 2019-05-09 13:00 +0200
            Re: Speed Problem Copying Files Lothar Schilling <ls@proasyl.de> - 2019-05-09 13:20 +0200
              Re: Speed Problem Copying Files Keith Christian <keith1christian@gmail.com> - 2019-05-09 13:30 +0200
                Re: Speed Problem Copying Files Lothar Schilling <ls@proasyl.de> - 2019-05-09 14:50 +0200
    Re: Speed Problem Copying Files Gene Heskett <gheskett@shentel.net> - 2019-05-09 16:20 +0200
      Re: Speed Problem Copying Files "Jeremy Nicoll" <jn.ml.dbn.25@letterboxes.org> - 2019-05-09 16:30 +0200
        Re: Speed Problem Copying Files Dan Ritter <dsr@randomstring.org> - 2019-05-09 17:00 +0200
        Re: Speed Problem Copying Files Gene Heskett <gheskett@shentel.net> - 2019-05-09 17:00 +0200
    Re: Speed Problem Copying Files David Christensen <dpchrist@holgerdanske.com> - 2019-05-10 07:50 +0200
      Re: Speed Problem Copying Files Lothar Schilling <ls@proasyl.de> - 2019-05-10 11:00 +0200
        Re: Speed Problem Copying Files Lothar Schilling <ls@proasyl.de> - 2019-05-13 10:40 +0200
          Re: Speed Problem Copying Files Lothar Schilling <ls@proasyl.de> - 2019-05-13 11:00 +0200
            Re: Speed Problem Copying Files Tixy <tixy@yxit.co.uk> - 2019-05-13 11:20 +0200
            Re: Speed Problem Copying Files Curt <curty@free.fr> - 2019-05-13 11:40 +0200
            Re: Speed Problem Copying Files Michael Stone <mstone@debian.org> - 2019-05-14 13:40 +0200
          Re: Speed Problem Copying Files Tixy <tixy@yxit.co.uk> - 2019-05-13 11:00 +0200
          Re: Speed Problem Copying Files David Christensen <dpchrist@holgerdanske.com> - 2019-05-14 02:30 +0200
            Re: Speed Problem Copying Files Lothar Schilling <ls@proasyl.de> - 2019-05-14 09:40 +0200
              Re: Speed Problem Copying Files David Christensen <dpchrist@holgerdanske.com> - 2019-05-14 18:30 +0200
      Re: Speed Problem Copying Files Michael Stone <mstone@debian.org> - 2019-05-14 13:30 +0200
        Re: Speed Problem Copying Files David Christensen <dpchrist@holgerdanske.com> - 2019-05-14 21:20 +0200
          Re: Speed Problem Copying Files Michael Stone <mstone@debian.org> - 2019-05-14 22:10 +0200

Page 2 of 2 — ← Prev page 1 [2]

#208610

FromLothar Schilling <ls@proasyl.de>
Date2019-05-13 10:40 +0200
Message-ID<xXeF3-yg-7@gated-at.bofh.it>
In reply to#208498
Am 10.05.2019 um 10:51 schrieb Lothar Schilling:
> Am 10.05.2019 um 07:48 schrieb David Christensen:
>> On 5/9/19 1:49 AM, Lothar Schilling wrote:
>>> Hi everybody,
>>>
>>> for years I have used CentOS for our server landscape. Now I decided to
>>> give Debian a try. I just set up a Stretch 9.8 system supposed to become
>>> our main backup server. So I set up a backup job wih rsync. But the
>>> going is really very very slooooow. Trying to figure out what's
>>> happening:
>>>
>>>    * iperf -c [host] => bandwith almost 1000 Mbit, that's fine.
>>>    * dd if=/dev/zero of=/daten/testfile bs=1G count=10 oflag=direct =>
>>>      10737418240 Bytes (11 GB, 10 GiB) kopiert, 40,4992 s, 265 MB/s, so
>>>      that's fine as well
>>>
>>> But whenever I try to rsync, cp or scp - whether copy files on the local
>>> hard disk only or over the network - speed goes down to 500 kB/s.
>>>
>>> Filesystem is ext4.
>>>
>>> I don't have any clue about what's going on. Any kind of help would be
>>> appreciated, thank you!
>>>
>>> Lothar
>>
>> On 5/9/19 2:46 AM, Lothar Schilling wrote:> Am 09.05.2019 um 11:14
>> schrieb Jonas Smedegaard:
>>>> Is _only_ transfer speed affected?
>>>>
>>>> I am no expert in this, but imagine that if you rsync massive amounts
>>>> involving hardlinks then memory becomes a problem too.
>>>>
>>>> Perhaps run atop to monitor bottlenecks live
>>>>
>>>>
>>>>    - Jonas
>>>>
>>> It is most definitely not a memory or cpu problem. It's a HP Proliant
>>> with 32 GB RAM and a Intel Xeon CPU 2.40GHz with 4 cores. Also the
>>> problem stays the same if I just copy one large file.
>>
>> On 5/9/19 3:26 AM, Lothar Schilling wrote:> Am 09.05.2019 um 11:51
>> schrieb Kevin DAGNEAUX:
>>>> Check if it's related to the disk speed (hdparm and/or iotop).
>>>>
>>>> Kevin
>>>>
>>> hdparm -tT /dev/sda
>>> /dev/sda:
>>>    Timing cached reads:   13348 MB in  2.00 seconds = 6683.42 MB/sec
>>>    Timing buffered disk reads: 1014 MB in  3.00 seconds = 337.72 MB/sec
>>>
>>> iotop -o (for rsync and cp)
>>> Total DISK READ :       0.00 B/s | Total DISK WRITE :     476.15 K/s
>>> Actual DISK READ:       0.00 B/s | Actual DISK WRITE:     487.86 K/s
>>>     TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
>>> 19531 be/4 root        0.00 B/s  476.15 K/s  0.00 % 99.24 % rsync
>>> --info=progress2 /daten/testfile /daten/testfile2
>>>
>>> iotop -o (for dd)
>>> Total DISK READ :       0.00 B/s | Total DISK WRITE :     297.68 M/s
>>> Actual DISK READ:       0.00 B/s | Actual DISK WRITE:     297.68 M/s
>>>     TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
>>> 19557 be/4 root        0.00 B/s  297.68 M/s  0.00 % 99.99 % dd
>>> if=/dev/zero of=/daten/testfile bs=1G count=10 oflag=direct
>>
>> On 5/9/19 5:43 AM, Lothar Schilling wrote:> Am 09.05.2019 um 13:27
>> schrieb Martin:
>>>> [..]
>>>>> hdparm -tT /dev/sda
>>>>> /dev/sda:
>>>>>    Timing cached reads:   13348 MB in  2.00 seconds = 6683.42 MB/sec
>>>>>    Timing buffered disk reads: 1014 MB in  3.00 seconds = 337.72
>> MB/sec
>>>>> iotop -o (for rsync and cp)
>>>>> Total DISK READ :       0.00 B/s | Total DISK WRITE :     476.15 K/s
>>>>> Actual DISK READ:       0.00 B/s | Actual DISK WRITE:     487.86 K/s
>>>>>     TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>   
>> COMMAND
>>>>> 19531 be/4 root        0.00 B/s  476.15 K/s  0.00 % 99.24 % rsync
>>>>> --info=progress2 /daten/testfile /daten/testfile2
>>>>>
>>>>> iotop -o (for dd)
>>>>> Total DISK READ :       0.00 B/s | Total DISK WRITE :     297.68 M/s
>>>>> Actual DISK READ:       0.00 B/s | Actual DISK WRITE:     297.68 M/s
>>>>>     TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>   
>> COMMAND
>>>>> 19557 be/4 root        0.00 B/s  297.68 M/s  0.00 % 99.99 % dd
>>>>> if=/dev/zero of=/daten/testfile bs=1G count=10 oflag=direct
>>>> Show us the 'dd if=/daten/testfile bs=1G oflag=direct
>> of=/dev/null', please.
>>>> If this is as slow as this ~480k/s above, check your disk's health
>> status. Like with smartmontools or some disk-utility software.
>>>> Martin
>>>>
>>> Fast enough...
>>>
>>> dd if=/daten/testfile bs=1G oflag=direct of=/daten/testfile2
>>> 10+0 Datensätze ein
>>> 10+0 Datensätze aus
>>> 10737418240 Bytes (11 GB, 10 GiB) kopiert, 72,7297 s, 148 MB/s
>>>
>>> dd if=/daten/testfile of=/dev/null
>>> 20971520+0 Datensätze ein
>>> 20971520+0 Datensätze aus
>>> 10737418240 Bytes (11 GB, 10 GiB) kopiert, 36,6887 s, 293 MB/s
>> Please reply with your *complete* console session (you can remove all
>> of the above in your reply).
>>
>> David
>>

# cat /etc/debian_version
9.9

# uname -a
Linux [my.server.com] 4.9.0-9-686-pae #1 SMP Debian 4.9.168-1
(2019-04-12) i686 GNU/Linux

# lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0   3,7T  0 disk
├─sda1   8:1    0   476M  0 part /boot
├─sda2   8:2    0  93,1G  0 part /
├─sda3   8:3    0  18,6G  0 part [SWAP]
└─sda4   8:4    0 931,3G  0 part /daten

# mount | egrep 'sd[a-z]'
/dev/sda2 on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/sda4 on /daten type ext4 (rw,relatime,data=ordered)
/dev/sda1 on /boot type ext4 (rw,relatime,data=ordered)

# df | egrep 'sd[a-z]'
/dev/sda2       95596964  2241952  88455840    3% /
/dev/sda4      960185376 41249236 870091648    5% /daten
/dev/sda1         463826    57063    378296   14% /boot

# time dd if=/dev/urandom of=test bs=1M count=100 conv=fsync
100+0 Datensätze ein
100+0 Datensätze aus
104857600 Bytes (105 MB, 100 MiB) kopiert, 192,778 s, 544 kB/s
real    3m12,781s
user    0m0,000s
sys     0m1,480s

# echo 3 > /proc/sys/vm/drop_caches && time dd if=test of=/dev/null
204800+0 Datensätze ein
204800+0 Datensätze aus
104857600 Bytes (105 MB, 100 MiB) kopiert, 0,600354 s, 175 MB/s
real    0m0,675s
user    0m0,048s
sys     0m0,340s

# echo 3 > /proc/sys/vm/drop_caches && f=test &&
uh=root@[my.other.server.com] && p=/tmp/test && ssh $uh rm -f $p/$g &&
rsync -a --stats $f $uh:$p
Number of files: 1 (reg: 1)
Number of created files: 1 (reg: 1)
Number of regular files transferred: 1
Total file size: 104,857,600 bytes
Total transferred file size: 104,857,600 bytes
Literal data: 104,857,600 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.021 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 104,883,277
Total bytes received: 34
sent 104,883,277 bytes  received 34 bytes  13,984,441.47 bytes/sec
total size is 104,857,600  speedup is 1.00

# echo 3 > /proc/sys/vm/drop_caches && f=test && p=/tmp && rsync -a
--stats $f $p
Number of files: 1 (reg: 1)
Number of created files: 1 (reg: 1)
Number of deleted files: 0
Number of regular files transferred: 1
Total file size: 104,857,600 bytes
Total transferred file size: 104,857,600 bytes
Literal data: 104,857,600 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.021 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 104,883,283
Total bytes received: 35
sent 104,883,283 bytes  received 35 bytes  629,929.84 bytes/sec
total size is 104,857,600  speedup is 1.00

[toc] | [prev] | [next] | [standalone]

#208611

FromLothar Schilling <ls@proasyl.de>
Date2019-05-13 11:00 +0200
Message-ID<xXeYp-F8-3@gated-at.bofh.it>
In reply to#208610
Am 13.05.2019 um 10:51 schrieb Tixy:
> On Mon, 2019-05-13 at 10:30 +0200, Lothar Schilling wrote:
> [...]
>> # uname -a
>> Linux [my.server.com] 4.9.0-9-686-pae #1 SMP Debian 4.9.168-1
>> (2019-04-12) i686 GNU/Linux
> So you're running a 32-bit system, not 64-bit. Is that because you're
> running on very old hardware or a decision for some other reason?
>
Neither old hardware nor deliberate decision - just plain stupid. But
would it really matter?

[toc] | [prev] | [next] | [standalone]

#208613

FromTixy <tixy@yxit.co.uk>
Date2019-05-13 11:20 +0200
Message-ID<xXfhL-10J-1@gated-at.bofh.it>
In reply to#208611
On Mon, 2019-05-13 at 10:58 +0200, Lothar Schilling wrote:
> Am 13.05.2019 um 10:51 schrieb Tixy:
> > On Mon, 2019-05-13 at 10:30 +0200, Lothar Schilling wrote:
> > [...]
> > > # uname -a
> > > Linux [my.server.com] 4.9.0-9-686-pae #1 SMP Debian 4.9.168-1
> > > (2019-04-12) i686 GNU/Linux
> > 
> > So you're running a 32-bit system, not 64-bit. Is that because
> > you're
> > running on very old hardware or a decision for some other reason?
> > 
> 
> Neither old hardware nor deliberate decision - just plain stupid. But
> would it really matter?

I don't know if it would, just thought I'd point it out in case it was
an accidental decision.

-- 
Tixy

[toc] | [prev] | [next] | [standalone]

#208615

FromCurt <curty@free.fr>
Date2019-05-13 11:40 +0200
Message-ID<xXfB7-17t-3@gated-at.bofh.it>
In reply to#208611
On 2019-05-13, Lothar Schilling <ls@proasyl.de> wrote:
> Am 13.05.2019 um 10:51 schrieb Tixy:
>> On Mon, 2019-05-13 at 10:30 +0200, Lothar Schilling wrote:
>> [...]
>>> # uname -a
>>> Linux [my.server.com] 4.9.0-9-686-pae #1 SMP Debian 4.9.168-1
>>> (2019-04-12) i686 GNU/Linux
>> So you're running a 32-bit system, not 64-bit. Is that because you're
>> running on very old hardware or a decision for some other reason?
>>
> Neither old hardware nor deliberate decision - just plain stupid. But
> would it really matter?
>

I'd give it a shot with 4 cores and 32 GB of ram. Why you wouldn't and
haven't is one of those mysteries that'll probably remain
unresolved in my lifetime.



-- 
“When he was dry, he believed it was alcohol he needed, but when he had a few
drinks in him, he knew it was something else, possibly a woman; and when he had
it all — cash, booze, and a wife — he couldn’t be distracted from the great
emptiness that was always falling through him and never hit the ground.” – Denis Johnson

[toc] | [prev] | [next] | [standalone]

#208633

FromMichael Stone <mstone@debian.org>
Date2019-05-14 13:40 +0200
Message-ID<xXDWO-7Sw-1@gated-at.bofh.it>
In reply to#208611
On Mon, May 13, 2019 at 10:58:45AM +0200, Lothar Schilling wrote:
>Am 13.05.2019 um 10:51 schrieb Tixy:
>> On Mon, 2019-05-13 at 10:30 +0200, Lothar Schilling wrote:
>> [...]
>>> # uname -a
>>> Linux [my.server.com] 4.9.0-9-686-pae #1 SMP Debian 4.9.168-1
>>> (2019-04-12) i686 GNU/Linux
>> So you're running a 32-bit system, not 64-bit. Is that because you're
>> running on very old hardware or a decision for some other reason?
>>
>Neither old hardware nor deliberate decision - just plain stupid. But
>would it really matter?

It can--when using that much RAM on a 32 bit kernel you need to use PAE, 
which adds significant overhead for certain operations vs a 64 bit 
kernel--especially if it's old enough that meltdown mitigations 
significantly impact context switching performance. "Intel Xeon CPU 
2.40GHz with 4 cores" applies to chips a decade old by now; what 
is the actual CPU model?

[toc] | [prev] | [next] | [standalone]

#208612

FromTixy <tixy@yxit.co.uk>
Date2019-05-13 11:00 +0200
Message-ID<xXeYp-F8-1@gated-at.bofh.it>
In reply to#208610
On Mon, 2019-05-13 at 10:30 +0200, Lothar Schilling wrote:
[...]
> # uname -a
> Linux [my.server.com] 4.9.0-9-686-pae #1 SMP Debian 4.9.168-1
> (2019-04-12) i686 GNU/Linux

So you're running a 32-bit system, not 64-bit. Is that because you're
running on very old hardware or a decision for some other reason?

-- 
Tixy

[toc] | [prev] | [next] | [standalone]

#208627

FromDavid Christensen <dpchrist@holgerdanske.com>
Date2019-05-14 02:30 +0200
Message-ID<xXtuq-1k3-1@gated-at.bofh.it>
In reply to#208610
On 5/13/19 1:30 AM, Lothar Schilling wrote:
> # cat /etc/debian_version
> 9.9

Okay.


> # uname -a
> Linux [my.server.com] 4.9.0-9-686-pae #1 SMP Debian 4.9.168-1
> (2019-04-12) i686 GNU/Linux

As other readers have noted, you are running 32-bit Debian GNU/Linux. 
It should not matter for what we're doing, but...


> # lsblk
> NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
> sda      8:0    0   3,7T  0 disk
> ├─sda1   8:1    0   476M  0 part /boot
> ├─sda2   8:2    0  93,1G  0 part /
> ├─sda3   8:3    0  18,6G  0 part [SWAP]
> └─sda4   8:4    0 931,3G  0 part /daten

So, your boot, root, swap, and data are all on a 4 TB drive.  I put my 
boot, swap, and root on a small "system" disk (SSD or USB flash drive) 
and my bulk data on large "data" disks (HDD; preferably RAID).  Again, 
it should not matter; but...


> # mount | egrep 'sd[a-z]'
> /dev/sda2 on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
> /dev/sda4 on /daten type ext4 (rw,relatime,data=ordered)
> /dev/sda1 on /boot type ext4 (rw,relatime,data=ordered)

Okay.


> # df | egrep 'sd[a-z]'
> /dev/sda2       95596964  2241952  88455840    3% /
> /dev/sda4      960185376 41249236 870091648    5% /daten
> /dev/sda1         463826    57063    378296   14% /boot

Okay.


> # time dd if=/dev/urandom of=test bs=1M count=100 conv=fsync
> 100+0 Datensätze ein
> 100+0 Datensätze aus
> 104857600 Bytes (105 MB, 100 MiB) kopiert, 192,778 s, 544 kB/s
> real    3m12,781s
> user    0m0,000s
> sys     0m1,480s

As you have redacted your prompt, I must assume your current working 
directory was within a file system on /dev/sda?


Write performance is bad.  But, now that you have a good test case for 
trouble-shooting.  :-)


> # echo 3 > /proc/sys/vm/drop_caches && time dd if=test of=/dev/null
> 204800+0 Datensätze ein
> 204800+0 Datensätze aus
> 104857600 Bytes (105 MB, 100 MiB) kopiert, 0,600354 s, 175 MB/s
> real    0m0,675s
> user    0m0,048s
> sys     0m0,340s

Read performance is questionable:

1.  The file may be small enough to fit entirely within the drive cache 
(e.g. 128 MB drive cache).  If so, I would have expected interface speed 
(e.g. ~600 MB/s for SATA 3).

2.  You forgot to specify 1 megabyte blocks 'bs=1M'.  dd defaults to 512 
byte blocks.  The drive probably uses 4 kB blocks.  Small blocks reduce 
performance for large files.

3.  Until we figure out the write problem(s), everything else is suspect.


Consider getting a power supply tester and checking your power supply. 
Computers behave very strangely when one power supply rail goes out but 
the others keep working.  I have an Antec, but I believe it is out of 
production:

https://www.amazon.com/Antec-ATX12V-Power-Supply-Tester/dp/B000BPNWWW


Download a bootable memory diagnostic tool, write it to media, boot it, 
and run it for at least an hour.  Some people have criticized the tool I 
use, so I won't make any recommendations.  If the tool finds memory 
problems, re-seat your memory modules and try again.


Download the diagnostic toolset from your disk drive manufacturer's web 
site, write it to media/ install it (Windows may be required), and run 
all the non-destructive tests.  If there are problems:

* Re-seat the drive power cable and try again.

* Re-seat both ends of the drive data cable and try again.

* Substitute another drive data cable and try again.

* Use a different drive port on your motherboard or interface card and 
try again.


Please reply with your findings.


David

[toc] | [prev] | [next] | [standalone]

#208629

FromLothar Schilling <ls@proasyl.de>
Date2019-05-14 09:40 +0200
Message-ID<xXAcy-5Dy-3@gated-at.bofh.it>
In reply to#208627
Thanks for your help, David. But to boot the machine from a live CD I
will have to pay a visit to the computing centre in Frankfurt anyway. So
instead of running a diagnostic tool I am going to install the system
from scratch, this time using a 64bit version.

[toc] | [prev] | [next] | [standalone]

#208644

FromDavid Christensen <dpchrist@holgerdanske.com>
Date2019-05-14 18:30 +0200
Message-ID<xXItr-2jk-1@gated-at.bofh.it>
In reply to#208629
On 5/14/19 12:34 AM, Lothar Schilling wrote:
> Thanks for your help, David. But to boot the machine from a live CD I
> will have to pay a visit to the computing centre in Frankfurt anyway. So
> instead of running a diagnostic tool I am going to install the system
> from scratch, this time using a 64bit version.

Okay.  Backup the data, backup the system configuration, and take an 
image of the system drive before starting.  I would test the hardware, 
just to be sure.  Again, I would recommend installing a small SSD as the 
system drive and using the HDD for data.  Take an image of the system 
drive after re-installing, but before first boot.  Configure the system. 
  Restore data.  Validate services, including backups of this machine. 
Take another image of the system drive before leaving.


David

[toc] | [prev] | [next] | [standalone]

#208632

FromMichael Stone <mstone@debian.org>
Date2019-05-14 13:30 +0200
Message-ID<xXDN8-7Py-9@gated-at.bofh.it>
In reply to#208494
On Thu, May 09, 2019 at 10:48:31PM -0700, David Christensen wrote:
>2019-05-09 22:00:27 root@po /mnt/scratch
># time dd if=/dev/urandom of=foo bs=1M count=1K conv=fsync

don't bother doing this, urandom will be the bottleneck and it will just 
confuse things

[toc] | [prev] | [next] | [standalone]

#208652

FromDavid Christensen <dpchrist@holgerdanske.com>
Date2019-05-14 21:20 +0200
Message-ID<xXL7X-42i-3@gated-at.bofh.it>
In reply to#208632
On 5/14/19 4:23 AM, Michael Stone wrote:
> On Thu, May 09, 2019 at 10:48:31PM -0700, David Christensen wrote:
>> 2019-05-09 22:00:27 root@po /mnt/scratch
>> # time dd if=/dev/urandom of=foo bs=1M count=1K conv=fsync
> 
> don't bother doing this, urandom will be the bottleneck and it will just 
> confuse things

That was intended more as a litmus test, rather than a benchmark.  I 
chose /dev/urandom, rather than /dev/zero, to preclude the possibility 
of caches or compression.


The test succeeded in finding a disk write problem:

On 5/13/19 1:30 AM, Lothar Schilling wrote:
 > # time dd if=/dev/urandom of=test bs=1M count=100 conv=fsync
 > 100+0 Datensätze ein
 > 100+0 Datensätze aus
 > 104857600 Bytes (105 MB, 100 MiB) kopiert, 192,778 s, 544 kB/s
 > real    3m12,781s
 > user    0m0,000s
 > sys     0m1,480s


David

[toc] | [prev] | [next] | [standalone]

#208653

FromMichael Stone <mstone@debian.org>
Date2019-05-14 22:10 +0200
Message-ID<xXLUm-4yh-5@gated-at.bofh.it>
In reply to#208652
On Tue, May 14, 2019 at 12:11:17PM -0700, David Christensen wrote:
>On 5/14/19 4:23 AM, Michael Stone wrote:
>>On Thu, May 09, 2019 at 10:48:31PM -0700, David Christensen wrote:
>>>2019-05-09 22:00:27 root@po /mnt/scratch
>>># time dd if=/dev/urandom of=foo bs=1M count=1K conv=fsync
>>
>>don't bother doing this, urandom will be the bottleneck and it will 
>>just confuse things
>
>That was intended more as a litmus test, rather than a benchmark.  I 
>chose /dev/urandom, rather than /dev/zero, to preclude the possibility 
>of caches or compression.

Just use /dev/zero, caching is exactly as relevant to either, and 
accidently compressing a disk isn't a likely scenario.

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | linux.debian.user

csiph-web