Re: Could RAM possibly be just 3-4 times faster than bare hdd writes and reads? or, is the Linux kernel doing its 'magic' in the bg? or, ...

From	David Christensen <dpchrist@holgerdanske.com>
Newsgroups	linux.debian.user
Subject	Re: Could RAM possibly be just 3-4 times faster than bare hdd writes and reads? or, is the Linux kernel doing its 'magic' in the bg? or, ...
Date	2020-06-17 21:20 +0200
Message-ID	<AiLLj-4TH-1@gated-at.bofh.it> (permalink)
References	<AiDkK-8kt-1@gated-at.bofh.it>
Organization	linux.* mail to news gateway

Show all headers | View raw

On 2020-06-17 03:14, Albretch Mueller wrote:
>   HDDs have their internal caching mechanism and I have heard that the
> Linux kernel uses RAM very effitiently, but to my understanding RAM
> being only 3-4 times faster doesn't make much sense, so I may be doing
> or understanding something not entirely right.
> 
>   does dd actually hit the bare metal drive or is it just reaching the
> disks cache
> 
>   This is what I am consistently getting from my code doing intesive IO
> on the RAM drive:
> 
> // __ write speed test
> #
> # time dd if=/dev/zero of="${_RAM_MNT}"/zero bs=4k count=100000
> 100000+0 DatensÃ¤tze ein
> 100000+0 DatensÃ¤tze aus
> 409600000 Bytes (410 MB, 391 MiB) kopiert, 0,756943 s, 541 MB/s
> 
> real    0m0,760s
> user    0m0,048s
> sys     0m0,680s
> #
> 
> // __ read speed test
> #
> # time dd if="${_RAM_MNT}"/zero of=/dev/null bs=4k count=100000
> 100000+0 DatensÃ¤tze ein
> 100000+0 DatensÃ¤tze aus
> 409600000 Bytes (410 MB, 391 MiB) kopiert, 0,36381 s, 1,1 GB/s
> 
> real    0m0,368s
> user    0m0,016s
> sys     0m0,344s
> #
> 
> // __ my code reports while using a ramdisk:
> 
> ...
> // __ |0| files not found or empty!
> // __ |8616768| bytes in |48| files offset processed (weighted and
> checked) in |720057| (ms), |11| (bytes/ms)
> 
> real    12m0,396s
> user    9m12,596s
> sys     3m11,524s

For background information on asynchronous input/ output, see:

https://en.wikipedia.org/wiki/Asynchronous_I/O

As others have mentioned:

1.  dd(1) with an option of 'bs=4k' will give lower performance that 
dd(1) with a larger blocksize.  (I believe the Linux kernel uses 128 kiB 
blocks for I/O; this value should work well on any Linux computer.  But, 
I use 'bs=1M' to make calculations easier.)

2.  AIUI dd(1) uses asynchronous (buffered) I/O unless told otherwise. 
If the total amount of data written fits into RAM, dd(1) with 
asynchronous writes will give higher performance that dd(1) with 
synchronous writes.  If the total amount of data written is much larger 
than RAM, the differences are less pronounced.

You seem to be comparing apples and oranges:

1.  The dash(1) and bash(1) built-in 'time' reports "accumulated user 
and system times for the shell and for processes run from the shell".  I 
tend to focus on the first value ("wall clock time").

2. dd(1) reports the number of full and partial records in, the number 
of full and partial records out, bytes copied, Megabytes copied, 
Mibabytes copied, copy time, and throughput.

3.  The statistics reported by your code are different values, units, 
and/or format.  This makes it hard to make comparisons against the dd(1) 
statistics.  I suggest that you rework your code to collect and output 
statistics in the same format as dd(1).

4.  It is unclear if your dd(1) invocations are representative of the 
I/O performed by your code.  I suggest that you match block size, block 
count, and I/O mode for both dd(1) and your code.

<snip>

> $ _HDD_DIR="/home/lbrtchx/cmllpz/prjx/kd/java/IO/ffst_bytes/logs/ffst_diffs_files"
> $
> $ time dd if=/dev/zero of="${_HDD_DIR}"/zero bs=4k count=100000
> 100000+0 DatensÃ¤tze ein
> 100000+0 DatensÃ¤tze aus
> 409600000 Bytes (410 MB, 391 MiB) kopiert, 1,92086 s, 213 MB/s
> 
> real    0m1,924s
> user    0m0,064s
> sys     0m1,468s

<snip>

> $ time dd if="${_HDD_DIR}"/zero of=/dev/null bs=4k count=100000
> 100000+0 DatensÃ¤tze ein
> 100000+0 DatensÃ¤tze aus
> 409600000 Bytes (410 MB, 391 MiB) kopiert, 0,521688 s, 785 MB/s
> 
> real    0m0,525s
> user    0m0,040s
> sys     0m0,308s

<snip>

> # time dd if=/dev/zero of="${_HDD_DIR}"/zero bs=4k count=100000
> 100000+0 DatensÃ¤tze ein
> 100000+0 DatensÃ¤tze aus
> 409600000 Bytes (410 MB, 391 MiB) kopiert, 1,94642 s, 210 MB/s
> 
> real    0m1,954s
> user    0m0,048s
> sys     0m1,484s
> #

<snip>

> # time dd if="${_HDD_DIR}"/zero of=/dev/null bs=4k count=100000
> 100000+0 DatensÃ¤tze ein
> 100000+0 DatensÃ¤tze aus
> 409600000 Bytes (410 MB, 391 MiB) kopiert, 0,735103 s, 557 MB/s
> 
> real    0m0,744s
> user    0m0,036s
> sys     0m0,324s
> #
> 
> // __ my code reports while using a hdd access:
> 
> ...
> 
> // __ |0| files not found or empty!
> // __ |8616768| bytes in |48| files offset processed (weighted and
> checked) in |2412518| (ms), |3| (bytes/ms)
> 
> real	40m12,727s
> user	10m42,576s
> sys	8m41,180s

Again, apples and oranges.  A minor point -- use one account for 
testing, not one user account and the root account.

Improving the efficiency and performance of software is both art and 
science.  I suggest that you look for a profiling tools for whatever 
programming language/ environment you are using (Java?).  Such tools can 
help you identify bottlenecks, evaluate alternatives, and make informed 
decisions.

David

Thread

csiph-web