Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #267276

Re: Unidentified subject!

From David Christensen <dpchrist@holgerdanske.com>
Newsgroups linux.debian.user
Subject Re: Unidentified subject!
Date 2024-02-11 11:10 +0100
Message-ID <I6eWC-9plk-7@gated-at.bofh.it> (permalink)
References <I656W-9jhs-11@gated-at.bofh.it> <I6dea-9oi8-3@gated-at.bofh.it>
Organization linux.* mail to news gateway

Show all headers | View raw

On 2/11/24 00:11, Thomas Schmitt wrote:
> Hi,
> 
> David Christensen wrote:
>> $ time dd if=/dev/urandom bs=8K count=128K | wc -c
>> [...]
>> 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.30652 s, 249 MB/s
> 
> This looks good enough for practical use on spinning rust and slow SSD.


Yes.


> Maybe the "wc" pipe slows it down ?
> ... not much on 4 GHz Xeon with Debian 11:
> 
>    $ time dd if=/dev/urandom bs=8K count=128K | wc -c
>    ...
>    1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.13074 s, 260 MB/s
>    $ time dd if=/dev/urandom bs=8K count=128K of=/dev/null
>    ...
>    1073741824 bytes (1.1 GB, 1.0 GiB) copied, 3.95569 s, 271 MB/s


My CPU has a Max Turbo Frequency of 3.3 GHz.  I would expect a 4 GHz 
processor to be ~21% faster, but apparently not.


Baseline with pipeline, wc(1), and bs=8K due to unknown Bash pipeline 
bottleneck (?):

2024-02-11 01:18:33 dpchrist@laalaa ~
$ dd if=/dev/urandom bs=8K count=128K | wc -c
1073741824
131072+0 records in
131072+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.27283 s, 251 MB/s


Eliminate pipeline and wc(1):

2024-02-11 01:18:44 dpchrist@laalaa ~
$ dd if=/dev/urandom of=/dev/null bs=8K count=128K
131072+0 records in
131072+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 3.75946 s, 286 MB/s


Increase block size:

2024-02-11 01:18:51 dpchrist@laalaa ~
$ dd if=/dev/urandom of=/dev/null bs=1M count=1K
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 3.62874 s, 296 MB/s


Concurrency:

threads	throughput
1	296 MB/s
2	285+286=571 MB/s
3	271+264+266=801 MB/s
4	249+250+241+262=1,002 MB/s
5	225+214+210+224+225=1,098 MB/s
6	223+199+199+204+213+205=1,243 MB/s
7	191+209+210+204+213+201+197=1,425 MB/s
8	205+198+180+195+205+184+184+189=1,540 MB/s


> Last time i tested /dev/urandom it was much slower on comparable machines
> and also became slower as the amount grew.


Did you figure out why the Linux random number subsystem slowed, and at 
what amount?


> Therefore i still have my amateur RNG which works with a little bit of
> MD5 and a lot of EXOR. It produces about 1100 MiB/s on the 4 GHz machine.
> No cryptographical strength, but chaotic enough to avoid any systematic
> pattern which could be recognized by a cheater and represented with
> some high compression factor.
> The original purpose was to avoid any systematic interference with the
> encoding of data blocks on optical media.
> 
> I am sure there are faster RNGs around with better random quality.


I assume the Linux kernel in Debian 11 is new enough to support RDRAND (?):

https://en.wikipedia.org/wiki/RdRand


But, my processor is too old to have Secure Key.


>> $ time perl -MMath::Random::ISAAC::XS -e '$i=Math::Random::ISAAC::XS->new(12345678); print pack 'L',$i->irand while 1' | dd bs=8K count=128K | wc -c
>> 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 82.6523 s, 13.0 MB/s
> 
> Now that's merely sufficient for shredding the content of a BD-RE medium
> or a slow USB stick.


Okay.


>> I suggest using /dev/urandom and tee(1) to write the same CSPRN
>> stream to all of the devices and using cmp(1) to validate.
> 
> I'd propose to use a checksummer like md5sum or sha256sum instead of cmp:
> 
>   $random_generator | tee $target | $checksummer
>   dd if=$target bs=... count=... | $checksummer
> 
> This way one can use unreproducible random streams and does not have to
> store the whole stream on a reliable device for comparison.


TIMTOWTDI.  :-)


David

Back to linux.debian.user | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Unidentified subject! gene heskett <gheskett@shentel.net> - 2024-02-07 21:40 +0100
  Re: Unidentified subject! Stefan Monnier <monnier@iro.umontreal.ca> - 2024-02-08 05:30 +0100
    Re: Unidentified subject! gene heskett <gheskett@shentel.net> - 2024-02-10 11:00 +0100
      Re: Unidentified subject! "Thomas Schmitt" <scdbackup@gmx.net> - 2024-02-10 11:40 +0100
        shred bug? [was: Unidentified subject!] <tomas@tuxteam.de> - 2024-02-10 13:50 +0100
          Re: shred bug? [was: Unidentified subject!] tomas@tuxteam.de - 2024-02-10 14:00 +0100
            Re: shred bug? "Thomas Schmitt" <scdbackup@gmx.net> - 2024-02-10 15:00 +0100
              Re: shred bug? <tomas@tuxteam.de> - 2024-02-10 15:40 +0100
          Re: shred bug? Gremlin <scott-andrews@columbus.rr.com> - 2024-02-10 14:40 +0100
          Re: shred bug? "Thomas Schmitt" <scdbackup@gmx.net> - 2024-02-10 14:40 +0100
          Re: shred bug? [was: Unidentified subject!] David Christensen <dpchrist@holgerdanske.com> - 2024-02-11 01:10 +0100
            Re: shred bug? [was: Unidentified subject!] Greg Wooledge <greg@wooledge.org> - 2024-02-11 01:20 +0100
              Re: shred bug? [was: Unidentified subject!] David Christensen <dpchrist@holgerdanske.com> - 2024-02-11 01:30 +0100
                Re: shred bug? [was: Unidentified subject!] debian-user@howorth.org.uk - 2024-02-11 14:00 +0100
                Re: shred bug? "Thomas Schmitt" <scdbackup@gmx.net> - 2024-02-11 14:30 +0100
              Re: shred bug? [was: Unidentified subject!] <tomas@tuxteam.de> - 2024-02-11 08:10 +0100
                Re: shred bug? [was: Unidentified subject!] Greg Wooledge <greg@wooledge.org> - 2024-02-11 15:40 +0100
                Re: shred bug? [was: Unidentified subject!] <tomas@tuxteam.de> - 2024-02-11 15:50 +0100
                Re: shred bug? [was: Unidentified subject!] Greg Wooledge <greg@wooledge.org> - 2024-02-11 16:00 +0100
                Re: shred bug? [was: Unidentified subject!] <tomas@tuxteam.de> - 2024-02-11 16:00 +0100
                Re: shred bug? [was: Unidentified subject!] Greg Wooledge <greg@wooledge.org> - 2024-02-12 18:00 +0100
                Re: shred bug? [was: Unidentified subject!] Curt <curty@free.fr> - 2024-02-12 18:00 +0100
                Re: shred bug? [was: Unidentified subject!] David Christensen <dpchrist@holgerdanske.com> - 2024-02-12 22:00 +0100
                Re: shred bug? [was: Unidentified subject!] "Thomas Schmitt" <scdbackup@gmx.net> - 2024-02-12 22:10 +0100
                Re: shred bug? [was: Unidentified subject!] <tomas@tuxteam.de> - 2024-02-13 06:50 +0100
                Re: shred bug? [was: Unidentified subject!] David Wright <deblis@lionunicorn.co.uk> - 2024-02-11 16:30 +0100
                Re: shred bug? [was: Unidentified subject!] Greg Wooledge <greg@wooledge.org> - 2024-02-13 13:20 +0100
                Re: shred bug? [was: Unidentified subject!] Greg Wooledge <greg@wooledge.org> - 2024-02-13 13:40 +0100
                Re: shred bug? [was: Unidentified subject!] <tomas@tuxteam.de> - 2024-02-13 13:50 +0100
                Re: shred bug? [was: Unidentified subject!] "Thomas Schmitt" <scdbackup@gmx.net> - 2024-02-13 14:00 +0100
                Re: shred bug? [was: Unidentified subject!] David Wright <deblis@lionunicorn.co.uk> - 2024-02-13 16:40 +0100
                Re: shred bug? [was: Unidentified subject!] Greg Wooledge <greg@wooledge.org> - 2024-02-13 17:30 +0100
                Re: shred bug? [was: Unidentified subject!] debian-user@howorth.org.uk - 2024-02-13 18:50 +0100
                Re: shred bug? [was: Unidentified subject!] David Christensen <dpchrist@holgerdanske.com> - 2024-02-13 22:10 +0100
                Re: shred bug? [was: Unidentified subject!] <tomas@tuxteam.de> - 2024-02-14 06:50 +0100
                Re: shred bug? [was: Unidentified subject!] "Thomas Schmitt" <scdbackup@gmx.net> - 2024-02-13 19:00 +0100
                Re: shred bug? [was: Unidentified subject!] Greg Wooledge <greg@wooledge.org> - 2024-02-13 20:00 +0100
                Re: shred bug? [was: Unidentified subject!] "Thomas Schmitt" <scdbackup@gmx.net> - 2024-02-13 20:40 +0100
                Re: shred bug? [was: Unidentified subject!] gene heskett <gheskett@shentel.net> - 2024-02-13 20:40 +0100
                Re: shred bug? [was: Unidentified subject!] "Thomas Schmitt" <scdbackup@gmx.net> - 2024-02-13 20:50 +0100
                Re: shred bug? [was: Unidentified subject!] gene heskett <gheskett@shentel.net> - 2024-02-13 21:30 +0100
                Re: shred bug? [was: Unidentified subject!] David Christensen <dpchrist@holgerdanske.com> - 2024-02-13 22:10 +0100
                Re: shred bug? [was: Unidentified subject!] gene heskett <gheskett@shentel.net> - 2024-02-13 22:50 +0100
                Re: shred bug? [was: Unidentified subject!] David Wright <deblis@lionunicorn.co.uk> - 2024-02-15 06:40 +0100
                Re: shred bug? [was: Unidentified subject!] David Christensen <dpchrist@holgerdanske.com> - 2024-02-11 23:50 +0100
                Re: shred bug? [was: Unidentified subject!] Max Nikulin <manikulin@gmail.com> - 2024-02-13 03:40 +0100
                Re: shred bug? [was: Unidentified subject!] "Thomas Schmitt" <scdbackup@gmx.net> - 2024-02-11 16:30 +0100
                Re: shred bug? [was: Unidentified subject!] Michael Stone <mstone@debian.org> - 2024-02-16 16:00 +0100
        Re: Unidentified subject! gene heskett <gheskett@shentel.net> - 2024-02-10 18:50 +0100
          Re: Unidentified subject! "Thomas Schmitt" <scdbackup@gmx.net> - 2024-02-10 19:40 +0100
            Re: Unidentified subject! gene heskett <gheskett@shentel.net> - 2024-02-10 20:30 +0100
            Re: Unidentified subject! David Christensen <dpchrist@holgerdanske.com> - 2024-02-11 00:50 +0100
              Re: Unidentified subject! "Thomas Schmitt" <scdbackup@gmx.net> - 2024-02-11 09:10 +0100
                Re: Unidentified subject! David Christensen <dpchrist@holgerdanske.com> - 2024-02-11 21:50 +0100
        Re: Unidentified subject! David Christensen <dpchrist@holgerdanske.com> - 2024-02-11 00:40 +0100
          Re: Unidentified subject! "Thomas Schmitt" <scdbackup@gmx.net> - 2024-02-11 09:20 +0100
            Re: Unidentified subject! David Christensen <dpchrist@holgerdanske.com> - 2024-02-11 11:10 +0100
              Fast Random Data Generation (Was: Re: Unidentified subject!) Linux-Fan <Ma_Sys.ma@web.de> - 2024-02-11 11:30 +0100
                Re: Fast Random Data Generation "Thomas Schmitt" <scdbackup@gmx.net> - 2024-02-11 12:30 +0100
                Re: Fast Random Data Generation (Was: Re: Unidentified subject!) Gremlin <scott-andrews@columbus.rr.com> - 2024-02-11 13:20 +0100
                Re: Fast Random Data Generation (Was: Re: Unidentified subject!) David Christensen <dpchrist@holgerdanske.com> - 2024-02-12 07:20 +0100
                Re: Fast Random Data Generation (Was: Re: Unidentified subject!) Linux-Fan <Ma_Sys.ma@web.de> - 2024-02-12 17:40 +0100
                Re: Fast Random Data Generation (Was: Re: Unidentified subject!) David Christensen <dpchrist@holgerdanske.com> - 2024-02-12 22:30 +0100
                Re: Fast Random Data Generation (Was: Re: Unidentified subject!) Linux-Fan <Ma_Sys.ma@web.de> - 2024-02-13 18:40 +0100
                Re: Fast Random Data Generation (Was: Re: Unidentified subject!) Jeffrey Walton <noloader@gmail.com> - 2024-02-12 22:30 +0100
              Re: Unidentified subject! "Thomas Schmitt" <scdbackup@gmx.net> - 2024-02-11 12:20 +0100
                Re: Unidentified subject! Jeffrey Walton <noloader@gmail.com> - 2024-02-11 16:20 +0100
                Re: Unidentified subject! David Christensen <dpchrist@holgerdanske.com> - 2024-02-11 23:00 +0100
      Re: Unidentified subject! Stefan Monnier <monnier@iro.umontreal.ca> - 2024-02-10 15:40 +0100
  Things I don't touch with a 3.048m barge pole: USB storage (Was Re:  Unidentified subject!) Andy Smith <andy@strugglers.net> - 2024-02-08 16:40 +0100
    Re: Things I don't touch with a 3.048m barge pole: USB storage (Was  Re: Unidentified subject!) Gremlin <scott-andrews@columbus.rr.com> - 2024-02-08 17:20 +0100
      Re: Things I don't touch with a 3.048m barge pole: USB storage (Was  Re: Unidentified subject!) Andy Smith <andy@strugglers.net> - 2024-02-08 17:30 +0100
      Re: Things I don't touch with a 3.048m barge pole: USB storage  (WasRe: Unidentified subject!) gene heskett <gheskett@shentel.net> - 2024-02-09 09:50 +0100
    Re: Things I don't touch with a 3.048m barge pole: USB storage (Was  Re: Unidentified subject!) Ralph Aichinger <ra@h5.or.at> - 2024-02-08 18:00 +0100
      Re: Things I don't touch with a 3.048m barge pole: USB storage (Was  Re: Unidentified subject!) Jeffrey Walton <noloader@gmail.com> - 2024-02-08 20:30 +0100
        Re: Things I don't touch with a 3.048m barge pole: USB storage (Was  Re: Unidentified subject!) Andy Smith <andy@strugglers.net> - 2024-02-08 21:50 +0100
        Re: Things I don't touch with a 3.048m barge pole: USB storage (Was  Re: Unidentified subject!) Andy Smith <andy@strugglers.net> - 2024-02-08 21:50 +0100
          Re: Things I don't touch with a 3.048m barge pole: USB storage (Was  Re: Unidentified subject!) Gremlin <scott-andrews@columbus.rr.com> - 2024-02-08 22:00 +0100
            Re: Things I don't touch with a 3.048m barge pole: USB storage (Was  Re: Unidentified subject!) Andy Smith <andy@strugglers.net> - 2024-02-08 22:20 +0100
              Re: Things I don't touch with a 3.048m barge pole: USB storage (Was  Re: Unidentified subject!) Andy Smith <andy@strugglers.net> - 2024-02-08 22:30 +0100
                Re: Things I don't touch with a 3.048m barge pole: USB storage (Was  Re: Unidentified subject!) Gremlin <scott-andrews@columbus.rr.com> - 2024-02-08 23:10 +0100
              Re: Things I don't touch with a 3.048m barge pole: USB storage (Was  Re: Unidentified subject!) Gremlin <scott-andrews@columbus.rr.com> - 2024-02-08 22:30 +0100
          Re: Things I don't touch with a 3.048m barge pole: USB storage  (WasRe: Unidentified subject!) gene heskett <gheskett@shentel.net> - 2024-02-09 14:00 +0100
            Re: Things I don't touch with a 3.048m barge pole "Thomas Schmitt" <scdbackup@gmx.net> - 2024-02-09 14:50 +0100
            Re: Things I don't touch with a 3.048m barge pole: USB storage  (WasRe: Unidentified subject!) David Christensen <dpchrist@holgerdanske.com> - 2024-02-10 07:00 +0100
              Re: Things I don't touch with a 3.048m barge pole: USB storage(WasRe:  Unidentified subject!) gene heskett <gheskett@shentel.net> - 2024-02-10 18:40 +0100
      Re: Things I don't touch with a 3.048m barge pole: USB storage (Was  Re: Unidentified subject!) Andy Smith <andy@strugglers.net> - 2024-02-08 21:40 +0100
        Re: Things I don't touch with a 3.048m barge pole: USB storage (Was  Re: Unidentified subject!) Arno Lehmann <al@its-lehmann.de> - 2024-02-09 09:50 +0100
    Re: Things I don't touch with a 3.048m barge pole: USB storage (Was  Re: Unidentified subject!) Max Nikulin <manikulin@gmail.com> - 2024-02-08 18:30 +0100
      Re: Things I don't touch with a 3.048m barge pole: USB storage (Was  Re: Unidentified subject!) Andy Smith <andy@strugglers.net> - 2024-02-08 21:40 +0100
        Re: Things I don't touch with a 3.048m barge pole: USB storage (Was  Re: Unidentified subject!) Gremlin <scott-andrews@columbus.rr.com> - 2024-02-08 22:10 +0100
          Re: Things I don't touch with a 3.048m barge pole: USB storage (Was  Re: Unidentified subject!) Andy Smith <andy@strugglers.net> - 2024-02-08 22:20 +0100
  Re: Unidentified subject! Richmond <dnomhcir@gmx.com> - 2024-02-08 23:50 +0100
    Re: Unidentified subject! Stefan Monnier <monnier@iro.umontreal.ca> - 2024-02-09 00:10 +0100
      Re: Unidentified subject! Charles Curley <charlescurley@charlescurley.com> - 2024-02-09 00:50 +0100
        Re: Unidentified subject! Richmond <dnomhcir@gmx.com> - 2024-02-09 05:50 +0100
          Re: Unidentified flying subject! Charles Curley <charlescurley@charlescurley.com> - 2024-02-09 07:50 +0100
            Re: Unidentified flying subject! Richmond <dnomhcir@gmx.com> - 2024-02-09 14:20 +0100

csiph-web