Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.debian.user > #267360
| From | Linux-Fan <Ma_Sys.ma@web.de> |
|---|---|
| Newsgroups | linux.debian.user |
| Subject | Re: Fast Random Data Generation (Was: Re: Unidentified subject!) |
| Date | 2024-02-13 18:40 +0100 |
| Message-ID | <I74Vc-9V9L-13@gated-at.bofh.it> (permalink) |
| References | (1 earlier) <I6eWC-9plk-7@gated-at.bofh.it> <I6ffX-9pry-7@gated-at.bofh.it> <I6xPA-9AWS-3@gated-at.bofh.it> <I6Hvz-9GDo-3@gated-at.bofh.it> <I6M2d-9Jqf-1@gated-at.bofh.it> |
| Organization | linux.* mail to news gateway |
[Multipart message — attachments visible in raw view] - view raw
David Christensen writes:
> On 2/12/24 08:30, Linux-Fan wrote:
>> David Christensen writes:
>>
>>> On 2/11/24 02:26, Linux-Fan wrote:
>>>> I wrote a program to automatically generate random bytes in multiple threads:
>>>> https://masysma.net/32/big4.xhtml
>>>
>>> What algorithm did you implement?
>>
>> I copied the algorithm from here:
>> https://www.javamex.com/tutorials/random_numbers/numerical_recipes.shtml
>
> That Java code uses locks, which implies it uses global state and cannot be
> run multi-threaded (?). (E.g. one process with one JVM.)
Indeed, the example code uses locks which is bad from a performance point of
view. That is why _my_ implementation works without this fine-grained
locking and instead ensures that each thread uses its own instance as to
avoid the lock. IOW: I copied the algorithm but of course adjusted the code
to my use case.
My version basically runs as follows:
* Create one queue of ByteBuffers
* Create multiple threads
* Each thread runs their own RNG instance
* Upon finishing the creation of a buffer, it enqueues
the resulting ByteBuffer into the queue (this is the only
part where multiple threads access concurrently)
* The main thread dequeues from the queue and writes the
buffers to the output file
> Is it possible to obtain parallel operation on an SMP machine with multiple
> virtual processors? (Other than multiple OS processes with one PRNG on one
> JVM each?)
Even the locked random could be instantiated multiple times (each instance
gets their own lock) and this could still be faster than running just
one of it. However, since the computation itself is fast, I suppose the
performance hit from managing the locks could be significant. Multiple OS
processes would also work, but is pretty uncommon in Java land AFAIR.
>> I found it during the development of another application where I needed a
>> lot of random data for simulation purposes :)
>>
>> My implementation code is here:
>> https://github.com/m7a/bo-big/blob/master/latest/Big4.java
See the end of that file to compare with the “Numerical Recipes” RNG linked
further above to observe the difference wrt. locking :)
>> If I were to do it again today, I'd probably switch to any of these PRNGS:
>>
>> * https://burtleburtle.net/bob/rand/smallprng.html
>> * https://www.pcg-random.org/
>
> Hard core. I'll let the experts figure it out; and then I will use their
> libraries and programs.
IIRC one of the findings of PCG was that the default RNGs of many
programming languages and environments are surprisingly bad. I only arrived at
using a non-default implementation after facing some issues with the Java
integrated ThreadLocalRandom ”back then” :)
It may indeed be worth pointing out (as Jeffrey Walton already mentioned in
another subthread) that these RNGs discussed here are _not_ cryptographic
RNGs. I think for disk testing purposes it is OK to use fast non-
cryptographic RNGs, but other applications may have higher demands on their
RNGs.
HTH
Linux-Fan
öö
[...]
Back to linux.debian.user | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread
Unidentified subject! gene heskett <gheskett@shentel.net> - 2024-02-07 21:40 +0100
Re: Unidentified subject! Stefan Monnier <monnier@iro.umontreal.ca> - 2024-02-08 05:30 +0100
Re: Unidentified subject! gene heskett <gheskett@shentel.net> - 2024-02-10 11:00 +0100
Re: Unidentified subject! "Thomas Schmitt" <scdbackup@gmx.net> - 2024-02-10 11:40 +0100
shred bug? [was: Unidentified subject!] <tomas@tuxteam.de> - 2024-02-10 13:50 +0100
Re: shred bug? [was: Unidentified subject!] tomas@tuxteam.de - 2024-02-10 14:00 +0100
Re: shred bug? "Thomas Schmitt" <scdbackup@gmx.net> - 2024-02-10 15:00 +0100
Re: shred bug? <tomas@tuxteam.de> - 2024-02-10 15:40 +0100
Re: shred bug? Gremlin <scott-andrews@columbus.rr.com> - 2024-02-10 14:40 +0100
Re: shred bug? "Thomas Schmitt" <scdbackup@gmx.net> - 2024-02-10 14:40 +0100
Re: shred bug? [was: Unidentified subject!] David Christensen <dpchrist@holgerdanske.com> - 2024-02-11 01:10 +0100
Re: shred bug? [was: Unidentified subject!] Greg Wooledge <greg@wooledge.org> - 2024-02-11 01:20 +0100
Re: shred bug? [was: Unidentified subject!] David Christensen <dpchrist@holgerdanske.com> - 2024-02-11 01:30 +0100
Re: shred bug? [was: Unidentified subject!] debian-user@howorth.org.uk - 2024-02-11 14:00 +0100
Re: shred bug? "Thomas Schmitt" <scdbackup@gmx.net> - 2024-02-11 14:30 +0100
Re: shred bug? [was: Unidentified subject!] <tomas@tuxteam.de> - 2024-02-11 08:10 +0100
Re: shred bug? [was: Unidentified subject!] Greg Wooledge <greg@wooledge.org> - 2024-02-11 15:40 +0100
Re: shred bug? [was: Unidentified subject!] <tomas@tuxteam.de> - 2024-02-11 15:50 +0100
Re: shred bug? [was: Unidentified subject!] Greg Wooledge <greg@wooledge.org> - 2024-02-11 16:00 +0100
Re: shred bug? [was: Unidentified subject!] <tomas@tuxteam.de> - 2024-02-11 16:00 +0100
Re: shred bug? [was: Unidentified subject!] Greg Wooledge <greg@wooledge.org> - 2024-02-12 18:00 +0100
Re: shred bug? [was: Unidentified subject!] Curt <curty@free.fr> - 2024-02-12 18:00 +0100
Re: shred bug? [was: Unidentified subject!] David Christensen <dpchrist@holgerdanske.com> - 2024-02-12 22:00 +0100
Re: shred bug? [was: Unidentified subject!] "Thomas Schmitt" <scdbackup@gmx.net> - 2024-02-12 22:10 +0100
Re: shred bug? [was: Unidentified subject!] <tomas@tuxteam.de> - 2024-02-13 06:50 +0100
Re: shred bug? [was: Unidentified subject!] David Wright <deblis@lionunicorn.co.uk> - 2024-02-11 16:30 +0100
Re: shred bug? [was: Unidentified subject!] Greg Wooledge <greg@wooledge.org> - 2024-02-13 13:20 +0100
Re: shred bug? [was: Unidentified subject!] Greg Wooledge <greg@wooledge.org> - 2024-02-13 13:40 +0100
Re: shred bug? [was: Unidentified subject!] <tomas@tuxteam.de> - 2024-02-13 13:50 +0100
Re: shred bug? [was: Unidentified subject!] "Thomas Schmitt" <scdbackup@gmx.net> - 2024-02-13 14:00 +0100
Re: shred bug? [was: Unidentified subject!] David Wright <deblis@lionunicorn.co.uk> - 2024-02-13 16:40 +0100
Re: shred bug? [was: Unidentified subject!] Greg Wooledge <greg@wooledge.org> - 2024-02-13 17:30 +0100
Re: shred bug? [was: Unidentified subject!] debian-user@howorth.org.uk - 2024-02-13 18:50 +0100
Re: shred bug? [was: Unidentified subject!] David Christensen <dpchrist@holgerdanske.com> - 2024-02-13 22:10 +0100
Re: shred bug? [was: Unidentified subject!] <tomas@tuxteam.de> - 2024-02-14 06:50 +0100
Re: shred bug? [was: Unidentified subject!] "Thomas Schmitt" <scdbackup@gmx.net> - 2024-02-13 19:00 +0100
Re: shred bug? [was: Unidentified subject!] Greg Wooledge <greg@wooledge.org> - 2024-02-13 20:00 +0100
Re: shred bug? [was: Unidentified subject!] "Thomas Schmitt" <scdbackup@gmx.net> - 2024-02-13 20:40 +0100
Re: shred bug? [was: Unidentified subject!] gene heskett <gheskett@shentel.net> - 2024-02-13 20:40 +0100
Re: shred bug? [was: Unidentified subject!] "Thomas Schmitt" <scdbackup@gmx.net> - 2024-02-13 20:50 +0100
Re: shred bug? [was: Unidentified subject!] gene heskett <gheskett@shentel.net> - 2024-02-13 21:30 +0100
Re: shred bug? [was: Unidentified subject!] David Christensen <dpchrist@holgerdanske.com> - 2024-02-13 22:10 +0100
Re: shred bug? [was: Unidentified subject!] gene heskett <gheskett@shentel.net> - 2024-02-13 22:50 +0100
Re: shred bug? [was: Unidentified subject!] David Wright <deblis@lionunicorn.co.uk> - 2024-02-15 06:40 +0100
Re: shred bug? [was: Unidentified subject!] David Christensen <dpchrist@holgerdanske.com> - 2024-02-11 23:50 +0100
Re: shred bug? [was: Unidentified subject!] Max Nikulin <manikulin@gmail.com> - 2024-02-13 03:40 +0100
Re: shred bug? [was: Unidentified subject!] "Thomas Schmitt" <scdbackup@gmx.net> - 2024-02-11 16:30 +0100
Re: shred bug? [was: Unidentified subject!] Michael Stone <mstone@debian.org> - 2024-02-16 16:00 +0100
Re: Unidentified subject! gene heskett <gheskett@shentel.net> - 2024-02-10 18:50 +0100
Re: Unidentified subject! "Thomas Schmitt" <scdbackup@gmx.net> - 2024-02-10 19:40 +0100
Re: Unidentified subject! gene heskett <gheskett@shentel.net> - 2024-02-10 20:30 +0100
Re: Unidentified subject! David Christensen <dpchrist@holgerdanske.com> - 2024-02-11 00:50 +0100
Re: Unidentified subject! "Thomas Schmitt" <scdbackup@gmx.net> - 2024-02-11 09:10 +0100
Re: Unidentified subject! David Christensen <dpchrist@holgerdanske.com> - 2024-02-11 21:50 +0100
Re: Unidentified subject! David Christensen <dpchrist@holgerdanske.com> - 2024-02-11 00:40 +0100
Re: Unidentified subject! "Thomas Schmitt" <scdbackup@gmx.net> - 2024-02-11 09:20 +0100
Re: Unidentified subject! David Christensen <dpchrist@holgerdanske.com> - 2024-02-11 11:10 +0100
Fast Random Data Generation (Was: Re: Unidentified subject!) Linux-Fan <Ma_Sys.ma@web.de> - 2024-02-11 11:30 +0100
Re: Fast Random Data Generation "Thomas Schmitt" <scdbackup@gmx.net> - 2024-02-11 12:30 +0100
Re: Fast Random Data Generation (Was: Re: Unidentified subject!) Gremlin <scott-andrews@columbus.rr.com> - 2024-02-11 13:20 +0100
Re: Fast Random Data Generation (Was: Re: Unidentified subject!) David Christensen <dpchrist@holgerdanske.com> - 2024-02-12 07:20 +0100
Re: Fast Random Data Generation (Was: Re: Unidentified subject!) Linux-Fan <Ma_Sys.ma@web.de> - 2024-02-12 17:40 +0100
Re: Fast Random Data Generation (Was: Re: Unidentified subject!) David Christensen <dpchrist@holgerdanske.com> - 2024-02-12 22:30 +0100
Re: Fast Random Data Generation (Was: Re: Unidentified subject!) Linux-Fan <Ma_Sys.ma@web.de> - 2024-02-13 18:40 +0100
Re: Fast Random Data Generation (Was: Re: Unidentified subject!) Jeffrey Walton <noloader@gmail.com> - 2024-02-12 22:30 +0100
Re: Unidentified subject! "Thomas Schmitt" <scdbackup@gmx.net> - 2024-02-11 12:20 +0100
Re: Unidentified subject! Jeffrey Walton <noloader@gmail.com> - 2024-02-11 16:20 +0100
Re: Unidentified subject! David Christensen <dpchrist@holgerdanske.com> - 2024-02-11 23:00 +0100
Re: Unidentified subject! Stefan Monnier <monnier@iro.umontreal.ca> - 2024-02-10 15:40 +0100
Things I don't touch with a 3.048m barge pole: USB storage (Was Re: Unidentified subject!) Andy Smith <andy@strugglers.net> - 2024-02-08 16:40 +0100
Re: Things I don't touch with a 3.048m barge pole: USB storage (Was Re: Unidentified subject!) Gremlin <scott-andrews@columbus.rr.com> - 2024-02-08 17:20 +0100
Re: Things I don't touch with a 3.048m barge pole: USB storage (Was Re: Unidentified subject!) Andy Smith <andy@strugglers.net> - 2024-02-08 17:30 +0100
Re: Things I don't touch with a 3.048m barge pole: USB storage (WasRe: Unidentified subject!) gene heskett <gheskett@shentel.net> - 2024-02-09 09:50 +0100
Re: Things I don't touch with a 3.048m barge pole: USB storage (Was Re: Unidentified subject!) Ralph Aichinger <ra@h5.or.at> - 2024-02-08 18:00 +0100
Re: Things I don't touch with a 3.048m barge pole: USB storage (Was Re: Unidentified subject!) Jeffrey Walton <noloader@gmail.com> - 2024-02-08 20:30 +0100
Re: Things I don't touch with a 3.048m barge pole: USB storage (Was Re: Unidentified subject!) Andy Smith <andy@strugglers.net> - 2024-02-08 21:50 +0100
Re: Things I don't touch with a 3.048m barge pole: USB storage (Was Re: Unidentified subject!) Andy Smith <andy@strugglers.net> - 2024-02-08 21:50 +0100
Re: Things I don't touch with a 3.048m barge pole: USB storage (Was Re: Unidentified subject!) Gremlin <scott-andrews@columbus.rr.com> - 2024-02-08 22:00 +0100
Re: Things I don't touch with a 3.048m barge pole: USB storage (Was Re: Unidentified subject!) Andy Smith <andy@strugglers.net> - 2024-02-08 22:20 +0100
Re: Things I don't touch with a 3.048m barge pole: USB storage (Was Re: Unidentified subject!) Andy Smith <andy@strugglers.net> - 2024-02-08 22:30 +0100
Re: Things I don't touch with a 3.048m barge pole: USB storage (Was Re: Unidentified subject!) Gremlin <scott-andrews@columbus.rr.com> - 2024-02-08 23:10 +0100
Re: Things I don't touch with a 3.048m barge pole: USB storage (Was Re: Unidentified subject!) Gremlin <scott-andrews@columbus.rr.com> - 2024-02-08 22:30 +0100
Re: Things I don't touch with a 3.048m barge pole: USB storage (WasRe: Unidentified subject!) gene heskett <gheskett@shentel.net> - 2024-02-09 14:00 +0100
Re: Things I don't touch with a 3.048m barge pole "Thomas Schmitt" <scdbackup@gmx.net> - 2024-02-09 14:50 +0100
Re: Things I don't touch with a 3.048m barge pole: USB storage (WasRe: Unidentified subject!) David Christensen <dpchrist@holgerdanske.com> - 2024-02-10 07:00 +0100
Re: Things I don't touch with a 3.048m barge pole: USB storage(WasRe: Unidentified subject!) gene heskett <gheskett@shentel.net> - 2024-02-10 18:40 +0100
Re: Things I don't touch with a 3.048m barge pole: USB storage (Was Re: Unidentified subject!) Andy Smith <andy@strugglers.net> - 2024-02-08 21:40 +0100
Re: Things I don't touch with a 3.048m barge pole: USB storage (Was Re: Unidentified subject!) Arno Lehmann <al@its-lehmann.de> - 2024-02-09 09:50 +0100
Re: Things I don't touch with a 3.048m barge pole: USB storage (Was Re: Unidentified subject!) Max Nikulin <manikulin@gmail.com> - 2024-02-08 18:30 +0100
Re: Things I don't touch with a 3.048m barge pole: USB storage (Was Re: Unidentified subject!) Andy Smith <andy@strugglers.net> - 2024-02-08 21:40 +0100
Re: Things I don't touch with a 3.048m barge pole: USB storage (Was Re: Unidentified subject!) Gremlin <scott-andrews@columbus.rr.com> - 2024-02-08 22:10 +0100
Re: Things I don't touch with a 3.048m barge pole: USB storage (Was Re: Unidentified subject!) Andy Smith <andy@strugglers.net> - 2024-02-08 22:20 +0100
Re: Unidentified subject! Richmond <dnomhcir@gmx.com> - 2024-02-08 23:50 +0100
Re: Unidentified subject! Stefan Monnier <monnier@iro.umontreal.ca> - 2024-02-09 00:10 +0100
Re: Unidentified subject! Charles Curley <charlescurley@charlescurley.com> - 2024-02-09 00:50 +0100
Re: Unidentified subject! Richmond <dnomhcir@gmx.com> - 2024-02-09 05:50 +0100
Re: Unidentified flying subject! Charles Curley <charlescurley@charlescurley.com> - 2024-02-09 07:50 +0100
Re: Unidentified flying subject! Richmond <dnomhcir@gmx.com> - 2024-02-09 14:20 +0100
csiph-web