Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #103349 > unrolled thread

Re: Make a unique filesystem path, without creating the file

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Make a unique filesystem path, without creating the file Ethan Furman <ethan@stoneleaf.us> - 2016-02-22 10:11 -0800
    Re: Make a unique filesystem path, without creating the file Jon Ribbens <jon+usenet@unequivocal.co.uk> - 2016-02-22 18:17 +0000
      Re: Make a unique filesystem path, without creating the file Chris Angelico <rosuav@gmail.com> - 2016-02-23 05:25 +1100
        Re: Make a unique filesystem path, without creating the file Jon Ribbens <jon+usenet@unequivocal.co.uk> - 2016-02-22 18:39 +0000
          Re: Make a unique filesystem path, without creating the file Marko Rauhamaa <marko@pacujo.net> - 2016-02-22 20:48 +0200
            Re: Make a unique filesystem path, without creating the file Steven D'Aprano <steve@pearwood.info> - 2016-02-23 10:37 +1100
              Re: Make a unique filesystem path, without creating the file Jon Ribbens <jon+usenet@unequivocal.co.uk> - 2016-02-23 00:08 +0000
                Re: Make a unique filesystem path, without creating the file Chris Angelico <rosuav@gmail.com> - 2016-02-23 11:18 +1100
                  Re: Make a unique filesystem path, without creating the file Jon Ribbens <jon+usenet@unequivocal.co.uk> - 2016-02-23 00:26 +0000
                    Re: Make a unique filesystem path, without creating the file Chris Angelico <rosuav@gmail.com> - 2016-02-23 11:33 +1100
                      Re: Make a unique filesystem path, without creating the file Jon Ribbens <jon+usenet@unequivocal.co.uk> - 2016-02-23 00:44 +0000
                        Re: Make a unique filesystem path, without creating the file Chris Angelico <rosuav@gmail.com> - 2016-02-23 11:56 +1100
          Re: Make a unique filesystem path, without creating the file Chris Angelico <rosuav@gmail.com> - 2016-02-23 06:04 +1100
            Re: Make a unique filesystem path, without creating the file Paul Rubin <no.email@nospam.invalid> - 2016-02-22 11:22 -0800
              Re: Make a unique filesystem path, without creating the file Steven D'Aprano <steve@pearwood.info> - 2016-02-23 10:45 +1100
            Re: Make a unique filesystem path, without creating the file Jon Ribbens <jon+usenet@unequivocal.co.uk> - 2016-02-22 19:22 +0000
              Re: Make a unique filesystem path, without creating the file Marko Rauhamaa <marko@pacujo.net> - 2016-02-22 21:32 +0200
                Re: Make a unique filesystem path, without creating the file Random832 <random832@fastmail.com> - 2016-02-22 14:41 -0500
                  Re: Make a unique filesystem path, without creating the file Marko Rauhamaa <marko@pacujo.net> - 2016-02-22 22:41 +0200
                    Re: Make a unique filesystem path, without creating the file Paul Rubin <no.email@nospam.invalid> - 2016-02-22 13:05 -0800
                      Re: Make a unique filesystem path, without creating the file Marko Rauhamaa <marko@pacujo.net> - 2016-02-22 23:22 +0200
                        Re: Make a unique filesystem path, without creating the file Paul Rubin <no.email@nospam.invalid> - 2016-02-22 15:26 -0800
                Re: Make a unique filesystem path, without creating the file Steven D'Aprano <steve@pearwood.info> - 2016-02-23 11:33 +1100
                  Re: Make a unique filesystem path, without creating the file Marko Rauhamaa <marko@pacujo.net> - 2016-02-23 08:54 +0200
                    Re: Make a unique filesystem path, without creating the file Paul Rubin <no.email@nospam.invalid> - 2016-02-22 23:18 -0800
                      Re: Make a unique filesystem path, without creating the file Marko Rauhamaa <marko@pacujo.net> - 2016-02-23 21:04 +0200
                    Re: Make a unique filesystem path, without creating the file Steven D'Aprano <steve@pearwood.info> - 2016-02-24 12:40 +1100
                      Re: Make a unique filesystem path, without creating the file Marko Rauhamaa <marko@pacujo.net> - 2016-02-24 09:20 +0200
                        Re: Make a unique filesystem path, without creating the file Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2016-02-25 16:38 +1100
                          Re: Make a unique filesystem path, without creating the file Marko Rauhamaa <marko@pacujo.net> - 2016-02-25 08:54 +0200
                            Re: Make a unique filesystem path, without creating the file Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2016-02-25 19:21 +1100
                          Re: Make a unique filesystem path, without creating the file Jon Ribbens <jon+usenet@unequivocal.co.uk> - 2016-02-25 10:05 +0000
              Re: Make a unique filesystem path, without creating the file Chris Angelico <rosuav@gmail.com> - 2016-02-23 06:37 +1100
              Re: Make a unique filesystem path, without creating the file Steven D'Aprano <steve@pearwood.info> - 2016-02-23 11:03 +1100
                Re: Make a unique filesystem path, without creating the file Jon Ribbens <jon+usenet@unequivocal.co.uk> - 2016-02-23 00:11 +0000
                Re: Make a unique filesystem path, without creating the file Paul Rubin <no.email@nospam.invalid> - 2016-02-22 18:27 -0800
                  Re: Make a unique filesystem path, without creating the file Chris Angelico <rosuav@gmail.com> - 2016-02-23 13:53 +1100
                    Re: Make a unique filesystem path, without creating the file Paul Rubin <no.email@nospam.invalid> - 2016-02-22 19:26 -0800
                  Re: Make a unique filesystem path, without creating the file Mark Lawrence <breamoreboy@yahoo.co.uk> - 2016-02-23 08:09 +0000
                    Re: Make a unique filesystem path, without creating the file Paul Rubin <no.email@nospam.invalid> - 2016-02-23 00:22 -0800
                      Re: Make a unique filesystem path, without creating the file Peter Otten <__peter__@web.de> - 2016-02-23 09:40 +0100
                      Re: Make a unique filesystem path, without creating the file Mark Lawrence <breamoreboy@yahoo.co.uk> - 2016-02-23 09:00 +0000
                        Re: Make a unique filesystem path, without creating the file Grant Edwards <invalid@invalid.invalid> - 2016-02-23 15:14 +0000
                      Re: Make a unique filesystem path, without creating the file Steven D'Aprano <steve@pearwood.info> - 2016-02-25 11:41 +1100
                      Re: Make a unique filesystem path, without creating the file Random832 <random832@fastmail.com> - 2016-02-25 10:03 -0500
      Re: Make a unique filesystem path, without creating the file Steven D'Aprano <steve@pearwood.info> - 2016-02-23 11:18 +1100

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

#103364

Paul Rubin <no.email@nospam.invalid>:

>>> http://www.2uo.de/myths-about-urandom/
>> Did you post the link because you agreed with the Web pamphlet?
>
> I don't know what web pamphlet you mean,

The only one linked above.

Cryptography is tricky business, indeed. I know enough about it not to
improvise too much. Infinitesimal weaknesses can make a difference
between feasible and unfeasible attacks.

> but the right thing to use now is getrandom(2).

getrandom(2) is a good interface that distinguishes between the flag
values

   0                            =>  /dev/urandom
   GRND_RANDOM                  =>  /dev/random
   GRND_RANDOM | GRND_NONBLOCK  =>  /dev/random (O_NONBLOCK)

However, although os.urandom() delegates to getrandom(), the
documentation suggests it uses the flag value 0 (/dev/urandom).

> The random/urandom interface was poorly designed and misleadingly
> documented.

It could be better I suppose, but I never found it particularly bad. The
nice thing about it is that it is readily usable in shell scripts.


Marko

[toc] | [prev] | [next] | [standalone]

#103371

Marko Rauhamaa <marko@pacujo.net> writes:
>>>> http://www.2uo.de/myths-about-urandom/
>> I don't know what web pamphlet you mean,
> The only one linked above.

Oh, I wouldn't have called that a pamphlet.  I could quibble with the
writing style but the points in the article are basically correct.

> getrandom(2) is a good interface that distinguishes between the flag
> values
>    0                            =>  /dev/urandom
>    GRND_RANDOM                  =>  /dev/random
>    GRND_RANDOM | GRND_NONBLOCK  =>  /dev/random (O_NONBLOCK)
> However, although os.urandom() delegates to getrandom(), the
> documentation suggests it uses the flag value 0 (/dev/urandom).

Flag value 0 does the right thing and blocks if the entropy pool is not
yet initialized, and doesn't block after that.  That fixes the errors of
both urandom (fails to block before there's enough entropy) and random
(blocks even after there's enough entropy).  The getrandom doc is also
misleading about the workings of the entropy pools but that's ok.  The
actual algorithm is described here:

  http://www.pinkas.net/PAPERS/gpr06.pdf

It's pretty clumsy but discussions about replacing it have gotten bogged
down several times.  OTOH maybe I'm out of date on this.

>> The random/urandom interface was poorly designed and misleadingly
>> documented.
> It could be better I suppose, but I never found it particularly bad. The
> nice thing about it is that it is readily usable in shell scripts.

DJB describes the problems:

https://groups.google.com/forum/#!msg/randomness-generation/4opmDHA6_3w/__TyKhbnNWsJ

Regarding shell scripts, it should be a simple matter to put a wrapper
around the system call.

[toc] | [prev] | [next] | [standalone]

#103381

On Tue, 23 Feb 2016 06:32 am, Marko Rauhamaa wrote:

> Jon Ribbens <jon+usenet@unequivocal.co.uk>:
> 
>> Suppose you had code like this:
>>
>>   filename = binascii.hexlify(os.urandom(16)).decode("ascii")
>>
>> Do we really think that is insecure or that there are any practical
>> attacks against it? It would be basically the same as saying that
>> urandom() is broken, surely?
> 
> urandom() is not quite random and so should not be considered
> cryptographically airtight.
> 
> Under Linux, /dev/random is the way to go when strong security is
> needed. Note that /dev/random is a scarce resource on ordinary systems.

That's actually incorrect, but you're not the only one to have been mislead
by the man pages.

http://sockpuppet.org/blog/2014/02/25/safely-generate-random-numbers/


On non-Linux Unixes, the difference between urandom and random is mostly, or
entirely, gone, in favour of urandom's non-blocking behaviour. And it's a
myth that the output of random is "more random" or "more pure" than
urandom's. In reality, on Linux both urandom and random use exactly the
same CSPRNG.

See also:

http://www.2uo.de/myths-about-urandom/


for a good explanation of how random and urandom actually work on Linux.






-- 
Steven

[toc] | [prev] | [next] | [standalone]

#103392

Steven D'Aprano <steve@pearwood.info>:

> On Tue, 23 Feb 2016 06:32 am, Marko Rauhamaa wrote:
>> Under Linux, /dev/random is the way to go when strong security is
>> needed. Note that /dev/random is a scarce resource on ordinary
>> systems.
>
> That's actually incorrect, but you're not the only one to have been
> mislead by the man pages.
>
> http://sockpuppet.org/blog/2014/02/25/safely-generate-random-numbers/

Still, mostly hypnotic repetitions.

However, it admits:

   But /dev/random also tries to keep track of how much entropy remains
   in its kernel pool, and will occasionally go on strike if it decides
   not enough remains.

That's the whole point. /dev/random will rather block the program than
lower the quality of the random numbers below a threshold. /dev/urandom
has no such qualms.

   If you use /dev/random instead of urandom, your program will
   unpredictably (or, if you’re an attacker, very predictably) hang when
   Linux gets confused about how its own RNG works.

Yes, possibly indefinitely, too.

   Using /dev/random will make your programs less stable, but it won’t
   make them any more cryptographically safe.

It is correct that you shouldn't use /dev/random as a routine source of
bulk random numbers. It is also correct that /dev/urandom depletes the
entropy pool as effectively as /dev/random. However, when you are
generating signing or encryption keys, you should use /dev/random.

As stated in <URL: https://lwn.net/Articles/606141/>:

   /dev/urandom should be used for essentially all random numbers
   required, but /dev/random is sometimes used for things like extremely
   sensitive, long-lived keys (e.g. GPG) or one-time pads.

> See also:
>
> http://www.2uo.de/myths-about-urandom/

Already addressed.


Marko

[toc] | [prev] | [next] | [standalone]

#103393

Marko Rauhamaa <marko@pacujo.net> writes:
> It is also correct that /dev/urandom depletes the entropy pool as
> effectively as /dev/random. 

I think see what's confusing you: the above is a misconception that is
probably held by lots of people.  Entropy is not water and from a
cryptographic standpoint there is essentially no such thing as
"depleting" an entropy pool.  There is either enough entropy (say 256
bits or more) in the PRNG or else there isn't.  If there's not enough,
urandom can misbehave by giving you bad output because it doesn't block
until more is gathered.  If there is enough, /dev/random misbehaves by
blocking under this bogus concept of "depletion".  If you have a seed
with 256 bits of entropy and you generate a gigabyte of random numbers
from it, you have not increased the predictability of the seed in any
significant way.

So once /dev/random unblocks, it should never again block, the behavior
of getrandom.  There used to be an article on David Wagner's web site
(cs.berkeley.edu/~daw) about the concept of "depleting" entropy by
iterated hashing, but I can't find it now.  That's unfortunate since it
might help cast light on the subject.

>> http://www.2uo.de/myths-about-urandom/
> Already addressed.

No really, all you've done is repeat bad advice.  The people cited in
that article are very knowledgeable and the stuff they say makes good
mathematical sense.  The stuff you say makes no sense and you haven't
given any convincing reason for anyone to listen to you.

[toc] | [prev] | [next] | [standalone]

#103414

Paul Rubin <no.email@nospam.invalid>:

> Marko Rauhamaa <marko@pacujo.net> writes:
>> It is also correct that /dev/urandom depletes the entropy pool as
>> effectively as /dev/random. 
>
> I think see what's confusing you: the above is a misconception that is
> probably held by lots of people.  Entropy is not water and from a
> cryptographic standpoint there is essentially no such thing as
> "depleting" an entropy pool.  There is either enough entropy (say 256
> bits or more) in the PRNG or else there isn't.  If there's not enough,
> urandom can misbehave by giving you bad output because it doesn't block
> until more is gathered.  If there is enough, /dev/random misbehaves by
> blocking under this bogus concept of "depletion".

You are making my point. /dev/random is correct to block until
top-quality random numbers can be supplied. That's not misbehaving.

> So once /dev/random unblocks, it should never again block, the behavior
> of getrandom.

What you are saying is that /dev/random has no reason to exist (and the
GRND_RANDOM flag to getrandom() is redundant).

I'm no cryptographer and can't judge that. However, as long as the
distinction is maintained, I have to abide by the documented
characteristics.

> No really, all you've done is repeat bad advice. The people cited in
> that article are very knowledgeable and the stuff they say makes good
> mathematical sense. The stuff you say makes no sense and you haven't
> given any convincing reason for anyone to listen to you.

Thing is, neither you nor me nor the cited articles has provided any
more info than insisting on a position, my position being relying on the
documented API.

So we have

 * /dev/urandom vs /dev/random

 * getrandom(0) vs getrandom(GRND_RANDOM)

 * GCRY_STRONG_RANDOM ("Use this level for session keys and similar
   purposes") vs GCRY_VERY_STRONG_RANDOM ("Use this level for long term
   key material") (in libgcrypt)

You don't need to convince me that that distinction is silly. You need
to convince the crypto facility providers.


Marko

[toc] | [prev] | [next] | [standalone]

#103424

On Tue, 23 Feb 2016 05:54 pm, Marko Rauhamaa wrote:

> Steven D'Aprano <steve@pearwood.info>:
> 
>> On Tue, 23 Feb 2016 06:32 am, Marko Rauhamaa wrote:
>>> Under Linux, /dev/random is the way to go when strong security is
>>> needed. Note that /dev/random is a scarce resource on ordinary
>>> systems.
>>
>> That's actually incorrect, but you're not the only one to have been
>> mislead by the man pages.
>>
>> http://sockpuppet.org/blog/2014/02/25/safely-generate-random-numbers/
> 
> Still, mostly hypnotic repetitions.

Repetition for the sake of emphasis, because there are so many misled and
confused people on the internet who misunderstand the difference between
urandom and random and consequently give bad advice. I believe that the
Linux man page for urandom is to blame, although I don't know why it hasn't
been fixed.

Possibly because it is *technically* correct, in the sense of "if you are
concerned by the risk of being hit by a meteorite, wearing a stainless
steel cooking pot on your head will give you some protection from meteorite
strikes to the head". Everything in it is technically correct, but
misleading.


> However, it admits:
> 
>    But /dev/random also tries to keep track of how much entropy remains
>    in its kernel pool, and will occasionally go on strike if it decides
>    not enough remains.
> 
> That's the whole point. 

Exactly, but you've missed the point. That is precisely why the blocking
random is HARMFUL and should not be used. There is one, and only one,
scenario when your CSPRNG should block: before the system has enough
entropy to securely seed the CSPRNG and it is at risk of returning
predictable numbers.

But after that point has passed, there is no test you can perform to
distinguish the outputs of /dev/random and /dev/urandom (apart from the
blocking behaviour itself). If I give you a million numbers, there is no
way you can tell whether I used random or urandom.

The important thing here is that there is no difference in "quality"
(whatever that means!) between the random numbers generated by urandom and
those generated by random. They are equally unpredictable. They pass the
same randomness tests. Neither is "better" or "worse" than the other,
because they are both generated by the same CSPRNG or HRNG.

Here is a summary of the random/urandom distinction on various Unixes:


Linux: 
    random blocks, urandom never blocks, both use the same CSPRNG 
    based on SHA-1 hashes, both will use a HRNG if available

FreeBSD:
    urandom is a link to random, which never blocks; uses 256-bit 
    Yarrow CSPRNG, will use a HRNG if available

OpenBSD:
    both never block; both use a variant of the RC4 CSPRNG 
    (misleadingly renamed ARC4 due to licencing issues), in
    newer versions use the ChaCha20 CSPRNG

OS X:
    both never block and use 160-bit Yarrow

NetBSD:
    random blocks, urandom never blocks, both use the same AES-128
    CSPRNG


The NetBSD man pages are quite scathing:


    "The entropy accounting described here is not grounded in any
    cryptography theory.  It is done because it was always done, 
    and because it gives people a warm fuzzy feeling about 
    information theory.

    ...

    History is littered with examples of broken entropy sources and
    failed system engineering for random number generators.  Nobody 
    has ever reported distinguishing AES ciphertext from uniform 
    random without side channels, nor reported computing SHA-1 
    preimages faster than brute force.  The folklore information-
    theoretic defence against computationally unbounded attackers 
    replaces system engineering that successfully defends against 
    realistic threat models by imaginary theory that defends only 
    against fantasy threat models."


To be clear, the "folklore information-theoretic defence" they are referring
to is /dev/random's blocking behaviour.

http://netbsd.gw.com/cgi-bin/man-cgi?rnd+4+NetBSD-current


The blocking behaviour of /dev/random (on Linux) doesn't solve any real
problems, but it *creates* new problems. /dev/random can block for minutes
or even hours, especially straight after booting a freshly installed OS.
This can be considered a Denial Of Service attack, and even if it isn't, it
encourages developers to "fix" the problem by using their own home-brewed
random numbers, weakening the security of the system.

There's even a minority viewpoint that constantly adding new entropy to the
CSPRNG is useless. Apart from collecting sufficient entropy for the initial
seed, you should never add new entropy to the CSPRNG. Your CSPRNG is either
cryptographically strong, or it isn't. If it is, then it is already
unpredictable and adding more entropy is a waste of time. If it isn't, then
adding more entropy isn't going to help you.

Adding entropy is just one more component that can contain bugs (see the
NetSBD comment about "broken entropy sources") or even allow an attack on
the CSPRNG:

http://blog.cr.yp.to/20140205-entropy.html

There's one good argument for adding new entropy: imagine an attacker who,
somehow, manages to get access to the internal state of your CSPRNG, but
nothing else. (If they have access to your whole system, they own you, and
the game is over.) In this scenario, instead of being able to predict the
random numbers you produce forever, they will only be able to predict them
until you've added sufficient fresh entropy into the system. Assuming they
can't read or modify the entropy going in.


> /dev/random will rather block the program than 
> lower the quality of the random numbers below a threshold. /dev/urandom
> has no such qualms.

No, that's wrong. There is no "lower the quality below a threshold".
Both /dev/random and /dev/urandom use the same CSPRNG on all the Unixes I
know of. So long as the CSPRNG has been seeded with enough entropy at the
start, they are both of equally good quality. If the OS supports a HRNG,
both will use it. On at least one Unix, FreeBSD, the two block devices are
literally identical, and one is a link to the other.


>    If you use /dev/random instead of urandom, your program will
>    unpredictably (or, if you’re an attacker, very predictably) hang when
>    Linux gets confused about how its own RNG works.
> 
> Yes, possibly indefinitely, too.
> 
>    Using /dev/random will make your programs less stable, but it won’t
>    make them any more cryptographically safe.
> 
> It is correct that you shouldn't use /dev/random as a routine source of
> bulk random numbers. It is also correct that /dev/urandom depletes the
> entropy pool as effectively as /dev/random. 

Correct so far. But then:

> However, when you are 
> generating signing or encryption keys, you should use /dev/random.

And that is where you repeat something which is rank superstition.


> As stated in <URL: https://lwn.net/Articles/606141/>:
> 
>    /dev/urandom should be used for essentially all random numbers
>    required, but /dev/random is sometimes used for things like extremely
>    sensitive, long-lived keys (e.g. GPG) or one-time pads.

That doesn't mean anything beyond a statement of fact that people sometimes
use /dev/random. Yeah, okay. So what? People sometimes use urandom for the
same purposes too. I'm sure that, somewhere in the world, some poor dofus
has used a lousy 16-bit PRNG to generate his GPG key. Doesn't mean that it
is good to do so, or necessary to do so.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#103429

Steven D'Aprano <steve@pearwood.info>:

> On Tue, 23 Feb 2016 05:54 pm, Marko Rauhamaa wrote:
>> However, when you are generating signing or encryption keys, you
>> should use /dev/random.
>
> And that is where you repeat something which is rank superstition.

Can you find info to back that up. All I've seen so far is forceful
claims that's superstition ("These are not the droids you're looking
for"). Even the ssh-keygen man page has:

    The reseeding of the OpenSSL random generator is usually done from
    /dev/urandom. If the SSH_USE_STRONG_RNG environment vari‐ able is
    set to value other than 0 the OpenSSL random generator is reseeded
    from /dev/random.


Marko

[toc] | [prev] | [next] | [standalone]

#103479

On Wednesday 24 February 2016 18:20, Marko Rauhamaa wrote:

> Steven D'Aprano <steve@pearwood.info>:
> 
>> On Tue, 23 Feb 2016 05:54 pm, Marko Rauhamaa wrote:
>>> However, when you are generating signing or encryption keys, you
>>> should use /dev/random.
>>
>> And that is where you repeat something which is rank superstition.
> 
> Can you find info to back that up. 

The links already provided go through the evidence. For example, they 
explain that /dev/random and /dev/urandom both use the exact same CSPRNG. If 
you don't believe that, you can actually read the source to Linux, FreeBSD, 
OpenBSD and NetBSD. (But not OS X, sorry.)

Put aside the known bug in Linux, where urandom will provide predictable 
values in the period following a fresh install before the OS has collected 
enough entropy. We all agree that's a problem, and that the right solution 
is to block. The question is, outside of that narrow set of circumstances, 
when it is appropriate to block?

As I mentioned, most Unixes don't block. urandom and random behave exactly 
the same way in three of the most popular Unixes (FreeBSD, OpenBSD, OS X):

https://en.wikipedia.org/wiki//dev/random

so let's consider just those that do block (Linux, and NetBSD). They both 
use the same CSPRNG. Do you dispute that? Then read the source. For one to 
be "better" than the other, there would need to be a detectable difference 
between the two. Nobody has ever found one, and nor will they, because 
they're both coming from the same CSPRNG (AES in the case of NetBSD, I'm not 
sure what in the case of Linux).

If you don't trust the CSPRNG, then you shouldn't trust it whether it comes 
from /dev/random or /dev/urandom. If you do trust it, then why would you 
want it to block? Blocking doesn't make it more random. That's not how it 
works. It just makes you vulnerable to a Denial Of Service attack.

There really doesn't seem to be any valid reason for random blocking. It's 
like warnings and timers on fans in South Korea to prevent fan death:

http://www.snopes.com/medical/freakish/fandeath.asp

(My favourite explanation is that the blades of the fan chop the oxygen 
molecules in two.)

I'm not surprised that there is so much misinformation about random/urandom. 
Here's a blog post by somebody wanting to port arc4 to Linux, so he clearly 
knows a few things about crypto. I can't judge whether arc4 is better or 
worse than what Linux already uses, but look at this quote:

http://insanecoding.blogspot.com.au/2014/05/a-good-idea-with-bad-usage-
devurandom.html

Quote:

    Linux is well known for inventing and supplying two default files,
    /dev/random and /dev/urandom (unlimited random). The former is 
    pretty much raw entropy, while the latter is the output of a CSPRNG
    function like OpenBSD's arc4random family. The former can be seen 
    as more random, and the latter as less random, but the differences
    are extremely hard to measure, which is why CSPRNGs work in the 
    first place. Since the former is only entropy, it is limited as to
    how much it can output, and one needing a lot of random data can be
    stuck waiting a while for it to fill up the random buffer. Since the
    latter is a CSPRNG, it can keep outputting data indefinitely, 
    without any significant waiting periods."

There's that myth about urandom being "less random" than random again, but 
even this guy admits that the difference is "extremely hard" (actually: 
impossible) to measure, and that CSPRNG's "work". Which is precisely why 
OpenBSD uses arc4random for their /dev/random and /dev/urandom, and 
presumably why he wants to bring it to Linux.

This author is *completely wrong* to say that /dev/random is "pretty much 
raw entropy". If it were, it would be biased, and easily manipulated by an 
attacker. Entropy is collected from (among other things) network traffic, 
which would allow an attacker to control at least one source of entropy and 
hence (in theory) make it easier to predict the output of /dev/random.

But fortunately it is not true. Linux's random system works like this:

- entropy is collected from various sources and fed into a pool;

- entropy from that pool is fed through a CSPRNG into two separate pools, 
  one each for /dev/random and /dev/urandom;

- when you read from /dev/random or urandom, they both collect entropy
  from their own pool, and again pass it through a CSPRNG;

- /dev/random has a throttle (it blocks if you take out too much);

- /dev/urandom doesn't have a throttle.

https://events.linuxfoundation.org/images/stories/pdf/lceu2012_anvin.pdf


Somebody criticized the author for spreading this misapprehension that 
/dev/random is "raw entropy" and here is his response:

    I tried giving an explanation which should be simple for a 
    layman to follow of what goes on. I wouldn't take it as precise 
    fact, especially when there's a washing machine involved in the
    explanation ;)


Or, in other words, "When I said the moon was made of green cheese, I was 
simplifying it for the layman. I didn't mean it was precisely green cheese. 
It's just like green cheese."

Sure. But it is *more like* rock and dust than green cheese. In fact, apart 
from both being made of atoms, it's hard to see any similarity between the 
moon and green cheese at all, just as there is no real similarity between 
what /dev/random actually does, and what this guy says it does.

The bottom line is, nobody can distinguish the output of urandom and random 
(apart from the blocking behaviour). Nobody has demonstrated any way to 
distinguish the output of either random or urandom from "actual randomness". 
There are theoretical attacks on urandom that random might be immune to, but 
if so, I haven't heard what they are.

Apart from pointing you at the source code, I don't know what else I can say 
to prove this. I can't prove that there's no known attack on Linux's CSPRNG, 
just as I can't prove that there's no tea cup in orbit around Jupiter, or 
that there wasn't a nuclear war between China and the USSR in 1985 and both 
countries covered it up. Maybe there was. But I wouldn't bet on it.


> All I've seen so far is forceful
> claims that's superstition ("These are not the droids you're looking
> for"). 
>
> Even the ssh-keygen man page has:
> 
>     The reseeding of the OpenSSL random generator is usually done from
>     /dev/urandom. If the SSH_USE_STRONG_RNG environment vari‐ able is
>     set to value other than 0 the OpenSSL random generator is reseeded
>     from /dev/random.

If they had called the environment variable SSH_MAKE_MONEY_FAST would you 
believe it?

What evidence do they give that /dev/urandom is weak? If it is weak, why are 
they using it as the default?



-- 
Steve

[toc] | [prev] | [next] | [standalone]

#103481

Steven D'Aprano <steve+comp.lang.python@pearwood.info>:

> On Wednesday 24 February 2016 18:20, Marko Rauhamaa wrote:
>> Steven D'Aprano <steve@pearwood.info>:
>>> And that is where you repeat something which is rank superstition.
>> 
>> Can you find info to back that up. 
>
> The links already provided go through the evidence. For example, they 
> explain that /dev/random and /dev/urandom both use the exact same
> CSPRNG.

A non-issue. The question is, after the initial entropy is collected and
used to seed the CSPRNG, is any further entropy needed for any
cryptographic purposes? Are there any nagging fears that weaknesses
could be found in the deterministic sequence?

/dev/random is supposed to be hardened against such concerns by stirring
the pot constantly (if rather slowly).

Here's what Linus Torvalds said on the matter years back:

   > No, it says /dev/random is primarily useful for generating large
   > (>>160 bit) keys.

   Which is exactly what something like sshd would want to use for
   generating keys for the machine, right? That is _the_ primary reason
   to use /dev/random.

   Yet apparently our /dev/random has been too conservative to be
   actually useful, because (as you point out somewhere else) even sshd
   uses /dev/urandom for the host key generation by default.

   That is really sad. That is the _one_ application that is common and
   that should really have a reason to maybe care about /dev/random vs
   urandom. And that application uses urandom. To me that says that
   /dev/random has turned out to be less than useful in real life.

   Is there anything that actually uses /dev/random at all (except for
   clueless programs that really don't need to)?

   <URL: http://article.gmane.org/gmane.linux.kernel/47437>

> If you don't trust the CSPRNG, then you shouldn't trust it whether it comes 
> from /dev/random or /dev/urandom. If you do trust it, then why would you 
> want it to block? Blocking doesn't make it more random.

It might not make it more secure cryptographically, but the point is
that it should make it more genuinely random.

> That's not how it works. It just makes you vulnerable to a Denial Of
> Service attack.

Understood. You should not use /dev/random for any reactive purposes
(like nonces or session encryption keys).

> There's that myth about urandom being "less random" than random again,
> but even this guy admits that the difference is "extremely hard"
> (actually: impossible) to measure, and that CSPRNG's "work". Which is
> precisely why OpenBSD uses arc4random for their /dev/random and
> /dev/urandom, and presumably why he wants to bring it to Linux.

That's for the cryptographic experts to judge. CSPRNG's aren't always as
CS as one would think:

   In December 2013, a Reuters news article alleged that in 2004, before
   NIST standardized Dual_EC_DRBG, NSA paid RSA Security $10 million in
   a secret deal to use Dual_EC_DRBG as the default in the RSA BSAFE
   cryptography library, which resulted in RSA Security becoming the
   most important distributor of the insecure algorithm.

   <URL: https://en.wikipedia.org/wiki/Dual_EC_DRBG>

> The bottom line is, nobody can distinguish the output of urandom and
> random (apart from the blocking behaviour). Nobody has demonstrated
> any way to distinguish the output of either random or urandom from
> "actual randomness". There are theoretical attacks on urandom that
> random might be immune to, but if so, I haven't heard what they are.

What I'm looking for is a cryptography mailing list (or equivalent)
giving their stamp of approval. As can be seen above, NIST ain't it.

It seems, though, that cryptography researchers are not ready to declare
any scheme void of vulnerabilities. At best they can mention that there
are no *known* vulnerabilities.

> What evidence do they give that /dev/urandom is weak? If it is weak,
> why are they using it as the default?

It's a big mess, but not a mess I would disentangle. Once the crypto
libraries, utilities, facilities and the OS come to a consensus, I can
hope they've done their homework. As it stands, the STRONG vs VERY
STRONG dichotomy seems to be alive all over the place.


Marko

[toc] | [prev] | [next] | [standalone]

#103488

On Thursday 25 February 2016 17:54, Marko Rauhamaa wrote:

> Steven D'Aprano <steve+comp.lang.python@pearwood.info>:
> 
>> On Wednesday 24 February 2016 18:20, Marko Rauhamaa wrote:
>>> Steven D'Aprano <steve@pearwood.info>:
>>>> And that is where you repeat something which is rank superstition.
>>> 
>>> Can you find info to back that up.
>>
>> The links already provided go through the evidence. For example, they
>> explain that /dev/random and /dev/urandom both use the exact same
>> CSPRNG.
> 
> A non-issue. The question is, after the initial entropy is collected and
> used to seed the CSPRNG, is any further entropy needed for any
> cryptographic purposes?

The short answer: "yes".

The long answer: "probably not, but it can't hurt".

The longer answer: "probably not, and it usually won't hurt, but it could".

If, somehow, an attacker manages to work out the state of your CSPRNG, 
including the entropy pool, then they can predict what values you get until 
they no longer know the state of the CSPRNG. The idea is that if, somehow, 
somebody knows the current state of the CSPRNG (including the entropy pool), 
but can't influence what future values go into the entropy pool, then they 
will only be able to predict the output values for a short time.

But it's hard to think of any actual attack where somebody can see what's in 
the entropy pool but can't influence the values going into it. It seems to 
me that this is an unrealistic attack:

"Assume that you're kidnapped by somebody with no arms or legs..."

The conventional wisdom is that adding poor sources of entropy into the pool 
will never hurt, but that is actually wrong. If an attacker knows what is in 
the entropy pool, and can craft the values going in, they can force the 
CSPRNG to return more predictable values. So sometimes adding more entropy 
can hurt. And it usually won't help. It *might* help if your system is 
compromised, but if so, it's not really clear how the attacker has 
compromised your current entropy pool but not the future ones.


> Are there any nagging fears that weaknesses
> could be found in the deterministic sequence?

Of course there are. Nobody really knows what capabilities the NSA have, but 
they almost surely aren't *that* advanced. CSPRNGs are subject to much the 
same sort of issues as other crypto, such as hash functions:

http://valerieaurora.org/hash.html

and encryption algorithms. (The main real difference between a hash function 
and encryption algorithm is that hashes don't have to be reversible.)

Expect the current crop of CSPRNGs (Yarrow, AES, whatever Linux uses) to be 
replaced long before there is a proven attack on them.


> /dev/random is supposed to be hardened against such concerns by stirring
> the pot constantly (if rather slowly).

As is /dev/urandom.


> Here's what Linus Torvalds said on the matter years back:
> 
>    > No, it says /dev/random is primarily useful for generating large
>    > (>>160 bit) keys.
> 
>    Which is exactly what something like sshd would want to use for
>    generating keys for the machine, right? That is _the_ primary reason
>    to use /dev/random.
> 
>    Yet apparently our /dev/random has been too conservative to be
>    actually useful, because (as you point out somewhere else) even sshd
>    uses /dev/urandom for the host key generation by default.
> 
>    That is really sad. That is the _one_ application that is common and
>    that should really have a reason to maybe care about /dev/random vs
>    urandom. And that application uses urandom. To me that says that
>    /dev/random has turned out to be less than useful in real life.
> 
>    Is there anything that actually uses /dev/random at all (except for
>    clueless programs that really don't need to)?

Most other Unixes have decided that /dev/random is unnecessary, and urandom 
is the right thing to do. SSH uses urandom by default, but allows the 
paranoid/clueless to use /dev/random if they insist. 



-- 
Steve

[toc] | [prev] | [next] | [standalone]

#103491

On 2016-02-25, Steven D'Aprano <steve+comp.lang.python@pearwood.info> wrote:
> The links already provided go through the evidence. For example, they 
> explain that /dev/random and /dev/urandom both use the exact same CSPRNG. If 
> you don't believe that, you can actually read the source to Linux, FreeBSD, 
> OpenBSD and NetBSD. (But not OS X, sorry.)

Actually yes OS X:

http://www.opensource.apple.com/source/xnu/xnu-3248.20.55/bsd/dev/random/
http://www.opensource.apple.com/source/xnu/xnu-3248.20.55/osfmk/prng/

[toc] | [prev] | [next] | [standalone]

#103360

On Tue, Feb 23, 2016 at 6:22 AM, Jon Ribbens
<jon+usenet@unequivocal.co.uk> wrote:
>> Maybe, if everyone's cooperating. I'm not sure how they fare in the
>> face of malice though.
>
> Suppose you had code like this:
>
>   filename = binascii.hexlify(os.urandom(16)).decode("ascii")
>
> Do we really think that is insecure or that there are any practical
> attacks against it? It would be basically the same as saying that
> urandom() is broken, surely?

Sure, that would be safe. But UUIDs aren't necessarily based on "give
me sixteen bytes from urandom". They can involve
potentially-predictable information such as MAC addresses, current
time of day, and so on, which gives them significantly less
randomness. In that kind of usage, they're not intended to be
cryptographically secure.

ChrisA

[toc] | [prev] | [next] | [standalone]

#103375

On Tue, 23 Feb 2016 06:22 am, Jon Ribbens wrote:

> Suppose you had code like this:
> 
> filename = binascii.hexlify(os.urandom(16)).decode("ascii")
> 
> Do we really think that is insecure or that there are any practical
> attacks against it? It would be basically the same as saying that
> urandom() is broken, surely?

Correct. Any attack against urandom would be an attack on this. You would
just have to trust that the kernel devs have made urandom as secure as
possible, and pay no attention to what the man page says, as its wrong.

By the way, Python 3.6 will have (once Guido formally approves it) a new
module, "secrets", for securely generating (pseudo)random tokens like this:

import secrets
filename = secrets.token_hex(16)


https://www.python.org/dev/peps/pep-0506/




-- 
Steven

[toc] | [prev] | [next] | [standalone]

#103377

On 2016-02-23, Steven D'Aprano <steve@pearwood.info> wrote:
> On Tue, 23 Feb 2016 06:22 am, Jon Ribbens wrote:
>> Suppose you had code like this:
>> 
>> filename = binascii.hexlify(os.urandom(16)).decode("ascii")
>> 
>> Do we really think that is insecure or that there are any practical
>> attacks against it? It would be basically the same as saying that
>> urandom() is broken, surely?
>
> Correct. Any attack against urandom would be an attack on this. You would
> just have to trust that the kernel devs have made urandom as secure as
> possible, and pay no attention to what the man page says, as its wrong.
>
> By the way, Python 3.6 will have (once Guido formally approves it) a new
> module, "secrets", for securely generating (pseudo)random tokens like this:
>
> import secrets
> filename = secrets.token_hex(16)

+1

[toc] | [prev] | [next] | [standalone]

#103388

Steven D'Aprano <steve@pearwood.info> writes:
> https://www.python.org/dev/peps/pep-0506/

I didn't know about this!  The discussion was all on mailing lists?

A few things I suggest changing:

  1) the default system RNG for Linux should be getrandom(2) on kernels
  that support it (3.17 and later).

  2) Some effort should be directed at simulating getrandom's behaviour
  on kernels that don't have it, using the /dev/random entropy estimator
  and the /dev/urandom interface.  I.e. it should block if the system
  hasn't seen enough entropy to get the CSPRNG started securely, and
  never block after that.

  3) The default token length should be long enough to not have to "change
  in the future".  If the user wants a shorter token, they ask for that,
  or can truncate a longer one that they receive from the default.

There are a few other choices in the PEP whose benefit is unclear to me,
but they aren't harmful, and I guess the decisions have already been
made.

[toc] | [prev] | [next] | [standalone]

#103389

On Tue, Feb 23, 2016 at 1:27 PM, Paul Rubin <no.email@nospam.invalid> wrote:
>   3) The default token length should be long enough to not have to "change
>   in the future".  If the user wants a shorter token, they ask for that,
>   or can truncate a longer one that they receive from the default.

How much future are you expecting?

ChrisA

[toc] | [prev] | [next] | [standalone]

#103390

Chris Angelico <rosuav@gmail.com> writes:
> How much future are you expecting?

This is old but its methodology still seems ok:

  http://saluc.engr.uconn.edu/refs/keymgr/blaze95minimalkeylength.pdf

I also like this:

  http://cr.yp.to/talks/2015.10.05/slides-djb-20151005-a4.pdf

Quote (slide 37): 

  The crypto users' fantasy is boring crypto: crypto that simply works,
  solidly resists attacks, never needs any upgrades.

HN discussion: https://news.ycombinator.com/item?id=10345965

[toc] | [prev] | [next] | [standalone]

#103395

On 23/02/2016 02:27, Paul Rubin wrote:
> Steven D'Aprano <steve@pearwood.info> writes:
>> https://www.python.org/dev/peps/pep-0506/
>
> I didn't know about this!  The discussion was all on mailing lists?

https://mail.python.org/pipermail/python-ideas/2015-September/036333.html then 
http://www.gossamer-threads.com/lists/python/dev/1223780

>
> A few things I suggest changing:
>
>    1) the default system RNG for Linux should be getrandom(2) on kernels
>    that support it (3.17 and later).
>
>    2) Some effort should be directed at simulating getrandom's behaviour
>    on kernels that don't have it, using the /dev/random entropy estimator
>    and the /dev/urandom interface.  I.e. it should block if the system
>    hasn't seen enough entropy to get the CSPRNG started securely, and
>    never block after that.
>
>    3) The default token length should be long enough to not have to "change
>    in the future".  If the user wants a shorter token, they ask for that,
>    or can truncate a longer one that they receive from the default.
>
> There are a few other choices in the PEP whose benefit is unclear to me,
> but they aren't harmful, and I guess the decisions have already been
> made.
>

The PEP status is draft so is subject to change.

-- 
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

[toc] | [prev] | [next] | [standalone]

#103396

Mark Lawrence <breamoreboy@yahoo.co.uk> writes:
> https://mail.python.org/pipermail/python-ideas/2015-September/036333.html
> then http://www.gossamer-threads.com/lists/python/dev/1223780

Thanks.  It would be nice if those were gatewayed to usenet like this
group is.  I can't bring myself to subscribe to mailing lists.

>> There are a few other choices in the PEP whose benefit is unclear to me,
>> but they aren't harmful, and I guess the decisions have already been
>> made.
> The PEP status is draft so is subject to change.

Well they might be changeable but it sounds like there's a level of
consensus by now, that wouldn't be helped by more bikeshedding over
relatively minor stuff.  I might write up some further comments and post
them here 

[toc] | [prev] | [next] | [standalone]

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

Back to top | Article view | comp.lang.python

csiph-web