Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > sci.crypt > #43418 > unrolled thread

Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function?

Back to article view | Back to sci.crypt

Contents

  Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? "Karl.Frank" <Karl.Frank@Freecx.co.uk> - 2018-05-15 18:07 +0200
    Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? "Karl.Frank" <Karl.Frank@Freecx.co.uk> - 2018-05-15 18:31 +0200
    Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? Bruce Stephens <bruce.r.stephens@gmail.com> - 2018-05-15 19:46 +0100
      Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? "Karl.Frank" <Karl.Frank@Freecx.co.uk> - 2018-05-15 20:52 +0200
    Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? Rich <rich@example.invalid> - 2018-05-15 18:49 +0000
      Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? "Karl.Frank" <Karl.Frank@Freecx.co.uk> - 2018-05-15 21:02 +0200
    Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? wizzofozz <oscxxxxxxxaas@gmail.com> - 2018-05-15 21:09 +0200
      Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? wizzofozz <oscxxxxxxxaas@gmail.com> - 2018-05-15 21:11 +0200
        Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? "Karl.Frank" <Karl.Frank@Freecx.co.uk> - 2018-05-15 21:25 +0200
      Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? "Karl.Frank" <Karl.Frank@Freecx.co.uk> - 2018-05-15 21:29 +0200
        Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? wizzofozz <oscxxxxxxxaas@gmail.com> - 2018-05-15 22:07 +0200
        Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? Rich <rich@example.invalid> - 2018-05-15 20:38 +0000
          Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? "Karl.Frank" <Karl.Frank@Freecx.co.uk> - 2018-05-15 22:57 +0200
    Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? "Karl.Frank" <Karl.Frank@Freecx.co.uk> - 2018-05-15 21:42 +0200
    Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? "Karl.Frank" <Karl.Frank@Freecx.co.uk> - 2018-05-15 23:37 +0200
      Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? "Karl.Frank" <Karl.Frank@Freecx.co.uk> - 2018-05-16 15:58 +0200
      Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? Robert Klemme <shortcutter@googlemail.com> - 2018-06-18 18:41 +0200
        Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? "Chris M. Thomasson" <invalid_chris_thomasson@invalid.invalid> - 2018-06-18 13:33 -0700
        Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? "Karl.Frank" <Karl.Frank@Freecx.co.uk> - 2018-06-19 12:54 +0200
          Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? Lothar Kimmeringer <news201705@kimmeringer.de> - 2018-06-19 22:18 +0200
            Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? Robert Klemme <shortcutter@googlemail.com> - 2018-06-19 23:20 +0200
            Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? "Karl.Frank" <Karl.Frank@Freecx.co.uk> - 2018-06-19 23:23 +0200
          Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? Robert Klemme <shortcutter@googlemail.com> - 2018-06-19 23:15 +0200
            Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? "Karl.Frank" <Karl.Frank@Freecx.co.uk> - 2018-06-20 00:03 +0200

Page 2 of 2 — ← Prev page 1 [2]

#43719

On 19.06.2018 22:18, Lothar Kimmeringer wrote:
> Karl.Frank wrote:
> 
>> The amount of byte to read is exactly defined by
>>
>>     int BuffSize = System.in.available();
>>     byte[] pwByte = new byte[BuffSize];
>>     System.in.read(pwByte, 0, BuffSize);
>>
>> How would the program read more byte than necessary?
> 
> InputStream.available only returns the number of bytes that
> can be read without blocking.

It returns an _estimate_ of that number.  A conforming implementation is 
even allowed to sometimes or always return 0.  And in fact this is what 
InputStream's own implementation does.

> Dependent on how the data ends
> up there there still might be data left to read that you
> missed because of that.
> 
> What you're doing here is a common programming error.

I hope not, but may be.

Cheers

	robert

-- 
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

[toc] | [prev] | [next] | [standalone]

#43720

On 19.06.18 22:18, Lothar Kimmeringer wrote:
> Karl.Frank wrote:
>
>> The amount of byte to read is exactly defined by
>>
>>     int BuffSize = System.in.available();
>>     byte[] pwByte = new byte[BuffSize];
>>     System.in.read(pwByte, 0, BuffSize);
>>
>> How would the program read more byte than necessary?
>
> InputStream.available only returns the number of bytes that
> can be read without blocking. Dependent on how the data ends
> up there there still might be data left to read that you
> missed because of that.
>
> What you're doing here is a common programming error.
>
>
> Cheers, Lothar

Well, well, well. The posted listing is just a short test program in
order to figure out why the results of the Java function were wrong.
It clearly does not expext a huge file being passed over by stdin.

Obviously in a program that need to handle large scale data one would
read in the data byte in chunks and process them subsequently like in
this short snipped


---------------------------------------8<----------

   int BuffSize = 8192;

   byte[] inByte  = new byte[BuffSize];
   byte[] outByte = new byte[BuffSize];

   int inB, outB, bufferRead;

   System.in.flush();
   System.out.flush();

   while ((bufferRead = System.in.read(inByte, 0, BuffSize)) != -1) {
     for (int i=0; i<bufferRead; i++){
       /*
         do something useful here
         for example XOR the byte

         inB  = (int)inByte[i];
         outB = (inB ^ *some CSPRNG byte output*) & 0xff;
         outByte[i] = (byte)outB;

         or similar...
       */
     }
     System.out.write(outByte, 0, bufferRead);
   }

   System.in.flush();
   System.out.flush();

   System.in.close();
   System.out.close();

--------------------------------------->8----------

So nothing to be worried about in regards of any unprocessed data left ;-)

Cheers,
Karl


-- 
cHNiMUBACG0HAAAAAAAAAAAAAABIZVbDdKVM0w1kM9vxQHw+bkLxsY/Z0czY0uv8/Ks6WULxJVua
zjvpoYvtEwDVhP7RGTCBVlzZ+VBWPHg5rqmKWvtzsuVmMSDxAIS6Db6YhtzT+RStzoG9ForBcG8k
G97Q3Jml/aBun8Kyf+XOBHpl5gNW4YqhiM0=

[toc] | [prev] | [next] | [standalone]

#43718

On 19.06.2018 12:54, Karl.Frank wrote:
> On 18.06.18 18:41, Robert Klemme wrote:
>> On 15.05.2018 23:37, Karl.Frank wrote:
>>> Holy moley! It's a buggy Java Library I'm currently using!!
>>
>> You are converting a byte array to a String without explicit mention of
>> the encoding. Maybe the way Java picks up the default encoding has
>> changed between versions. Anyway, this is an unsafe practice.
>>
> Robert, probably that might be the case.

It is, it is.

> In regards of cryptography I always tend to stay on the byte level. This
> way my own implementation of HMAC and PBKDF2 has no need to bother with
> string encoding when reading in from stdin or files. Curiously Java's
> PBKDF2 expect a string or char array rather than a byte array and
> therefore I need the extra conversion. Actually from now on my own
> functions will be used instead.

The point is not your usage of byte array nor your need for a 
conversion.  The point is that your conversion is unsafe because the 
default encoding is used implicitly instead of making your choice of 
encoding explicit.  There are various factors that might influence the 
JVM's choice of default encoding and they might not be the same as 
govern the encoding of the user input (what you send via echo).  This is 
a source of errors and the bad thing is it might actually work in one 
environment - but not in others.  Such bugs can go unnoticed for a long 
time before they hit you.  And then you're left with wrongly encoded 
data which can be very painful.

> It might be a different matter if the user type in password and/or salt
> in some GUI. In this case however I would define the kind of strings
> accepted and later convert them into byte arrays before processing them
> further.

Graphical inputs usually emit String, char[] or CharSequence so you can 
be sure the input is properly encoded (modulo some issues) and you can 
pick whatever encoding you want for the transformation into bytes.

>> Also, you are reading only once, there is the chance that there is still
>> data in the pipe which you will miss. Even worse, depending on how data
>> is written to stdin you might even get different results. This might not
>> be an issue for your test program, but could be if you use this code in
>> production.
>>
> Well, yes I read only once from stdin and store the result in a byte
> array. How do you suppose to read twice?
> 
> The amount of byte to read is exactly defined by
> 
>    int BuffSize = System.in.available();
>    byte[] pwByte = new byte[BuffSize];
>    System.in.read(pwByte, 0, BuffSize);
> 
> How would the program read more byte than necessary?

Please read carefully: I wrote it might read _less_ bytes.  You should 
really read the JavaDocs of InputStream#available().  For example it is 
perfectly legal for an implementation to always return 0 yet still yield 
data via read().  Basically, available() is quite useless most of the 
time if you want to read the complete input anyway.

> As data on stdin is always byte and all my programs read in byte and
> never ever strings in my opinion it doesn't matter how these byte are
> written and piped by any previous instance or program.

Again, the point is a different one: your reading does not ensure you 
get the complete input.  For this you need a loop that reads until EOF. 
Using just a single read() is not sufficient as an implementation is 
free to choose how many bytes to return for a single read().

There is no way you can get away without a loop.

final byte[] buffer = new byte[1024];
final ByteArrayOutputStream out = new ByteArrayOutputStream();

for (int len; (len = in.read(buffer)) != -1;) {
     out.write(buffer, 0, len);
}

byte[] data = out.toByteArray();

There are other more elegant variants depending on which libs you use 
and on which Java version you are.

>>> So my own implementation works correctly - what a relieve!
>>
>> There are cases where wrong code emits correct results - and this
>> surfaces only later, when the input or some other environmental factor
>> changes.
>>
> That's one reason why I tend to run large and extensive tests before my
> program go into a production environment. Until today I've never
> experienced any real problem. Even in this case I'd implement my own
> HMAC and PBKDF2 and therefore found the "buggy" Java lib. Sadly the
> developer of the original program some years ago didn't check that and
> now everyone is baffled that we can't decrypt old files with the
> correctly working PBKDF2 implementation.

I am not convinced you have identified a "'buggy' Java lib".

Kind regards

	robert

-- 
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

[toc] | [prev] | [next] | [standalone]

#43721

On 19.06.18 23:15, Robert Klemme wrote:
> On 19.06.2018 12:54, Karl.Frank wrote:
>> On 18.06.18 18:41, Robert Klemme wrote:
>>> On 15.05.2018 23:37, Karl.Frank wrote:
>>>> Holy moley! It's a buggy Java Library I'm currently using!!
>>>
>>> You are converting a byte array to a String without explicit mention of
>>> the encoding. Maybe the way Java picks up the default encoding has
>>> changed between versions. Anyway, this is an unsafe practice.
>>>
>> Robert, probably that might be the case.
>
> It is, it is.
>
>> In regards of cryptography I always tend to stay on the byte level. This
>> way my own implementation of HMAC and PBKDF2 has no need to bother with
>> string encoding when reading in from stdin or files. Curiously Java's
>> PBKDF2 expect a string or char array rather than a byte array and
>> therefore I need the extra conversion. Actually from now on my own
>> functions will be used instead.
>
> The point is not your usage of byte array nor your need for a
> conversion. The point is that your conversion is unsafe because the
> default encoding is used implicitly instead of making your choice of
> encoding explicit. There are various factors that might influence the
> JVM's choice of default encoding and they might not be the same as
> govern the encoding of the user input (what you send via echo). This is
> a source of errors and the bad thing is it might actually work in one
> environment - but not in others. Such bugs can go unnoticed for a long
> time before they hit you. And then you're left with wrongly encoded data
> which can be very painful.
>
Whatever comes in from stdin has to be handled and converted by the
piping program as the reading program normally does expect byte - so is
my understanding and experience in the past with C, Python and other
languages.

In fact my test program get the byte properly from stdin because they
are converted by the piping program - in case of the example by "echo".

Due to the fact that the PBKDF2 function of Java expect a string which
has to be converted to a char array for the password (but a byte array
for the salt) I need to convert the byte array first. Unclear for me
however why the mixup of char array and byte array is necessary here.
But to be honest, I'm not really keen to dig deeper because my own
implementations are working properly.


   public static String byteArrToString(byte[] byteArr) {
     String bs = new String(byteArr);

     return bs;
   }

   Password = byteArrToString(pwByte);

And here it comes:

certainly I do *not* know in advance what kind of string conversion to 
choose in the byteArrToString() function simply because I do not know 
what kind of data the piping program might pass over.

For that reason I'll *always* read in byte and never ever anything else, 
because one thing I can be sure, the piped data will be byte. So never 
ever read in data as strings or what ever else crude format they might be.


>> It might be a different matter if the user type in password and/or salt
>> in some GUI. In this case however I would define the kind of strings
>> accepted and later convert them into byte arrays before processing them
>> further.
>
> Graphical inputs usually emit String, char[] or CharSequence so you can
> be sure the input is properly encoded (modulo some issues) and you can
> pick whatever encoding you want for the transformation into bytes.
>
That's what I prefer. If I'd had to read in user typed data I define the 
encoding in the GUI forms input field.


>>> Also, you are reading only once, there is the chance that there is still
>>> data in the pipe which you will miss. Even worse, depending on how data
>>> is written to stdin you might even get different results. This might not
>>> be an issue for your test program, but could be if you use this code in
>>> production.
>>>
>> Well, yes I read only once from stdin and store the result in a byte
>> array. How do you suppose to read twice?
>>
>> The amount of byte to read is exactly defined by
>>
>>   int BuffSize = System.in.available();
>>   byte[] pwByte = new byte[BuffSize];
>>   System.in.read(pwByte, 0, BuffSize);
>>
>> How would the program read more byte than necessary?
>
> Please read carefully: I wrote it might read _less_ bytes. You should
> really read the JavaDocs of InputStream#available(). For example it is
> perfectly legal for an implementation to always return 0 yet still yield
> data via read(). Basically, available() is quite useless most of the
> time if you want to read the complete input anyway.
>
Right, missed that. Explained that in another answer in this thread
already. In case of the test program it seems fairly sufficient to use
System.in.available().


>> As data on stdin is always byte and all my programs read in byte and
>> never ever strings in my opinion it doesn't matter how these byte are
>> written and piped by any previous instance or program.
>
> Again, the point is a different one: your reading does not ensure you
> get the complete input. For this you need a loop that reads until EOF.
> Using just a single read() is not sufficient as an implementation is
> free to choose how many bytes to return for a single read().
>
> There is no way you can get away without a loop.
>
> final byte[] buffer = new byte[1024];
> final ByteArrayOutputStream out = new ByteArrayOutputStream();
>
> for (int len; (len = in.read(buffer)) != -1;) {
> out.write(buffer, 0, len);
> }
>
> byte[] data = out.toByteArray();
>
> There are other more elegant variants depending on which libs you use
> and on which Java version you are.
>
To be honest again, all these different versions and libs coming with
Java are a real pain in the ass. And exactly these incompatibilities are
the reason why I try to rely as much as possible only on the bare
minimum of Java's inherent commands and implement critical functions on
my own in pure Java.


>>>> So my own implementation works correctly - what a relieve!
>>>
>>> There are cases where wrong code emits correct results - and this
>>> surfaces only later, when the input or some other environmental factor
>>> changes.
>>>
>> That's one reason why I tend to run large and extensive tests before my
>> program go into a production environment. Until today I've never
>> experienced any real problem. Even in this case I'd implement my own
>> HMAC and PBKDF2 and therefore found the "buggy" Java lib. Sadly the
>> developer of the original program some years ago didn't check that and
>> now everyone is baffled that we can't decrypt old files with the
>> correctly working PBKDF2 implementation.
>
> I am not convinced you have identified a "'buggy' Java lib".
>
Well, I am.

> Kind regards
>
> robert
>

Cheers,
Karl


-- 
cHNiMUBACG0HAAAAAAAAAAAAAABIZVbDdKVM0w1kM9vxQHw+bkLxsY/Z0czY0uv8/Ks6WULxJVua
zjvpoYvtEwDVhP7RGTCBVlzZ+VBWPHg5rqmKWvtzsuVmMSDxAIS6Db6YhtzT+RStzoG9ForBcG8k
G97Q3Jml/aBun8Kyf+XOBHpl5gNW4YqhiM0=

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | sci.crypt

csiph-web