Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > sci.crypt > #43418 > unrolled thread

Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function?

Back to article view | Back to sci.crypt

Contents

  Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? "Karl.Frank" <Karl.Frank@Freecx.co.uk> - 2018-05-15 18:07 +0200
    Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? "Karl.Frank" <Karl.Frank@Freecx.co.uk> - 2018-05-15 18:31 +0200
    Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? Bruce Stephens <bruce.r.stephens@gmail.com> - 2018-05-15 19:46 +0100
      Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? "Karl.Frank" <Karl.Frank@Freecx.co.uk> - 2018-05-15 20:52 +0200
    Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? Rich <rich@example.invalid> - 2018-05-15 18:49 +0000
      Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? "Karl.Frank" <Karl.Frank@Freecx.co.uk> - 2018-05-15 21:02 +0200
    Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? wizzofozz <oscxxxxxxxaas@gmail.com> - 2018-05-15 21:09 +0200
      Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? wizzofozz <oscxxxxxxxaas@gmail.com> - 2018-05-15 21:11 +0200
        Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? "Karl.Frank" <Karl.Frank@Freecx.co.uk> - 2018-05-15 21:25 +0200
      Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? "Karl.Frank" <Karl.Frank@Freecx.co.uk> - 2018-05-15 21:29 +0200
        Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? wizzofozz <oscxxxxxxxaas@gmail.com> - 2018-05-15 22:07 +0200
        Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? Rich <rich@example.invalid> - 2018-05-15 20:38 +0000
          Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? "Karl.Frank" <Karl.Frank@Freecx.co.uk> - 2018-05-15 22:57 +0200
    Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? "Karl.Frank" <Karl.Frank@Freecx.co.uk> - 2018-05-15 21:42 +0200
    Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? "Karl.Frank" <Karl.Frank@Freecx.co.uk> - 2018-05-15 23:37 +0200
      Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? "Karl.Frank" <Karl.Frank@Freecx.co.uk> - 2018-05-16 15:58 +0200
      Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? Robert Klemme <shortcutter@googlemail.com> - 2018-06-18 18:41 +0200
        Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? "Chris M. Thomasson" <invalid_chris_thomasson@invalid.invalid> - 2018-06-18 13:33 -0700
        Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? "Karl.Frank" <Karl.Frank@Freecx.co.uk> - 2018-06-19 12:54 +0200
          Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? Lothar Kimmeringer <news201705@kimmeringer.de> - 2018-06-19 22:18 +0200
            Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? Robert Klemme <shortcutter@googlemail.com> - 2018-06-19 23:20 +0200
            Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? "Karl.Frank" <Karl.Frank@Freecx.co.uk> - 2018-06-19 23:23 +0200
          Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? Robert Klemme <shortcutter@googlemail.com> - 2018-06-19 23:15 +0200
            Re: Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function? "Karl.Frank" <Karl.Frank@Freecx.co.uk> - 2018-06-20 00:03 +0200

Page 1 of 2  [1] 2  Next page →

#43418 — Why is Java 7 converting the Password toCharArray instead of Byte in it's PBKDF2 function?

I've stuck with getting potential wrong results from a Java 7
implementation.

If I call a test function like

echo -n "Lsda[¥~§#€]" | java -jar pbkdf2_stdin.jar "Salt" 4096 16 sha-1

I get

Password   : 4c7364615bc2a57ec2a723e282ac5d
Salt       : 53616c74
PBKDF2     : aba223d6679ad1e8101c01bf860f788b
PBKDF2_java: 461f05924a7945a652bcabb30d87fad3

whereupon PBKDF2 is the result given be my own PBKDF2 implementation and
PBKDF2_java what I get using the javax library.

So I'm wondering if converting the Password into a char array instead of
a byte array might be the correct approach.

Verified the PBKDF2 result at 
https://asecuritysite.com/encryption/PBKDF2z and I get: 
aba223d6679ad1e8101c01bf860f788b

Can anyone give me a hint on this matter?




-- 
cHNiMUBACG0HAAAAAAAAAAAAAABIZVbDdKVM0w1kM9vxQHw+bkLxsY/Z0czY0uv8/Ks6WULxJVua
zjvpoYvtEwDVhP7RGTCBVlzZ+VBWPHg5rqmKWvtzsuVmMSDxAIS6Db6YhtzT+RStzoG9ForBcG8k
G97Q3Jml/aBun8Kyf+XOBHpl5gNW4YqhiM0=

[toc] | [next] | [standalone]

#43420

On 15.05.18 18:07, Karl.Frank wrote:
> I've stuck with getting potential wrong results from a Java 7
> implementation.
>
> If I call a test function like
>
> echo -n "Lsda[¥~§#€]" | java -jar pbkdf2_stdin.jar "Salt" 4096 16 sha-1
>
> I get
>
> Password : 4c7364615bc2a57ec2a723e282ac5d
> Salt : 53616c74
> PBKDF2 : aba223d6679ad1e8101c01bf860f788b
> PBKDF2_java: 461f05924a7945a652bcabb30d87fad3
>
> whereupon PBKDF2 is the result given be my own PBKDF2 implementation and
> PBKDF2_java what I get using the javax library.
>
> So I'm wondering if converting the Password into a char array instead of
> a byte array might be the correct approach.
>
> Verified the PBKDF2 result at
> https://asecuritysite.com/encryption/PBKDF2z and I get:
> aba223d6679ad1e8101c01bf860f788b
>
> Can anyone give me a hint on this matter?
>
>
>
>

I suppose it might be necessary to explain how I call the functions.

---------8<-----------------------------------------

   basicFunctions  bf = new basicFunctions();
   cryptoFunctions cf = new cryptoFunctions();

   if (System.in.available() > 0) {
     String hashType, PBKDF2_Result, Password, Salt;
     int dkLen, c;
     int BuffSize = System.in.available();

     byte[] pwByte = new byte[BuffSize];
     System.in.read(pwByte, 0, BuffSize);
     Password = bf.byteArrToString(pwByte);
     Salt     = argv[0];
     c        = Integer.parseInt(argv[1]);
     dkLen    = Integer.parseInt(argv[2]);
     hashType = argv[3];

     System.err.println("Password   : " + 
bf.byteArrToHexString(Password.getBytes()));
     System.err.println("Salt       : " + 
bf.byteArrToHexString(Salt.getBytes()));

     PBKDF2_Result = bf.byteArrToHexString( cf.PBKDF2(hashType, 
Password.getBytes(), Salt.getBytes(), c, dkLen) );
     System.err.println("PBKDF2     : " + PBKDF2_Result);

     PBKDF2_Result = bf.byteArrToHexString( pbkdf2_javax(hashType, 
Password, Salt, c, dkLen) );
     System.err.println("PBKDF2_java: " + PBKDF2_Result);

     }
   else {
     System.err.printf("Pass data via stdin.\n");
   }

--------->8-----------------------------------------

Inhere the password is read from stdin into a byte array (pwByte), then
converted into a string (Password), because the javax PBKDF2 function
does not accept a byte array for the password parameter, whereas it does
for the salt.

My own PBKDF2 implementation however is only accepting byte arrays for
password and salt, because I prefer staying on the byte level in regards
of cryptography.

bf.byteArrToString(), bf.byteArrToHexString() and cf.PBKDF2 are some
functions of my own functions lib that simply do as they are named.


In addtion this is the PBKDF2 function of javax

---------8<-----------------------------------------

   public static byte[] pbkdf2_javax(String hashType, String password, 
String salt, int iterations, int dkLen) {
     byte[] derivedKey = new byte[dkLen];

     if ( Integer.parseInt(hashType.substring(4)) > 1 ) return derivedKey;

     try {
       SecretKeyFactory f = 
SecretKeyFactory.getInstance("PBKDF2WithHmacSHA1");
       KeySpec ks  = new PBEKeySpec(password.toCharArray(), 
salt.getBytes(), iterations, dkLen*8);
       SecretKey s = f.generateSecret(ks);
       derivedKey  = s.getEncoded();
       }
     catch (Exception e) {
       e.printStackTrace();
     }

     return derivedKey;
   }

--------->8-----------------------------------------

For example calling the test program like below does give the correct 
result for both functions output.

echo -n "The quick brown fox jumps over the lazy dog" | java -jar 
pbkdf2_stdin.jar "Salt" 4096 16 sha-1

PBKDF2     : cf93699a512044c1bc788699cc986f88
PBKDF2_java: cf93699a512044c1bc788699cc986f88



-- 
cHNiMUBACG0HAAAAAAAAAAAAAABIZVbDdKVM0w1kM9vxQHw+bkLxsY/Z0czY0uv8/Ks6WULxJVua
zjvpoYvtEwDVhP7RGTCBVlzZ+VBWPHg5rqmKWvtzsuVmMSDxAIS6Db6YhtzT+RStzoG9ForBcG8k
G97Q3Jml/aBun8Kyf+XOBHpl5gNW4YqhiM0=

[toc] | [prev] | [next] | [standalone]

#43428

On 15/05/2018 17:07, Karl.Frank wrote:
> So I'm wondering if converting the Password into a char array instead of
> a byte array might be the correct approach.

Might be, but it'll depend on what's happening underneath, I think. I 
don't see a clear spec.

If you want something to compare your implementation against, you might 
try Bouncy Castle's instead. That explicitly provides 
PBKDF2WithHmacSHA1AndUTF8 and things. 
(https://www.bouncycastle.org/specifications.html)

Looks like an underspecified API, and as you say, would be better as 
bytes (with maybe options for passing UTF-8 char array for convenience).

[toc] | [prev] | [next] | [standalone]

#43430

On 15.05.18 20:46, Bruce Stephens wrote:
> On 15/05/2018 17:07, Karl.Frank wrote:
>> So I'm wondering if converting the Password into a char array instead of
>> a byte array might be the correct approach.
>
> Might be, but it'll depend on what's happening underneath, I think. I
> don't see a clear spec.
>
> If you want something to compare your implementation against, you might
> try Bouncy Castle's instead. That explicitly provides
> PBKDF2WithHmacSHA1AndUTF8 and things.
> (https://www.bouncycastle.org/specifications.html)
>
> Looks like an underspecified API, and as you say, would be better as
> bytes (with maybe options for passing UTF-8 char array for convenience).

All functions used in my test program are available here as a
none-bloated source code

/* Java Basic Functions Source Code */
http://paste.debian.net/1024821

/* Java Crypto Functions Source Code */
http://paste.debian.net/1024828



-- 
cHNiMUBACG0HAAAAAAAAAAAAAABIZVbDdKVM0w1kM9vxQHw+bkLxsY/Z0czY0uv8/Ks6WULxJVua
zjvpoYvtEwDVhP7RGTCBVlzZ+VBWPHg5rqmKWvtzsuVmMSDxAIS6Db6YhtzT+RStzoG9ForBcG8k
G97Q3Jml/aBun8Kyf+XOBHpl5gNW4YqhiM0=

[toc] | [prev] | [next] | [standalone]

#43429

Karl.Frank <Karl.Frank@freecx.co.uk> wrote:
> I've stuck with getting potential wrong results from a Java 7
> implementation.
> 
> If I call a test function like
> 
> echo -n "Lsda[¥~§#?]" | java -jar pbkdf2_stdin.jar "Salt" 4096 16 sha-1
> 
> I get
> 
> Password   : 4c7364615bc2a57ec2a723e282ac5d
> Salt       : 53616c74
> PBKDF2     : aba223d6679ad1e8101c01bf860f788b
> PBKDF2_java: 461f05924a7945a652bcabb30d87fad3
> 
> whereupon PBKDF2 is the result given be my own PBKDF2 implementation and
> PBKDF2_java what I get using the javax library.
> 
> So I'm wondering if converting the Password into a char array instead of
> a byte array might be the correct approach.
> 
> Verified the PBKDF2 result at 
> https://asecuritysite.com/encryption/PBKDF2z and I get: 
> aba223d6679ad1e8101c01bf860f788b
> 
> Can anyone give me a hint on this matter?

Gut feeling (note, I know zero java, and don't want to ever know any
java) -- character encoding issues.

You've got this character in your echo: "¥", but the bytes that end up
getting processed may be different due to character encodings (i.e.,
that character in the 8859 encoding is a different set of bytes than
that same character in the UTF-8 encoding.

[toc] | [prev] | [next] | [standalone]

#43431

On 15.05.18 20:49, Rich wrote:
> Karl.Frank<Karl.Frank@freecx.co.uk>  wrote:
>> I've stuck with getting potential wrong results from a Java 7
>> implementation.
>>
>> If I call a test function like
>>
>> echo -n "Lsda[¥~§#?]" | java -jar pbkdf2_stdin.jar "Salt" 4096 16 sha-1
>>
>> I get
>>
>> Password   : 4c7364615bc2a57ec2a723e282ac5d
>> Salt       : 53616c74
>> PBKDF2     : aba223d6679ad1e8101c01bf860f788b
>> PBKDF2_java: 461f05924a7945a652bcabb30d87fad3
>>
>> whereupon PBKDF2 is the result given be my own PBKDF2 implementation and
>> PBKDF2_java what I get using the javax library.
>>
>> So I'm wondering if converting the Password into a char array instead of
>> a byte array might be the correct approach.
>>
>> Verified the PBKDF2 result at
>> https://asecuritysite.com/encryption/PBKDF2z and I get:
>> aba223d6679ad1e8101c01bf860f788b
>>
>> Can anyone give me a hint on this matter?
>
> Gut feeling (note, I know zero java, and don't want to ever know any
> java) -- character encoding issues.
>
> You've got this character in your echo: "¥", but the bytes that end up
> getting processed may be different due to character encodings (i.e.,
> that character in the 8859 encoding is a different set of bytes than
> that same character in the UTF-8 encoding.

echo -n "Lsda[¥~§#€]" | xxd -g1
0000000: 4c 73 64 61 5b c2 a5 7e c2 a7 23 e2 82 ac 5d     Lsda[..~..#...]

it give the same byte output as with my test program.

Additionally I've added a line to visually verify that the conversion
from byte to string will print the same result

System.err.println("Password   : " + bf.byteArrToString(pwByte));

which will read

Password   : Lsda[¥~§#€]
Password   : 4c7364615bc2a57ec2a723e282ac5d
Salt       : 53616c74
PBKDF2     : aba223d6679ad1e8101c01bf860f788b
PBKDF2_java: 461f05924a7945a652bcabb30d87fad3

As it seem my functions are working correctly, but instead the javax
call of password.toCharArray() get it wrong.



-- 
cHNiMUBACG0HAAAAAAAAAAAAAABIZVbDdKVM0w1kM9vxQHw+bkLxsY/Z0czY0uv8/Ks6WULxJVua
zjvpoYvtEwDVhP7RGTCBVlzZ+VBWPHg5rqmKWvtzsuVmMSDxAIS6Db6YhtzT+RStzoG9ForBcG8k
G97Q3Jml/aBun8Kyf+XOBHpl5gNW4YqhiM0=

[toc] | [prev] | [next] | [standalone]

#43433

On 15-5-2018 18:07, Karl.Frank wrote:
> I've stuck with getting potential wrong results from a Java 7
> implementation.
> 
> If I call a test function like
> 
> echo -n "Lsda[¥~§#€]" | java -jar pbkdf2_stdin.jar "Salt" 4096 16 sha-1
>

If I do

wizzofozz@cuckoosnest:~$ echo -n 'Lsda[¥~§#.]' | hexdump -C
00000000  4c 73 64 61 5b a5 7e a7  23 2e 5d                 |Lsda[.~.#.]|

Compare with:

> I get
> 
> Password   : 4c7364615bc2a57ec2a723e282ac5d

You can see the interleaved c2's in your password bytes.

Not sure what's happening, but it seems character encoding related (to
UTF8). (http://www.fileformat.info/info/unicode/char/A5/index.htm)
But you probably already figured that out.
I would put more "byteArrToHexString" calls for debugging if I were you.

Cheers
Ozz

[toc] | [prev] | [next] | [standalone]

#43434

On 15-5-2018 21:09, wizzofozz wrote:
> On 15-5-2018 18:07, Karl.Frank wrote:
>> I've stuck with getting potential wrong results from a Java 7
>> implementation.
>>
>> If I call a test function like
>>

<snipped redundant stuff>

sorry, didn't see all the other posts which probably were posted when I
wrote the previous post.

Cheers
Ozz

[toc] | [prev] | [next] | [standalone]

#43435

On 15.05.18 21:11, wizzofozz wrote:
> On 15-5-2018 21:09, wizzofozz wrote:
>> On 15-5-2018 18:07, Karl.Frank wrote:
>>> I've stuck with getting potential wrong results from a Java 7
>>> implementation.
>>>
>>> If I call a test function like
>>>
>
> <snipped redundant stuff>
>
> sorry, didn't see all the other posts which probably were posted when I
> wrote the previous post.
>
> Cheers
> Ozz
>
Nothing to apologise for ;-)

It's some kind of a either a wired behaviour of the javax lib or some
kind of error in mine.



-- 
cHNiMUBACG0HAAAAAAAAAAAAAABIZVbDdKVM0w1kM9vxQHw+bkLxsY/Z0czY0uv8/Ks6WULxJVua
zjvpoYvtEwDVhP7RGTCBVlzZ+VBWPHg5rqmKWvtzsuVmMSDxAIS6Db6YhtzT+RStzoG9ForBcG8k
G97Q3Jml/aBun8Kyf+XOBHpl5gNW4YqhiM0=

[toc] | [prev] | [next] | [standalone]

#43436

On 15.05.18 21:09, wizzofozz wrote:
> On 15-5-2018 18:07, Karl.Frank wrote:
>> I've stuck with getting potential wrong results from a Java 7
>> implementation.
>>
>> If I call a test function like
>>
>> echo -n "Lsda[¥~§#€]" | java -jar pbkdf2_stdin.jar "Salt" 4096 16 sha-1
>>
>
> If I do
>
> wizzofozz@cuckoosnest:~$ echo -n 'Lsda[¥~§#.]' | hexdump -C
> 00000000  4c 73 64 61 5b a5 7e a7  23 2e 5d                 |Lsda[.~.#.]|
>
> Compare with:
>
>> I get
>>
>> Password   : 4c7364615bc2a57ec2a723e282ac5d
>
> You can see the interleaved c2's in your password bytes.
>
> Not sure what's happening, but it seems character encoding related (to
> UTF8). (http://www.fileformat.info/info/unicode/char/A5/index.htm)
> But you probably already figured that out.
> I would put more "byteArrToHexString" calls for debugging if I were you.
>
> Cheers
> Ozz
>

Just realised that your echo statement is missing the EUR symbol and
therefore your byte result is different than mine.

ist should read

echo -n 'Lsda[¥~§#<< EUR Symbol here >>]' | hexdump -C

and you'll get

00000000  4c 73 64 61 5b c2 a5 7e  c2 a7 23 e2 82 ac 5d 
|Lsda[..~..#...]|
0000000f



-- 
cHNiMUBACG0HAAAAAAAAAAAAAABIZVbDdKVM0w1kM9vxQHw+bkLxsY/Z0czY0uv8/Ks6WULxJVua
zjvpoYvtEwDVhP7RGTCBVlzZ+VBWPHg5rqmKWvtzsuVmMSDxAIS6Db6YhtzT+RStzoG9ForBcG8k
G97Q3Jml/aBun8Kyf+XOBHpl5gNW4YqhiM0=

[toc] | [prev] | [next] | [standalone]

#43440

On 15-5-2018 21:29, Karl.Frank wrote:
> On 15.05.18 21:09, wizzofozz wrote:
>> On 15-5-2018 18:07, Karl.Frank wrote:
>>> I've stuck with getting potential wrong results from a Java 7
> Just realised that your echo statement is missing the EUR symbol and
> therefore your byte result is different than mine.
> 
> ist should read
> 
> echo -n 'Lsda[¥~§#<< EUR Symbol here >>]' | hexdump -C
> 
> and you'll get
> 
> 00000000  4c 73 64 61 5b c2 a5 7e  c2 a7 23 e2 82 ac 5d |Lsda[..~..#...]|
> 0000000f
> 

You're right. I didn't notice that the euro wasn't copied.
Changed my settings in putty to UTF8, and now I get the same results.
I'll remember that for next time :-)

Ozz

[toc] | [prev] | [next] | [standalone]

#43441

Karl.Frank <Karl.Frank@freecx.co.uk> wrote:
> On 15.05.18 21:09, wizzofozz wrote:
>> On 15-5-2018 18:07, Karl.Frank wrote:
>>> I've stuck with getting potential wrong results from a Java 7
>>> implementation.
>>>
>>> If I call a test function like
>>>
>>> echo -n "Lsda[¥~§#?]" | java -jar pbkdf2_stdin.jar "Salt" 4096 16 sha-1
>>>
>>
>> If I do
>>
>> wizzofozz@cuckoosnest:~$ echo -n 'Lsda[¥~§#.]' | hexdump -C
>> 00000000  4c 73 64 61 5b a5 7e a7  23 2e 5d                 |Lsda[.~.#.]|
>>
>> Compare with:
>>
>>> I get
>>>
>>> Password   : 4c7364615bc2a57ec2a723e282ac5d
>>
>> You can see the interleaved c2's in your password bytes.
>>
>> Not sure what's happening, but it seems character encoding related (to
>> UTF8). (http://www.fileformat.info/info/unicode/char/A5/index.htm)
>> But you probably already figured that out.
>> I would put more "byteArrToHexString" calls for debugging if I were you.
>>
>> Cheers
>> Ozz
>>
> 
> Just realised that your echo statement is missing the EUR symbol and
> therefore your byte result is different than mine.
> 
> ist should read
> 
> echo -n 'Lsda[¥~§#<< EUR Symbol here >>]' | hexdump -C
> 
> and you'll get
> 
> 00000000  4c 73 64 61 5b c2 a5 7e  c2 a7 23 e2 82 ac 5d 
> |Lsda[..~..#...]|
> 0000000f

That "c2 a5" (and the c2 a7) is likely what is causing the issue.

c2 a5 is the UTF-8 encoding for the Yen symbol:
http://www.fileformat.info/info/unicode/char/a5/index.htm

It is how character "a5" is represented when encoded as UTF-8.

c2 a7 is the UTF-8 encoding for the section symbol:
http://www.fileformat.info/info/unicode/char/a7/index.htm

c2 a7 means display character "a7" on the display.

So, my guess at what is happening:

1) your code is performing a UTF-8 decode, and the bytes your code
feeds to the algorithm are a5 and a7.  Meanwhile the other Java code is
not performing a UTF-8 decode, and so the bytes it sees are c2 a5 and
c2 a7 and naturally a different result occurs.

or

2) your code is not performing a UTF-8 decode, while the Java library
/is/ performing a UTF-8 decode, and your code is seeing c2 a5 and c2 a7
where the Java library is seeing just a5 and a7.

[toc] | [prev] | [next] | [standalone]

#43443

On 15.05.18 22:38, Rich wrote:
> Karl.Frank<Karl.Frank@freecx.co.uk>  wrote:
>> On 15.05.18 21:09, wizzofozz wrote:
>>> On 15-5-2018 18:07, Karl.Frank wrote:
>>>> I've stuck with getting potential wrong results from a Java 7
>>>> implementation.
>>>>
>>>> If I call a test function like
>>>>
>>>> echo -n "Lsda[¥~§#?]" | java -jar pbkdf2_stdin.jar "Salt" 4096 16 sha-1
>>>>
>>>
>>> If I do
>>>
>>> wizzofozz@cuckoosnest:~$ echo -n 'Lsda[¥~§#.]' | hexdump -C
>>> 00000000  4c 73 64 61 5b a5 7e a7  23 2e 5d                 |Lsda[.~.#.]|
>>>
>>> Compare with:
>>>
>>>> I get
>>>>
>>>> Password   : 4c7364615bc2a57ec2a723e282ac5d
>>>
>>> You can see the interleaved c2's in your password bytes.
>>>
>>> Not sure what's happening, but it seems character encoding related (to
>>> UTF8). (http://www.fileformat.info/info/unicode/char/A5/index.htm)
>>> But you probably already figured that out.
>>> I would put more "byteArrToHexString" calls for debugging if I were you.
>>>
>>> Cheers
>>> Ozz
>>>
>>
>> Just realised that your echo statement is missing the EUR symbol and
>> therefore your byte result is different than mine.
>>
>> ist should read
>>
>> echo -n 'Lsda[¥~§#<<  EUR Symbol here>>]' | hexdump -C
>>
>> and you'll get
>>
>> 00000000  4c 73 64 61 5b c2 a5 7e  c2 a7 23 e2 82 ac 5d
>> |Lsda[..~..#...]|
>> 0000000f
>
> That "c2 a5" (and the c2 a7) is likely what is causing the issue.
>
> c2 a5 is the UTF-8 encoding for the Yen symbol:
> http://www.fileformat.info/info/unicode/char/a5/index.htm
>
> It is how character "a5" is represented when encoded as UTF-8.
>
> c2 a7 is the UTF-8 encoding for the section symbol:
> http://www.fileformat.info/info/unicode/char/a7/index.htm
>
> c2 a7 means display character "a7" on the display.
>
> So, my guess at what is happening:
>
> 1) your code is performing a UTF-8 decode, and the bytes your code
> feeds to the algorithm are a5 and a7.  Meanwhile the other Java code is
> not performing a UTF-8 decode, and so the bytes it sees are c2 a5 and
> c2 a7 and naturally a different result occurs.
>
> or
>
> 2) your code is not performing a UTF-8 decode, while the Java library
> /is/ performing a UTF-8 decode, and your code is seeing c2 a5 and c2 a7
> where the Java library is seeing just a5 and a7.

Thanks Rich, surely something I need to dig in.


-- 
cHNiMUBACG0HAAAAAAAAAAAAAABIZVbDdKVM0w1kM9vxQHw+bkLxsY/Z0czY0uv8/Ks6WULxJVua
zjvpoYvtEwDVhP7RGTCBVlzZ+VBWPHg5rqmKWvtzsuVmMSDxAIS6Db6YhtzT+RStzoG9ForBcG8k
G97Q3Jml/aBun8Kyf+XOBHpl5gNW4YqhiM0=

[toc] | [prev] | [next] | [standalone]

#43438

Here's the source code of test program in question

http://paste.debian.net/1024838


-- 
cHNiMUBACG0HAAAAAAAAAAAAAABIZVbDdKVM0w1kM9vxQHw+bkLxsY/Z0czY0uv8/Ks6WULxJVua
zjvpoYvtEwDVhP7RGTCBVlzZ+VBWPHg5rqmKWvtzsuVmMSDxAIS6Db6YhtzT+RStzoG9ForBcG8k
G97Q3Jml/aBun8Kyf+XOBHpl5gNW4YqhiM0=

[toc] | [prev] | [next] | [standalone]

#43444

Holy moley! It's a buggy Java Library I'm currently using!!

###

echo $LANG
en_US.UTF-8

echo -n '¥' | xxd -g1
0000000: c2 a5

###

java version "1.6.0_17"
Java(TM) SE Runtime Environment (build 1.6.0_17-b04-248-9M3125)
Java HotSpot(TM) 64-Bit Server VM (build 14.3-b01-101, mixed mode)

echo -n "Lsda[¥~§#€]" | java -jar pbkdf2_stdin.jar "Salt" 4096 16 sha-1
Password   : Lsda[¥~§#€]
Password   : 4c7364615bc2a57ec2a723e282ac5d
Salt       : 53616c74
PBKDF2     : aba223d6679ad1e8101c01bf860f788b
PBKDF2_java: 461f05924a7945a652bcabb30d87fad3

###

java -version
openjdk version "1.7.0-u80-unofficial"
OpenJDK Runtime Environment (build 1.7.0-u80-unofficial-b32)
OpenJDK 64-Bit Server VM (build 24.80-b11, mixed mode)

echo -n "Lsda[¥~§#€]" | java -jar pbkdf2_stdin.jar "Salt" 4096 16 sha-1
Password   : Lsda[¥~§#€]
Password   : 4c7364615bc2a57ec2a723e282ac5d
Salt       : 53616c74
PBKDF2     : aba223d6679ad1e8101c01bf860f788b
PBKDF2_java: aba223d6679ad1e8101c01bf860f788b

###

java -version
java version "1.6.0_41"
OpenJDK Runtime Environment (IcedTea6 1.13.13) (rhel-1.13.13.1.el6_8-x86_64)
OpenJDK 64-Bit Server VM (build 23.41-b41, mixed mode)

echo -n "Lsda[¥~§#€]" | java -jar pbkdf2_stdin.jar "Salt" 4096 16 sha-1
Password   : Lsda[¥~§#€]
Password   : 4c7364615bc2a57ec2a723e282ac5d
Salt       : 53616c74
PBKDF2     : aba223d6679ad1e8101c01bf860f788b
PBKDF2_java: aba223d6679ad1e8101c01bf860f788b

###

openjdk version "1.8.0_171"
OpenJDK Runtime Environment (build 1.8.0_171-b10)
OpenJDK 64-Bit Server VM (build 25.171-b10, mixed mode)

echo -n "Lsda[¥~§#€]" | java -jar pbkdf2_stdin.jar "Salt" 4096 16 sha-1
Password   : Lsda[¥~§#€]
Password   : 4c7364615bc2a57ec2a723e282ac5d
Salt       : 53616c74
PBKDF2     : aba223d6679ad1e8101c01bf860f788b
PBKDF2_java: aba223d6679ad1e8101c01bf860f788b


So my own implementation works correctly - what a relieve!


-- 
cHNiMUBACG0HAAAAAAAAAAAAAABIZVbDdKVM0w1kM9vxQHw+bkLxsY/Z0czY0uv8/Ks6WULxJVua
zjvpoYvtEwDVhP7RGTCBVlzZ+VBWPHg5rqmKWvtzsuVmMSDxAIS6Db6YhtzT+RStzoG9ForBcG8k
G97Q3Jml/aBun8Kyf+XOBHpl5gNW4YqhiM0=

[toc] | [prev] | [next] | [standalone]

#43458

On 15.05.18 23:37, Karl.Frank wrote:
> Holy moley! It's a buggy Java Library I'm currently using!!
>
> ###
>
> echo $LANG
> en_US.UTF-8
>
> echo -n '¥' | xxd -g1
> 0000000: c2 a5
>
> ###
>
> java version "1.6.0_17"
> Java(TM) SE Runtime Environment (build 1.6.0_17-b04-248-9M3125)
> Java HotSpot(TM) 64-Bit Server VM (build 14.3-b01-101, mixed mode)
>
> echo -n "Lsda[¥~§#€]" | java -jar pbkdf2_stdin.jar "Salt" 4096 16 sha-1
> Password : Lsda[¥~§#€]
> Password : 4c7364615bc2a57ec2a723e282ac5d
> Salt : 53616c74
> PBKDF2 : aba223d6679ad1e8101c01bf860f788b
> PBKDF2_java: 461f05924a7945a652bcabb30d87fad3
>

For God sake, all files encrypted by the application based on the buggy
Java version have a wrong encryption and MAC key now. So I can't
simply update the old application as expected. I can't believe how such
a Java version could ever being published at all!

A conclusion might be implementing all crypto primitives needed by a
certain program from scratch and run intense vector and other useful
tests in order to avoid such a mess in the future by any potential buggy
Java update.

I'm really pissed. :-(


-- 
cHNiMUBACG0HAAAAAAAAAAAAAABIZVbDdKVM0w1kM9vxQHw+bkLxsY/Z0czY0uv8/Ks6WULxJVua
zjvpoYvtEwDVhP7RGTCBVlzZ+VBWPHg5rqmKWvtzsuVmMSDxAIS6Db6YhtzT+RStzoG9ForBcG8k
G97Q3Jml/aBun8Kyf+XOBHpl5gNW4YqhiM0=

[toc] | [prev] | [next] | [standalone]

#43711

On 15.05.2018 23:37, Karl.Frank wrote:
> Holy moley! It's a buggy Java Library I'm currently using!!

You are converting a byte array to a String without explicit mention of 
the encoding.  Maybe the way Java picks up the default encoding has 
changed between versions.  Anyway, this is an unsafe practice.

Also, you are reading only once, there is the chance that there is still 
data in the pipe which you will miss.  Even worse, depending on how data 
is written to stdin you might even get different results.  This might 
not be an issue for your test program, but could be if you use this code 
in production.

Btw. I find your habit of using names beginning with an uppercase letter 
for variables confusing.  Conventionally these are reserved for classes 
and interfaces.

> So my own implementation works correctly - what a relieve!

There are cases where wrong code emits correct results - and this 
surfaces only later, when the input or some other environmental factor 
changes.

Regards

	robert


-- 
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

[toc] | [prev] | [next] | [standalone]

#43714

On 6/18/2018 9:41 AM, Robert Klemme wrote:
> On 15.05.2018 23:37, Karl.Frank wrote:
>> Holy moley! It's a buggy Java Library I'm currently using!!
> 
> You are converting a byte array to a String without explicit mention of 
> the encoding.  Maybe the way Java picks up the default encoding has 
> changed between versions.  Anyway, this is an unsafe practice.
[...]

Yup. For instance, I had to alter my Python test vector to use 
bytearrays without default encoding in order for it go get in perfect 
sync with a C analog.

https://groups.google.com/forum/#!original/sci.crypt/2K33_4GjkVs/Ws6bWE-aAQAJ
(read all for context...)

The C code:

https://pastebin.com/raw/feUnA3kP

The Python Test vector:

https://pastebin.com/raw/NAnsBJAZ

Notice the following code in the Python version:
___________________________________
if (M == False):
     U = bytearray([P_byte, C_I_P]);
     H.update(U);
else:
     U = bytearray([C_I_P, P_byte]);
     H.update(U);
___________________________________

Here is the equivalent C code in the ct_crypt_round function:
___________________________________
if (M == 0)
{
     unsigned char update[2] = {
         P_byte, C_byte
     };

     hmac_sha256_update(&H, update, 2);
}

else
{
     unsigned char update[2] = {
         C_byte, P_byte
     };

     hmac_sha256_update(&H, update, 2);
}
___________________________________


Now, there is a way to get a different result in the Python test 
vector... Use the default encoding! Afaict, it can change on different 
impls. Example:
___________________________________
if (M == False):
     H.update(P.bytes[I_P].encode());
     H.update(chr(C_I_P).encode());
else:
     H.update(chr(C_I_P).encode());
     H.update(P.bytes[I_P].encode());
___________________________________

NO! This is unsafe. I need to correct my code in my paper:

http://funwithfractals.atspace.cc/ct_cipher/

that uses the .encode() function to the one that uses bytearrays. Damn 
it.. Sorry!

I had a weary feeling about using default encoding, but finally verified 
it when I worked on getting the C version and Python test vector in 
perfect sync. Please keep in mind that one needs to define 
PYTHON_TEST_VECTOR in my C code to sync it up with the Python test vector.

[toc] | [prev] | [next] | [standalone]

#43716

On 18.06.18 18:41, Robert Klemme wrote:
> On 15.05.2018 23:37, Karl.Frank wrote:
>> Holy moley! It's a buggy Java Library I'm currently using!!
>
> You are converting a byte array to a String without explicit mention of
> the encoding. Maybe the way Java picks up the default encoding has
> changed between versions. Anyway, this is an unsafe practice.
>
Robert, probably that might be the case.

In regards of cryptography I always tend to stay on the byte level. This
way my own implementation of HMAC and PBKDF2 has no need to bother with
string encoding when reading in from stdin or files. Curiously Java's
PBKDF2 expect a string or char array rather than a byte array and
therefore I need the extra conversion. Actually from now on my own
functions will be used instead.

It might be a different matter if the user type in password and/or salt
in some GUI. In this case however I would define the kind of strings
accepted and later convert them into byte arrays before processing them
further.


> Also, you are reading only once, there is the chance that there is still
> data in the pipe which you will miss. Even worse, depending on how data
> is written to stdin you might even get different results. This might not
> be an issue for your test program, but could be if you use this code in
> production.
>
Well, yes I read only once from stdin and store the result in a byte
array. How do you suppose to read twice?

The amount of byte to read is exactly defined by

   int BuffSize = System.in.available();
   byte[] pwByte = new byte[BuffSize];
   System.in.read(pwByte, 0, BuffSize);

How would the program read more byte than necessary?

As data on stdin is always byte and all my programs read in byte and
never ever strings in my opinion it doesn't matter how these byte are
written and piped by any previous instance or program.


> Btw. I find your habit of using names beginning with an uppercase letter
> for variables confusing. Conventionally these are reserved for classes
> and interfaces.
>
Yeah, it's old habit. Manly because I do not program much in Java
actually. What I do is translating my important crypto functions for
example into Java or Python in order to check that they will produce the
exact same results and comply with given test vectors.


>> So my own implementation works correctly - what a relieve!
>
> There are cases where wrong code emits correct results - and this
> surfaces only later, when the input or some other environmental factor
> changes.
>
That's one reason why I tend to run large and extensive tests before my
program go into a production environment. Until today I've never
experienced any real problem. Even in this case I'd implement my own
HMAC and PBKDF2 and therefore found the "buggy" Java lib. Sadly the
developer of the original program some years ago didn't check that and
now everyone is baffled that we can't decrypt old files with the
correctly working PBKDF2 implementation.


> Regards
>
> robert
>
>
Cheers,
Karl



-- 
cHNiMUBACG0HAAAAAAAAAAAAAABIZVbDdKVM0w1kM9vxQHw+bkLxsY/Z0czY0uv8/Ks6WULxJVua
zjvpoYvtEwDVhP7RGTCBVlzZ+VBWPHg5rqmKWvtzsuVmMSDxAIS6Db6YhtzT+RStzoG9ForBcG8k
G97Q3Jml/aBun8Kyf+XOBHpl5gNW4YqhiM0=

[toc] | [prev] | [next] | [standalone]

#43717

Karl.Frank wrote:

> The amount of byte to read is exactly defined by
> 
>    int BuffSize = System.in.available();
>    byte[] pwByte = new byte[BuffSize];
>    System.in.read(pwByte, 0, BuffSize);
> 
> How would the program read more byte than necessary?

InputStream.available only returns the number of bytes that
can be read without blocking. Dependent on how the data ends
up there there still might be data left to read that you
missed because of that.

What you're doing here is a common programming error.


Cheers, Lothar
-- 
Lothar Kimmeringer                E-Mail: spamfang@kimmeringer.de
               PGP-encrypted mails preferred (Key-ID: 0x8BC3CD81)

Always remember: The answer is forty-two, there can only be wrong
                 questions!

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | sci.crypt

csiph-web