Path: csiph.com!usenet.pasdenom.info!aioe.org!.POSTED!not-for-mail From: qwertmonkey@syberianoutpost.ru Newsgroups: comp.lang.java.programmer Subject: greatly differing processing time between java and Linux while calculating hashes? Date: Sun, 9 Sep 2012 04:40:28 +0000 (UTC) Organization: Aioe.org NNTP Server Lines: 276 Message-ID: NNTP-Posting-Host: z9Rq+Ge+SLJPEkk9TGcnaw.user.speranza.aioe.org X-Complaints-To: abuse@aioe.org X-Notice: Filtered by postfilter v. 0.8.2 X-Newsreader: NetComponents Xref: csiph.com comp.lang.java.programmer:18622 Roedy, I did play with the size of the input block used for the input stream and the files lenghts themselves and the results where the same, so 3 doesn't seem to be the case. ~ There must be something causing it, of course! but to me a difference of 25% (and the diff of more than 50% of the flip!) is way too much. Especially if it is an algorithm that MUST have been optimized (many libraries use them), besides those checksums algos, all they do is crunching data and they were developed and maintained by the same coders ~ $ md5sum --version md5sum (GNU coreutils) 8.13 Copyright (C) 2011 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Written by Ulrich Drepper, Scott Miller, and David Madore. $ sha512sum --version sha512sum (GNU coreutils) 8.13 Copyright (C) 2011 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Written by Ulrich Drepper, Scott Miller, and David Madore. ~ > I think we could use some URLs here ... ~ as I said: ~ youtube.com/watch?v=DQfUaXLk_sw ~ youtube.com/watch?v=0buBJlPo9us ~ you can go: ~ youtube-dl --no-overwrites --continue --no-progress --console-title --audio-format best --max-quality mp4 "http://www.youtube.com/watch?v=...&hd=1" ~ and here you have your cake ;-) ~ #!/bin/bash echo "// __ uname -a:" uname -a echo "// __ java -version:" java -version array=( DQfUaXLk_sw.mp4 0buBJlPo9us.flv ) for i in "${array[@]}" do ls -l $i echo "// __ checksumming using SHA-512" date; time java CheckSum00Test SHA-512 $i; date; date; time sha512sum -b $i; date; echo "~" echo "// __ checksumming using SHA-384" date; time java CheckSum00Test SHA-384 $i; date; date; time sha384sum -b $i; date; echo "~" echo "// __ checksumming using SHA-256" date; time java CheckSum00Test SHA-256 $i; date; date; time sha256sum -b $i; date; echo "~" echo "// __ checksumming using SHA-1" date; time java CheckSum00Test SHA-1 $i; date; date; time sha1sum -b $i; date; echo "~" echo "// __ checksumming using MD5" date; time java CheckSum00Test MD5 $i; date; date; time md5sum -b $i; date; echo "~" done ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ $ sh test_bash_java_dif00.sh // __ uname -a: Linux Microknoppix 3.3.7 #38 SMP PREEMPT Tue May 22 06:21:01 CEST 2012 i686 GNU/Linux // __ java -version: java version "1.7.0_07" Java(TM) SE Runtime Environment (build 1.7.0_07-b10) Java HotSpot(TM) Client VM (build 23.3-b01, mixed mode) -rw-r--r-- 1 knoppix knoppix 628588285 Apr 1 18:09 DQfUaXLk_sw.mp4 // __ checksumming using SHA-512 Sun Sep 9 00:20:10 UTC 2012 // __ SHA-512 encrypting "DQfUaXLk_sw.mp4": "f6f2f2b88faf3cef5a5de0cb0bc8f1ed15969490ef785673d3f7e325ab947c8df8b30edf663665 7e945b27eb5202300244b8acef874a11e630b2053f956982d8" took: 45386 (ms) real 0m45.659s user 0m43.291s sys 0m2.083s Sun Sep 9 00:20:55 UTC 2012 Sun Sep 9 00:20:55 UTC 2012 f6f2f2b88faf3cef5a5de0cb0bc8f1ed15969490ef785673d3f7e325ab947c8df8b30edf6636657 e945b27eb5202300244b8acef874a11e630b2053f956982d8 *DQfUaXLk_sw.mp4 real 1m2.410s user 1m1.336s sys 0m0.800s Sun Sep 9 00:21:58 UTC 2012 ~ // __ checksumming using SHA-384 Sun Sep 9 00:21:58 UTC 2012 // __ SHA-384 encrypting "DQfUaXLk_sw.mp4":"41d9cc52c2908f2b70f002fcf64a9365b4 3905173e37b2867e90d9ebb6f95033cd5f0a3126bd6021fa914ae7ba0d4f41" took: 45131 (ms) real 0m45.469s user 0m43.227s sys 0m1.920s Sun Sep 9 00:22:43 UTC 2012 Sun Sep 9 00:22:43 UTC 2012 41d9cc52c2908f2b70f002fcf64a9365b43905173e37b2867e90d9ebb6f95033cd5f0a3126bd602 1fa914ae7ba0d4f41 *DQfUaXLk_sw.mp4 real 1m2.677s user 1m1.476s sys 0m0.740s Sun Sep 9 00:23:46 UTC 2012 ~ // __ checksumming using SHA-256 Sun Sep 9 00:23:46 UTC 2012 // __ SHA-256 encrypting "DQfUaXLk_sw.mp4":"495656683bbf7d5cac6e5a48853bf18b6f 6e73a782bf0c65dc03bce0b5e814a5" took: 18655 (ms) real 0m18.978s user 0m17.409s sys 0m1.527s Sun Sep 9 00:24:05 UTC 2012 Sun Sep 9 00:24:05 UTC 2012 495656683bbf7d5cac6e5a48853bf18b6f6e73a782bf0c65dc03bce0b5e814a5 *DQfUaXLk_sw.mp4 real 0m8.509s user 0m8.093s sys 0m0.363s Sun Sep 9 00:24:13 UTC 2012 ~ // __ checksumming using SHA-1 Sun Sep 9 00:24:13 UTC 2012 // __ SHA-1 encrypting "DQfUaXLk_sw.mp4":"afc351c4ea13ac59c0181b6fd66ccb21c60b 0c48" took: 13338 (ms) real 0m13.606s user 0m12.113s sys 0m1.523s Sun Sep 9 00:24:27 UTC 2012 Sun Sep 9 00:24:27 UTC 2012 afc351c4ea13ac59c0181b6fd66ccb21c60b0c48 *DQfUaXLk_sw.mp4 real 0m4.287s user 0m3.990s sys 0m0.270s Sun Sep 9 00:24:31 UTC 2012 ~ // __ checksumming using MD5 Sun Sep 9 00:24:31 UTC 2012 // __ MD5 encrypting "DQfUaXLk_sw.mp4":"ecbcbe3d0a2fda4ebfddc41836cee528" took: 5450 (ms) real 0m5.729s user 0m4.123s sys 0m1.637s Sun Sep 9 00:24:37 UTC 2012 Sun Sep 9 00:24:37 UTC 2012 ecbcbe3d0a2fda4ebfddc41836cee528 *DQfUaXLk_sw.mp4 real 0m1.878s user 0m1.467s sys 0m0.367s Sun Sep 9 00:24:39 UTC 2012 ~ -rw-r--r-- 1 knoppix knoppix 475965918 Aug 31 2010 0buBJlPo9us.flv // __ checksumming using SHA-512 Sun Sep 9 00:24:39 UTC 2012 // __ SHA-512 encrypting "0buBJlPo9us.flv":"c77b1694fe82f14235a81f32201790a703 3a6d1600446017d66ba1f595ceb54960b3b2322546ee6676eb4f878dd8cd83b974738c560614d85 212bf83c4aee9f3" took: 34330 (ms) real 0m34.611s user 0m32.995s sys 0m1.480s Sun Sep 9 00:25:14 UTC 2012 Sun Sep 9 00:25:14 UTC 2012 c77b1694fe82f14235a81f32201790a7033a6d1600446017d66ba1f595ceb54960b3b2322546ee6 676eb4f878dd8cd83b974738c560614d85212bf83c4aee9f3 *0buBJlPo9us.flv real 0m47.409s user 0m46.517s sys 0m0.620s Sun Sep 9 00:26:01 UTC 2012 ~ // __ checksumming using SHA-384 Sun Sep 9 00:26:01 UTC 2012 // __ SHA-384 encrypting "0buBJlPo9us.flv":"cbf8770296c529f19786abac324b7a857e 2949ca5248d0fc45d767c27b323686279915da10d6710dfd9156f0fff0f30b" took: 34892 (ms) real 0m35.625s user 0m33.291s sys 0m1.767s Sun Sep 9 00:26:37 UTC 2012 Sun Sep 9 00:26:37 UTC 2012 cbf8770296c529f19786abac324b7a857e2949ca5248d0fc45d767c27b323686279915da10d6710 dfd9156f0fff0f30b *0buBJlPo9us.flv real 0m46.871s user 0m46.360s sys 0m0.350s Sun Sep 9 00:27:24 UTC 2012 ~ // __ checksumming using SHA-256 Sun Sep 9 00:27:24 UTC 2012 // __ SHA-256 encrypting "0buBJlPo9us.flv":"a49a51c7a3e1a5fcbe5bca5a7814697842 0fdbb2c70467873512fd215b31a644" took: 14187 (ms) real 0m14.790s user 0m13.106s sys 0m1.403s Sun Sep 9 00:27:39 UTC 2012 Sun Sep 9 00:27:39 UTC 2012 a49a51c7a3e1a5fcbe5bca5a78146978420fdbb2c70467873512fd215b31a644 *0buBJlPo9us.flv real 0m6.432s user 0m6.160s sys 0m0.250s Sun Sep 9 00:27:45 UTC 2012 ~ // __ checksumming using SHA-1 Sun Sep 9 00:27:45 UTC 2012 // __ SHA-1 encrypting "0buBJlPo9us.flv":"895c3a1c5979ab7d5bacd9eaafcadea40393 9da5" took: 10197 (ms) real 0m10.466s user 0m9.216s sys 0m1.287s Sun Sep 9 00:27:55 UTC 2012 Sun Sep 9 00:27:55 UTC 2012 895c3a1c5979ab7d5bacd9eaafcadea403939da5 *0buBJlPo9us.flv real 0m3.270s user 0m2.970s sys 0m0.280s Sun Sep 9 00:27:59 UTC 2012 ~ // __ checksumming using MD5 Sun Sep 9 00:27:59 UTC 2012 // __ MD5 encrypting "0buBJlPo9us.flv":"e96442f896fad41903016882b352fdd9" took: 4238 (ms) real 0m4.523s user 0m3.303s sys 0m1.223s Sun Sep 9 00:28:03 UTC 2012 Sun Sep 9 00:28:03 UTC 2012 e96442f896fad41903016882b352fdd9 *0buBJlPo9us.flv real 0m1.421s user 0m1.127s sys 0m0.267s Sun Sep 9 00:28:05 UTC 2012 ~