Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.os.linux.advocacy > #354064

Re: man, am I sick of md5

Show all headers | View raw

owl <owl@rooftop.invalid> wrote:
> vallor <vallor@cultnix.org> wrote:
>> On Sat, 07 May 2016 12:30:01 +0000, meat wrote:
>> 
>>> On Sat, 07 May 2016 02:59:11 +0000, vallor wrote:
>>> 
>>>> On Sat, 07 May 2016 02:40:04 +0000, vallor wrote:
>>>>> But look at that bootiful sorted data:
>>>>> 
>>>>> $ head -10 md5_ipv4_rainbow.hex
>>>>> 0000000191fab544b163e31753366fae9c8380c2
>>>>> 0000000384db5f2b13aae0c719c21aa002e6c0b2
>>>> 
>>>>> fffffffcfef441108d45e6ea4c38b0a7c3fe66b0
>>>> 
>>>> I just showed this to someone looking over my shoulder, and they said,
>>>> "You mean to tell me you have a file with all the Internet addresses in
>>>> the world?  Like, even foreign countries?"
>>> 
>>> That's what I was thinking!
>>> I suspect it's human nature to assume something like that would require
>>> the NSA and a super computer to churn out. Kind of like calculating pi
>>> to the 1000000 trillionth digit or something.
>>> 
>>> It kind of puts a new face on government spying doesn't it?
>> 
>> BTW, if you've done the math on my file size, you'll discover that there 
>> are 16 records missing.  They are for the following IP addresses:
>> 
>> 0.0.0.0, 28.0.0.0, 42.0.0.0, 56.0.0.0,
>> 70.0.0.0, 84.0.0.0, 98.0.0.0, 112.0.0.0,
>> 126.0.0.0, 140.0.0.0, 154.0.0.0, 168.0.0.0,
>> 182.0.0.0, 196.0.0.0, 210.0.0.0, 224.0.0.0
>> 
>> It was an off-by-one error in the shell script that I used to run the C 
>> program to generate each block.  (I used 16 processors on my video 
>> superserver.)
>> 
>> I'm probably going to just hard-code these hashes into the search 
>> program, because I don't want to re-generate the files, convert them, 
>> sort them, merge them, then convert them back to binary...it takes a 
>> while, as you can imagine.
>> 
> 
> I'm decided to try a different approach today.  I set up a btrfs
> partition.  btrfs does not suffer the inode limit that you get with ext4,
> so billions of directories should be doable.  It takes a bit of time to
> generate the tree -- I'm guessing about 1.7 hours per /8 -- but seek times
> seem to be very fast with testing so far.  I pulled a recursive mkdir()
> function from the web and merged it to my existing code.  The ip address
> for a given hash will end up as a node directory.
> 
> anon@lowtide:/media/anon/f617b3c2-e000-437f-9c75-04474c001d92/ips/ipdir$ echo -n 0.3.100.198 | md5sum
> ffff32641ebadad6de6b77dc19699e82  -
> anon@lowtide:/media/anon/f617b3c2-e000-437f-9c75-04474c001d92/ips/ipdir$ time ls f/f/f/f/3/2/6/4/1/e/b/a/d/a/d/6/d/e/6/b/7/7/d/c/1/9/6/9/9/e/8/2
> 0.3.100.198
> 
> real    0m0.003s
> user    0m0.000s
> sys     0m0.000s
> anon@lowtide:/media/anon/f617b3c2-e000-437f-9c75-04474c001d92/ips/ipdir$
> 
> This was not with the whole tree built yet.  I'm still waiting on the
> first /8 to complete to get a better idea of the total build time.
> 

Damn.  It's been running for quite a while now creating tree for
0.0.0.0-0.255.255.255.  Just now hit 0.9.0.0 level, but speeds are
still holding pretty good.

anon@lowtide:/media/anon/f617b3c2-e000-437f-9c75-04474c001d92/ips/ipdir$ echo -n 0.9.0.0 | md5sum
14637e918fe0e9edd63634b57cfe809b  -
anon@lowtide:/media/anon/f617b3c2-e000-437f-9c75-04474c001d92/ips/ipdir$ time ./doit.sh 14637e918fe0e9edd63634b57cfe809b
0.9.0.0

real	0m0.005s
user	0m0.000s
sys	0m0.000s
anon@lowtide:/media/anon/f617b3c2-e000-437f-9c75-04474c001d92/ips/ipdir$ 

This is doit.sh:

#!/bin/bash

if [ ${#} -ne 1 ];then
  echo "need a hash"
  exit
fi

ls $(./haship2dir ${1})

--------
haship2dir just creates the directory from the provided hash.
At this rate, I think we're back to the multiple weeks for tree
generation, but "decode" speeds are near instantaneous, at least
so far.  And all this assumes btrfs doesn't crap out before it's
done.

Back to comp.os.linux.advocacy | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Re: man, am I sick of md5 meat <meat.stretcher@linuxmail.org> - 2016-05-07 12:30 +0000
  Re: man, am I sick of md5 vallor <vallor@cultnix.org> - 2016-05-07 18:17 +0000
    Re: man, am I sick of md5 owl <owl@rooftop.invalid> - 2016-05-07 19:53 +0000
      Re: man, am I sick of md5 owl <owl@rooftop.invalid> - 2016-05-07 21:47 +0000
        Re: man, am I sick of md5 owl <owl@rooftop.invalid> - 2016-05-07 22:43 +0000
          Re: man, am I sick of md5 vallor <vallor@cultnix.org> - 2016-05-08 19:38 +0000
            Re: man, am I sick of md5 owl <owl@rooftop.invalid> - 2016-05-09 22:43 +0000
              Re: man, am I sick of md5 Peter Köhlmann <peter-koehlmann@t-online.de> - 2016-05-10 00:58 +0200
                Re: man, am I sick of md5 owl <owl@rooftop.invalid> - 2016-05-10 00:02 +0000
              Re: man, am I sick of md5 vallor <vallor@cultnix.org> - 2016-05-10 00:53 +0000
                Re: man, am I sick of md5 owl <owl@rooftop.invalid> - 2016-05-13 03:52 +0000
                Re: man, am I sick of md5 DFS <nospam@dfs.com> - 2016-05-13 00:06 -0400
                Re: man, am I sick of md5 owl <owl@rooftop.invalid> - 2016-05-13 04:27 +0000
                Re: man, am I sick of md5 vallor <vallor@cultnix.org> - 2016-05-18 18:30 +0000
                Re: man, am I sick of md5 DFS <nospam@dfs.com> - 2016-05-18 16:55 -0400
                Re: man, am I sick of md5 vallor <vallor@cultnix.org> - 2016-05-17 20:22 +0000
                Re: man, am I sick of md5 owl <owl@rooftop.invalid> - 2016-05-18 00:44 +0000
    Re: man, am I sick of md5 vallor <vallor@cultnix.org> - 2016-05-07 20:02 +0000
      Re: man, am I sick of md5 Snit <usenet@gallopinginsanity.com> - 2016-05-07 13:40 -0700

csiph-web