Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.os.linux.advocacy > #354065

Re: man, am I sick of md5

Show all headers | View raw

owl <owl@rooftop.invalid> wrote:
> owl <owl@rooftop.invalid> wrote:
>> vallor <vallor@cultnix.org> wrote:
>>> On Sat, 07 May 2016 12:30:01 +0000, meat wrote:
>>> 
>>>> On Sat, 07 May 2016 02:59:11 +0000, vallor wrote:
>>>> 
>>>>> On Sat, 07 May 2016 02:40:04 +0000, vallor wrote:
>>>>>> But look at that bootiful sorted data:
>>>>>> 
>>>>>> $ head -10 md5_ipv4_rainbow.hex
>>>>>> 0000000191fab544b163e31753366fae9c8380c2
>>>>>> 0000000384db5f2b13aae0c719c21aa002e6c0b2
>>>>> 
>>>>>> fffffffcfef441108d45e6ea4c38b0a7c3fe66b0
>>>>> 
>>>>> I just showed this to someone looking over my shoulder, and they said,
>>>>> "You mean to tell me you have a file with all the Internet addresses in
>>>>> the world?  Like, even foreign countries?"
>>>> 
>>>> That's what I was thinking!
>>>> I suspect it's human nature to assume something like that would require
>>>> the NSA and a super computer to churn out. Kind of like calculating pi
>>>> to the 1000000 trillionth digit or something.
>>>> 
>>>> It kind of puts a new face on government spying doesn't it?
>>> 
>>> BTW, if you've done the math on my file size, you'll discover that there 
>>> are 16 records missing.  They are for the following IP addresses:
>>> 
>>> 0.0.0.0, 28.0.0.0, 42.0.0.0, 56.0.0.0,
>>> 70.0.0.0, 84.0.0.0, 98.0.0.0, 112.0.0.0,
>>> 126.0.0.0, 140.0.0.0, 154.0.0.0, 168.0.0.0,
>>> 182.0.0.0, 196.0.0.0, 210.0.0.0, 224.0.0.0
>>> 
>>> It was an off-by-one error in the shell script that I used to run the C 
>>> program to generate each block.  (I used 16 processors on my video 
>>> superserver.)
>>> 
>>> I'm probably going to just hard-code these hashes into the search 
>>> program, because I don't want to re-generate the files, convert them, 
>>> sort them, merge them, then convert them back to binary...it takes a 
>>> while, as you can imagine.
>>> 
>> 
>> I'm decided to try a different approach today.  I set up a btrfs
>> partition.  btrfs does not suffer the inode limit that you get with ext4,
>> so billions of directories should be doable.  It takes a bit of time to
>> generate the tree -- I'm guessing about 1.7 hours per /8 -- but seek times
>> seem to be very fast with testing so far.  I pulled a recursive mkdir()
>> function from the web and merged it to my existing code.  The ip address
>> for a given hash will end up as a node directory.
>> 
>> anon@lowtide:/media/anon/f617b3c2-e000-437f-9c75-04474c001d92/ips/ipdir$ echo -n 0.3.100.198 | md5sum
>> ffff32641ebadad6de6b77dc19699e82  -
>> anon@lowtide:/media/anon/f617b3c2-e000-437f-9c75-04474c001d92/ips/ipdir$ time ls f/f/f/f/3/2/6/4/1/e/b/a/d/a/d/6/d/e/6/b/7/7/d/c/1/9/6/9/9/e/8/2
>> 0.3.100.198
>> 
>> real    0m0.003s
>> user    0m0.000s
>> sys     0m0.000s
>> anon@lowtide:/media/anon/f617b3c2-e000-437f-9c75-04474c001d92/ips/ipdir$
>> 
>> This was not with the whole tree built yet.  I'm still waiting on the
>> first /8 to complete to get a better idea of the total build time.
>> 
> 
> Damn.  It's been running for quite a while now creating tree for
> 0.0.0.0-0.255.255.255.  Just now hit 0.9.0.0 level, but speeds are
> still holding pretty good.
> 
> anon@lowtide:/media/anon/f617b3c2-e000-437f-9c75-04474c001d92/ips/ipdir$ echo -n 0.9.0.0 | md5sum
> 14637e918fe0e9edd63634b57cfe809b  -
> anon@lowtide:/media/anon/f617b3c2-e000-437f-9c75-04474c001d92/ips/ipdir$ time ./doit.sh 14637e918fe0e9edd63634b57cfe809b
> 0.9.0.0
> 
> real    0m0.005s
> user    0m0.000s
> sys     0m0.000s
> anon@lowtide:/media/anon/f617b3c2-e000-437f-9c75-04474c001d92/ips/ipdir$ 
> 
> This is doit.sh:
> 
> #!/bin/bash
> 
> if [ ${#} -ne 1 ];then
>   echo "need a hash"
>   exit
> fi
> 
> ls $(./haship2dir ${1})
> 
> --------
> haship2dir just creates the directory from the provided hash.
> At this rate, I think we're back to the multiple weeks for tree
> generation, but "decode" speeds are near instantaneous, at least
> so far.  And all this assumes btrfs doesn't crap out before it's
> done.
> 

Clarification: haship2dir creates the directory *name* from the
provided hash.  The directory would have been created already.
This is just to eliminate having to re-type the hash with slashes
when you want to search.

anon@lowtide:/media/anon/f617b3c2-e000-437f-9c75-04474c001d92/ips/ipdir$ time ./doit.sh c0f68c9df008b6ba890bf03f84d4c251
0.10.0.0

real	0m0.009s
user	0m0.000s
sys	0m0.004s
anon@lowtide:/media/anon/f617b3c2-e000-437f-9c75-04474c001d92/ips/ipdir$ time ./doit.sh c0f68c9df008b6ba890bf03f84d4c25f
ls: cannot access c/0/f/6/8/c/9/d/f/0/0/8/b/6/b/a/8/9/0/b/f/0/3/f/8/4/d/4/c/2/5/f/: No such file or directory

real	0m0.009s
user	0m0.008s
sys	0m0.000s
anon@lowtide:/media/anon/f617b3c2-e000-437f-9c75-04474c001d92/ips/ipdir$ 
 

Back to comp.os.linux.advocacy | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Re: man, am I sick of md5 meat <meat.stretcher@linuxmail.org> - 2016-05-07 12:30 +0000
  Re: man, am I sick of md5 vallor <vallor@cultnix.org> - 2016-05-07 18:17 +0000
    Re: man, am I sick of md5 owl <owl@rooftop.invalid> - 2016-05-07 19:53 +0000
      Re: man, am I sick of md5 owl <owl@rooftop.invalid> - 2016-05-07 21:47 +0000
        Re: man, am I sick of md5 owl <owl@rooftop.invalid> - 2016-05-07 22:43 +0000
          Re: man, am I sick of md5 vallor <vallor@cultnix.org> - 2016-05-08 19:38 +0000
            Re: man, am I sick of md5 owl <owl@rooftop.invalid> - 2016-05-09 22:43 +0000
              Re: man, am I sick of md5 Peter Köhlmann <peter-koehlmann@t-online.de> - 2016-05-10 00:58 +0200
                Re: man, am I sick of md5 owl <owl@rooftop.invalid> - 2016-05-10 00:02 +0000
              Re: man, am I sick of md5 vallor <vallor@cultnix.org> - 2016-05-10 00:53 +0000
                Re: man, am I sick of md5 owl <owl@rooftop.invalid> - 2016-05-13 03:52 +0000
                Re: man, am I sick of md5 DFS <nospam@dfs.com> - 2016-05-13 00:06 -0400
                Re: man, am I sick of md5 owl <owl@rooftop.invalid> - 2016-05-13 04:27 +0000
                Re: man, am I sick of md5 vallor <vallor@cultnix.org> - 2016-05-18 18:30 +0000
                Re: man, am I sick of md5 DFS <nospam@dfs.com> - 2016-05-18 16:55 -0400
                Re: man, am I sick of md5 vallor <vallor@cultnix.org> - 2016-05-17 20:22 +0000
                Re: man, am I sick of md5 owl <owl@rooftop.invalid> - 2016-05-18 00:44 +0000
    Re: man, am I sick of md5 vallor <vallor@cultnix.org> - 2016-05-07 20:02 +0000
      Re: man, am I sick of md5 Snit <usenet@gallopinginsanity.com> - 2016-05-07 13:40 -0700

csiph-web