Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.os.linux.advocacy > #354054

Re: man, am I sick of md5

Show all headers | View raw

vallor <vallor@cultnix.org> wrote:
> On Sat, 07 May 2016 12:30:01 +0000, meat wrote:
> 
>> On Sat, 07 May 2016 02:59:11 +0000, vallor wrote:
>> 
>>> On Sat, 07 May 2016 02:40:04 +0000, vallor wrote:
>>>> But look at that bootiful sorted data:
>>>> 
>>>> $ head -10 md5_ipv4_rainbow.hex
>>>> 0000000191fab544b163e31753366fae9c8380c2
>>>> 0000000384db5f2b13aae0c719c21aa002e6c0b2
>>> 
>>>> fffffffcfef441108d45e6ea4c38b0a7c3fe66b0
>>> 
>>> I just showed this to someone looking over my shoulder, and they said,
>>> "You mean to tell me you have a file with all the Internet addresses in
>>> the world?  Like, even foreign countries?"
>> 
>> That's what I was thinking!
>> I suspect it's human nature to assume something like that would require
>> the NSA and a super computer to churn out. Kind of like calculating pi
>> to the 1000000 trillionth digit or something.
>> 
>> It kind of puts a new face on government spying doesn't it?
> 
> BTW, if you've done the math on my file size, you'll discover that there 
> are 16 records missing.  They are for the following IP addresses:
> 
> 0.0.0.0, 28.0.0.0, 42.0.0.0, 56.0.0.0,
> 70.0.0.0, 84.0.0.0, 98.0.0.0, 112.0.0.0,
> 126.0.0.0, 140.0.0.0, 154.0.0.0, 168.0.0.0,
> 182.0.0.0, 196.0.0.0, 210.0.0.0, 224.0.0.0
> 
> It was an off-by-one error in the shell script that I used to run the C 
> program to generate each block.  (I used 16 processors on my video 
> superserver.)
> 
> I'm probably going to just hard-code these hashes into the search 
> program, because I don't want to re-generate the files, convert them, 
> sort them, merge them, then convert them back to binary...it takes a 
> while, as you can imagine.
> 

I'm decided to try a different approach today.  I set up a btrfs
partition.  btrfs does not suffer the inode limit that you get with ext4,
so billions of directories should be doable.  It takes a bit of time to
generate the tree -- I'm guessing about 1.7 hours per /8 -- but seek times
seem to be very fast with testing so far.  I pulled a recursive mkdir()
function from the web and merged it to my existing code.  The ip address
for a given hash will end up as a node directory.

anon@lowtide:/media/anon/f617b3c2-e000-437f-9c75-04474c001d92/ips/ipdir$ echo -n 0.3.100.198 | md5sum
ffff32641ebadad6de6b77dc19699e82  -
anon@lowtide:/media/anon/f617b3c2-e000-437f-9c75-04474c001d92/ips/ipdir$ time ls f/f/f/f/3/2/6/4/1/e/b/a/d/a/d/6/d/e/6/b/7/7/d/c/1/9/6/9/9/e/8/2
0.3.100.198

real	0m0.003s
user	0m0.000s
sys	0m0.000s
anon@lowtide:/media/anon/f617b3c2-e000-437f-9c75-04474c001d92/ips/ipdir$

This was not with the whole tree built yet.  I'm still waiting on the
first /8 to complete to get a better idea of the total build time.

Back to comp.os.linux.advocacy | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Re: man, am I sick of md5 meat <meat.stretcher@linuxmail.org> - 2016-05-07 12:30 +0000
  Re: man, am I sick of md5 vallor <vallor@cultnix.org> - 2016-05-07 18:17 +0000
    Re: man, am I sick of md5 owl <owl@rooftop.invalid> - 2016-05-07 19:53 +0000
      Re: man, am I sick of md5 owl <owl@rooftop.invalid> - 2016-05-07 21:47 +0000
        Re: man, am I sick of md5 owl <owl@rooftop.invalid> - 2016-05-07 22:43 +0000
          Re: man, am I sick of md5 vallor <vallor@cultnix.org> - 2016-05-08 19:38 +0000
            Re: man, am I sick of md5 owl <owl@rooftop.invalid> - 2016-05-09 22:43 +0000
              Re: man, am I sick of md5 Peter Köhlmann <peter-koehlmann@t-online.de> - 2016-05-10 00:58 +0200
                Re: man, am I sick of md5 owl <owl@rooftop.invalid> - 2016-05-10 00:02 +0000
              Re: man, am I sick of md5 vallor <vallor@cultnix.org> - 2016-05-10 00:53 +0000
                Re: man, am I sick of md5 owl <owl@rooftop.invalid> - 2016-05-13 03:52 +0000
                Re: man, am I sick of md5 DFS <nospam@dfs.com> - 2016-05-13 00:06 -0400
                Re: man, am I sick of md5 owl <owl@rooftop.invalid> - 2016-05-13 04:27 +0000
                Re: man, am I sick of md5 vallor <vallor@cultnix.org> - 2016-05-18 18:30 +0000
                Re: man, am I sick of md5 DFS <nospam@dfs.com> - 2016-05-18 16:55 -0400
                Re: man, am I sick of md5 vallor <vallor@cultnix.org> - 2016-05-17 20:22 +0000
                Re: man, am I sick of md5 owl <owl@rooftop.invalid> - 2016-05-18 00:44 +0000
    Re: man, am I sick of md5 vallor <vallor@cultnix.org> - 2016-05-07 20:02 +0000
      Re: man, am I sick of md5 Snit <usenet@gallopinginsanity.com> - 2016-05-07 13:40 -0700

csiph-web