Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.java.programmer > #28147 > unrolled thread

64-bit hashing function

Back to article view | Back to comp.lang.java.programmer

Contents

  64-bit hashing function Roedy Green <see_website@mindprod.com.invalid> - 2014-04-20 15:15 -0700
    Re: 64-bit hashing function Arne Vajhøj <arne@vajhoej.dk> - 2014-04-20 19:37 -0400
    Re: 64-bit hashing function Roedy Green <see_website@mindprod.com.invalid> - 2014-04-20 22:18 -0700
    Re: 64-bit hashing function Marcel Müller <news.5.maazl@spamgourmet.org> - 2014-04-21 09:19 +0200
      Re: 64-bit hashing function Roedy Green <see_website@mindprod.com.invalid> - 2014-04-21 00:51 -0700
        Re: 64-bit hashing function Roedy Green <see_website@mindprod.com.invalid> - 2014-04-21 02:05 -0700
          Re: 64-bit hashing function taqmcg@gmail.com - 2014-04-21 11:49 -0700
            Re: 64-bit hashing function taqmcg@gmail.com - 2014-04-21 12:01 -0700
          Re: 64-bit hashing function wolfgang zeidler <invalid@invalid.invalid> - 2014-04-22 20:42 +0200
        Re: 64-bit hashing function Marcel Müller <news.5.maazl@spamgourmet.org> - 2014-04-21 12:51 +0200
          Re: 64-bit hashing function Roedy Green <see_website@mindprod.com.invalid> - 2014-04-21 06:31 -0700
            Re: 64-bit hashing function Marcel Müller <news.5.maazl@spamgourmet.org> - 2014-04-21 16:40 +0200
              Re: 64-bit hashing function Roedy Green <see_website@mindprod.com.invalid> - 2014-04-21 20:45 -0700
                Re: 64-bit hashing function Marcel Müller <news.5.maazl@spamgourmet.org> - 2014-04-22 09:46 +0200
          Re: 64-bit hashing function Roedy Green <see_website@mindprod.com.invalid> - 2014-04-21 06:38 -0700
            Re: 64-bit hashing function Roedy Green <see_website@mindprod.com.invalid> - 2014-04-21 06:55 -0700
          Re: 64-bit hashing function Roedy Green <see_website@mindprod.com.invalid> - 2014-04-21 10:09 -0700
            Re: 64-bit hashing function markspace <markspace@nospam.nospam> - 2014-04-21 11:46 -0700
              Re: 64-bit hashing function Roedy Green <see_website@mindprod.com.invalid> - 2014-04-21 20:55 -0700
                Re: 64-bit hashing function markspace <markspace@nospam.nospam> - 2014-04-22 08:39 -0700
              Re: 64-bit hashing function Roedy Green <see_website@mindprod.com.invalid> - 2014-04-21 20:56 -0700
              Re: 64-bit hashing function Roedy Green <see_website@mindprod.com.invalid> - 2014-04-21 21:05 -0700
                Re: 64-bit hashing function markspace <markspace@nospam.nospam> - 2014-04-22 08:15 -0700
              Re: 64-bit hashing function Roedy Green <see_website@mindprod.com.invalid> - 2014-04-25 02:29 -0700
            Re: 64-bit hashing function Roedy Green <see_website@mindprod.com.invalid> - 2014-04-21 20:53 -0700
            Re: 64-bit hashing function Roedy Green <see_website@mindprod.com.invalid> - 2014-05-08 10:12 -0700
              Re: 64-bit hashing function Simon Lewis <simonlewis2001@gmail.com> - 2014-05-08 21:27 +0200
    Re: 64-bit hashing function rossum <rossum48@coldmail.com> - 2014-04-23 09:57 +0100
      Re: 64-bit hashing function Roedy Green <see_website@mindprod.com.invalid> - 2014-04-25 04:08 -0700
        Re: 64-bit hashing function Mike Amling <mamling@chaff.us> - 2014-05-05 11:03 -0500
          Re: 64-bit hashing function Roedy Green <see_website@mindprod.com.invalid> - 2014-05-29 16:00 -0700
          Re: 64-bit hashing function Roedy Green <see_website@mindprod.com.invalid> - 2014-05-29 16:16 -0700

Page 2 of 2 — ← Prev page 1 [2]

#28174

On Mon, 21 Apr 2014 11:46:06 -0700, markspace
<markspace@nospam.nospam> wrote, quoted or indirectly quoted someone
who said :

>1. Way too much static initialization.  Just referencing the class in 
>any fashion will cause the entire cache to be read into memory.

If you have methods like load and save, you can make the client call
them, but it just one more thing to go wrong.  What are the
alternatives?
-- 
Roedy Green Canadian Mind Products http://mindprod.com
"Don't worry about people stealing an idea; if it's original, you'll
 have to shove it down their throats." 
~ Howard Aiken (born: 1900-03-08 died: 1973-03-14 at age: 73)

[toc] | [prev] | [next] | [standalone]

#28175

On Mon, 21 Apr 2014 11:46:06 -0700, markspace
<markspace@nospam.nospam> wrote, quoted or indirectly quoted someone
who said :

>Way too many static methods here.  What happened to instantiating a 
>class like normal folks?

Because I am not prepared at this early stage to think about multiple
instances.  Before I released it, I would have to do that.

A released version would need a callback for compacting the filename.
It would need the name of the cache file as a parameter.

I tend to think in concrete terms, then when all is working, gradually
add the generalisations.
-- 
Roedy Green Canadian Mind Products http://mindprod.com
"Don't worry about people stealing an idea; if it's original, you'll
 have to shove it down their throats." 
~ Howard Aiken (born: 1900-03-08 died: 1973-03-14 at age: 73)

[toc] | [prev] | [next] | [standalone]

#28181

On 4/21/2014 9:05 PM, Roedy Green wrote:
> On Mon, 21 Apr 2014 11:46:06 -0700, markspace
> <markspace@nospam.nospam> wrote, quoted or indirectly quoted someone
> who said :
>
>> Way too many static methods here.  What happened to instantiating a
>> class like normal folks?
>
> Because I am not prepared at this early stage to think about multiple
> instances.  Before I released it, I would have to do that.

I don't see how you test though, given the way you've designed your code.

Re. your previous question, "What are the alternatives?", just make an 
object that calls the methods in its ctor:

public class Lazy {

   public Lazy() {
       load();
       addShutdownHook();
   }

   private load() {
      // load database...
   }

   private addShutdownHook() {
      // add hook...
   }

}

The next thing I would do is make "load()" package private and final, 
and take a parameter for an InputStream, so I could inject some test 
vectors into it.  Stuff like that makes testing much easier and faster.

[toc] | [prev] | [next] | [standalone]

#28226

On Mon, 21 Apr 2014 11:46:06 -0700, markspace
<markspace@nospam.nospam> wrote, quoted or indirectly quoted someone
who said :

>3. You should leave yourself some wiggle room to move from flat files to 
>a simple database in the future.  Since it's your code you can just gut 
>the thing and re-write it but it doesn't look easy.

If you redid the load and save methods with a database, the code would
be quite different.  About all that would be left of save is the
Iterator and the hash= entry.getKey timestamp=entry.getvalue

What sort of thing might I do to increase wiggle room?
-- 
Roedy Green Canadian Mind Products http://mindprod.com
Computers are like Old Testament gods; lots of rules and no mercy. 
~ Joseph Campbell (born: 1904-03-26 died: 1987-10-31 at age: 83)

[toc] | [prev] | [next] | [standalone]

#28172

On Mon, 21 Apr 2014 10:09:04 -0700, Roedy Green
<see_website@mindprod.com.invalid> wrote, quoted or indirectly quoted
someone who said :

>here is my first cut at the code. I have not run it yet 
 
I think by collision handling is a bit naive. I need to think about it
a bit more.  If I can get the number of collisions down to once every
1000 years, I will live with it since any software change flushes the
cache.
-- 
Roedy Green Canadian Mind Products http://mindprod.com
"Don't worry about people stealing an idea; if it's original, you'll
 have to shove it down their throats." 
~ Howard Aiken (born: 1900-03-08 died: 1973-03-14 at age: 73)

[toc] | [prev] | [next] | [standalone]

#28412

On Mon, 21 Apr 2014 10:09:04 -0700, Roedy Green
<see_website@mindprod.com.invalid> wrote, quoted or indirectly quoted
someone who said :

>here is my first cut at the code. I have not run it ye

Here is what it looks like now. I am using hit in generating my web
pages.

/*
 * [Lazy.java]
 *
 * Summary: Lets us avoid the work of expanding macros if they were
done successfully earlier.
 *
 * Copyright: (c) 2012-2014 Roedy Green, Canadian Mind Products,
http://mindprod.com
 *
 * Licence: This software may be copied and used freely for any
purpose but military.
 *          http://mindprod.com/contact/nonmil.html
 *
 * Requires: JDK 1.7+
 *
 * Created with: JetBrains IntelliJ IDEA IDE
http://www.jetbrains.com/idea/
 *
 * Version History:
 *  1.0 2014-04-21 initial version.
 */
package com.mindprod.htmlmacros.support;

import com.mindprod.common17.FNV1a64;
import com.mindprod.common17.Misc;
import com.mindprod.common17.ST;

import java.io.DataInputStream;
import java.io.DataOutputStream;
import java.io.EOFException;
import java.io.File;
import java.io.IOException;
import java.util.HashMap;
import java.util.Map.Entry;
import java.util.concurrent.TimeUnit;

import static java.lang.System.err;

/**
 * Lets us avoid the work of expanding macros if they were done
successfully earlier.
 * To use it:
 * 1. Call the constructor to create a Lazy object.
 * 2. call lazy.open
 * 3. for each file you are considering processing call
lazy.isAlreadyDone
 * 4. after you have processed each file call lazy.markStatus.
 * 5. when you are done call lazy.close
 *
 * @author Roedy Green, Canadian Mind Products
 * @version 1.0 2014-04-21 initial version
 * @since 2014-04-21
 */
public class Lazy
    {
    // declarations

    /**
     * size of buffer in bytes
     */
    private static final int BUFFERSIZE_IN_BYTES = 64 * 1024;

    /**
     * allow 5 seconds of slop in matching dates
     */
    private static final long SLOP = TimeUnit.SECONDS.toMillis( 5 );

    /**
     * length of each record in the serialised cache file in bytes.
     * two 64-bit longs.
     */
    private static int RECORD_LENGTH = ( 64 + 64 ) / 8;

    /**
     * lead ignorable string on most files
     */

    private String filenamePrefix;

    /**
     * trailing ignorable string on most files
     */
    private String filenameSuffix;

    /**
     * binary file where we store state between runs.
     */
    private File cacheFile;

    /**
     * look up filename hash-32 to timestamp
     */
    private HashMap<Long, Long> hashToTimestamp;

    // end declarations

    /**
     * true if this Lasy is open
     */
    private boolean isOpen;
    // methods

    /**
     * constructor
     */
    public Lazy()
        {
        isOpen = false;
        }

    /**
     * save the contents of the lookup cacheFIle into
embellishments/cacheFIle.bin
     * It is a binary file of pairs hash-32, timestamp-64
     */
    public void close()
        {
        if ( !isOpen )
            {
            return;
            }

        try
            {
            // O P E N

            final DataOutputStream dos = Misc.getDataOutputStream(
this.cacheFile, BUFFERSIZE_IN_BYTES );
            for ( Entry<Long, Long> entry : hashToTimestamp.entrySet()
)
                {
                // W R I T E
                final long hash = entry.getKey();
                final long timestamp = entry.getValue();
                // writing in big-endian binary to be compact and
fast.
                dos.writeLong( hash ); // we write int and long, not
Integer and Long.
                dos.writeLong( timestamp );
                }
            // C L O S E
            dos.close();
            } // end if
        catch ( IOException e )
            {
            err.println( ">>> Warning. Unable to write " +
this.cacheFile.getAbsolutePath() + " file "
                         + e.getMessage() );
            }

        isOpen = false;
        }// end method

    /**
     * has this file already been processed and is unchanged since
that time?
     *
     * @param file file we are processing.
     *
     * @return true if the file has already been successfully
processed.
     */
    public boolean isAlreadyDone( File file )
        {
        if ( !isOpen )
            {
            throw new IllegalArgumentException( "Lazy.open() has not
yet been called." );
            }
        final long hash = calcHash( file );
        final Long timestampL = hashToTimestamp.get( hash );
        // if no entry, it was not registered as done.
        if ( timestampL == null )
            {
            return false;
            }
        // if all is well ,the last modified date should not have
changed since we recorded the file as
        // successfully processed.
        if ( file.lastModified() > timestampL + SLOP )
            {
            // the file has been modified since we last processed it.
            // we will have to reprocess it.
            // This cacheFile entry is useless. We might as well get
rid of it now to save some space.
            hashToTimestamp.remove( hash );
            return false;
            }
        else
            {
            // it has not been touched since we last successfully
processed it.
            // the cacheFile entry is fine as is. This is the whole
point, to save reprocessing it.
            return true;
            }
        }// end method

    /**
     * Mark the status of this file.
     *
     * @param file   file we are processing.
     * @param status true= file successfully processed, false=file was
not successfully processed.
     */
    public void markStatus( File file, boolean status )
        {
        if ( !isOpen )
            {
            throw new IllegalArgumentException( "Lazy.open() has not
yet been called." );
            }
        final long hash = calcHash( file );
        if ( status )
            {
            // GOOD
            // we record the fact by leaving an entry with hash/Now.
            // file was just given or will soon be given a timestamp
close to this.
            hashToTimestamp.put( hash, System.currentTimeMillis() );
            // collisions are so rare, we do not worry about them. Two
files sharing a hash.
            }
        else
            {
            // BAD
            // erase all record of it.  There may be no record already
            hashToTimestamp.remove( hash );
            }
        }// end method

    /**
     * Open and read the cacheFIle file.
     *
     * @param cacheFile      cacheFile file with state from last stime
storted. If the file does not exist,
     *                       we start over. e.g. new File(
"E:\\mindprod\\embellishments\\cacheFIle.bin")
     * @param filePrefix     If nearly all filenames start the same
way, the common lead string, null or  "" otherwise.
     *                       e.g.  "E:\mindprod\".  Use \ or / to
match the way you specify the files
     *                       you feed to markStatus.
     * @param fileSuffix     If nearly all filenames end the same way,
the common trailing string, null or "" otherwise.
     *                       e.g.  ".html".  Use \ or / to match the
way you specify the files
     *                       you feed to markStatus.
     * @param estimatedFiles estimate of how many files we will
process
     */
    public void open( final File cacheFile, final String filePrefix,
final String fileSuffix, final int estimatedFiles )
        {
        if ( isOpen )
            {
            return;
            }
        this.cacheFile = cacheFile;
        this.filenamePrefix = ST.canonical( filePrefix );
        this.filenameSuffix = ST.canonical( fileSuffix );
        if ( cacheFile.exists() && cacheFile.canRead() )
            {
            // load up the HashMap we use to track when files were
last successfully processed.
            final int elts = Math.max( estimatedFiles, ( int )
cacheFile.length() / RECORD_LENGTH );
            // allow some padding to avoid collisions
            hashToTimestamp = new HashMap<>( elts + elts / 4 );  //
25% padding
            //  load binary long pairs from it.
            DataInputStream dis = null;
            try
                {
                try
                    {
                    // O P E N

                    dis = Misc.getDataInputStream( cacheFile,
BUFFERSIZE_IN_BYTES );
                    while ( true )
                        {
                        // R E A D  pairs hash-64, timestamp-64
                        long hash = dis.readLong();
                        long timestamp = dis.readLong();
                        hashToTimestamp.put( hash, timestamp );
                        } // end loop
                    } // end  inner try
                catch ( EOFException e )
                    {
                    // nothing to do
                    }
                finally
                    {
                    // C L O S E
                    if ( dis != null )
                        {
                        dis.close();
                        }
                    }
                } // end outer try
            catch ( IOException e )
                {
                err.println( ">>> Warning. Unable to read " +
cacheFile.getAbsolutePath() + " file" );
                // we carry on, using as much as we could read.
                }
            } // end if
        else
            {
            hashToTimestamp = new HashMap<>( estimatedFiles +
estimatedFiles / 4 );
            }
        isOpen = true;
        }// end method

    /**
     * calculate a hash-64 of the name of the file, not its contents
     *
     * @param file file to be processed
     *
     * @return 64-bit FNV1a64 hash.
     */
    private long calcHash( final File file )
        {
        // prune down if possible.
        final String chopped = ST.chopLeadingString(
ST.chopTrailingString( file.getAbsolutePath(), this.filenameSuffix ),
this.filenamePrefix );
        return FNV1a64.computeHash( chopped );
        }// end method
    // end methods
    } // end class


-- 
Roedy Green Canadian Mind Products http://mindprod.com
Culture is your operating system. 
~ Terence McKenna (born: 1946-11-16 died: 2000-04-03 at age: 53)

[toc] | [prev] | [next] | [standalone]

#28414

Roedy Green <see_website@mindprod.com.invalid> writes:

> On Mon, 21 Apr 2014 10:09:04 -0700, Roedy Green
> <see_website@mindprod.com.invalid> wrote, quoted or indirectly quoted
> someone who said :
>
>>here is my first cut at the code. I have not run it ye
>
> Here is what it looks like now. I am using hit in generating my web
> pages.
>
> /*
>  * [Lazy.java]
>  *
>  * Summary: Lets us avoid the work of expanding macros if they were
> done successfully earlier.
>  *
>  * Copyright: (c) 2012-2014 Roedy Green, Canadian Mind Products,
> http://mindprod.com
>  *
>  * Licence: This software may be copied and used freely for any
> purpose but military.
>  *          http://mindprod.com/contact/nonmil.html
>  *
>  * Requires: JDK 1.7+
>  *
>  * Created with: JetBrains IntelliJ IDEA IDE
> http://www.jetbrains.com/idea/
>  *
>  * Version History:
>  *  1.0 2014-04-21 initial version.
>  */
> package com.mindprod.htmlmacros.support;
>
> import com.mindprod.common17.FNV1a64;
> import com.mindprod.common17.Misc;
> import com.mindprod.common17.ST;
>
> import java.io.DataInputStream;
> import java.io.DataOutputStream;
> import java.io.EOFException;
> import java.io.File;
> import java.io.IOException;
> import java.util.HashMap;
> import java.util.Map.Entry;
> import java.util.concurrent.TimeUnit;
>
> import static java.lang.System.err;
>
> /**
>  * Lets us avoid the work of expanding macros if they were done
> successfully earlier.
>  * To use it:
>  * 1. Call the constructor to create a Lazy object.
>  * 2. call lazy.open
>  * 3. for each file you are considering processing call
> lazy.isAlreadyDone
>  * 4. after you have processed each file call lazy.markStatus.
>  * 5. when you are done call lazy.close
>  *
>  * @author Roedy Green, Canadian Mind Products
>  * @version 1.0 2014-04-21 initial version
>  * @since 2014-04-21
>  */
> public class Lazy
>     {
>     // declarations
>
>     /**
>      * size of buffer in bytes
>      */
>     private static final int BUFFERSIZE_IN_BYTES = 64 * 1024;
>
>     /**
>      * allow 5 seconds of slop in matching dates
>      */
>     private static final long SLOP = TimeUnit.SECONDS.toMillis( 5 );
>
>     /**
>      * length of each record in the serialised cache file in bytes.
>      * two 64-bit longs.
>      */
>     private static int RECORD_LENGTH = ( 64 + 64 ) / 8;
>
>     /**
>      * lead ignorable string on most files
>      */
>
>     private String filenamePrefix;
>
>     /**
>      * trailing ignorable string on most files
>      */
>     private String filenameSuffix;
>
>     /**
>      * binary file where we store state between runs.
>      */
>     private File cacheFile;
>
>     /**
>      * look up filename hash-32 to timestamp
>      */
>     private HashMap<Long, Long> hashToTimestamp;
>
>     // end declarations
>
>     /**
>      * true if this Lasy is open
>      */
>     private boolean isOpen;
>     // methods
>
>     /**
>      * constructor
>      */
>     public Lazy()
>         {
>         isOpen = false;
>         }
>
>     /**
>      * save the contents of the lookup cacheFIle into
> embellishments/cacheFIle.bin
>      * It is a binary file of pairs hash-32, timestamp-64
>      */
>     public void close()
>         {
>         if ( !isOpen )
>             {
>             return;
>             }
>
>         try
>             {
>             // O P E N
>
>             final DataOutputStream dos = Misc.getDataOutputStream(
> this.cacheFile, BUFFERSIZE_IN_BYTES );
>             for ( Entry<Long, Long> entry : hashToTimestamp.entrySet()
> )
>                 {
>                 // W R I T E
>                 final long hash = entry.getKey();
>                 final long timestamp = entry.getValue();
>                 // writing in big-endian binary to be compact and
> fast.
>                 dos.writeLong( hash ); // we write int and long, not
> Integer and Long.
>                 dos.writeLong( timestamp );
>                 }
>             // C L O S E
>             dos.close();
>             } // end if
>         catch ( IOException e )
>             {
>             err.println( ">>> Warning. Unable to write " +
> this.cacheFile.getAbsolutePath() + " file "
>                          + e.getMessage() );
>             }
>
>         isOpen = false;
>         }// end method
>
>     /**
>      * has this file already been processed and is unchanged since
> that time?
>      *
>      * @param file file we are processing.
>      *
>      * @return true if the file has already been successfully
> processed.
>      */
>     public boolean isAlreadyDone( File file )
>         {
>         if ( !isOpen )
>             {
>             throw new IllegalArgumentException( "Lazy.open() has not
> yet been called." );
>             }
>         final long hash = calcHash( file );
>         final Long timestampL = hashToTimestamp.get( hash );
>         // if no entry, it was not registered as done.
>         if ( timestampL == null )
>             {
>             return false;
>             }
>         // if all is well ,the last modified date should not have
> changed since we recorded the file as
>         // successfully processed.
>         if ( file.lastModified() > timestampL + SLOP )
>             {
>             // the file has been modified since we last processed it.
>             // we will have to reprocess it.
>             // This cacheFile entry is useless. We might as well get
> rid of it now to save some space.
>             hashToTimestamp.remove( hash );
>             return false;
>             }
>         else
>             {
>             // it has not been touched since we last successfully
> processed it.
>             // the cacheFile entry is fine as is. This is the whole
> point, to save reprocessing it.
>             return true;
>             }
>         }// end method
>
>     /**
>      * Mark the status of this file.
>      *
>      * @param file   file we are processing.
>      * @param status true= file successfully processed, false=file was
> not successfully processed.
>      */
>     public void markStatus( File file, boolean status )
>         {
>         if ( !isOpen )
>             {
>             throw new IllegalArgumentException( "Lazy.open() has not
> yet been called." );
>             }
>         final long hash = calcHash( file );
>         if ( status )
>             {
>             // GOOD
>             // we record the fact by leaving an entry with hash/Now.
>             // file was just given or will soon be given a timestamp
> close to this.
>             hashToTimestamp.put( hash, System.currentTimeMillis() );
>             // collisions are so rare, we do not worry about them. Two
> files sharing a hash.
>             }
>         else
>             {
>             // BAD
>             // erase all record of it.  There may be no record already
>             hashToTimestamp.remove( hash );
>             }
>         }// end method
>
>     /**
>      * Open and read the cacheFIle file.
>      *
>      * @param cacheFile      cacheFile file with state from last stime
> storted. If the file does not exist,
>      *                       we start over. e.g. new File(
> "E:\\mindprod\\embellishments\\cacheFIle.bin")
>      * @param filePrefix     If nearly all filenames start the same
> way, the common lead string, null or  "" otherwise.
>      *                       e.g.  "E:\mindprod\".  Use \ or / to
> match the way you specify the files
>      *                       you feed to markStatus.
>      * @param fileSuffix     If nearly all filenames end the same way,
> the common trailing string, null or "" otherwise.
>      *                       e.g.  ".html".  Use \ or / to match the
> way you specify the files
>      *                       you feed to markStatus.
>      * @param estimatedFiles estimate of how many files we will
> process
>      */
>     public void open( final File cacheFile, final String filePrefix,
> final String fileSuffix, final int estimatedFiles )
>         {
>         if ( isOpen )
>             {
>             return;
>             }
>         this.cacheFile = cacheFile;
>         this.filenamePrefix = ST.canonical( filePrefix );
>         this.filenameSuffix = ST.canonical( fileSuffix );
>         if ( cacheFile.exists() && cacheFile.canRead() )
>             {
>             // load up the HashMap we use to track when files were
> last successfully processed.
>             final int elts = Math.max( estimatedFiles, ( int )
> cacheFile.length() / RECORD_LENGTH );
>             // allow some padding to avoid collisions
>             hashToTimestamp = new HashMap<>( elts + elts / 4 );  //
> 25% padding
>             //  load binary long pairs from it.
>             DataInputStream dis = null;
>             try
>                 {
>                 try
>                     {
>                     // O P E N
>
>                     dis = Misc.getDataInputStream( cacheFile,
> BUFFERSIZE_IN_BYTES );
>                     while ( true )
>                         {
>                         // R E A D  pairs hash-64, timestamp-64
>                         long hash = dis.readLong();
>                         long timestamp = dis.readLong();
>                         hashToTimestamp.put( hash, timestamp );
>                         } // end loop
>                     } // end  inner try
>                 catch ( EOFException e )
>                     {
>                     // nothing to do
>                     }
>                 finally
>                     {
>                     // C L O S E
>                     if ( dis != null )
>                         {
>                         dis.close();
>                         }
>                     }
>                 } // end outer try
>             catch ( IOException e )
>                 {
>                 err.println( ">>> Warning. Unable to read " +
> cacheFile.getAbsolutePath() + " file" );
>                 // we carry on, using as much as we could read.
>                 }
>             } // end if
>         else
>             {
>             hashToTimestamp = new HashMap<>( estimatedFiles +
> estimatedFiles / 4 );
>             }
>         isOpen = true;
>         }// end method
>
>     /**
>      * calculate a hash-64 of the name of the file, not its contents
>      *
>      * @param file file to be processed
>      *
>      * @return 64-bit FNV1a64 hash.
>      */
>     private long calcHash( final File file )
>         {
>         // prune down if possible.
>         final String chopped = ST.chopLeadingString(
> ST.chopTrailingString( file.getAbsolutePath(), this.filenameSuffix ),
> this.filenamePrefix );
>         return FNV1a64.computeHash( chopped );
>         }// end method
>     // end methods
>     } // end class

Way to put off a code review. That's some seriously horrible coding
standard you use for brace bracketing there. No wonder you seem to have
so much problems navigating braces. Why so much white space?!? There is
no benefit in having all braces on their own lines. It's truly horrible
and wastes far too much valuable screen estate when debugging or
browsing the code.

Something like this:

>                 finally
>                     {
>                     // C L O S E
>                     if ( dis != null )
>                         {
>                         dis.close();
>                         }
>                     }

should, IMO, be

,----
| finally{ // close
|     if (dis!=null)
|         dis.close();
| }
`----

I realise it's all down to individuals taste but seriously that is one
awful looking listing that appears to be a first year pascal project
from back in the 80s.

[toc] | [prev] | [next] | [standalone]

#28196

On Sun, 20 Apr 2014 15:15:07 -0700, Roedy Green
<see_website@mindprod.com.invalid> wrote:

>What is your favourite 64-bit hashing function/digest? This is not for
>cryptography.  Ideally there is Java source available and it is fast. 
>
>I will be using it to collapse filenames down.  There is no malice or
>privacy to worry about.
>
>I would like similar filenames to generate different results.
I usually suggest the FNV hash for this sort of situation:
http://isthe.com/chongo/tech/comp/fnv/

It is a non-crypto (and hence fast) hash in sizes from 32 bit to 1024
bit and is simple to program:

  hash = offset_basis
  for each octet_of_data to be hashed
    hash = hash xor octet_of_data
    hash = hash * FNV_prime
  endfor
  return hash

For the 64 bit version:

  offset_basis = 14695981039346656037
  FNV_prime = 1099511628211
  hash is an unsigned 64 bit integer.

HTH

rossum

[toc] | [prev] | [next] | [standalone]

#28227

On Wed, 23 Apr 2014 09:57:11 +0100, rossum <rossum48@coldmail.com>
wrote, quoted or indirectly quoted someone who said :

>I usually suggest the FNV hash for this sort of situation:
>http://isthe.com/chongo/tech/comp/fnv/


I have posted Java code for the 32 and 64-bit version at
http://mindprod.com/jgloss/digest.html

The only tricky part is defining a long unsigned magic number.

They should show up within the hour.
-- 
Roedy Green Canadian Mind Products http://mindprod.com
Computers are like Old Testament gods; lots of rules and no mercy. 
~ Joseph Campbell (born: 1904-03-26 died: 1987-10-31 at age: 83)

[toc] | [prev] | [next] | [standalone]

#28374

On 4/25/14 6:08 AM, Roedy Green wrote:
> On Wed, 23 Apr 2014 09:57:11 +0100, rossum <rossum48@coldmail.com>
> wrote, quoted or indirectly quoted someone who said :
>
>> I usually suggest the FNV hash for this sort of situation:
>> http://isthe.com/chongo/tech/comp/fnv/
>
>
> I have posted Java code for the 32 and 64-bit version at
> http://mindprod.com/jgloss/digest.html
>
> The only tricky part is defining a long unsigned magic number.
>
> They should show up within the hour.

Have you tested that code? You have

hash^=b;

Isn't that going to sign-extend the byte before doing the XOR? IIRC, FNV 
treats the byte as unsigned. To be compatible with other FNV1a 
implementations, wouldn't you need

hash^=(b & 0xFF);

or equivalent? Of course, if interoperability is irrelevant, you're free 
to do as you please.

Mike Amling
--
VGhlIFJlaWduIG9mIEVycm9y

[toc] | [prev] | [next] | [standalone]

#28606

On Mon, 05 May 2014 11:03:33 -0500, Mike Amling <mamling@chaff.us>
wrote, quoted or indirectly quoted someone who said :

>Have you tested that code? You have
>
>hash^=b;

I have fixed that. I did not test it for compatibility, just that it
was evenly distributed. 
-- 
Roedy Green Canadian Mind Products http://mindprod.com
Young man, in mathematics you don't understand things.
You just get used to them. 
~ John von Neumann (born: 1903-12-28 died: 1957-02-08 at age: 53)

[toc] | [prev] | [next] | [standalone]

#28607

On Mon, 05 May 2014 11:03:33 -0500, Mike Amling <mamling@chaff.us>
wrote, quoted or indirectly quoted someone who said :

>hash^=(b & 0xFF);

Both would work equally well. They may have intended either.  Do you
know of a test digest I could compare?
-- 
Roedy Green Canadian Mind Products http://mindprod.com
Young man, in mathematics you don't understand things.
You just get used to them. 
~ John von Neumann (born: 1903-12-28 died: 1957-02-08 at age: 53)

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | comp.lang.java.programmer

csiph-web