Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.java.programmer > #28163

Re: 64-bit hashing function

Show all headers | View raw

On 21.04.14 15.31, Roedy Green wrote:
> On Mon, 21 Apr 2014 12:51:32 +0200, Marcel Müller
> <news.5.maazl@spamgourmet.org>  wrote, quoted or indirectly quoted
> someone who said :
>
>> I still can't see any reasonable improvement of using hash codes instead
>> of file names. If you need to process a file you need the file name anyway.
>
> If I used a HashMap of filename ->  timestamp,
> I would have 20,000 x ( 15*2  + 8 ) = 767,436 bytes.

You are not really talking about 1MB memory nowadays, aren't you?

Btw. Where did you get 15*2 from? File names take about 32 bytes + twice 
the string length, depending on the platform (32/64 bit, JVM etc.).

> If I use a HashMap of digest ->  timestamp,
> I would have 20,000 x ( 4 + 2 ) = 120,000 bytes
>
> The difference is even more pronounced when you consider the overhead
> of wrapping each digest/timestamp in an object.

I don't see why this second map needs no boxing in Java. Also your 
timestamp shrink to 2 bytes.

> I only need look at one filename at time. I always have the current
> filename.  I don't need the HashMap to give me a filename.

OK.

So you basically need a (persistent) decision what file names in which 
revisions are already processed. (Any version control software has 
similar requirements.)

I would not recommend to use your digest here. How would you ever be 
able to clean up the hash map? You don't know the hash code of the 
vanished file names so you can't know what to remove. And since your 
operation is not a LUW with respect to all files in a folder (as 
mentioned in your other post) you cannot simply remove all untouched 
keys during a run.

Furthermore the result of filename -> timestamp is by far more human 
readable than hash keys. This will simplify testing and serviceability. 
If you care about the MB storage in the file system then zip the content 
of your persistent information file (will likely shrink some factor 2+x) 
or use a smarter file format that does not repeat the same strings over 
and over. E.g.
   Folder1
   File1	timestamp
   File2	timestamp
   Folder2
   ...
This will save approx. another factor 2 with many files in deep folders.


Marcel

Back to comp.lang.java.programmer | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

64-bit hashing function Roedy Green <see_website@mindprod.com.invalid> - 2014-04-20 15:15 -0700
  Re: 64-bit hashing function Arne Vajhøj <arne@vajhoej.dk> - 2014-04-20 19:37 -0400
  Re: 64-bit hashing function Roedy Green <see_website@mindprod.com.invalid> - 2014-04-20 22:18 -0700
  Re: 64-bit hashing function Marcel Müller <news.5.maazl@spamgourmet.org> - 2014-04-21 09:19 +0200
    Re: 64-bit hashing function Roedy Green <see_website@mindprod.com.invalid> - 2014-04-21 00:51 -0700
      Re: 64-bit hashing function Roedy Green <see_website@mindprod.com.invalid> - 2014-04-21 02:05 -0700
        Re: 64-bit hashing function taqmcg@gmail.com - 2014-04-21 11:49 -0700
          Re: 64-bit hashing function taqmcg@gmail.com - 2014-04-21 12:01 -0700
        Re: 64-bit hashing function wolfgang zeidler <invalid@invalid.invalid> - 2014-04-22 20:42 +0200
      Re: 64-bit hashing function Marcel Müller <news.5.maazl@spamgourmet.org> - 2014-04-21 12:51 +0200
        Re: 64-bit hashing function Roedy Green <see_website@mindprod.com.invalid> - 2014-04-21 06:31 -0700
          Re: 64-bit hashing function Marcel Müller <news.5.maazl@spamgourmet.org> - 2014-04-21 16:40 +0200
            Re: 64-bit hashing function Roedy Green <see_website@mindprod.com.invalid> - 2014-04-21 20:45 -0700
              Re: 64-bit hashing function Marcel Müller <news.5.maazl@spamgourmet.org> - 2014-04-22 09:46 +0200
        Re: 64-bit hashing function Roedy Green <see_website@mindprod.com.invalid> - 2014-04-21 06:38 -0700
          Re: 64-bit hashing function Roedy Green <see_website@mindprod.com.invalid> - 2014-04-21 06:55 -0700
        Re: 64-bit hashing function Roedy Green <see_website@mindprod.com.invalid> - 2014-04-21 10:09 -0700
          Re: 64-bit hashing function markspace <markspace@nospam.nospam> - 2014-04-21 11:46 -0700
            Re: 64-bit hashing function Roedy Green <see_website@mindprod.com.invalid> - 2014-04-21 20:55 -0700
              Re: 64-bit hashing function markspace <markspace@nospam.nospam> - 2014-04-22 08:39 -0700
            Re: 64-bit hashing function Roedy Green <see_website@mindprod.com.invalid> - 2014-04-21 20:56 -0700
            Re: 64-bit hashing function Roedy Green <see_website@mindprod.com.invalid> - 2014-04-21 21:05 -0700
              Re: 64-bit hashing function markspace <markspace@nospam.nospam> - 2014-04-22 08:15 -0700
            Re: 64-bit hashing function Roedy Green <see_website@mindprod.com.invalid> - 2014-04-25 02:29 -0700
          Re: 64-bit hashing function Roedy Green <see_website@mindprod.com.invalid> - 2014-04-21 20:53 -0700
          Re: 64-bit hashing function Roedy Green <see_website@mindprod.com.invalid> - 2014-05-08 10:12 -0700
            Re: 64-bit hashing function Simon Lewis <simonlewis2001@gmail.com> - 2014-05-08 21:27 +0200
  Re: 64-bit hashing function rossum <rossum48@coldmail.com> - 2014-04-23 09:57 +0100
    Re: 64-bit hashing function Roedy Green <see_website@mindprod.com.invalid> - 2014-04-25 04:08 -0700
      Re: 64-bit hashing function Mike Amling <mamling@chaff.us> - 2014-05-05 11:03 -0500
        Re: 64-bit hashing function Roedy Green <see_website@mindprod.com.invalid> - 2014-05-29 16:00 -0700
        Re: 64-bit hashing function Roedy Green <see_website@mindprod.com.invalid> - 2014-05-29 16:16 -0700

csiph-web