Re: Checking large numbers of files

From	Don Y <blockedofcourse@foo.invalid>
Newsgroups	comp.arch.embedded
Subject	Re: Checking large numbers of files
Date	2016-10-27 16:48 -0700
Organization	A noiseless patient Spider
Message-ID	<nuu3mu$fp2$1@dont-email.me> (permalink)
References	<nutuoo$k2$1@dont-email.me> <5812842a$0$63774$c3e8da3$38634283@news.astraweb.com>

Show all headers | View raw

On 10/27/2016 3:48 PM, Clifford Heath wrote:
> On 28/10/16 09:23, Don Y wrote:
>> I currently "verify" my data archive with an algorithm similar to:
> ...
>> Are there any "tricks" I can use, here?
>
> Git is very fast at verifying files. You might find that
> "git annex" is just right for maintaining your archives.
> It's built for just this, and more.

There's more to it than just verifying a file hasn't changed
(I can do that by comparing its current hash -- pick your
favorite algorithm -- to a stored hash).  There are innumerable
tools for doing that, synchronizing different copies on
different volumes (even if a volume is offline), etc.

The point of my post was to avoid computing hashes of terabytes
of files when I can discover obvious changes by looking at file
sizes, dates, names, etc.

OTOH, as certain metadata can be too easily corrupted (e.g., dates),
incorporating them in a strategy can result in too many false
positives: "The timestamp on 'foo' has changed.  I know you
'verified' the contents of the entire file YESTERDAY, but the
changed timestamp suggests you might want to reverify it, today!"

[I also want the "process" to automatically pause and resume as
I asynchronously add and remove volumes -- possibly idling
for weeks or months before having access to a volume, again]

Thread

Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-27 15:23 -0700
  Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-28 09:48 +1100
    Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-27 16:48 -0700
      Re: Checking large numbers of files Dimiter_Popoff <dp@tgi-sci.com> - 2016-10-28 03:17 +0300
        Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-27 18:50 -0700
          Re: Checking large numbers of files Dimiter_Popoff <dp@tgi-sci.com> - 2016-10-28 05:38 +0300
            Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-27 21:33 -0700
      Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-28 11:40 +1100
        Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-27 18:52 -0700
  Re: Checking large numbers of files kalvin.news@gmail.com - 2016-10-28 03:50 -0700
    Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-28 10:49 -0700
      Re: Checking large numbers of files kalvin.news@gmail.com - 2016-10-29 05:17 -0700
        Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 13:15 -0700
          Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 09:47 +1100
            Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 16:19 -0700
              Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 16:36 -0700
              Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 12:00 +1100
                Re: Checking large numbers of files Dimiter_Popoff <dp@tgi-sci.com> - 2016-10-30 04:05 +0200
                Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 14:06 +1100
                Re: Checking large numbers of files Dimiter_Popoff <dp@tgi-sci.com> - 2016-10-30 05:48 +0200
                Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 22:33 -0700
                Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 17:00 +1100
                Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-10-31 04:43 -0700
                Re: Checking large numbers of files (long) David Brown <david.brown@hesbynett.no> - 2016-10-31 14:20 +0100
                Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-10-31 08:02 -0700
                Re: Checking large numbers of files (long) David Brown <david.brown@hesbynett.no> - 2016-11-01 09:47 +0100
                Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-11-01 04:36 -0700
                Re: Checking large numbers of files (long) David Brown <david.brown@hesbynett.no> - 2016-11-01 14:29 +0100
                Re: Checking large numbers of files (long) David Brown <david.brown@hesbynett.no> - 2016-11-01 15:16 +0100
                Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-11-02 11:42 -0700
                Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 22:15 -0700
                Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 21:50 -0700
                Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 17:14 +1100
                Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-10-31 03:36 -0700
                Re: Checking large numbers of files (long) Clifford Heath <no.spam@please.net> - 2016-11-01 09:20 +1100
                Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-11-01 02:23 -0700
                Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 21:20 -0700
                Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 17:25 +1100
          Re: Checking large numbers of files Robert Wessel <robertwessel2@yahoo.com> - 2016-10-30 00:14 -0500
            Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 23:02 -0700
          Re: Checking large numbers of files David Brown <david.brown@hesbynett.no> - 2016-10-31 12:19 +0100
            Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-31 07:00 -0700
  Re: Checking large numbers of files Ali <abdulrazaq@gmail.com> - 2016-10-30 15:45 -0700
    Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-31 03:45 -0700

csiph-web