Checking large numbers of files

From	Don Y <blockedofcourse@foo.invalid>
Newsgroups	comp.arch.embedded
Subject	Checking large numbers of files
Date	2016-10-27 15:23 -0700
Organization	A noiseless patient Spider
Message-ID	<nutuoo$k2$1@dont-email.me> (permalink)

Show all headers | View raw

I currently "verify" my data archive with an algorithm similar to:

ONCE: con 0;
Success: con 0;
Removed, Inaccessible, NoRecord, BadSize, BadHash: con Success+1+iota;

verify(files: list of string ): list of (string, reason)
{
    failures := list of (string, reason);

    while (files != nil) {
       theFile := hd files;
       (size, signature, result) := examine_file(theFile);
       (Size, Signature, Result) := query_DBMS(theFile);
       do {
          if (NotFound == result)
             {Result = Removed;      break;}
          if (BadRead == result)
             {Result = Inaccessible; break;}

          // theFile exists!
          if (NotFound == Result)
             {Result = NoRecord;     break;}
          if (Size != size)
             {Result = BadSize;      break;}
          if (Signature != signature)
             {Result = BadHash;      break;}
       } while (ONCE);

       if (Success != Result) {
          failures = (theFile, Result) :: failures;
       }

       files = tl files;
    }
    return( failures );
}

[apologies for any typos; too early in the day to be working!]

The problem this has is examine-file() is expensive -- for large
numbers of large files!

I'd like to short-circuit this to provide an early indication of
possible problems.  E.g., stat(2) each "theFile" to identify any
that are "Removed" without incurring the cost of examining the
file in its entirety.

[Of course, all files MUST be eventually examined as you can't
verify accessibility (BadRead), actual size (BadSize) or correct
signature (BadHash) without looking at every byte!  But, you can
get the simple checks out of the way expeditiously to alert you
to "obvious" problems!]

I can get an approximate indication of "actual size" just by
stat(2)-ing the file, as well -- relying on examine_file() to
later confirm that figure.

The hash, of course, can't be guesstimated without trodding through
the file in its entirety.

The problem with *this* approach is I'd have to lock the entire
object to ensure the integrity of the "early results" agrees with
that of the "later results"  (the original approach only requires
taking a lock on the file being examined *while* it is being
examined).

Are there any "tricks" I can use, here?  E.g., the equivalent of
CoW would ensure the integrity of my data (for the *original*
instance) in the presence of such a global lock.  But, I think only
the Bullet Server would give me any behavior approaching this...?

Also, how wary would one be relying on the files' timestamps as a
(crude) indication of "alteration"?  (It seems far too easy to
alter a timestamp without altering content/size so I've been leary
of relying on that as having any "worth").

Back to comp.arch.embedded | Previous | Next — Next in thread | Find similar | Unroll thread

Thread

Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-27 15:23 -0700
  Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-28 09:48 +1100
    Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-27 16:48 -0700
      Re: Checking large numbers of files Dimiter_Popoff <dp@tgi-sci.com> - 2016-10-28 03:17 +0300
        Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-27 18:50 -0700
          Re: Checking large numbers of files Dimiter_Popoff <dp@tgi-sci.com> - 2016-10-28 05:38 +0300
            Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-27 21:33 -0700
      Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-28 11:40 +1100
        Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-27 18:52 -0700
  Re: Checking large numbers of files kalvin.news@gmail.com - 2016-10-28 03:50 -0700
    Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-28 10:49 -0700
      Re: Checking large numbers of files kalvin.news@gmail.com - 2016-10-29 05:17 -0700
        Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 13:15 -0700
          Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 09:47 +1100
            Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 16:19 -0700
              Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 16:36 -0700
              Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 12:00 +1100
                Re: Checking large numbers of files Dimiter_Popoff <dp@tgi-sci.com> - 2016-10-30 04:05 +0200
                Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 14:06 +1100
                Re: Checking large numbers of files Dimiter_Popoff <dp@tgi-sci.com> - 2016-10-30 05:48 +0200
                Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 22:33 -0700
                Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 17:00 +1100
                Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-10-31 04:43 -0700
                Re: Checking large numbers of files (long) David Brown <david.brown@hesbynett.no> - 2016-10-31 14:20 +0100
                Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-10-31 08:02 -0700
                Re: Checking large numbers of files (long) David Brown <david.brown@hesbynett.no> - 2016-11-01 09:47 +0100
                Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-11-01 04:36 -0700
                Re: Checking large numbers of files (long) David Brown <david.brown@hesbynett.no> - 2016-11-01 14:29 +0100
                Re: Checking large numbers of files (long) David Brown <david.brown@hesbynett.no> - 2016-11-01 15:16 +0100
                Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-11-02 11:42 -0700
                Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 22:15 -0700
                Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 21:50 -0700
                Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 17:14 +1100
                Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-10-31 03:36 -0700
                Re: Checking large numbers of files (long) Clifford Heath <no.spam@please.net> - 2016-11-01 09:20 +1100
                Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-11-01 02:23 -0700
                Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 21:20 -0700
                Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 17:25 +1100
          Re: Checking large numbers of files Robert Wessel <robertwessel2@yahoo.com> - 2016-10-30 00:14 -0500
            Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 23:02 -0700
          Re: Checking large numbers of files David Brown <david.brown@hesbynett.no> - 2016-10-31 12:19 +0100
            Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-31 07:00 -0700
  Re: Checking large numbers of files Ali <abdulrazaq@gmail.com> - 2016-10-30 15:45 -0700
    Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-31 03:45 -0700

csiph-web