Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.arch.embedded > #25263

Re: Checking large numbers of files

Show all headers | View raw

On 10/28/2016 3:50 AM, kalvin.news@gmail.com wrote:
> For a very crude check you can check that the file size hasn't changed in
> addition to file time stamp.

"File size" just tells me the *stored* metadata hasn't "been altered"
for the file.  There is still the possibility that the actual size
*has*, effectively, changed (think: access errors, sparse file support,
etc.).  Knowing the object's size, with certainty, still requires its
more thorough examination.

I am leary of using timestamps as they are too easily altered.  E.g.,
I've seen build scripts that deliberately, unconditionally, touch(1)
some "marker file" just to effectively record their actions -- despite
the contents of the file being unaltered.  (e.g., consider a file
named "MostRecentBuildTime" -- the timestamp of which actually
contains the information the file's presence is intended to convey).

As I am looking for information that can help me INTELLIGENTLY decide
where more careful scrutiny should be INITIALLY focused, I'd ideally
like things that can't be as easily altered *without* indicating real
potential changes to the object's content.  E.g., altering the file's
size can't be accomplished without actually altering the file's content!

[However, if I am only using the timestamp to tell if *this* file has
changed from *its* previous state (i.e., not using it to tell if this
file is DIFFERENT/changed from the original from which it was copied),
it potentially offers *some* information.]

> Then, you could possibly compute a fast hash
> for each file and use it as the first line of verification instead of using
> more expensive hash algorithms. If you just need to check if the file has
> been altered a simple 64-bit CRC might be usable.

I've benchmarked a variety of different hashes to get a feel for how
expensive they are.  There is a HUGE range of performance -- like
2 orders of magnitude (neglecting hardware accelerators)!

But, the real pissers are the access and transport delays for the raw file
data; getting it off the media and *to* the "process" that is actually
examining it!  In many cases, the volume is an external drive (which can
mean the interface is the bottleneck), a "consumer" drive (intended to be
inexpensive, but not very "performant"), uses an old interface (e.g.,
USB2) or is inherently slow (e.g., optical media).

So, looking at the file contents is the *last* thing I'd want to do
to get a feel for the file's integrity -- it is the FINAL assessment!
Ideally, I want indicators of "bad data" that are far less expensive
(e.g., if a file SHOULD be 16KB and it stat(2)'s as 30GB, I sure as
hell don't want to waste time hashing 30 billion bytes before deciding
that the file (or its metadata) has been corrupted!

> You should of course
> evaluate the implications of using hash algorithms that can produce
> collisions and that are not considered safe and robust.

Yes.  But, the hash isn't trying to protect against an active adversary
attempting to deliberately manipulate the files contents in a selective
manner to preserve the original hash.  Instead, I just have to evaluate
likely media failure modes that could potentially alter the data in ways
that would NOT be detected by the hash.

> Some file systems
> provide hooks that can monitor file alterations. Creating an observer which
> will monitor any file changes, you will be notified if any file should be
> altered.

This requires the filesystem/media to be visible to such an observer at all
times.  That would force all the media into a live repository, of sorts
(regardless of whether it is centralized or distributed).

And, it doesn't address the potential for bitrot.

I'm looking for something where the media can be offline and/or potentially
manipulated "unsupervised" (and, while not *maliciously*, perhaps prone
to missteps).  I also have to tolerate R/O media as well as a prohibition
against adding "breadcrumbs" to the monitored media (i.e., no "control files"
littering the monitored media).

[E.g., to take mass-produced optical media and verify *it* is intact
as well as R/W media for which the client does not have write/create
privileges -- or that may have write-protect tabs]

Back to comp.arch.embedded | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-27 15:23 -0700
  Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-28 09:48 +1100
    Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-27 16:48 -0700
      Re: Checking large numbers of files Dimiter_Popoff <dp@tgi-sci.com> - 2016-10-28 03:17 +0300
        Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-27 18:50 -0700
          Re: Checking large numbers of files Dimiter_Popoff <dp@tgi-sci.com> - 2016-10-28 05:38 +0300
            Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-27 21:33 -0700
      Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-28 11:40 +1100
        Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-27 18:52 -0700
  Re: Checking large numbers of files kalvin.news@gmail.com - 2016-10-28 03:50 -0700
    Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-28 10:49 -0700
      Re: Checking large numbers of files kalvin.news@gmail.com - 2016-10-29 05:17 -0700
        Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 13:15 -0700
          Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 09:47 +1100
            Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 16:19 -0700
              Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 16:36 -0700
              Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 12:00 +1100
                Re: Checking large numbers of files Dimiter_Popoff <dp@tgi-sci.com> - 2016-10-30 04:05 +0200
                Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 14:06 +1100
                Re: Checking large numbers of files Dimiter_Popoff <dp@tgi-sci.com> - 2016-10-30 05:48 +0200
                Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 22:33 -0700
                Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 17:00 +1100
                Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-10-31 04:43 -0700
                Re: Checking large numbers of files (long) David Brown <david.brown@hesbynett.no> - 2016-10-31 14:20 +0100
                Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-10-31 08:02 -0700
                Re: Checking large numbers of files (long) David Brown <david.brown@hesbynett.no> - 2016-11-01 09:47 +0100
                Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-11-01 04:36 -0700
                Re: Checking large numbers of files (long) David Brown <david.brown@hesbynett.no> - 2016-11-01 14:29 +0100
                Re: Checking large numbers of files (long) David Brown <david.brown@hesbynett.no> - 2016-11-01 15:16 +0100
                Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-11-02 11:42 -0700
                Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 22:15 -0700
                Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 21:50 -0700
                Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 17:14 +1100
                Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-10-31 03:36 -0700
                Re: Checking large numbers of files (long) Clifford Heath <no.spam@please.net> - 2016-11-01 09:20 +1100
                Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-11-01 02:23 -0700
                Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 21:20 -0700
                Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 17:25 +1100
          Re: Checking large numbers of files Robert Wessel <robertwessel2@yahoo.com> - 2016-10-30 00:14 -0500
            Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 23:02 -0700
          Re: Checking large numbers of files David Brown <david.brown@hesbynett.no> - 2016-10-31 12:19 +0100
            Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-31 07:00 -0700
  Re: Checking large numbers of files Ali <abdulrazaq@gmail.com> - 2016-10-30 15:45 -0700
    Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-31 03:45 -0700

csiph-web