Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.arch.embedded > #25256

Re: Checking large numbers of files

Show all headers | View raw

>> [Still 35C!  Cripes!!]
>
> we switched from walking around in shorts to winter uniform and
> house heating almost a month ago...

We're still running the air conditioning.  Unusual to be using it
this long.  I think the *average* high temp for October, this year,
was 33C.  I know we've had 20+ days above 90F (and usually WELL
above) and it's only the 27th...

>> First, timestamps on "containers" (e.g., things like "folders")
>> don't mean anything.  If /foo/A and /foo/B are "being tracked"
>> but other contents of /foo are not, then the timestamp of /foo
>> can change even though NEITHER A nor B have been altered!
>
> Well yes, the last modification date of a directory may be
> impractical to be kept up to date. I made this optional (by changing
> the directory file type) some 20+ years ago...

And, even if ALL of the directory's contents are "being tracked",
you can do:
    mv * ../X
    mv ../X/* .
to effectively leave teh directory "as it was" -- yet the timestamp
is modified.

> But the file "last modification date" is pretty stable, can't see
> it change for no good reason. I often do copy with the option to
> overwrite only files which are older than the source to be copied,
> has not bitten me so far.

Pull the disk out of your computer and put it in another computer
(which is what you effectively do when you access it over the wire
*or* physically move an external medium).

Or, "touch *" -- none of the files' contents/names have been altered
but all of their timestamps have.  Which timestamp is now correct:
the timestamp of the touched file(s) or the "original" timestamps
(that have been lost)?

Remember, any application that accesses the files can also dick with
the timestamps.  *You* can vouch for all of YOUR applications (having
written them all).  But, *I* am not keen on trying to quantify that
behavior for those, here.

>> Also, I access/mount the volumes from a variety of different hosts
>> and "maintain" the files with a variety of different utilities.
>> The clocks on all of these hosts aren't guaranteed to be synchronized.
>> So, you can't infer chronology from the timestamps.
>
> Hmmm, when copying a file its timestamp should normally be copied
> as well so local clocks should not matter re file contents.

Yes, if you are copying it on a *local* machine.  But, what happens
to the timestamp of an object that you fetch from a web server? :>

>> And, the mechanisms that I use to move files around may not
>> preserve original timestamps, accurately.
>>
>> E.g., I may move files onto a volume over FTP -- using a client
>> that doesn't propagate the timestamp of the *original* file.
>
> This is an issue as old as ftp is, yes... If you use GET with
> ftp the file timestamp can come correctly across (MLST). Still
> no standard way to propagate the correct timestamp when you "STOR".

So, do I "certify" every web/ftp/scp/fxp/etc client?  Or,
manually verify the timestamps are "approximately" correct
(i.e., 201610272110 is PROBABLY wrong for a local file)?

One of the graphical file synchronization tools I use highlights
differences with color.  It's not uncommon for me to see *mirrored*
filestores claiming to have multiple "discrepancies".  On closer
examination, it's almost always a timestamp error.

FAT12 forces timestamps to a 2 second granularity (my PROM
programmer still uses floppies as does at least one of my logic
analyzers and a DSO) -- something to remember when comparing
files on floppies to the original "archive images"!

>> So, that instance of the file now has a different timestamp
>> from the original FROM WHICH IT WAS COPIED.  Yet, I *know*
>> they are identical, in contents.
>
> Well yes, using ftp "put" does that indeed - but I can't think
> of another cause so it is soft of avoidable.
>
>> [Imagine a scenario where a volume is failing; you'd much rather know
>> about a file that is NOT present on a "backup copy" than be reassured
>> that the contents of all the OTHER files are intact; you KNOW you
>> will lose data due to the missing file but MIGHT not lose anything
>> from those other files if you've not yet wasted time checking them!]
>
> Well searching the whole tree for a particular name would be painful
> but still faster than copying it all. I personally tend to know which
> file belongs where so locating it is easy - but this is not much help
> if you look into an archive you are not familiar with.

I have one half of the dataset readily available in the DBMS:
I know which files should exist at each level of the hierarchy,
their sizes, names, containers, and signatures -- BEFORE the drive
is even on-line.

So, I can just walk through the file hierarchy on the medium
and build an unordered list of pathnames/filenames.  Then, sort
this and compare it to the same sorted list from the DBMS.
Superficial differences (names, dates, sizes) will stand out
(the DBMS query can find them all for me faster than I could
find them myself as it can exploit indexes its precomputed).

The real costs come in:
- verifying the *actual* size of the file (i.e., a disk error may
   effectively truncate it or otherwise rendered parts of it inaccessible)
- verifying the actual signature (hash) of the file

So, I should "do the easy checks" first (for ALL contents) instead
of processing files COMPLETELY, but one-at-a-time -- *if* I want to
expeditiously locate potential problems.  (I just think timestamps
would result in me chasing problems that weren't actually problems)

Check your mail for more detail (look at Subject lines as I've
been changing email accounts, lately)

Back to comp.arch.embedded | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-27 15:23 -0700
  Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-28 09:48 +1100
    Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-27 16:48 -0700
      Re: Checking large numbers of files Dimiter_Popoff <dp@tgi-sci.com> - 2016-10-28 03:17 +0300
        Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-27 18:50 -0700
          Re: Checking large numbers of files Dimiter_Popoff <dp@tgi-sci.com> - 2016-10-28 05:38 +0300
            Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-27 21:33 -0700
      Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-28 11:40 +1100
        Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-27 18:52 -0700
  Re: Checking large numbers of files kalvin.news@gmail.com - 2016-10-28 03:50 -0700
    Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-28 10:49 -0700
      Re: Checking large numbers of files kalvin.news@gmail.com - 2016-10-29 05:17 -0700
        Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 13:15 -0700
          Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 09:47 +1100
            Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 16:19 -0700
              Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 16:36 -0700
              Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 12:00 +1100
                Re: Checking large numbers of files Dimiter_Popoff <dp@tgi-sci.com> - 2016-10-30 04:05 +0200
                Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 14:06 +1100
                Re: Checking large numbers of files Dimiter_Popoff <dp@tgi-sci.com> - 2016-10-30 05:48 +0200
                Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 22:33 -0700
                Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 17:00 +1100
                Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-10-31 04:43 -0700
                Re: Checking large numbers of files (long) David Brown <david.brown@hesbynett.no> - 2016-10-31 14:20 +0100
                Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-10-31 08:02 -0700
                Re: Checking large numbers of files (long) David Brown <david.brown@hesbynett.no> - 2016-11-01 09:47 +0100
                Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-11-01 04:36 -0700
                Re: Checking large numbers of files (long) David Brown <david.brown@hesbynett.no> - 2016-11-01 14:29 +0100
                Re: Checking large numbers of files (long) David Brown <david.brown@hesbynett.no> - 2016-11-01 15:16 +0100
                Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-11-02 11:42 -0700
                Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 22:15 -0700
                Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 21:50 -0700
                Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 17:14 +1100
                Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-10-31 03:36 -0700
                Re: Checking large numbers of files (long) Clifford Heath <no.spam@please.net> - 2016-11-01 09:20 +1100
                Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-11-01 02:23 -0700
                Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 21:20 -0700
                Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 17:25 +1100
          Re: Checking large numbers of files Robert Wessel <robertwessel2@yahoo.com> - 2016-10-30 00:14 -0500
            Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 23:02 -0700
          Re: Checking large numbers of files David Brown <david.brown@hesbynett.no> - 2016-10-31 12:19 +0100
            Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-31 07:00 -0700
  Re: Checking large numbers of files Ali <abdulrazaq@gmail.com> - 2016-10-30 15:45 -0700
    Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-31 03:45 -0700

csiph-web