Path: csiph.com!newsfeed.hal-mli.net!feeder3.hal-mli.net!newsfeed.hal-mli.net!feeder2.hal-mli.net!de-l.enfer-du-nord.net!feeder2.enfer-du-nord.net!cs.uu.nl!news.stack.nl!.POSTED!not-for-mail
From: Willem <willem@turtle.stack.nl>
Newsgroups: comp.programming
Subject: Re: Do integrity checks by file comparison really work?
Date: Mon, 14 Jan 2013 18:23:39 +0000 (UTC)
Organization: Stack Usenet News Service
Lines: 71
Message-ID: <slrnkf8j9a.tgv.willem@turtle.stack.nl>
References: <50ec0b68$0$17952$4fafbaef@reader1.news.tin.it>
NNTP-Posting-Host: turtle.stack.nl
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
X-Trace: mud.stack.nl 1358187819 69888 2001:610:1108:5010::132 (14 Jan 2013 18:23:39 GMT)
X-Complaints-To: abuse@stack.nl
NNTP-Posting-Date: Mon, 14 Jan 2013 18:23:39 +0000 (UTC)
User-Agent: slrn/0.9.9p1 (FreeBSD)
Xref: csiph.com comp.programming:2814

Thomas B wrote:
) Hello,
)
) I'm sorry if this post is not really programming-related, but I thought 
) that programmers could know about it.
)
) I use a program (Beyond Compare) which has a Folder Compare module, with 
) a function for doing a binary comparison between the files inside two 
) folders. I use it to check the integrity of my backups, by comparing my 
) main hard disk files, with the ones on the backup hard disks. If two 
) files get recognized as being different, then one of them should be 
) corrupt (if they didn't get modified "normally").
)
) My question is: will this method work to detect real file corruptions? 

Yes, it should.

) Will it detect any kind of corruption, both copy corruptions, and "bit 
) rot" corruptions? Will there be no problem regarding read cache?

Well, if you just wrote it, it might remain in the cache and not get read
from disk.  But not what you meant below.

) I mean: 
) if the first file to compare gets read from disk, and is kept in a read 
) cache (by Windows or by the hard disk), will the second file get read 
) from the same cache?

No.

) Do Windows and hard disks have some kind of 
) procedure to detect if a file to read from disk is already available in 
) cache, even if it is in a different folder than the "original" one? 

No.

) Perhaps some kind of file-checksum-system which decides that the files 
) are same? (And this system would not notice if the file to compare is 
) corrupt). If this would be true, then integrity checks by file 
) comparison would not work, because in practice the same file would be 
) read twice (first from disk, and then from cache), instead of reading 
) both the two files to be compared from disk.

The only thing that could happen (but not on Windows) is that the file
system decides that two files are the same and then uses the same disk
space to store them in.  Which also means that if the disk gets corrupted,
both files are bad.

) I have already done test by manually "corrupting" files (changing 
) slightly the contents, while keeping size and timestamp the same), and 
) it works (the files get recognized as different). But I'm not sure if it 
) will work also with "real" corrupt files.

If you read immediately after writing, it might be read out of the write
cache.  It could even be that it hasn't actually been written to disk yet(*).

Other than that, there are no issues.


*) Some newer file systems keep small files in the write cache for a longer
   time, because it anticipates that those are temporary files which will
   be deleted soon anyway.  So they never make it to disk.


SaSW, Willem
-- 
Disclaimer: I am in no way responsible for any of the statements
            made in the above text. For all I know I might be
            drugged or something..
            No I'm not paranoid. You all think I'm paranoid, don't you !
#EOT