Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.programming > #2755 > unrolled thread

Do integrity checks by file comparison really work?

Back to article view | Back to comp.programming

Contents

  Do integrity checks by file comparison really work? Thomas B <thomasb@nospam.it> - 2013-01-08 13:04 +0100
    Re: Do integrity checks by file comparison really work? Robert Wessel <robertwessel2@yahoo.com> - 2013-01-13 05:04 -0600
      Re: Do integrity checks by file comparison really work? Mark F <mark53916@gmail.com> - 2013-01-13 09:26 -0500
        Re: Do integrity checks by file comparison really work? Mark F <mark53916@gmail.com> - 2013-01-13 09:40 -0500
      Re: Do integrity checks by file comparison really work? Thomas B <thomasb@nospam.it> - 2013-01-16 11:38 +0100
        Re: Do integrity checks by file comparison really work? Robert Wessel <robertwessel2@yahoo.com> - 2013-01-18 15:07 -0600
          Re: Do integrity checks by file comparison really work? Ian Collins <ian-news@hotmail.com> - 2013-01-19 17:12 +1300
            Re: Do integrity checks by file comparison really work? Robert Wessel <robertwessel2@yahoo.com> - 2013-01-18 22:37 -0600
              Re: Do integrity checks by file comparison really work? Ian Collins <ian-news@hotmail.com> - 2013-01-19 18:36 +1300
          Re: Do integrity checks by file comparison really work? Thomas B <thomasb@nospam.it> - 2013-01-19 14:06 +0100
            Re: Do integrity checks by file comparison really work? Robert Wessel <robertwessel2@yahoo.com> - 2013-01-21 03:40 -0600
              Re: Do integrity checks by file comparison really work? Thomas B <thomasb@nospam.it> - 2013-01-21 17:41 +0100
                Re: Do integrity checks by file comparison really work? Robert Wessel <robertwessel2@yahoo.com> - 2013-01-21 15:28 -0600
    Re: Do integrity checks by file comparison really work? Willem <willem@turtle.stack.nl> - 2013-01-14 18:23 +0000
      Re: Do integrity checks by file comparison really work? Thomas B <thomasb@nospam.it> - 2013-01-16 16:46 +0100
        Re: Do integrity checks by file comparison really work? Robert Wessel <robertwessel2@yahoo.com> - 2013-01-18 15:09 -0600

#2755 — Do integrity checks by file comparison really work?

Hello,

I'm sorry if this post is not really programming-related, but I thought 
that programmers could know about it.

I use a program (Beyond Compare) which has a Folder Compare module, with 
a function for doing a binary comparison between the files inside two 
folders. I use it to check the integrity of my backups, by comparing my 
main hard disk files, with the ones on the backup hard disks. If two 
files get recognized as being different, then one of them should be 
corrupt (if they didn't get modified "normally").

My question is: will this method work to detect real file corruptions? 
Will it detect any kind of corruption, both copy corruptions, and "bit 
rot" corruptions? Will there be no problem regarding read cache? I mean: 
if the first file to compare gets read from disk, and is kept in a read 
cache (by Windows or by the hard disk), will the second file get read 
from the same cache? Do Windows and hard disks have some kind of 
procedure to detect if a file to read from disk is already available in 
cache, even if it is in a different folder than the "original" one? 
Perhaps some kind of file-checksum-system which decides that the files 
are same? (And this system would not notice if the file to compare is 
corrupt). If this would be true, then integrity checks by file 
comparison would not work, because in practice the same file would be 
read twice (first from disk, and then from cache), instead of reading 
both the two files to be compared from disk.

I have already done test by manually "corrupting" files (changing 
slightly the contents, while keeping size and timestamp the same), and 
it works (the files get recognized as different). But I'm not sure if it 
will work also with "real" corrupt files.

I'm interested mostly about Windows 8 Pro 64bit and NTFS (but would like 
to know also in general).
Thanks.

[toc] | [next] | [standalone]

#2803

On Tue, 08 Jan 2013 13:04:49 +0100, Thomas B <thomasb@nospam.it>
wrote:

>Hello,
>
>I'm sorry if this post is not really programming-related, but I thought 
>that programmers could know about it.
>
>I use a program (Beyond Compare) which has a Folder Compare module, with 
>a function for doing a binary comparison between the files inside two 
>folders. I use it to check the integrity of my backups, by comparing my 
>main hard disk files, with the ones on the backup hard disks. If two 
>files get recognized as being different, then one of them should be 
>corrupt (if they didn't get modified "normally").
>
>My question is: will this method work to detect real file corruptions? 
>Will it detect any kind of corruption, both copy corruptions, and "bit 
>rot" corruptions? Will there be no problem regarding read cache? I mean: 
>if the first file to compare gets read from disk, and is kept in a read 
>cache (by Windows or by the hard disk), will the second file get read 
>from the same cache? Do Windows and hard disks have some kind of 
>procedure to detect if a file to read from disk is already available in 
>cache, even if it is in a different folder than the "original" one? 
>Perhaps some kind of file-checksum-system which decides that the files 
>are same? (And this system would not notice if the file to compare is 
>corrupt). If this would be true, then integrity checks by file 
>comparison would not work, because in practice the same file would be 
>read twice (first from disk, and then from cache), instead of reading 
>both the two files to be compared from disk.
>
>I have already done test by manually "corrupting" files (changing 
>slightly the contents, while keeping size and timestamp the same), and 
>it works (the files get recognized as different). But I'm not sure if it 
>will work also with "real" corrupt files.
>
>I'm interested mostly about Windows 8 Pro 64bit and NTFS (but would like 
>to know also in general).


Mostly yes.  I've never seen an OS that would share file buffers
between two files like that, although I suppose it is possible.

Now that doesn't necessarily apply to situations where the two files
are not really different (*nix hard links, for example).  It also has
a issue that you might not detect a write error if the read to that
device is satisfied from the original buffer use to write.  Also, you
might now have gotten the buffers written at all, if the OS is being
somewhat lazy about writing dirty buffers to disk.

But so long as you're on different volumes, and you arrange to discard
all data in disk cache before you do the compare (which may require
powering off the system and restarting it in some cases), you should
be fine.  Or if you can ask the OS to address a file with physical
reads at all times.

OTOH, as a practical matter, this is silly, hard disks and file system
caches are reliable enough that the above doesn't really present and
meaningful exposure in the vast majority of cases.  And if you're that
worried, you should be using some removable media anyway.

[toc] | [prev] | [next] | [standalone]

#2804

On Tue, 08 Jan 2013 13:04:49 +0100, Thomas B <thomasb@nospam.it>
wrote in part:
> >I use a program (Beyond Compare) which has a Folder Compare module, with 
> >a function for doing a binary comparison between the files inside two 
> >folders. I use it to check the integrity of my backups, by comparing my 
> >main hard disk files, with the ones on the backup hard disks. If two 
> >files get recognized as being different, then one of them should be 
> >corrupt (if they didn't get modified "normally").
> >
> >My question is: will this method work to detect real file corruptions? 
> >Will it detect any kind of corruption, both copy corruptions, and "bit 
> >rot" corruptions?
There are also issues as to what is actually copied and what is 
compared.
 Does Beyond Compare compare NTFS Alternate Data Streams?
  http://en.wikipedia.org/wiki/NTFS#Alternate_data_streams_.28ADS.29
 All of the ADS's should be included in the copy and compared

 How does Beyond Compare handle sparse files?
  http://en.wikipedia.org/wiki/NTFS#Sparse_files
 
 Which of the NTFS date fields are compared?
  The user normally sees Created, Accessed, and Modified,
  but there are several other dates that I don't know the definition
  of.  Some of these alternate dates probably need to be maintained
  if a copy is to be considered correct.

Encrypted files that the user is not able to decrypt are also an
issue.

[toc] | [prev] | [next] | [standalone]

#2805

On Sun, 13 Jan 2013 09:26:30 -0500, Mark F <mark53916@gmail.com>
wrote:

> On Tue, 08 Jan 2013 13:04:49 +0100, Thomas B <thomasb@nospam.it>
> wrote in part:
> > >I use a program (Beyond Compare) which has a Folder Compare module, with 
> > >a function for doing a binary comparison between the files inside two 
> > >folders. I use it to check the integrity of my backups, by comparing my 
> > >main hard disk files, with the ones on the backup hard disks. If two 
> > >files get recognized as being different, then one of them should be 
> > >corrupt (if they didn't get modified "normally").
> > >
> > >My question is: will this method work to detect real file corruptions? 
> > >Will it detect any kind of corruption, both copy corruptions, and "bit 
> > >rot" corruptions?
> There are also issues as to what is actually copied and what is 
> compared.
>  Does Beyond Compare compare NTFS Alternate Data Streams?
>   http://en.wikipedia.org/wiki/NTFS#Alternate_data_streams_.28ADS.29
>  All of the ADS's should be included in the copy and compared
> 
>  How does Beyond Compare handle sparse files?
>   http://en.wikipedia.org/wiki/NTFS#Sparse_files
>  
>  Which of the NTFS date fields are compared?
>   The user normally sees Created, Accessed, and Modified,
>   but there are several other dates that I don't know the definition
>   of.  Some of these alternate dates probably need to be maintained
>   if a copy is to be considered correct.
> 
I left out:
  How does Beyond Compare handle compressed files?
   For instance, is maintaining compression state optionally
    checked for if both sides are compressed is it optionally
    required that both compressed forms of the data match?
> Encrypted files that the user is not able to decrypt are also an
> issue.

[toc] | [prev] | [next] | [standalone]

#2822

Thanks to everybody who has replied.

[...]

On 13/01/2013 12.04, Robert Wessel wrote:

> It also has
> a issue that you might not detect a write error if the read to that
> device is satisfied from the original buffer use to write.  Also, you
> might now have gotten the buffers written at all, if the OS is being
> somewhat lazy about writing dirty buffers to disk.

Yes, I was aware of this.

> But so long as you're on different volumes, and you arrange to discard
> all data in disk cache before you do the compare (which may require
> powering off the system and restarting it in some cases), you should
> be fine.  Or if you can ask the OS to address a file with physical
> reads at all times.

I don't understand what you are saying about "different volumes". Which 
would be the problem when comparing two files on the same volume?

You wrote: "I've never seen an OS that would share file buffers between 
two files like that", so I suppose this should be the same for files on 
the same volume.

Regarding discarding/flushing the disk cache, should a simple system 
reboot (instead of powering off) do the job, with most "normal" hard disks?

> OTOH, as a practical matter, this is silly, hard disks and file system
> caches are reliable enough that the above doesn't really present and
> meaningful exposure in the vast majority of cases.  And if you're that
> worried, you should be using some removable media anyway.

I do backups both to an internal hard drive, and to external USB hard 
drives. Why are you suggesting to use some removable media (do you mean 
a USB hard drive, or optical disks, or other?). I don't understand what 
you mean, which is the advantage.

Thanks,

Thomas

[toc] | [prev] | [next] | [standalone]

#2856

On Wed, 16 Jan 2013 11:38:54 +0100, Thomas B <thomasb@nospam.it>
wrote:

>Thanks to everybody who has replied.
>
>[...]
>
>On 13/01/2013 12.04, Robert Wessel wrote:
>
>> It also has
>> a issue that you might not detect a write error if the read to that
>> device is satisfied from the original buffer use to write.  Also, you
>> might now have gotten the buffers written at all, if the OS is being
>> somewhat lazy about writing dirty buffers to disk.
>
>Yes, I was aware of this.
>
>> But so long as you're on different volumes, and you arrange to discard
>> all data in disk cache before you do the compare (which may require
>> powering off the system and restarting it in some cases), you should
>> be fine.  Or if you can ask the OS to address a file with physical
>> reads at all times.
>
>I don't understand what you are saying about "different volumes". Which 
>would be the problem when comparing two files on the same volume?
>
>You wrote: "I've never seen an OS that would share file buffers between 
>two files like that", so I suppose this should be the same for files on 
>the same volume.


Some OS's/filesystems can do a copy with the equivalent of a *nix hard
link, and then do a copy-on-write procedure if the file is ever
modified.  In that case you may have only a single copy of the file.
It can't do that to you across volumes.


>Regarding discarding/flushing the disk cache, should a simple system 
>reboot (instead of powering off) do the job, with most "normal" hard disks?


Usually.  Depending on how the drive is configured, it may still have
data in (device) cache not yet written to disk.  The OS should force a
cache flush on the device when it shuts down (on SCSI, a
Synchronize-Cache, for example).  But a reboot without a clean
shutdown might not flush things in all cases.  Higher end devices
would have non-volatile write buffers, which makes the issue mostly
moot (unless there's a time limit on the non-volatility, and you leave
the drive powered down for more than that).


>> OTOH, as a practical matter, this is silly, hard disks and file system
>> caches are reliable enough that the above doesn't really present and
>> meaningful exposure in the vast majority of cases.  And if you're that
>> worried, you should be using some removable media anyway.
>
>I do backups both to an internal hard drive, and to external USB hard 
>drives. Why are you suggesting to use some removable media (do you mean 
>a USB hard drive, or optical disks, or other?). I don't understand what 
>you mean, which is the advantage.


An external USB drive is fine.  It certainly counts as removable.  And
the OS should flush buffers when you dismount it properly.

[toc] | [prev] | [next] | [standalone]

#2863

Robert Wessel wrote:
> On Wed, 16 Jan 2013 11:38:54 +0100, Thomas B <thomasb@nospam.it>
> wrote:
>>
>> You wrote: "I've never seen an OS that would share file buffers between
>> two files like that", so I suppose this should be the same for files on
>> the same volume.
>
>
> Some OS's/filesystems can do a copy with the equivalent of a *nix hard
> link, and then do a copy-on-write procedure if the file is ever
> modified.  In that case you may have only a single copy of the file.
> It can't do that to you across volumes.

ZFS is a good example.  If you snapshot or clone a filesystem, no extra 
space is used until it (a clone) or the source are modified.

ZFS also checksums its data, so there is no need to use another tool to 
validate copes.

-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

#2864

On Sat, 19 Jan 2013 17:12:24 +1300, Ian Collins <ian-news@hotmail.com>
wrote:

>Robert Wessel wrote:
>> On Wed, 16 Jan 2013 11:38:54 +0100, Thomas B <thomasb@nospam.it>
>> wrote:
>>>
>>> You wrote: "I've never seen an OS that would share file buffers between
>>> two files like that", so I suppose this should be the same for files on
>>> the same volume.
>>
>>
>> Some OS's/filesystems can do a copy with the equivalent of a *nix hard
>> link, and then do a copy-on-write procedure if the file is ever
>> modified.  In that case you may have only a single copy of the file.
>> It can't do that to you across volumes.
>
>ZFS is a good example.  If you snapshot or clone a filesystem, no extra 
>space is used until it (a clone) or the source are modified.
>
>ZFS also checksums its data, so there is no need to use another tool to 
>validate copes.


That's not completely true - a copy can go bad while you're not
looking.  The ZFS checksums will catch that right away when you do go
to read it (so actually comparing it to the original is theoretically
not necessary), a compare after writing a copy could add a level of
confidence to the backup.  Of course nothing prevents sectors on the
backup disk from going bad immediately after you do the compare.

Another reason to do a compare is if you're paranoid enough to not
completely trust the backup software/filesystem/I/O subsystem/disk
drive.  We've all seen cases where the software and/or hardware did
peculiar things.

Turning on read-after-write verification helps with many possible
hardware issues, but it's often not on for performance reasons.

[toc] | [prev] | [next] | [standalone]

#2865

Robert Wessel wrote:
> On Sat, 19 Jan 2013 17:12:24 +1300, Ian Collins <ian-news@hotmail.com>
> wrote:
>
>> Robert Wessel wrote:
>>> On Wed, 16 Jan 2013 11:38:54 +0100, Thomas B <thomasb@nospam.it>
>>> wrote:
>>>>
>>>> You wrote: "I've never seen an OS that would share file buffers between
>>>> two files like that", so I suppose this should be the same for files on
>>>> the same volume.
>>>
>>>
>>> Some OS's/filesystems can do a copy with the equivalent of a *nix hard
>>> link, and then do a copy-on-write procedure if the file is ever
>>> modified.  In that case you may have only a single copy of the file.
>>> It can't do that to you across volumes.
>>
>> ZFS is a good example.  If you snapshot or clone a filesystem, no extra
>> space is used until it (a clone) or the source are modified.
>>
>> ZFS also checksums its data, so there is no need to use another tool to
>> validate copes.
>
>
> That's not completely true - a copy can go bad while you're not
> looking.  The ZFS checksums will catch that right away when you do go
> to read it (so actually comparing it to the original is theoretically
> not necessary), a compare after writing a copy could add a level of
> confidence to the backup.  Of course nothing prevents sectors on the
> backup disk from going bad immediately after you do the compare.

Paranoia makes for a good sys-admin!  That's one reason why ZFS has the 
scrub feature which reads all of the data in a pool.

> Another reason to do a compare is if you're paranoid enough to not
> completely trust the backup software/filesystem/I/O subsystem/disk
> drive.  We've all seen cases where the software and/or hardware did
> peculiar things.

There were a number of cases of virtual environments causing trouble 
with ZFS by claiming data was written to disk when it wasn't.

-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

#2866

Il 18/01/2013 22.07, Robert Wessel wrote:

> Some OS's/filesystems can do a copy with the equivalent of a *nix hard
> link, and then do a copy-on-write procedure if the file is ever
> modified.  In that case you may have only a single copy of the file.
> It can't do that to you across volumes.

I'm using Windows 8 Pro 64bit and NTFS. This shouldn't happen with this 
OS and file-system, right?

>> Regarding discarding/flushing the disk cache, should a simple system
>> reboot (instead of powering off) do the job, with most "normal" hard disks?
>
> Usually.  Depending on how the drive is configured, it may still have
> data in (device) cache not yet written to disk.  The OS should force a
> cache flush on the device when it shuts down (on SCSI, a
> Synchronize-Cache, for example).  But a reboot without a clean
> shutdown might not flush things in all cases.  Higher end devices
> would have non-volatile write buffers, which makes the issue mostly
> moot (unless there's a time limit on the non-volatility, and you leave
> the drive powered down for more than that).

I never "brutally" reboot my computer.
I always click on "restart" inside Windows (currently version 8 Pro 
64bit). This is a "clean shutdown" as you intend it?
Restarting the computer this way should flush all caches, in my case 
(with normal hard disks)?

> An external USB drive is fine.  It certainly counts as removable.  And
> the OS should flush buffers when you dismount it properly.

Are you recommending removable media (e.g. USB hard disk) because its 
cache is easier to flush, or for which other reason?
Anyway with USB hard disks I just have to turn them off (with "safely 
remove hardware" option) and then on, in order to flush all related 
caches? I'm always speaking about Windows 8, if it makes any difference.

Thanks,

Thomas

[toc] | [prev] | [next] | [standalone]

#2875

On Sat, 19 Jan 2013 14:06:14 +0100, Thomas B <thomasb@nospam.it>
wrote:

>Il 18/01/2013 22.07, Robert Wessel wrote:
>
>> Some OS's/filesystems can do a copy with the equivalent of a *nix hard
>> link, and then do a copy-on-write procedure if the file is ever
>> modified.  In that case you may have only a single copy of the file.
>> It can't do that to you across volumes.
>
>I'm using Windows 8 Pro 64bit and NTFS. This shouldn't happen with this 
>OS and file-system, right?


Not for local drives.


>>> Regarding discarding/flushing the disk cache, should a simple system
>>> reboot (instead of powering off) do the job, with most "normal" hard disks?
>>
>> Usually.  Depending on how the drive is configured, it may still have
>> data in (device) cache not yet written to disk.  The OS should force a
>> cache flush on the device when it shuts down (on SCSI, a
>> Synchronize-Cache, for example).  But a reboot without a clean
>> shutdown might not flush things in all cases.  Higher end devices
>> would have non-volatile write buffers, which makes the issue mostly
>> moot (unless there's a time limit on the non-volatility, and you leave
>> the drive powered down for more than that).
>
>I never "brutally" reboot my computer.


Except when the local power company "brutally" reboots your computer
for you.


>I always click on "restart" inside Windows (currently version 8 Pro 
>64bit). This is a "clean shutdown" as you intend it?
>Restarting the computer this way should flush all caches, in my case 
>(with normal hard disks)?


It should, excepting some (arguably broken) hardware which caches
despite user requests.  Some VMs have been caught caching disk pages
even when a guest restarts.  Also some hardware buffers writes, but
has non-volatile storage that should maintain that data across a power
failure - it doesn't always work, and the NV storage (battery backed
RAM, for example) often has a maximum lifetime if you don't get it
back on power.


>> An external USB drive is fine.  It certainly counts as removable.  And
>> the OS should flush buffers when you dismount it properly.
>
>Are you recommending removable media (e.g. USB hard disk) because its 
>cache is easier to flush, or for which other reason?
>Anyway with USB hard disks I just have to turn them off (with "safely 
>remove hardware" option) and then on, in order to flush all related 
>caches? I'm always speaking about Windows 8, if it makes any difference.


Mainly because it's easy to flush the cache without rebooting the
computer.

[toc] | [prev] | [next] | [standalone]

#2879

Il 21/01/2013 10.40, Robert Wessel ha scritto:

> Some VMs have been caught caching disk pages
> even when a guest restarts.

"VM" means Virtual Machine?

>>> An external USB drive is fine.  It certainly counts as removable.  And
>>> the OS should flush buffers when you dismount it properly.
>>
>> Are you recommending removable media (e.g. USB hard disk) because its
>> cache is easier to flush, or for which other reason?
>> Anyway with USB hard disks I just have to turn them off (with "safely
>> remove hardware" option) and then on, in order to flush all related
>> caches? I'm always speaking about Windows 8, if it makes any difference.
>
> Mainly because it's easy to flush the cache without rebooting the
> computer.

So in order to flush the cache of an USB hard disk, I just have to turn 
it off and then on, without rebooting the PC?
Will this flush both the cache inside the hard disk, and all other 
caches related to that hard disk (e.g. caches inside Windows)?

Thanks,

Thomas

[toc] | [prev] | [next] | [standalone]

#2880

On Mon, 21 Jan 2013 17:41:12 +0100, Thomas B <thomasb@nospam.it>
wrote:

>Il 21/01/2013 10.40, Robert Wessel ha scritto:
>
>> Some VMs have been caught caching disk pages
>> even when a guest restarts.
>
>"VM" means Virtual Machine?


Yes.


>>>> An external USB drive is fine.  It certainly counts as removable.  And
>>>> the OS should flush buffers when you dismount it properly.
>>>
>>> Are you recommending removable media (e.g. USB hard disk) because its
>>> cache is easier to flush, or for which other reason?
>>> Anyway with USB hard disks I just have to turn them off (with "safely
>>> remove hardware" option) and then on, in order to flush all related
>>> caches? I'm always speaking about Windows 8, if it makes any difference.
>>
>> Mainly because it's easy to flush the cache without rebooting the
>> computer.
>
>So in order to flush the cache of an USB hard disk, I just have to turn 
>it off and then on, without rebooting the PC?
>Will this flush both the cache inside the hard disk, and all other 
>caches related to that hard disk (e.g. caches inside Windows)?


Do the "safely remove" thing in Windows (which forces the OS to flush
all of its caches for the device), and then unplug the USB cable and
power cycle the thing.

Now that still wouldn't help you if the device is using (say) battery
backed NV RAM to cache writes, and you then leave it powered down for
long enough that the battery dies, and the device lies in response to
the SCSI SYNCHRONIZE_CACHE Windows would be sending at that time.  But
I know of no USB hard drives that actually do that.

[toc] | [prev] | [next] | [standalone]

#2814

Thomas B wrote:
) Hello,
)
) I'm sorry if this post is not really programming-related, but I thought 
) that programmers could know about it.
)
) I use a program (Beyond Compare) which has a Folder Compare module, with 
) a function for doing a binary comparison between the files inside two 
) folders. I use it to check the integrity of my backups, by comparing my 
) main hard disk files, with the ones on the backup hard disks. If two 
) files get recognized as being different, then one of them should be 
) corrupt (if they didn't get modified "normally").
)
) My question is: will this method work to detect real file corruptions? 

Yes, it should.

) Will it detect any kind of corruption, both copy corruptions, and "bit 
) rot" corruptions? Will there be no problem regarding read cache?

Well, if you just wrote it, it might remain in the cache and not get read
from disk.  But not what you meant below.

) I mean: 
) if the first file to compare gets read from disk, and is kept in a read 
) cache (by Windows or by the hard disk), will the second file get read 
) from the same cache?

No.

) Do Windows and hard disks have some kind of 
) procedure to detect if a file to read from disk is already available in 
) cache, even if it is in a different folder than the "original" one? 

No.

) Perhaps some kind of file-checksum-system which decides that the files 
) are same? (And this system would not notice if the file to compare is 
) corrupt). If this would be true, then integrity checks by file 
) comparison would not work, because in practice the same file would be 
) read twice (first from disk, and then from cache), instead of reading 
) both the two files to be compared from disk.

The only thing that could happen (but not on Windows) is that the file
system decides that two files are the same and then uses the same disk
space to store them in.  Which also means that if the disk gets corrupted,
both files are bad.

) I have already done test by manually "corrupting" files (changing 
) slightly the contents, while keeping size and timestamp the same), and 
) it works (the files get recognized as different). But I'm not sure if it 
) will work also with "real" corrupt files.

If you read immediately after writing, it might be read out of the write
cache.  It could even be that it hasn't actually been written to disk yet(*).

Other than that, there are no issues.


*) Some newer file systems keep small files in the write cache for a longer
   time, because it anticipates that those are temporary files which will
   be deleted soon anyway.  So they never make it to disk.


SaSW, Willem
-- 
Disclaimer: I am in no way responsible for any of the statements
            made in the above text. For all I know I might be
            drugged or something..
            No I'm not paranoid. You all think I'm paranoid, don't you !
#EOT

[toc] | [prev] | [next] | [standalone]

#2824

Il 14/01/2013 19.23, Willem ha scritto:

[...]

Thanks for your answers.
Just to be sure: will integrity checks by file comparison detect any 
kind of corruption? (Copy corruption, "bit-rot" corruption, head-crash 
corruption, etc.)
I think so.

Thanks,

Thomas

[toc] | [prev] | [next] | [standalone]

#2857

On Wed, 16 Jan 2013 16:46:38 +0100, Thomas B <thomasb@nospam.it>
wrote:

>Il 14/01/2013 19.23, Willem ha scritto:
>
>[...]
>
>Thanks for your answers.
>Just to be sure: will integrity checks by file comparison detect any 
>kind of corruption? (Copy corruption, "bit-rot" corruption, head-crash 
>corruption, etc.)


Yes, assuming you've actually ended up with two copies of the data,
and you're actually reading the two files from disk.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.programming

csiph-web