Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.arch.embedded > #25247 > unrolled thread
| Started by | Don Y <blockedofcourse@foo.invalid> |
|---|---|
| First post | 2016-10-27 15:23 -0700 |
| Last post | 2016-10-31 03:45 -0700 |
| Articles | 4 on this page of 44 — 7 participants |
Back to article view | Back to comp.arch.embedded
Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-27 15:23 -0700
Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-28 09:48 +1100
Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-27 16:48 -0700
Re: Checking large numbers of files Dimiter_Popoff <dp@tgi-sci.com> - 2016-10-28 03:17 +0300
Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-27 18:50 -0700
Re: Checking large numbers of files Dimiter_Popoff <dp@tgi-sci.com> - 2016-10-28 05:38 +0300
Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-27 21:33 -0700
Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-28 11:40 +1100
Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-27 18:52 -0700
Re: Checking large numbers of files kalvin.news@gmail.com - 2016-10-28 03:50 -0700
Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-28 10:49 -0700
Re: Checking large numbers of files kalvin.news@gmail.com - 2016-10-29 05:17 -0700
Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 13:15 -0700
Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 09:47 +1100
Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 16:19 -0700
Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 16:36 -0700
Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 12:00 +1100
Re: Checking large numbers of files Dimiter_Popoff <dp@tgi-sci.com> - 2016-10-30 04:05 +0200
Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 14:06 +1100
Re: Checking large numbers of files Dimiter_Popoff <dp@tgi-sci.com> - 2016-10-30 05:48 +0200
Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 22:33 -0700
Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 17:00 +1100
Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-10-31 04:43 -0700
Re: Checking large numbers of files (long) David Brown <david.brown@hesbynett.no> - 2016-10-31 14:20 +0100
Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-10-31 08:02 -0700
Re: Checking large numbers of files (long) David Brown <david.brown@hesbynett.no> - 2016-11-01 09:47 +0100
Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-11-01 04:36 -0700
Re: Checking large numbers of files (long) David Brown <david.brown@hesbynett.no> - 2016-11-01 14:29 +0100
Re: Checking large numbers of files (long) David Brown <david.brown@hesbynett.no> - 2016-11-01 15:16 +0100
Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-11-02 11:42 -0700
Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 22:15 -0700
Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 21:50 -0700
Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 17:14 +1100
Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-10-31 03:36 -0700
Re: Checking large numbers of files (long) Clifford Heath <no.spam@please.net> - 2016-11-01 09:20 +1100
Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-11-01 02:23 -0700
Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 21:20 -0700
Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 17:25 +1100
Re: Checking large numbers of files Robert Wessel <robertwessel2@yahoo.com> - 2016-10-30 00:14 -0500
Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 23:02 -0700
Re: Checking large numbers of files David Brown <david.brown@hesbynett.no> - 2016-10-31 12:19 +0100
Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-31 07:00 -0700
Re: Checking large numbers of files Ali <abdulrazaq@gmail.com> - 2016-10-30 15:45 -0700
Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-31 03:45 -0700
Page 3 of 3 — ← Prev page 1 2 [3]
| From | David Brown <david.brown@hesbynett.no> |
|---|---|
| Date | 2016-10-31 12:19 +0100 |
| Message-ID | <nv79af$2pg$1@dont-email.me> |
| In reply to | #25277 |
On 29/10/16 22:15, Don Y wrote:
> On 10/29/2016 5:17 AM, kalvin.news@gmail.com wrote:
Don, let me try to understand the basis of your problem here. Correct
me at any point that I am wrong (I am writing in many small paragraphs
to make that easier). I have also added a few comments inline below my
main post.
1. You have lots of files, held on computers, and you want to make sure
they don't get changed or corrupted.
2. You don't trust "metadata" methods such as sizes and timestamps,
because files could be changed or corrupted without affecting these.
3. You don't want to read through the entire file, because they are big.
4. You want some sort of algorithm for detecting file change or
corruption, given 2 and 3.
If that is the case, then you will be disappointed - no such algorithm
could exist.
So now we have to consider what would give you a convenient, economical
and reliable way to store your files safely.
First, I assume that your old existing systems are irrelevant except as
a source to copy from. Spending time and effort on maintaining outdated
hardware is a waste - I expect you to set up a new system.
Second, I expect you to have off-site backups. This could be done by
duplicating the main system, or by simply having a number of off-site
copies on DVD, archived hard disks, cloud storage, etc.
Third, I expect you to leave behind your prejudices and knowledge of
limitations of old systems, and to look forward.
Files change, get lost, or corrupted due to four main causes:
1. A user changes them in some way (intentionally or unintentionally).
2. Malware or hackers change them.
3. Bit rot or sector wear causes some data to be lost on a disk.
4. Serious hardware failure, such as a disk head crash losing the whole
disk.
I expect you to handle 1 and 2 mainly through good security and regular
backups. I strongly recommend using a filesystem or setup with cheap
snapshots - then you can take them regularly.
There is a myth that hard disk sectors can get corrupted and return
incorrect data without an error indication - the so-called "undetected
read error". In theory, this /can/ occur - but unless you are using the
data quantities of CERN or Google, you will never see it in your
lifetime (that's why I call it a myth). You /might/ come across
firmware errors in disks that give such undetected read errors, though
it is highly unlikely. "Unrecoverable read errors", on the other hand,
are not as uncommon as one would like - sometimes a disk is simply
unable to return the data you ask for, but it can at least inform you of
the failure.
The general method to protect against failures is redundancy combined
with systems to spot errors. For storage, that is almost invariably
done using RAID.
So your basic setup here is to have two drives with a RAID-1 mirror.
Anything that is written to the mirror gets written to both drives. If
a sector read fails (despite the drive's extensive error checking and
correcting codes), the RAID controller (hardware or software) will read
the second copy from the other disk. It will then re-write this correct
data to the first disk, which the drive will map to a spare sector. If
a drive fails entirely, the controller will read from the other drive.
"Bit rot", or slow bit corruption on disk sectors, is prevented by
scrubbing. You regularly read through the entire array, for both disks.
If a sector has had corruption on some bits, the drives ECC coding will
usually let it correct the error automatically, and it will re-write the
sector (with re-mapping if necessary) to refresh the data. If the
corruption is too great, the RAID controller will re-write the sector
based on the second copy.
Thus RAID keeps your data reliable, at least until a catastrophic
failure (such as the whole server dying) - that's why you have backups.
For more features, I would recommend using BTRFS or ZFS to handle your
RAID mirror. These filesystems give you additional checksumming (for
protection against drive firmware errors, or extremely rare undetected
read errors), as well as very cheap snapshots and other goodies.
On the other hand, if you want the simplest "will work on anything, even
old Linux systems or Windows systems" solution, then use Linux md RAID
with metadata format 1.0. This puts the RAID information at the end of
the disk - all the rest of the disk appears /exactly/ as a normal
non-raid disk, and can be read from anything else.
>> Disclaimer: I am not an expert on the data storage, but here are my
>> thoughts:
>>
>> - Use a RAID system which has enough redundancy to detect and correct
>> defects in the media. Depending of the RAID level used, the capability to
>> data recovery varies.
>
> I started out using RAID5 "hardware" arrays (and other, more sophisticated
> configurations). This requires you to have multiple spindles running
> concurrently (else the array can't *detect* discrepancies). Compact
> arrays also tend to suffer from cooling problems.
You will need two spindles (or SSDs) concurrently - that's the only way
to get two copies of your data on two disks.
Don't bother with RAID-5. Either use RAID-1 if your data fits
comfortably on a single disk, or use RAID-6. Modern disks have less
cooling requirements than old ones, if you go for RAID-6.
>
> It also ties you to an often proprietary (or, at least, poorly documented)
> implementation ("volume format"). So, a failure of the hardware can leave
> you with possibly intact volume(s) and no straightforward means of
> accessing their contents.
Use software RAID - either Linux, or BSD if you prefer. Avoid
proprietary hardware solutions.
>
> Moving volumes between arrays is seldom supported (unless you have
> identical
> arrays -- hardware/software -- AND a well considered means of doing this.
> This effectively precludes having volumes off-line. Introducing a
> "foreign"
> volume to an array will often result in the array prompting you to
> "initialize" it (wipe it clean!).
>
>> - Duplicate or mirror the data across different physical media.
>
> This is RAID1. Many of the above issues still apply. In any case,
> the mirrors must be spinning concurrently for the array to function.
> (It has no way of knowing what the "other" copy of the "file"
> looks like unless it can *see* it, now, in its entirety).
If you are really desperate to avoid having two disks spinning at the
same time, it is possible with Linux md raid to designate one drive as
"write mostly". It will then only be used when writing (obviously),
when scrubbing (obviously), or when reading if the first drive fails to
return the correct data. I cannot think of any real benefit of this,
but it is possible.
>
> This begs two crucial questions:
No, it /raises/ two questions.
> - are BOTH copies of the file examined when you try to access it?
No. That would be highly inefficient, and does not contribute to
improved reliability.
> Or, is a "primary" copy accessed and the "secondary" copy only accessed
> if the primary throws a read error?
Yes.
> - If a discrepancy is found between the TWO instances, how is the
> "winner" selected?
Discrepancies don't occur with non-negligible probability unless you
have had a massive hardware failure, or power-failure during writing.
See <http://neil.brown.name/blog/20100211050355> for a discussion on the
topic.
>
> Plus, the more subtle question seldom considered:
> - are files that are not "manually" accessed EVER verified? Or, do you
> have to wait until you need/want a file to discover if it is intact??
That's what "scrubbing" is for. Hardware raid controllers should do
this automatically, with software raid it is usually done with a cronjob.
> (Do *you* periodically check the contents of the batteries in your
> flashlight to verify they are functioning BEFORE you need them?
> Does the *flashlight* take on the responsibility of performing this
> test on your behalf??)
A torch is not a raid device.
>
> Consider the above when you have many terabytes of data at stake.
RAID like this is /exactly/ what people use when they have TB of data at
stake.
When they have more than that, they use redundant servers, or split the
two halves of their software RAID-1 pairs across different boxes.
> Do you want ALL of it "spinning" whenever you need to access *any*
> of it?
Why is that a problem? Most hard disks have no problem spinning all the
time.
> How long will it take for a proactive array to examine/verify
> ALL of the data just to reassure you? (will you have the ENTIRE
> array "up" for that length of time? Or, will you power it up just
> long enough to access the file(s) of interest, then power it back down,
> again? Or, will you power up ONLY the portion of the array known to
> contain the file(s) you seek?)
Again, what is the problem?
If this is an array you rarely access, you can spin down all the disks
when it is not in use.
>
> As I said, I've been down this route. Ever hear what twenty to
> fifty 10-15K drives sound like spinning at the same time?
> <http://techreport.com/blog/13849/behold-thumper-sun-sunfire-x4500-storage-server>.
> Even a 12-14 drive "shelf" is loud:
> <http://www.amtecgroup.co.uk/products/external-disk-storag-1071/sun-storedge-a1000-12-bay-hdd-rack-array-raid-ctrl-2x-psu-1502.aspx>.
> Remember,
> they're all throwing off heat in a relatively dense package so you
> have lots of (redundant!) fans in place to keep them from melting...
>
Why are you considering a decade old high-end datacenter storage system?
These are designed for installations where noise doesn't matter, and
you have massive cooling systems.
Just get a simple little server box with space for a few disks. I
really like the HP Microservers - Gen 8 devices are dirt cheap,
reliable, and have four neat 3.5" drive bays behind a door in the front.
They are perfect for file servers unless you need really high-end
performance.
> The "problem" with these solutions is they are designed for high
> performance.
> Serving files continuously, typically, to an "organization". And, with a
> high degree of reliability/availability.
>
So don't get one.
> For an *archive*, you tend to need lots of capacity but comparably little
> bandwidth! (how *often* will you access its contents? how quickly will
> you need the file "delivered" to you?) And, you can trade reliability and
> availability for bandwidth and cost. Do you keep your 2013 tax returns
> in your top left desk drawer (where they are easily accessible)? Or, have
> you moth-balled them to a more capacious site that's, perhaps, a bit more
> difficult to access??
What capacity are you looking for here?
[toc] | [prev] | [next] | [standalone]
| From | Don Y <blockedofcourse@foo.invalid> |
|---|---|
| Date | 2016-10-31 07:00 -0700 |
| Message-ID | <nv7ipe$3fc$1@dont-email.me> |
| In reply to | #25307 |
Hi David,
On 10/31/2016 4:19 AM, David Brown wrote:
> On 29/10/16 22:15, Don Y wrote:
>> On 10/29/2016 5:17 AM, kalvin.news@gmail.com wrote:
>
> Don, let me try to understand the basis of your problem here. Correct
> me at any point that I am wrong (I am writing in many small paragraphs
> to make that easier). I have also added a few comments inline below my
> main post.
>
> 1. You have lots of files, held on computers, and you want to make sure
> they don't get changed or corrupted.
I want to ensure they "remain accessible". I'm guarding against data loss.
Both for "old" data and "freshly minted" data. I.e., wanting to ensure
the work I do TODAY is preserved without my having to take deliberate
steps to ensure it ends up "where it belongs".
> 2. You don't trust "metadata" methods such as sizes and timestamps,
> because files could be changed or corrupted without affecting these.
No, I don't trust *certain* metadata (notably timestamps). I am
willing to rely on a dirent's "size" to be reflective of the files
actual size. Whether I can *access* that many bytes is a different
issue -- something I can only determine by trying to access those
bytes.
As a medium can "fail" in such a way that the dirent is munged -- but
the data may still be intact (just not accessible via the normal
mechanisms), if the dirent says the file name is now "boogie"
instead of "myfile", then I have no choice but to assume "myfile"
is gone -- I'm not trying to implement a "data recovery service".
> 3. You don't want to read through the entire file, because they are big.
Correct. And, data rates may be slow. Imagine trying to "verify"
CD's at a few megabytes per second!
And, if I have information that suggests the file is NOT "as it should be",
then why undertake the effort to PROVE that it isn't? "Gee, this file
is supposed to be 32GB but I stat(2) it at just 5KB! Either the dirent
is munged -- the file really IS 32GB but the dirent is in error -- OR,
the file has been altered/replaced/truncated/etc."
> 4. You want some sort of algorithm for detecting file change or
> corruption, given 2 and 3.
Also, some way of verifying NON change/corruption. There are many
ways to ask the question and different approaches to getting an
answer. "Has ANYTHING changed, here?", on a 4TB drive, can yield
an answer in a few SECONDS -- if the first file you go looking for is
found to be absent (have a changed name) or of a different size.
OTOH, "is *everything* UNCHANGED, here" takes a lot longer to answer
in the affirmative; you have to perform all of the tests on all of the
objects!
> If that is the case, then you will be disappointed - no such algorithm
> could exist.
See above.
Think about the ways that "file collections" can be altered. And, think
of the relative likelihoods of those changes!
By far, I suspect accident user actions are the biggest issue. Someone
deletes something that they shouldn't have. Or, moves something and
the operation is aborted without their noticing it (media full, etc.).
Similarly, if users are responsible for maintaining the file sets,
then its possible a user updates X and forgets to update Y. Or,
X *instead* of Y.
Or, a buggy protocol implementation (an aborted automated transfer that
leaves a partial result)
After that, perhaps a total drive failure (e.g., not spinning up).
Somewhere down the list are local, uncorrectable errors that affect
some portion of the drive surface. These would tend to increase
in frequency, be more widely distributed through the dataset and
PROBABLY be noticeable if someone is aggressively watching for them.
For example, I have noticed that cheap consumer drives (e.g., 4TB)
tend to have dramatically different performance/throughput as they
"get full". Presumably, fewer opportunities for "reliable"
sectors to be encountered vs. faulty sectors accumulating at the
unused portion of the medium.
> So now we have to consider what would give you a convenient, economical
> and reliable way to store your files safely.
>
> First, I assume that your old existing systems are irrelevant except as
> a source to copy from. Spending time and effort on maintaining outdated
> hardware is a waste - I expect you to set up a new system.
I've already pulled the files off "old media". There are some
obsolescent media that I have no choice but to support: e.g.
my DSO and EPROM programmer both rely on 3" floppy media. To
work around that, I'd have to buy new equipment -- a silly
"optimization" (easier to keep a floppy drive operational AND
capture the images off the "factory media" before it degrades!
What is considered "outdated" is subject to personal opinion.
I see no reason to replace a machine that still runs "modern"
OS's -- even if faster machines are available. With that
criteria, you'd have to update hardware yearly -- and update
all licensed software at the same rate!
> Second, I expect you to have off-site backups. This could be done by
> duplicating the main system, or by simply having a number of off-site
> copies on DVD, archived hard disks, cloud storage, etc.
Nope. I've satisfied all of my contractual/legal obligations so
I'm only "on the hook" for things that have value to me, personally.
There is some value to having "past projects" accessible as
references. But, if the house burned to the ground, the LAST
thing I would worry about would be my possessions -- both
tangible and intellectual.
[As an aside, a routine exercise I perform is querying myself
re: how I'd handle such an event: "If the house was on fire
and you could retrieve ONE item, what would that item be?"
My goal, for as long as I've been asking myself that question,
has been to be COMFORTABLE answering "nothing". I have been
at that point for quite some time! :> Of course, that doesn't
mean I'd be eager to lose *anything*!!]
> Third, I expect you to leave behind your prejudices and knowledge of
> limitations of old systems, and to look forward.
I've only mentioned old systems to indicate how I got here.
How my archive maintenance philosophy has evolved to its
current set of practices WITHOUT foreknowledge of future
developments.
E.g., dealing with RAID5 and $1K drives and why RAID1 was
"an improvement" (for me) in that timeframe. Then, why
"manual mirroring" was an improvement beyond that (no
special hardware OR software required).
And, using SCSI as a basis for removable disks before USB
existed to provide that capability (so, I can keep a stack
of SCSI disks "cold" on a shelf and swap them into a
system as needed instead of having ALL of them spinning
ALL THE TIME).
Then, the downside (discipline required) for these approaches.
Finally, how to avoid that aspect (by automating the process
taking advantage of more modern hardware and less expensive
components).
> Files change, get lost, or corrupted due to four main causes:
>
> 1. A user changes them in some way (intentionally or unintentionally).
Exactly.
> 2. Malware or hackers change them.
Not a problem, here. My systems are offline. In the odd chance that
something was "infected" and spread to items CURRENTLY on-line,
my solution would detect those changes by noting the differences
in file hashes during periodic "audits".
And, as most of my archive is offline at any given time, I'd probably
notice any widespread "corruption" BEFORE mounting yet another volume.
> 3. Bit rot or sector wear causes some data to be lost on a disk.
Yes. Nothing I can do about that short of buying better quality
components. In practice, I've not seen this to be a real problem
(noting the consumer drive issue mentioned above). Other folks
have reported poor durability of optical media -- I've not found
that despite using "commodity media". I've heard of horror
stories with disk failures but have been spared that problem
(touch wood!).
I suspect part of this is due to the fact that I distribute my work
among many different machines. So, total power on hours tends to
stay low. For machines and their peripherals. I suspect I've only
a hundred hours on my "most used" laptop -- cuz I don't use laptops
unless traveling or away from home (in the field, customer site, etc.)
> 4. Serious hardware failure, such as a disk head crash losing the whole
> disk.
Along the same lines, I HAVE encountered a catastrophic failure due to
a *driver* bug! I watched a disk get "scrambled" and assumed the
problem was a disk failure (cuz everyone claims these are so common!).
Then, mounted the bytewise backup of the same drive -- and saw *it*
get scrambled just as quickly!
[No, I don't buy the idea of two independant drives dying at the same
time -- esp when they've both been sitting on a shelf for most of
their lifetimes! What's changed? Ah, I updated the OS yesterday...
let's roll things back to the previous release and try that, again.
Dig out the MO disks to recreate the corrupted drives (painfully
slow as the media is inherently slower than winchesters).]
> I expect you to handle 1 and 2 mainly through good security and regular
> backups. I strongly recommend using a filesystem or setup with cheap
> snapshots - then you can take them regularly.
The second isn't a problem. The first always is, potentially.
I only access the archive AS the superuser -- to ensure "noone"
else can screw up (i.e., me in my non-privileged guise). So,
everything is exposed to my potential folly.
For example, I recently pruned all the "foreign language" updates from
my MS update archive. ARA, JAP, POR, etc. Just save ENU! Click.
Click. Delete. Done.
Ooops! "GLB" isn;t an abbreviation for a localization but,
rather "common to all"! (sigh) Better go back and redownload that
stuff, again!
Even automating this leaves opportunity for screwups -- if I specify
the wrong criteria/options for a particular object (and the system
dutifully follows my ill-conceived directives!)
> There is a myth that hard disk sectors can get corrupted and return
> incorrect data without an error indication - the so-called "undetected
> read error". In theory, this /can/ occur - but unless you are using the
> data quantities of CERN or Google, you will never see it in your
> lifetime (that's why I call it a myth). You /might/ come across
> firmware errors in disks that give such undetected read errors, though
> it is highly unlikely. "Unrecoverable read errors", on the other hand,
> are not as uncommon as one would like - sometimes a disk is simply
> unable to return the data you ask for, but it can at least inform you of
> the failure.
I'm not worried about disks misreporting data/errors.
Likewise, I'm not worried about hash collisions -- there's
no active adversary involved. Sure, some random set of
events could THEORETICALLY result in all those magnetic
domains flipping to some OTHER combination of states
AND still yield the same hash.
But, the Sun will burn cold long before that happens!
I'm willing to "risk" accessing that file at some
later date and wondering why its completely munged -- yet
still hashing to the correct value!
> The general method to protect against failures is redundancy combined
> with systems to spot errors. For storage, that is almost invariably
> done using RAID.
RAID is predicated on you wanting "online" verification of EVERY
data access. And, semi-seamless correction of any failures that
turn up. All happening at "regular disk speeds" on volumes that
are PROBABLY seeing some significant amount of traffic.
It would be foolish to keep such an array spinning 24/7/365 for
the *rare* access that might occur. When was the last time you
needed to access the install media for application XYZ? Do
you REALLY need it to be available in a fraction of a second?
Are you THAT impatient?
The last time I accessed my 1976 tax forms was... in 1977! Do
I *need* to keep them? No. Should I discard them? Why? Are
those few magnetic domains THAT valuable to me that I need
to salvage them??
You can use the redundancy aspects of RAID *without* using a
RAID appliance. You just trade-off performance for "accessibility"
and "performance".
> So your basic setup here is to have two drives with a RAID-1 mirror.
> Anything that is written to the mirror gets written to both drives. If
> a sector read fails (despite the drive's extensive error checking and
> correcting codes), the RAID controller (hardware or software) will read
> the second copy from the other disk. It will then re-write this correct
> data to the first disk, which the drive will map to a spare sector. If
> a drive fails entirely, the controller will read from the other drive.
Yes, but you can adopt a similar strategy without having both drives
on-line at the same instant. This exposes a window in which things
can go south. But, they are "unlikely" to do so.
E.g., when I moved from RAID1 to JBOD, I simply copied every file to two
different places -- under two parallel mount points. The two drives
are not sector-wise images of each other. But, at the filesystem
level, they contain the same files in the same logical places in their
hierarchies. I could close my eyes while you rearranged the drives
and I'd not be able to tell if you'd swapped them -- or not.
If you have a means of improving your confidence in a particular
instance of a file (i.e., if you record name, size, hash in a
reliable medium and then verify this when you later access
that instance), then you can use that instance in lieu of the
original instance in creating the "mirror" copy of the file.
E.g.,
# cp file /Archive/1
# md5 file > hash
# rm file
# md5 /Archive/1/file | cmp hash -
# cp /Archive/1/file /Archive/2/file
is effectively (to "many 9's") the same as:
# cp file /Archive/1
# cp file /Archive/2
# rm file
*if* the operations are reasonably close together in time. A RAID
array cuts the time to microseconds. But, realistically, would it
be much different if the RAID controller ran a delay loop for
some number of seconds between each write? (obviously, throughput
would suffer!)
My current approach exploits this. So, a file can be copied to a volume
and its "signature" reliably recorded. Later, that instance can be
used (after verifying it against that stored signature) to create
another instance of the file on another medium.
And, both instances can be "verified" at any time (assuming they
are available, online, at the time) -- by double-checking the
"signature", size, name, etc.
> "Bit rot", or slow bit corruption on disk sectors, is prevented by
> scrubbing. You regularly read through the entire array, for both disks.
Yes. But, just reading doesn't PREVENT the problem. A disk can
degrade to a point where scrubbing just keeps revealing errors.
There;s a point when a drive's reliability is not recoverable.
The more the utilization, the less ability to adapt to grown defects.
> If a sector has had corruption on some bits, the drives ECC coding will
> usually let it correct the error automatically, and it will re-write the
> sector (with re-mapping if necessary) to refresh the data. If the
> corruption is too great, the RAID controller will re-write the sector
> based on the second copy.
And, the same sort of algorithm can be applied to multiple copies of a file
accessed at different points in time! RAID keeps wanting to do everything
"now". It doesn't have a deep memory for errors -- fix it and forget it.
And, if *the* copy of *the* file becomes irrecoverably corrupted, its
not smart enough to know that another copy might exist -- on this same
medium OR a different medium... possibly at the other end of a network
connection!
> Thus RAID keeps your data reliable, at least until a catastrophic
> failure (such as the whole server dying) - that's why you have backups.
>
> For more features, I would recommend using BTRFS or ZFS to handle your
> RAID mirror. These filesystems give you additional checksumming (for
> protection against drive firmware errors, or extremely rare undetected
> read errors), as well as very cheap snapshots and other goodies.
>
> On the other hand, if you want the simplest "will work on anything, even
> old Linux systems or Windows systems" solution, then use Linux md RAID
> with metadata format 1.0. This puts the RAID information at the end of
> the disk - all the rest of the disk appears /exactly/ as a normal
> non-raid disk, and can be read from anything else.
You can't then accommodate R/O media. E.g., I want one instance of
a particular set of files to be the "Dell Windows 7 Reinstallation DVD"
that I have in my desk drawer. I want the *other*, probably more
vulnerable copy of that file to be "Dell_Windows_7_Reinstallation_DVD.iso"
So, I can call up the ISO file relatively easily (power up a particular
disk drive and CD to the correct directory) and access its contents.
But, being on writeable media, its conceivable that this .iso file
can get corrupted, overwritten, replaced, etc. So, I want to be able to
verify that it is intact. And/or "recover" it -- from the original
medium!
Or, if the original medium gets munged (scratched), to be able to recreate
a NEW physical DVD from the ISO file -- which can later be used to verify
the ISO's integrity...
>>> Disclaimer: I am not an expert on the data storage, but here are my
>>> thoughts:
>>>
>>> - Use a RAID system which has enough redundancy to detect and correct
>>> defects in the media. Depending of the RAID level used, the capability to
>>> data recovery varies.
>>
>> I started out using RAID5 "hardware" arrays (and other, more sophisticated
>> configurations). This requires you to have multiple spindles running
>> concurrently (else the array can't *detect* discrepancies). Compact
>> arrays also tend to suffer from cooling problems.
>
> You will need two spindles (or SSDs) concurrently - that's the only way
> to get two copies of your data on two disks.
Yes, though you can cache those files that you know will need to
be coped to a yet-to-be-mounted volume. You know what their signatures,
sizes, names are and SHOULD BE.
So, when the desired target volume is made available to you (perhaps
because you requested the user mount it on your behalf!), you can
"resume" the DEFERRED copy to that second spindle. I.e., where you
cache the file need not be one of the spindles IN the archive.
> Don't bother with RAID-5. Either use RAID-1 if your data fits
> comfortably on a single disk, or use RAID-6. Modern disks have less
> cooling requirements than old ones, if you go for RAID-6.
I made the cooling point to indicate why LARGE RAID arrays (12 and
48 drives) were problematic when I was using them. They wanted to
be compact -- but 20W drives packed side-by-side in a small cabinet
is a lot of BTUs to disperse.
[nowadays, you can buy disks that are 10 - 1000 times larger
so you;ve already cut the equivalent power-per-gigabyte-capacity
significantly]
>> It also ties you to an often proprietary (or, at least, poorly documented)
>> implementation ("volume format"). So, a failure of the hardware can leave
>> you with possibly intact volume(s) and no straightforward means of
>> accessing their contents.
>
> Use software RAID - either Linux, or BSD if you prefer. Avoid
> proprietary hardware solutions.
>
>> Moving volumes between arrays is seldom supported (unless you have
>> identical
>> arrays -- hardware/software -- AND a well considered means of doing this.
>> This effectively precludes having volumes off-line. Introducing a
>> "foreign"
>> volume to an array will often result in the array prompting you to
>> "initialize" it (wipe it clean!).
>>
>>> - Duplicate or mirror the data across different physical media.
>>
>> This is RAID1. Many of the above issues still apply. In any case,
>> the mirrors must be spinning concurrently for the array to function.
>> (It has no way of knowing what the "other" copy of the "file"
>> looks like unless it can *see* it, now, in its entirety).
>
> If you are really desperate to avoid having two disks spinning at the
> same time, it is possible with Linux md raid to designate one drive as
> "write mostly". It will then only be used when writing (obviously),
> when scrubbing (obviously), or when reading if the first drive fails to
> return the correct data. I cannot think of any real benefit of this,
> but it is possible.
I'm not averse to running multiple spindles. I often have 2 or 4
external drives spinning so I can easily move a single set of
files onto "a volume and its 'mirror'".
But, I shouldn't NEED to have two or more spindles up at the same time!
There's already a DBMS running to do all the bookkeeping (similar to
what the RAID has to do). If you overprovision this machine -- by
even a small amount -- then you can set aside a staging area to cache
those files that you are adding or altering in the "system" -- even
if the actual medium supporting it isn't available, presently.
Because you have the DBMS accessible, you can make a notation for the
file that you will *ultimately* be updating: "When I see this
volume hosting this file in the future, I need to replace it with
the file currently hiding in the 'pending file cache'".
At the same time, you can choose to notify the user that you;d
eventually like to have that volume mounted -- and why its important
to you!
And, if you ever have to purge the file from that cache, you still have
a record of the fact that an update is pending for the/those other
copies of it! So, you can remind the user that the array is
still in an inconsistent (degraded) state.
I'm doing the same thing that a RAID1 array would do -- but allowing
the timescale to be more elastic. Letting the user decide how
long he is willing to operate in those "degraded" states.
E.g., perhaps the file needs to be updated in *3* places! You;ve
successfully updated the first, "original" (insomuch as any copy can
be considered the "original"). You cache a copy in the holding area.
And, tickle the user to mount the one or two volumes that host
these two other, degraded copies.
Eventually, you encounter one of those columes and surreptitiously
update the copy while the user is fetching some OTHER files off the
medium. You update the database to record the fact that this
copy is now in agreement with the "original" and no longer in
need of an update.
And, you continue to cache the file as you wait for the OTHER
medium to come along for that third copy!
You continue to pester the user -- but, can now remind him that
he no loner is operating in *AS* DEGRADED a state as previously;
he now has TWO copies of the file available! But, you'd
really like to get that third one updated, as well!
Regardless of how much time passes, you will continue to remember the
"state" of this file and its counterparts -- because the DBMS is persistent!
Some day, the file may disappear from the cache -- pushed out by some newer
"change" that represents a more serious degradation problem: only ONE
instance of that new file exists in the array at this time! (the previously
cached file has TWO copies available, despite needing a third update)
Some day, you may encounter one of these first two copies (of the first
file). You can surreptitiously reacquire a copy to have a fresh cached
copy. And, verify that it is intact -- in the hope that you might
encounter that third volume, RSN.
[See?]
>> This begs two crucial questions:
>
> No, it /raises/ two questions.
>
>> - are BOTH copies of the file examined when you try to access it?
>
> No. That would be highly inefficient, and does not contribute to
> improved reliability.
>
>> Or, is a "primary" copy accessed and the "secondary" copy only accessed
>> if the primary throws a read error?
>
> Yes.
These were rhetorical questions intended to highlight the fact that
having two copies doesn't really mean what you think! *If* you get
a requested file from the array, there is no guarantee that both
copies are intact. No guarantee that the copy you got HAS a
"backup" in case it is lost.
And it highlights the fact that you can only get an ambiguous
proclamation of a file's integrity with two instances:
if they agree, then, chances are, they are intact. If they
DISAGREE, you have no way of knowing which -- if either -- is
correct.
Realistically, the failures are one (or both) copies being inaccessible.
Underlying ECC makes the chances of both drives CLAIMING to have
reliable data and that data being DIFFERENT is virtually impossible.
For similar reasons, if one of my volumes returns "intact" data,
the hash is only there to verify the file's contents haven't been
altered "while the system wasn't paying attention" (e.g., take
the external drive and mount it DIRECTLY on a machine for ease
of access. Alter its contents willingly. Then, reintroduce it
to the archive system -- which will have no knowledge of your
"offline" (as far as IT is concerned) modifications. Until,
of course, the file is scrubbed and verified.
Because of this -- the fact that a drive can be accessed outside
of the control of the archive system -- I can have two files
that claim to be correct (no ECC errors, no RECORDED differences
in their hashes) yet are very different!
>> - If a discrepancy is found between the TWO instances, how is the
>> "winner" selected?
>
> Discrepancies don't occur with non-negligible probability unless you
> have had a massive hardware failure, or power-failure during writing.
> See <http://neil.brown.name/blog/20100211050355> for a discussion on the
> topic.
>
>> Plus, the more subtle question seldom considered:
>> - are files that are not "manually" accessed EVER verified? Or, do you
>> have to wait until you need/want a file to discover if it is intact??
>
> That's what "scrubbing" is for. Hardware raid controllers should do
> this automatically, with software raid it is usually done with a cronjob.
>
>> (Do *you* periodically check the contents of the batteries in your
>> flashlight to verify they are functioning BEFORE you need them?
>> Does the *flashlight* take on the responsibility of performing this
>> test on your behalf??)
>
> A torch is not a raid device.
>
>> Consider the above when you have many terabytes of data at stake.
>
> RAID like this is /exactly/ what people use when they have TB of data at
> stake.
As volume sizes increase, the problem of a second crash during a
rebuild becomes tangible. The fact that rebuild times increase
as the sizes of the volumes increases leads to longer "windows of
vulnerability".
> When they have more than that, they use redundant servers, or split the
> two halves of their software RAID-1 pairs across different boxes.
which is what I am doing. But, not REQUIRING the two halves to
be online concurrently. The DBMS "vouches" for the missing party
(i.e., instead of comparing A to B, you compare A to a function of B
(which also happens to be the same function of A). Then, later,
when A is gone, do the same with B.
>> Do you want ALL of it "spinning" whenever you need to access *any*
>> of it?
>
> Why is that a problem? Most hard disks have no problem spinning all the
> time.
Cheap consumer disks tend not to be well cooled and effectively
disposable. And, there's no reason to have them spinning when
you don't need to access them.
Do you leave your computers on all the time -- in case you might want to
walk up and do something "in an instant" without waiting for it to
boot? Do you let your disk spin down after an idle period? Do you
let drives idle at ~1W in case you might want them to spin up, now?
(so, if you have 50 such disks, you've got all that power being
dissipated keeping wall warts warm AND electronics connected to a
potential path for lightning strikes and line disturbances)
>> How long will it take for a proactive array to examine/verify
>> ALL of the data just to reassure you? (will you have the ENTIRE
>> array "up" for that length of time? Or, will you power it up just
>> long enough to access the file(s) of interest, then power it back down,
>> again? Or, will you power up ONLY the portion of the array known to
>> contain the file(s) you seek?)
>
> Again, what is the problem?
>
> If this is an array you rarely access, you can spin down all the disks
> when it is not in use.
I power down (external) drives when not in use -- unplug them from the
AC mains. If I'm going to use a higher performance drive that's part
of an array, I will often "pop" the unused drives out of that array
to reduce the cooling load and drive noise -- as well as wear and
tear on drives that aren't DOING anything besides "keeping warm".
>> As I said, I've been down this route. Ever hear what twenty to
>> fifty 10-15K drives sound like spinning at the same time?
>> <http://techreport.com/blog/13849/behold-thumper-sun-sunfire-x4500-storage-server>.
>> Even a 12-14 drive "shelf" is loud:
>> <http://www.amtecgroup.co.uk/products/external-disk-storag-1071/sun-storedge-a1000-12-bay-hdd-rack-array-raid-ctrl-2x-psu-1502.aspx>.
>> Remember,
>> they're all throwing off heat in a relatively dense package so you
>> have lots of (redundant!) fans in place to keep them from melting...
>
> Why are you considering a decade old high-end datacenter storage system?
> These are designed for installations where noise doesn't matter, and
> you have massive cooling systems.
Reread the explanation. Its a box that I used more than 20 years ago.
Before there were small alternatives. And, big disks.
> Just get a simple little server box with space for a few disks. I
> really like the HP Microservers - Gen 8 devices are dirt cheap,
> reliable, and have four neat 3.5" drive bays behind a door in the front.
> They are perfect for file servers unless you need really high-end
> performance.
Ive adopted the approach of using external drives -- so I can move
them, readily, between hosts, power them down individually, hotplug,
etc. The cheap consumer drives are ~$25/TB. I can afford to buy
two -- and replace one of EACH PAIR every year and still end up ahead!
I can put 4 drives on a single box and selectively power up
any combination of them. No fans, no heat, etc. I can easily get
20MB/s off the drives and over the wire. So, I may have to wait
a fraction of a minute to get a CD image. <shrug> If I'm just
looking for individual files, I can't click fast enough to get ahead
of them!
>> The "problem" with these solutions is they are designed for high
>> performance.
>> Serving files continuously, typically, to an "organization". And, with a
>> high degree of reliability/availability.
>
> So don't get one.
I haven't!
>> For an *archive*, you tend to need lots of capacity but comparably little
>> bandwidth! (how *often* will you access its contents? how quickly will
>> you need the file "delivered" to you?) And, you can trade reliability and
>> availability for bandwidth and cost. Do you keep your 2013 tax returns
>> in your top left desk drawer (where they are easily accessible)? Or, have
>> you moth-balled them to a more capacious site that's, perhaps, a bit more
>> difficult to access??
>
> What capacity are you looking for here?
I can easily account for ~30TB of "consumer grade" external drives without
considering optical media "originals", "burned" and the slew of 500G and 1T
drives around here. Or, the higher performance drives in the Live repository,
VM server, individual workstations, etc.
As I said, at $25/TB, its not worth the time to THINK about trimming
your archive; just shell out another $200 and buy another REDUNDANT 4TB
of capacity (i.e., 8TB).
[I have a LOT of stuff in my archive as it is SO much easier to access,
there, than on DVD spindles or in specific workstations/servers]
[toc] | [prev] | [next] | [standalone]
| From | Ali <abdulrazaq@gmail.com> |
|---|---|
| Date | 2016-10-30 15:45 -0700 |
| Message-ID | <a2ea0f11-07c6-453b-917c-c3ec388df73d@googlegroups.com> |
| In reply to | #25247 |
On Thursday, October 27, 2016 at 11:23:58 PM UTC+1, Don Y wrote:
> I currently "verify" my data archive with an algorithm similar to:
>
> ONCE: con 0;
> Success: con 0;
> Removed, Inaccessible, NoRecord, BadSize, BadHash: con Success+1+iota;
>
> verify(files: list of string ): list of (string, reason)
> {
> failures := list of (string, reason);
>
> while (files != nil) {
> theFile := hd files;
> (size, signature, result) := examine_file(theFile);
> (Size, Signature, Result) := query_DBMS(theFile);
> do {
> if (NotFound == result)
> {Result = Removed; break;}
> if (BadRead == result)
> {Result = Inaccessible; break;}
>
> // theFile exists!
> if (NotFound == Result)
> {Result = NoRecord; break;}
> if (Size != size)
> {Result = BadSize; break;}
> if (Signature != signature)
> {Result = BadHash; break;}
> } while (ONCE);
>
> if (Success != Result) {
> failures = (theFile, Result) :: failures;
> }
>
> files = tl files;
> }
> return( failures );
> }
>
> [apologies for any typos; too early in the day to be working!]
>
> The problem this has is examine-file() is expensive -- for large
> numbers of large files!
>
> I'd like to short-circuit this to provide an early indication of
> possible problems. E.g., stat(2) each "theFile" to identify any
> that are "Removed" without incurring the cost of examining the
> file in its entirety.
>
> [Of course, all files MUST be eventually examined as you can't
> verify accessibility (BadRead), actual size (BadSize) or correct
> signature (BadHash) without looking at every byte! But, you can
> get the simple checks out of the way expeditiously to alert you
> to "obvious" problems!]
>
> I can get an approximate indication of "actual size" just by
> stat(2)-ing the file, as well -- relying on examine_file() to
> later confirm that figure.
>
> The hash, of course, can't be guesstimated without trodding through
> the file in its entirety.
>
> The problem with *this* approach is I'd have to lock the entire
> object to ensure the integrity of the "early results" agrees with
> that of the "later results" (the original approach only requires
> taking a lock on the file being examined *while* it is being
> examined).
>
> Are there any "tricks" I can use, here? E.g., the equivalent of
> CoW would ensure the integrity of my data (for the *original*
> instance) in the presence of such a global lock. But, I think only
> the Bullet Server would give me any behavior approaching this...?
>
> Also, how wary would one be relying on the files' timestamps as a
> (crude) indication of "alteration"? (It seems far too easy to
> alter a timestamp without altering content/size so I've been leary
> of relying on that as having any "worth").
I would suggest to compartmentalize this function. There are unnecessary conditional operators stacked to achieve a ‘something’ common.
How about breaking apart file handler and DB results? That should provide two clear paths to work with.
//late to party!
[toc] | [prev] | [next] | [standalone]
| From | Don Y <blockedofcourse@foo.invalid> |
|---|---|
| Date | 2016-10-31 03:45 -0700 |
| Message-ID | <nv77b4$rmr$1@dont-email.me> |
| In reply to | #25300 |
On 10/30/2016 3:45 PM, Ali wrote:
> On Thursday, October 27, 2016 at 11:23:58 PM UTC+1, Don Y wrote:
>> The problem this has is examine-file() is expensive -- for large numbers
>> of large files!
>>
>> I'd like to short-circuit this to provide an early indication of possible
>> problems. E.g., stat(2) each "theFile" to identify any that are
>> "Removed" without incurring the cost of examining the file in its
>> entirety.
>>
>> [Of course, all files MUST be eventually examined as you can't verify
>> accessibility (BadRead), actual size (BadSize) or correct signature
>> (BadHash) without looking at every byte! But, you can get the simple
>> checks out of the way expeditiously to alert you to "obvious" problems!]
>>
>> I can get an approximate indication of "actual size" just by stat(2)-ing
>> the file, as well -- relying on examine_file() to later confirm that
>> figure.
>>
>> The hash, of course, can't be guesstimated without trodding through the
>> file in its entirety.
>>
>> The problem with *this* approach is I'd have to lock the entire object to
>> ensure the integrity of the "early results" agrees with that of the
>> "later results" (the original approach only requires taking a lock on the
>> file being examined *while* it is being examined).
>>
>> Are there any "tricks" I can use, here? E.g., the equivalent of CoW
>> would ensure the integrity of my data (for the *original* instance) in
>> the presence of such a global lock. But, I think only the Bullet Server
>> would give me any behavior approaching this...?
>>
>> Also, how wary would one be relying on the files' timestamps as a (crude)
>> indication of "alteration"? (It seems far too easy to alter a timestamp
>> without altering content/size so I've been leary of relying on that as
>> having any "worth").
>
> I would suggest to compartmentalize this function. There are unnecessary
> conditional operators stacked to achieve a ‘something’ common.
Its just pseudo-code -- intended to show what has to happen, overall.
I.e., if a particular file in the list presented is NOT found on
the media "as expected", then it should be considered as having been
REMOVED (deleted) from the medium -- naming it in the list of files
presupposes that it SHOULD be there. If it *is* there but can;t
be read/examined, then it is INACCESSIBLE -- naming it in the list
of files presupposes that it WILL be accessible (i.e., no read/seek
errors involved in examining its entire content).
Having verified the file is present and accessible, a record for it
should be available in the database -- yet another presupposition.
When a record IS available, the size recorded should coincide with
the size of the actual file encountered on the medium. And, having
the correct size should also lead to the contents having hashed
to the correct "signature" recorded.
Failing any of these conditions is an indication that the file presented
has not been encountered on the media in the form expected.
You can promote the "not found in database" test to precede the
examination of the file's contents. But, that test will rarely fail;
the list of files to be checked comes FROM the database! The test
just ensures the database hasn't changed in that regard wrt THIS
particular file in the interval between when the list of files
was prepared and when this particular file happened to be examined
(checking a file can take MINUTES or more; plenty of time for the
database to encounter changes -- unless I take a lock on it for
the duration of the testing process. Or, interleave the file list
creation with the actual verification)
> How about breaking apart file handler and DB results? That should provide
> two clear paths to work with. //late to party!
You can't compare the "actual" OBSERVATIONS of the file to the
stored EXPECTATIONS until you have examined the file.
The question is how much "examine_file()" can be decomposed and
the consequences of that decomposition.
E.g., if implemented, instead, as (I'm being lazy in my rewrite and trying to
maintain a resemblance to the original code so this is a quick cut-n-paste):
verify(files: list of string ): list of (string, reason)
{
failures := list of (string, reason);
while (files != nil) {
theFile := hd files;
do {
(result) := check_file_exists(theFile);
if (NotFound == result)
{Result = Removed; break;}
(Result, Size, Signature) := query_DBMS(theFile);
if (NotFound == Result)
{Result = NoRecord; break;}
// assume file must remain in the database, now that
// it has been spotted there; likewise, assume file
// must persist on the medium as it has been seen there
// previously! Otherwise, rely on a lock, transaction
// or some explicit "recovery" algorithm
(size) := check_file_size(theFile);
if (Size != size)
{Result = BadSize; break;}
// same assumptions as above!
(result) = verify_file_accesible(theFile);
// of course, this would actually be folded into the
// operation below; you'd not need to make two passes
// over the file to verify it can be read in its
// entirety and THEN compute the hash in a separate pass
if (BadRead == result)
{Result = Inaccessible; break;}
// same assumptions as above!
(signature) := check_file_signature(theFile);
if (Signature != signature)
{Result = BadHash; break;}
} while (ONCE);
if (Success != Result) {
failures = (theFile, Result) :: failures;
}
files = tl files;
}
return( failures );
}
The problem with this approach is it gives you several windows where the
state of the file, the filesystem (the metadata accessed by check_file_size
and check_file_exists) itself and the DBMS can get out of sync. You
have to take a lock on the database and the filesystem!
[A "transactional filesystem" has been discussed in this context; so,
like the DBMS, you can rollback operations on the filesystem to ensure
it always remains in sync with the data you've stored regarding its
state (you have two copies of that data and no other way to ensure they
REMAIN in sync while you perform these lengthy tests]
The bigger problem lies in structuring the loop to process each file
in the list in its entirety -- before advancing to the next file
in the list.
Imagine, instead:
verify(files: list of string ): list of (string, reason)
{
failures := list of (string, reason);
non_existent := verify_existence(files)
failures = non_existent :: failures; // unsupported syntax
// FAIL: should have removed non_existent from files!
not_recorded := verify_record(files);
failures = not_recorded :: failures; // unsupported syntax
// FAIL: should have removed not_recorded from files!
wrong_sized := verify_sizes(files);
failures = wrong_sized :: failures; // unsupported syntax
// FAIL: should have removed wrong_sized from files!
inaccessible := verify_read(files);
failures = not_readable :: failures; // unsupported syntax
// FAIL: should have removed inaccessible from files!
counterfeit := verify_content(files);
failures = counterfeit :: failures; // unsupported syntax
return( failures )
}
verify_existence(files: list of string ): list of (string, reason)
{
failures := list of (string, reason);
while (files != nil) {
theFile := hd files;
(result) := check_file_exists(theFile);
if (NotFound == result)
failures = (theFile, Removed) :: failures;
files = tl files;
}
return( failures );
}
verify_record(files: list of string ): list of (string, reason)
{
failures := list of (string, reason);
while (files != nil) {
theFile := hd files;
(Result, Size, Signature) := query_DBMS(theFile);
if (NotFound == Result)
failures = (theFile, NoRecord) :: failures;
files = tl files;
}
return( failures );
}
verify_sizes(files: list of string ): list of (string, reason)
{
failures := list of (string, reason);
while (files != nil) {
theFile := hd files;
(Result, Size, Signature) := query_DBMS(theFile);
(size) := check_file_size(theFile);
if (Size != size)
failures = (theFile, BadSize) :: failures;
files = tl files;
}
return( failures );
}
etc. (hopefully you can see the pattern emerging, here...)
There's a subtle but important difference. The END RESULT is
POTENTIALLY the same list of failures. There can be subtle
differences in the list as different aspects of each file
(and the database entries) will be processed at different
relative wall-clock times. E.g, the signature of the first
file won't be checked at the same time relative to the
invocation of the main function. Indeed, it may not be
checked AT ALL -- based on which preceding tests might fail!
(compared to my initial implementation which examined ALL
aspects of a file before advancing to the next)
Another subtlety is that at each point where "failures" is
updated, the presence of ANY "theFile" in that list is a
positive indication that the file in question is *NOT*
intact. And, by extension, that the list of files is not
completely intact!
The caller only has to wait for the *final* assessment to
be "assured" that the ENTIRE list is INTACT (in so far as
these tests can vouch). Other information can be leaked,
sooner (as shown)
So, if you have a medium whose integrity is suspect, you
get an early indication that it *is*, in fact, corrupted!
You aren't forced to wait for a THOROUGH examination of
all files preceding the KNOWN OFFENDER (i.e., the file
that has a size of 3 bytes instead of 283922; or the file
that has "gone missing"; or the dirent that can't be
accessed; etc.) before that determination can be made!
"Crap! This DVD+R is showing signs of bitrot!"
"Crap! I must have accidentally deleted part of this medium!"
The remedial actions you take when armed with this information
sooner (vs. later) can be very different.
E.g., suspicious of a failing medium, I might not want to wait
hours for its entire contents to be ROUTINELY VERIFIED before
taking action to preserve those objects that might ONLY exist
on that medium. Or, the objects that are of "true value".
ESPECIALLY if the device is throwing read errors and incurring
lengthy timeout/retries (e.g., an optical medium).
If, as is probably the MOST COMMON CASE, a set of objects had been
accidentally DELETED, then waiting until they were encountered in
the "list of files" could potentially be after many (thousand, millions?)
of files have been THOROUGHLY examined -- and found to be intact
(i.e., The Norm for any reliable medium!) Checking for the existence
of ALL files before checking the content of EACH file would make
that information known sooner rather than later. This affords
you the opportunity to restore those files as the removed copies
may have been your sole "backup"; the originals are at risk until
you can recreate those backup copies!
Of course, the "hazards" and races that this approach presents differ
significantly from the previous -- and original! -- approaches.
So, while the results may be the same if done during "regularly scheduled
system maintenance" (like a commercial IT department would do), they
won't be the same if the database and the filesystems are being actively
used when the tests are performed.
[Indeed, when the filesystems are NOT being used, the media are
typically POWERED DOWN in my case! No dedicated period for
"routine system maintenance"; that has to happen alongside my
normal usage -- and ADAPT to those usage patterns!]
[toc] | [prev] | [standalone]
Page 3 of 3 — ← Prev page 1 2 [3]
Back to top | Article view | comp.arch.embedded
csiph-web