Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.arch.embedded > #25313

Re: Checking large numbers of files

Show all headers | View raw

Hi David,

On 10/31/2016 4:19 AM, David Brown wrote:
> On 29/10/16 22:15, Don Y wrote:
>> On 10/29/2016 5:17 AM, kalvin.news@gmail.com wrote:
>
> Don, let me try to understand the basis of your problem here.  Correct
> me at any point that I am wrong (I am writing in many small paragraphs
> to make that easier).  I have also added a few comments inline below my
> main post.
>
> 1. You have lots of files, held on computers, and you want to make sure
> they don't get changed or corrupted.

I want to ensure they "remain accessible".  I'm guarding against data loss.
Both for "old" data and "freshly minted" data.  I.e., wanting to ensure
the work I do TODAY is preserved without my having to take deliberate
steps to ensure it ends up "where it belongs".

> 2. You don't trust "metadata" methods such as sizes and timestamps,
> because files could be changed or corrupted without affecting these.

No, I don't trust *certain* metadata (notably timestamps).  I am
willing to rely on a dirent's "size" to be reflective of the files
actual size.  Whether I can *access* that many bytes is a different
issue -- something I can only determine by trying to access those
bytes.

As a medium can "fail" in such a way that the dirent is munged -- but
the data may still be intact (just not accessible via the normal
mechanisms), if the dirent says the file name is now "boogie"
instead of "myfile", then I have no choice but to assume "myfile"
is gone -- I'm not trying to implement a "data recovery service".

> 3. You don't want to read through the entire file, because they are big.

Correct.  And, data rates may be slow.  Imagine trying to "verify"
CD's at a few megabytes per second!

And, if I have information that suggests the file is NOT "as it should be",
then why undertake the effort to PROVE that it isn't?  "Gee, this file
is supposed to be 32GB but I stat(2) it at just 5KB!  Either the dirent
is munged -- the file really IS 32GB but the dirent is in error -- OR,
the file has been altered/replaced/truncated/etc."

> 4. You want some sort of algorithm for detecting file change or
> corruption, given 2 and 3.

Also, some way of verifying NON change/corruption.  There are many
ways to ask the question and different approaches to getting an
answer.  "Has ANYTHING changed, here?", on a 4TB drive, can yield
an answer in a few SECONDS -- if the first file you go looking for is
found to be absent (have a changed name) or of a different size.

OTOH, "is *everything* UNCHANGED, here" takes a lot longer to answer
in the affirmative; you have to perform all of the tests on all of the
objects!

> If that is the case, then you will be disappointed - no such algorithm
> could exist.

See above.

Think about the ways that "file collections" can be altered.  And, think
of the relative likelihoods of those changes!

By far, I suspect accident user actions are the biggest issue.  Someone
deletes something that they shouldn't have.  Or, moves something and
the operation is aborted without their noticing it (media full, etc.).

Similarly, if users are responsible for maintaining the file sets,
then its possible a user updates X and forgets to update Y.  Or,
X *instead* of Y.

Or, a buggy protocol implementation (an aborted automated transfer that
leaves a partial result)

After that, perhaps a total drive failure (e.g., not spinning up).

Somewhere down the list are local, uncorrectable errors that affect
some portion of the drive surface.  These would tend to increase
in frequency, be more widely distributed through the dataset and
PROBABLY be noticeable if someone is aggressively watching for them.

For example, I have noticed that cheap consumer drives (e.g., 4TB)
tend to have dramatically different performance/throughput as they
"get full".  Presumably, fewer opportunities for "reliable"
sectors to be encountered vs. faulty sectors accumulating at the
unused portion of the medium.

> So now we have to consider what would give you a convenient, economical
> and reliable way to store your files safely.
>
> First, I assume that your old existing systems are irrelevant except as
> a source to copy from.  Spending time and effort on maintaining outdated
> hardware is a waste - I expect you to set up a new system.

I've already pulled the files off "old media".  There are some
obsolescent media that I have no choice but to support:  e.g.
my DSO and EPROM programmer both rely on 3" floppy media.  To
work around that, I'd have to buy new equipment -- a silly
"optimization" (easier to keep a floppy drive operational AND
capture the images off the "factory media" before it degrades!

What is considered "outdated" is subject to personal opinion.
I see no reason to replace a machine that still runs "modern"
OS's -- even if faster machines are available.  With that
criteria, you'd have to update hardware yearly -- and update
all licensed software at the same rate!

> Second, I expect you to have off-site backups.  This could be done by
> duplicating the main system, or by simply having a number of off-site
> copies on DVD, archived hard disks, cloud storage, etc.

Nope.  I've satisfied all of my contractual/legal obligations so
I'm only "on the hook" for things that have value to me, personally.
There is some value to having "past projects" accessible as
references.  But, if the house burned to the ground, the LAST
thing I would worry about would be my possessions -- both
tangible and intellectual.

[As an aside, a routine exercise I perform is querying myself
re: how I'd handle such an event:  "If the house was on fire
and you could retrieve ONE item, what would that item be?"
My goal, for as long as I've been asking myself that question,
has been to be COMFORTABLE answering "nothing".  I have been
at that point for quite some time!  :>  Of course, that doesn't
mean I'd be eager to lose *anything*!!]

> Third, I expect you to leave behind your prejudices and knowledge of
> limitations of old systems, and to look forward.

I've only mentioned old systems to indicate how I got here.
How my archive maintenance philosophy has evolved to its
current set of practices WITHOUT foreknowledge of future
developments.

E.g., dealing with RAID5 and $1K drives and why RAID1 was
"an improvement" (for me) in that timeframe.  Then, why
"manual mirroring" was an improvement beyond that (no
special hardware OR software required).

And, using SCSI as a basis for removable disks before USB
existed to provide that capability (so, I can keep a stack
of SCSI disks "cold" on a shelf and swap them into a
system as needed instead of having ALL of them spinning
ALL THE TIME).

Then, the downside (discipline required) for these approaches.

Finally, how to avoid that aspect (by automating the process
taking advantage of more modern hardware and less expensive
components).

> Files change, get lost, or corrupted due to four main causes:
>
> 1. A user changes them in some way (intentionally or unintentionally).

Exactly.

> 2. Malware or hackers change them.

Not a problem, here.  My systems are offline.  In the odd chance that
something was "infected" and spread to items CURRENTLY on-line,
my solution would detect those changes by noting the differences
in file hashes during periodic "audits".

And, as most of my archive is offline at any given time, I'd probably
notice any widespread "corruption" BEFORE mounting yet another volume.

> 3. Bit rot or sector wear causes some data to be lost on a disk.

Yes.  Nothing I can do about that short of buying better quality
components.  In practice, I've not seen this to be a real problem
(noting the consumer drive issue mentioned above).  Other folks
have reported poor durability of optical media -- I've not found
that despite using "commodity media".  I've heard of horror
stories with disk failures but have been spared that problem
(touch wood!).

I suspect part of this is due to the fact that I distribute my work
among many different machines.  So, total power on hours tends to
stay low.  For machines and their peripherals.  I suspect I've only
a hundred hours on my "most used" laptop -- cuz I don't use laptops
unless traveling or away from home (in the field, customer site, etc.)

> 4. Serious hardware failure, such as a disk head crash losing the whole
> disk.

Along the same lines, I HAVE encountered a catastrophic failure due to
a *driver* bug!  I watched a disk get "scrambled" and assumed the
problem was a disk failure (cuz everyone claims these are so common!).
Then, mounted the bytewise backup of the same drive -- and saw *it*
get scrambled just as quickly!

[No, I don't buy the idea of two independant drives dying at the same
time -- esp when they've both been sitting on a shelf for most of
their lifetimes!  What's changed?  Ah, I updated the OS yesterday...
let's roll things back to the previous release and try that, again.
Dig out the MO disks to recreate the corrupted drives (painfully
slow as the media is inherently slower than winchesters).]

> I expect you to handle 1 and 2 mainly through good security and regular
> backups.  I strongly recommend using a filesystem or setup with cheap
> snapshots - then you can take them regularly.

The second isn't a problem.  The first always is, potentially.
I only access the archive AS the superuser -- to ensure "noone"
else can screw up (i.e., me in my non-privileged guise).  So,
everything is exposed to my potential folly.

For example, I recently pruned all the "foreign language" updates from
my MS update archive.  ARA, JAP, POR, etc.  Just save ENU!  Click.
Click.  Delete.  Done.

Ooops!  "GLB" isn;t an abbreviation for a localization but,
rather "common to all"!  (sigh)  Better go back and redownload that
stuff, again!

Even automating this leaves opportunity for screwups -- if I specify
the wrong criteria/options for a particular object (and the system
dutifully follows my ill-conceived directives!)

> There is a myth that hard disk sectors can get corrupted and return
> incorrect data without an error indication - the so-called "undetected
> read error".  In theory, this /can/ occur - but unless you are using the
> data quantities of CERN or Google, you will never see it in your
> lifetime (that's why I call it a myth).  You /might/ come across
> firmware errors in disks that give such undetected read errors, though
> it is highly unlikely.  "Unrecoverable read errors", on the other hand,
> are not as uncommon as one would like - sometimes a disk is simply
> unable to return the data you ask for, but it can at least inform you of
> the failure.

I'm not worried about disks misreporting data/errors.
Likewise, I'm not worried about hash collisions -- there's
no active adversary involved.  Sure, some random set of
events could THEORETICALLY result in all those magnetic
domains flipping to some OTHER combination of states
AND still yield the same hash.

But, the Sun will burn cold long before that happens!
I'm willing to "risk" accessing that file at some
later date and wondering why its completely munged -- yet
still hashing to the correct value!

> The general method to protect against failures is redundancy combined
> with systems to spot errors.  For storage, that is almost invariably
> done using RAID.

RAID is predicated on you wanting "online" verification of EVERY
data access.  And, semi-seamless correction of any failures that
turn up.  All happening at "regular disk speeds" on volumes that
are PROBABLY seeing some significant amount of traffic.

It would be foolish to keep such an array spinning 24/7/365 for
the *rare* access that might occur.  When was the last time you
needed to access the install media for application XYZ?  Do
you REALLY need it to be available in a fraction of a second?
Are you THAT impatient?

The last time I accessed my 1976 tax forms was... in 1977!  Do
I *need* to keep them?  No.  Should I discard them?  Why?  Are
those few magnetic domains THAT valuable to me that I need
to salvage them??

You can use the redundancy aspects of RAID *without* using a
RAID appliance.  You just trade-off performance for "accessibility"
and "performance".

> So your basic setup here is to have two drives with a RAID-1 mirror.
> Anything that is written to the mirror gets written to both drives.  If
> a sector read fails (despite the drive's extensive error checking and
> correcting codes), the RAID controller (hardware or software) will read
> the second copy from the other disk.  It will then re-write this correct
> data to the first disk, which the drive will map to a spare sector.  If
> a drive fails entirely, the controller will read from the other drive.

Yes, but you can adopt a similar strategy without having both drives
on-line at the same instant.  This exposes a window in which things
can go south.  But, they are "unlikely" to do so.

E.g., when I moved from RAID1 to JBOD, I simply copied every file to two
different places -- under two parallel mount points.  The two drives
are not sector-wise images of each other.  But, at the filesystem
level, they contain the same files in the same logical places in their
hierarchies.  I could close my eyes while you rearranged the drives
and I'd not be able to tell if you'd swapped them -- or not.

If you have a means of improving your confidence in a particular
instance of a file (i.e., if you record name, size, hash in a
reliable medium and then verify this when you later access
that instance), then you can use that instance in lieu of the
original instance in creating the "mirror" copy of the file.

E.g.,

# cp file /Archive/1
# md5 file > hash
# rm file
# md5 /Archive/1/file | cmp hash -
# cp /Archive/1/file /Archive/2/file

is effectively (to "many 9's") the same as:

# cp file /Archive/1
# cp file /Archive/2
# rm file

*if* the operations are reasonably close together in time.  A RAID
array cuts the time to microseconds.  But, realistically, would it
be much different if the RAID controller ran a delay loop for
some number of seconds between each write?  (obviously, throughput
would suffer!)

My current approach exploits this.  So, a file can be copied to a volume
and its "signature" reliably recorded.  Later, that instance can be
used (after verifying it against that stored signature) to create
another instance of the file on another medium.

And, both instances can be "verified" at any time (assuming they
are available, online, at the time) -- by double-checking the
"signature", size, name, etc.

> "Bit rot", or slow bit corruption on disk sectors, is prevented by
> scrubbing.  You regularly read through the entire array, for both disks.

Yes.  But, just reading doesn't PREVENT the problem.  A disk can
degrade to a point where scrubbing just keeps revealing errors.
There;s a point when a drive's reliability is not recoverable.
The more the utilization, the less ability to adapt to grown defects.

>   If a sector has had corruption on some bits, the drives ECC coding will
> usually let it correct the error automatically, and it will re-write the
> sector (with re-mapping if necessary) to refresh the data.  If the
> corruption is too great, the RAID controller will re-write the sector
> based on the second copy.

And, the same sort of algorithm can be applied to multiple copies of a file
accessed at different points in time!  RAID keeps wanting to do everything
"now".  It doesn't have a deep memory for errors -- fix it and forget it.

And, if *the* copy of *the* file becomes irrecoverably corrupted, its
not smart  enough to know that another copy might exist -- on this same
medium OR a different medium... possibly at the other end of a network
connection!

> Thus RAID keeps your data reliable, at least until a catastrophic
> failure (such as the whole server dying) - that's why you have backups.
>
> For more features, I would recommend using BTRFS or ZFS to handle your
> RAID mirror.  These filesystems give you additional checksumming (for
> protection against drive firmware errors, or extremely rare undetected
> read errors), as well as very cheap snapshots and other goodies.
>
> On the other hand, if you want the simplest "will work on anything, even
> old Linux systems or Windows systems" solution, then use Linux md RAID
> with metadata format 1.0.  This puts the RAID information at the end of
> the disk - all the rest of the disk appears /exactly/ as a normal
> non-raid disk, and can be read from anything else.

You can't then accommodate R/O media.  E.g., I want one instance of
a particular set of files to be the "Dell Windows 7 Reinstallation DVD"
that I have in my desk drawer.  I want the *other*, probably more
vulnerable copy of that file to be "Dell_Windows_7_Reinstallation_DVD.iso"
So, I can call up the ISO file relatively easily (power up a particular
disk drive and CD to the correct directory) and access its contents.

But, being on writeable media, its conceivable that this .iso file
can get corrupted, overwritten, replaced, etc.  So, I want to be able to
verify that it is intact.  And/or "recover" it -- from the original
medium!

Or, if the original medium gets munged (scratched), to be able to recreate
a NEW physical DVD from the ISO file -- which can later be used to verify
the ISO's integrity...

>>> Disclaimer: I am not an expert on the data storage, but here are my
>>> thoughts:
>>>
>>> - Use a RAID system which has enough redundancy to detect and correct
>>> defects in the media. Depending of the RAID level used, the capability to
>>> data recovery varies.
>>
>> I started out using RAID5 "hardware" arrays (and other, more sophisticated
>> configurations).  This requires you to have multiple spindles running
>> concurrently (else the array can't *detect* discrepancies).  Compact
>> arrays also tend to suffer from cooling problems.
>
> You will need two spindles (or SSDs) concurrently - that's the only way
> to get two copies of your data on two disks.

Yes, though you can cache those files that you know will need to
be coped to a yet-to-be-mounted volume.  You know what their signatures,
sizes, names are and SHOULD BE.

So, when the desired target volume is made available to you (perhaps
because you requested the user mount it on your behalf!), you can
"resume" the DEFERRED copy to that second spindle.  I.e., where you
cache the file need not be one of the spindles IN the archive.

> Don't bother with RAID-5.  Either use RAID-1 if your data fits
> comfortably on a single disk, or use RAID-6.  Modern disks have less
> cooling requirements than old ones, if you go for RAID-6.

I made the cooling point to indicate why LARGE RAID arrays (12 and
48 drives) were problematic when I was using them.  They wanted to
be compact -- but 20W drives packed side-by-side in a small cabinet
is a lot of BTUs to disperse.

[nowadays, you can buy disks that are 10 - 1000 times larger
so you;ve already cut the equivalent power-per-gigabyte-capacity
significantly]

>> It also ties you to an often proprietary (or, at least, poorly documented)
>> implementation ("volume format").  So, a failure of the hardware can leave
>> you with possibly intact volume(s) and no straightforward means of
>> accessing their contents.
>
> Use software RAID - either Linux, or BSD if you prefer.  Avoid
> proprietary hardware solutions.
>
>> Moving volumes between arrays is seldom supported (unless you have
>> identical
>> arrays -- hardware/software -- AND a well considered means of doing this.
>> This effectively precludes having volumes off-line.  Introducing a
>> "foreign"
>> volume to an array will often result in the array prompting you to
>> "initialize" it (wipe it clean!).
>>
>>> - Duplicate or mirror the data across different physical media.
>>
>> This is RAID1.  Many of the above issues still apply.  In any case,
>> the mirrors must be spinning concurrently for the array to function.
>> (It has no way of knowing what the "other" copy of the "file"
>> looks like unless it can *see* it, now, in its entirety).
>
> If you are really desperate to avoid having two disks spinning at the
> same time, it is possible with Linux md raid to designate one drive as
> "write mostly".  It will then only be used when writing (obviously),
> when scrubbing (obviously), or when reading if the first drive fails to
> return the correct data.  I cannot think of any real benefit of this,
> but it is possible.

I'm not averse to running multiple spindles.  I often have 2 or 4
external drives spinning so I can easily move a single set of
files onto "a volume and its 'mirror'".

But, I shouldn't NEED to have two or more spindles up at the same time!

There's already a DBMS running to do all the bookkeeping (similar to
what the RAID has to do).  If you overprovision this machine  -- by
even a small amount -- then you can set aside a staging area to cache
those files that you are adding or altering in the "system" -- even
if the actual medium supporting it isn't available, presently.

Because you have the DBMS accessible, you can make a notation for the
file that you will *ultimately* be updating:  "When I see this
volume hosting this file in the future, I need to replace it with
the file currently hiding in the 'pending file cache'".

At the same time, you can choose to notify the user that you;d
eventually like to have that volume mounted -- and why its important
to you!

And, if you ever have to purge the file from that cache, you still have
a record of the fact that an update is pending for the/those other
copies of it!  So, you can remind the user that the array is
still in an inconsistent (degraded) state.

I'm doing the same thing that a RAID1 array would do -- but allowing
the timescale to be more elastic.  Letting the user decide how
long he is willing to operate in those "degraded" states.

E.g., perhaps the file needs to be updated in *3* places!  You;ve
successfully updated the first, "original" (insomuch as any copy can
be considered the "original").  You cache a copy in the holding area.
And, tickle the user to mount the one or two volumes that host
these two other, degraded copies.

Eventually, you encounter one of those columes and surreptitiously
update the copy while the user is fetching some OTHER files off the
medium.  You update the database to record the fact that this
copy is now in agreement with the "original" and no longer in
need of an update.

And, you continue to cache the file as you wait for the OTHER
medium to come along for that third copy!

You continue to pester the user -- but, can now remind him that
he no loner is operating in *AS* DEGRADED a state as previously;
he now has TWO copies of the file available!  But, you'd
really like to get that third one updated, as well!

Regardless of how much time passes, you will continue to remember the
"state" of this file and its counterparts -- because the DBMS is persistent!

Some day, the file may disappear from the cache -- pushed out by some newer
"change" that represents a more serious degradation problem:  only ONE
instance of that new file exists in the array at this time!  (the previously
cached file has TWO copies available, despite needing a third update)

Some day, you may encounter one of these first two copies (of the first
file).  You can surreptitiously reacquire a copy to have a fresh cached
copy.  And, verify that it is intact -- in the hope that you might
encounter that third volume, RSN.

[See?]

>> This begs two crucial questions:
>
> No, it /raises/ two questions.
>
>> - are BOTH copies of the file examined when you try to access it?
>
> No.  That would be highly inefficient, and does not contribute to
> improved reliability.
>
>>    Or, is a "primary" copy accessed and the "secondary" copy only accessed
>>    if the primary throws a read error?
>
> Yes.

These were rhetorical questions intended to highlight the fact that
having two copies doesn't really mean what you think!  *If* you get
a requested file from the array, there is no guarantee that both
copies are intact.  No guarantee that the copy you got HAS a
"backup" in case it is lost.

And it highlights the fact that you can only get an ambiguous
proclamation of a file's integrity with two instances:
if they agree, then, chances are, they are intact.  If they
DISAGREE, you have no way of knowing which -- if either -- is
correct.

Realistically, the failures are one (or both) copies being inaccessible.
Underlying ECC makes the chances of both drives CLAIMING to have
reliable data and that data being DIFFERENT is virtually impossible.

For similar reasons, if one of my volumes returns "intact" data,
the hash is only there to verify the file's contents haven't been
altered "while the system wasn't paying attention"  (e.g., take
the external drive and mount it DIRECTLY on a machine for ease
of access.  Alter its contents willingly.  Then, reintroduce it
to the archive system -- which will have no knowledge of your
"offline" (as far as IT is concerned) modifications.  Until,
of course, the file is scrubbed and verified.

Because of this -- the fact that a drive can be accessed outside
of the control of the archive system -- I can have two files
that claim to be correct (no ECC errors, no RECORDED differences
in their hashes) yet are very different!

>> - If a discrepancy is found between the TWO instances, how is the
>>    "winner" selected?
>
> Discrepancies don't occur with non-negligible probability unless you
> have had a massive hardware failure, or power-failure during writing.
> See <http://neil.brown.name/blog/20100211050355> for a discussion on the
> topic.
>
>> Plus, the more subtle question seldom considered:
>> - are files that are not "manually" accessed EVER verified?  Or, do you
>>    have to wait until you need/want a file to discover if it is intact??
>
> That's what "scrubbing" is for.  Hardware raid controllers should do
> this automatically, with software raid it is usually done with a cronjob.
>
>>    (Do *you* periodically check the contents of the batteries in your
>>    flashlight to verify they are functioning BEFORE you need them?
>>    Does the *flashlight* take on the responsibility of performing this
>>    test on your behalf??)
>
> A torch is not a raid device.
>
>> Consider the above when you have many terabytes of data at stake.
>
> RAID like this is /exactly/ what people use when they have TB of data at
> stake.

As volume sizes increase, the problem of a second crash during a
rebuild becomes tangible.  The fact that rebuild times increase
as the sizes of the volumes increases leads to longer "windows of
vulnerability".

> When they have more than that, they use redundant servers, or split the
> two halves of their software RAID-1 pairs across different boxes.

which is what I am doing.  But, not REQUIRING the two halves to
be online concurrently.  The DBMS "vouches" for the missing party
(i.e., instead of comparing A to B, you compare A to a function of B
(which also happens to be the same function of A).  Then, later,
when A is gone, do the same with B.

>> Do you want ALL of it "spinning" whenever you need to access *any*
>> of it?
>
> Why is that a problem?  Most hard disks have no problem spinning all the
> time.

Cheap consumer disks tend not to be well cooled and effectively
disposable.  And, there's no reason to have them spinning when
you don't need to access them.

Do you leave your computers on all the time -- in case you might want to
walk up and do something "in an instant" without waiting for it to
boot?  Do you let your disk spin down after an idle period?  Do you
let drives idle at ~1W in case you might want them to spin up, now?
(so, if you have 50 such disks, you've got all that power being
dissipated keeping wall warts warm AND electronics connected to a
potential path for lightning strikes and line disturbances)

>> How long will it take for a proactive array to examine/verify
>> ALL of the data just to reassure you?  (will you have the ENTIRE
>> array "up" for that length of time?  Or, will you power it up just
>> long enough to access the file(s) of interest, then power it back down,
>> again?  Or, will you power up ONLY the portion of the array known to
>> contain the file(s) you seek?)
>
> Again, what is the problem?
>
> If this is an array you rarely access, you can spin down all the disks
> when it is not in use.

I power down (external) drives when not in use -- unplug them from the
AC mains.  If I'm going to use a higher performance drive that's part
of an array, I will often "pop" the unused drives out of that array
to reduce the cooling load and drive noise -- as well as wear and
tear on drives that aren't DOING anything besides "keeping warm".

>> As I said, I've been down this route.  Ever hear what twenty to
>> fifty 10-15K drives sound like spinning at the same time?
>> <http://techreport.com/blog/13849/behold-thumper-sun-sunfire-x4500-storage-server>.
>>   Even a 12-14 drive "shelf" is loud:
>> <http://www.amtecgroup.co.uk/products/external-disk-storag-1071/sun-storedge-a1000-12-bay-hdd-rack-array-raid-ctrl-2x-psu-1502.aspx>.
>>   Remember,
>> they're all throwing off heat in a relatively dense package so you
>> have lots of (redundant!) fans in place to keep them from melting...
>
> Why are you considering a decade old high-end datacenter storage system?
>   These are designed for installations where noise doesn't matter, and
> you have massive cooling systems.

Reread the explanation.  Its a box that I used more than 20 years ago.
Before there were small alternatives.  And, big disks.

> Just get a simple little server box with space for a few disks.  I
> really like the HP Microservers - Gen 8 devices are dirt cheap,
> reliable, and have four neat 3.5" drive bays behind a door in the front.
>   They are perfect for file servers unless you need really high-end
> performance.

Ive adopted the approach of using external drives -- so I can move
them, readily, between hosts, power them down individually, hotplug,
etc.  The cheap consumer drives are ~$25/TB.  I can afford to buy
two -- and replace one of EACH PAIR every year and still end up ahead!

I can put 4 drives on a single box and selectively power up
any combination of them.  No fans, no heat, etc.  I can easily get
20MB/s off the drives and over the wire.  So, I may have to wait
a fraction of a minute to get a CD image.  <shrug>  If I'm just
looking for individual files, I can't click fast enough to get ahead
of them!

>> The "problem" with these solutions is they are designed for high
>> performance.
>> Serving files continuously, typically, to an "organization".  And, with a
>> high degree of reliability/availability.
>
> So don't get one.

I haven't!

>> For an *archive*, you tend to need lots of capacity but comparably little
>> bandwidth!  (how *often* will you access its contents?  how quickly will
>> you need the file "delivered" to you?)  And, you can trade reliability and
>> availability for bandwidth and cost.  Do you keep your 2013 tax returns
>> in your top left desk drawer (where they are easily accessible)?  Or, have
>> you moth-balled them to a more capacious site that's, perhaps, a bit more
>> difficult to access??
>
> What capacity are you looking for here?

I can easily account for ~30TB of "consumer grade" external drives without
considering optical media "originals", "burned" and the slew of 500G and 1T
drives around here.  Or, the higher performance drives in the Live repository,
VM server, individual workstations, etc.

As I said, at $25/TB, its not worth the time to THINK about trimming
your archive; just shell out another $200 and buy another REDUNDANT 4TB
of capacity (i.e., 8TB).

[I have a LOT of stuff in my archive as it is SO much easier to access,
there, than on DVD spindles or in specific workstations/servers]

Back to comp.arch.embedded | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-27 15:23 -0700
  Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-28 09:48 +1100
    Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-27 16:48 -0700
      Re: Checking large numbers of files Dimiter_Popoff <dp@tgi-sci.com> - 2016-10-28 03:17 +0300
        Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-27 18:50 -0700
          Re: Checking large numbers of files Dimiter_Popoff <dp@tgi-sci.com> - 2016-10-28 05:38 +0300
            Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-27 21:33 -0700
      Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-28 11:40 +1100
        Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-27 18:52 -0700
  Re: Checking large numbers of files kalvin.news@gmail.com - 2016-10-28 03:50 -0700
    Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-28 10:49 -0700
      Re: Checking large numbers of files kalvin.news@gmail.com - 2016-10-29 05:17 -0700
        Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 13:15 -0700
          Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 09:47 +1100
            Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 16:19 -0700
              Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 16:36 -0700
              Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 12:00 +1100
                Re: Checking large numbers of files Dimiter_Popoff <dp@tgi-sci.com> - 2016-10-30 04:05 +0200
                Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 14:06 +1100
                Re: Checking large numbers of files Dimiter_Popoff <dp@tgi-sci.com> - 2016-10-30 05:48 +0200
                Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 22:33 -0700
                Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 17:00 +1100
                Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-10-31 04:43 -0700
                Re: Checking large numbers of files (long) David Brown <david.brown@hesbynett.no> - 2016-10-31 14:20 +0100
                Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-10-31 08:02 -0700
                Re: Checking large numbers of files (long) David Brown <david.brown@hesbynett.no> - 2016-11-01 09:47 +0100
                Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-11-01 04:36 -0700
                Re: Checking large numbers of files (long) David Brown <david.brown@hesbynett.no> - 2016-11-01 14:29 +0100
                Re: Checking large numbers of files (long) David Brown <david.brown@hesbynett.no> - 2016-11-01 15:16 +0100
                Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-11-02 11:42 -0700
                Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 22:15 -0700
                Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 21:50 -0700
                Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 17:14 +1100
                Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-10-31 03:36 -0700
                Re: Checking large numbers of files (long) Clifford Heath <no.spam@please.net> - 2016-11-01 09:20 +1100
                Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-11-01 02:23 -0700
                Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 21:20 -0700
                Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 17:25 +1100
          Re: Checking large numbers of files Robert Wessel <robertwessel2@yahoo.com> - 2016-10-30 00:14 -0500
            Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 23:02 -0700
          Re: Checking large numbers of files David Brown <david.brown@hesbynett.no> - 2016-10-31 12:19 +0100
            Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-31 07:00 -0700
  Re: Checking large numbers of files Ali <abdulrazaq@gmail.com> - 2016-10-30 15:45 -0700
    Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-31 03:45 -0700

csiph-web