Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.arch.embedded > #25307

Re: Checking large numbers of files

Show all headers | View raw

On 29/10/16 22:15, Don Y wrote:
> On 10/29/2016 5:17 AM, kalvin.news@gmail.com wrote:

Don, let me try to understand the basis of your problem here.  Correct
me at any point that I am wrong (I am writing in many small paragraphs
to make that easier).  I have also added a few comments inline below my
main post.

1. You have lots of files, held on computers, and you want to make sure
they don't get changed or corrupted.

2. You don't trust "metadata" methods such as sizes and timestamps,
because files could be changed or corrupted without affecting these.

3. You don't want to read through the entire file, because they are big.

4. You want some sort of algorithm for detecting file change or
corruption, given 2 and 3.

If that is the case, then you will be disappointed - no such algorithm
could exist.


So now we have to consider what would give you a convenient, economical
and reliable way to store your files safely.

First, I assume that your old existing systems are irrelevant except as
a source to copy from.  Spending time and effort on maintaining outdated
hardware is a waste - I expect you to set up a new system.

Second, I expect you to have off-site backups.  This could be done by
duplicating the main system, or by simply having a number of off-site
copies on DVD, archived hard disks, cloud storage, etc.

Third, I expect you to leave behind your prejudices and knowledge of
limitations of old systems, and to look forward.


Files change, get lost, or corrupted due to four main causes:

1. A user changes them in some way (intentionally or unintentionally).

2. Malware or hackers change them.

3. Bit rot or sector wear causes some data to be lost on a disk.

4. Serious hardware failure, such as a disk head crash losing the whole
disk.

I expect you to handle 1 and 2 mainly through good security and regular
backups.  I strongly recommend using a filesystem or setup with cheap
snapshots - then you can take them regularly.

There is a myth that hard disk sectors can get corrupted and return
incorrect data without an error indication - the so-called "undetected
read error".  In theory, this /can/ occur - but unless you are using the
data quantities of CERN or Google, you will never see it in your
lifetime (that's why I call it a myth).  You /might/ come across
firmware errors in disks that give such undetected read errors, though
it is highly unlikely.  "Unrecoverable read errors", on the other hand,
are not as uncommon as one would like - sometimes a disk is simply
unable to return the data you ask for, but it can at least inform you of
the failure.


The general method to protect against failures is redundancy combined
with systems to spot errors.  For storage, that is almost invariably
done using RAID.

So your basic setup here is to have two drives with a RAID-1 mirror.
Anything that is written to the mirror gets written to both drives.  If
a sector read fails (despite the drive's extensive error checking and
correcting codes), the RAID controller (hardware or software) will read
the second copy from the other disk.  It will then re-write this correct
data to the first disk, which the drive will map to a spare sector.  If
a drive fails entirely, the controller will read from the other drive.

"Bit rot", or slow bit corruption on disk sectors, is prevented by
scrubbing.  You regularly read through the entire array, for both disks.
 If a sector has had corruption on some bits, the drives ECC coding will
usually let it correct the error automatically, and it will re-write the
sector (with re-mapping if necessary) to refresh the data.  If the
corruption is too great, the RAID controller will re-write the sector
based on the second copy.

Thus RAID keeps your data reliable, at least until a catastrophic
failure (such as the whole server dying) - that's why you have backups.


For more features, I would recommend using BTRFS or ZFS to handle your
RAID mirror.  These filesystems give you additional checksumming (for
protection against drive firmware errors, or extremely rare undetected
read errors), as well as very cheap snapshots and other goodies.

On the other hand, if you want the simplest "will work on anything, even
old Linux systems or Windows systems" solution, then use Linux md RAID
with metadata format 1.0.  This puts the RAID information at the end of
the disk - all the rest of the disk appears /exactly/ as a normal
non-raid disk, and can be read from anything else.


>> Disclaimer: I am not an expert on the data storage, but here are my
>> thoughts:
>>
>> - Use a RAID system which has enough redundancy to detect and correct
>> defects in the media. Depending of the RAID level used, the capability to
>> data recovery varies.
> 
> I started out using RAID5 "hardware" arrays (and other, more sophisticated
> configurations).  This requires you to have multiple spindles running
> concurrently (else the array can't *detect* discrepancies).  Compact
> arrays also tend to suffer from cooling problems.

You will need two spindles (or SSDs) concurrently - that's the only way
to get two copies of your data on two disks.

Don't bother with RAID-5.  Either use RAID-1 if your data fits
comfortably on a single disk, or use RAID-6.  Modern disks have less
cooling requirements than old ones, if you go for RAID-6.

> 
> It also ties you to an often proprietary (or, at least, poorly documented)
> implementation ("volume format").  So, a failure of the hardware can leave
> you with possibly intact volume(s) and no straightforward means of
> accessing their contents.

Use software RAID - either Linux, or BSD if you prefer.  Avoid
proprietary hardware solutions.

> 
> Moving volumes between arrays is seldom supported (unless you have
> identical
> arrays -- hardware/software -- AND a well considered means of doing this.
> This effectively precludes having volumes off-line.  Introducing a
> "foreign"
> volume to an array will often result in the array prompting you to
> "initialize" it (wipe it clean!).
> 
>> - Duplicate or mirror the data across different physical media.
> 
> This is RAID1.  Many of the above issues still apply.  In any case,
> the mirrors must be spinning concurrently for the array to function.
> (It has no way of knowing what the "other" copy of the "file"
> looks like unless it can *see* it, now, in its entirety).

If you are really desperate to avoid having two disks spinning at the
same time, it is possible with Linux md raid to designate one drive as
"write mostly".  It will then only be used when writing (obviously),
when scrubbing (obviously), or when reading if the first drive fails to
return the correct data.  I cannot think of any real benefit of this,
but it is possible.

> 
> This begs two crucial questions:

No, it /raises/ two questions.

> - are BOTH copies of the file examined when you try to access it?

No.  That would be highly inefficient, and does not contribute to
improved reliability.

>   Or, is a "primary" copy accessed and the "secondary" copy only accessed
>   if the primary throws a read error?

Yes.

> - If a discrepancy is found between the TWO instances, how is the
>   "winner" selected?

Discrepancies don't occur with non-negligible probability unless you
have had a massive hardware failure, or power-failure during writing.
See <http://neil.brown.name/blog/20100211050355> for a discussion on the
topic.

> 
> Plus, the more subtle question seldom considered:
> - are files that are not "manually" accessed EVER verified?  Or, do you
>   have to wait until you need/want a file to discover if it is intact??

That's what "scrubbing" is for.  Hardware raid controllers should do
this automatically, with software raid it is usually done with a cronjob.

>   (Do *you* periodically check the contents of the batteries in your
>   flashlight to verify they are functioning BEFORE you need them?
>   Does the *flashlight* take on the responsibility of performing this
>   test on your behalf??)

A torch is not a raid device.

> 
> Consider the above when you have many terabytes of data at stake.

RAID like this is /exactly/ what people use when they have TB of data at
stake.

When they have more than that, they use redundant servers, or split the
two halves of their software RAID-1 pairs across different boxes.

> Do you want ALL of it "spinning" whenever you need to access *any*
> of it?  

Why is that a problem?  Most hard disks have no problem spinning all the
time.

> How long will it take for a proactive array to examine/verify
> ALL of the data just to reassure you?  (will you have the ENTIRE
> array "up" for that length of time?  Or, will you power it up just
> long enough to access the file(s) of interest, then power it back down,
> again?  Or, will you power up ONLY the portion of the array known to
> contain the file(s) you seek?)

Again, what is the problem?

If this is an array you rarely access, you can spin down all the disks
when it is not in use.

> 
> As I said, I've been down this route.  Ever hear what twenty to
> fifty 10-15K drives sound like spinning at the same time?
> <http://techreport.com/blog/13849/behold-thumper-sun-sunfire-x4500-storage-server>.
>  Even a 12-14 drive "shelf" is loud:
> <http://www.amtecgroup.co.uk/products/external-disk-storag-1071/sun-storedge-a1000-12-bay-hdd-rack-array-raid-ctrl-2x-psu-1502.aspx>.
>  Remember,
> they're all throwing off heat in a relatively dense package so you
> have lots of (redundant!) fans in place to keep them from melting...
> 

Why are you considering a decade old high-end datacenter storage system?
 These are designed for installations where noise doesn't matter, and
you have massive cooling systems.

Just get a simple little server box with space for a few disks.  I
really like the HP Microservers - Gen 8 devices are dirt cheap,
reliable, and have four neat 3.5" drive bays behind a door in the front.
 They are perfect for file servers unless you need really high-end
performance.

> The "problem" with these solutions is they are designed for high
> performance.
> Serving files continuously, typically, to an "organization".  And, with a
> high degree of reliability/availability.
> 

So don't get one.

> For an *archive*, you tend to need lots of capacity but comparably little
> bandwidth!  (how *often* will you access its contents?  how quickly will
> you need the file "delivered" to you?)  And, you can trade reliability and
> availability for bandwidth and cost.  Do you keep your 2013 tax returns
> in your top left desk drawer (where they are easily accessible)?  Or, have
> you moth-balled them to a more capacious site that's, perhaps, a bit more
> difficult to access??

What capacity are you looking for here?

Back to comp.arch.embedded | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-27 15:23 -0700
  Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-28 09:48 +1100
    Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-27 16:48 -0700
      Re: Checking large numbers of files Dimiter_Popoff <dp@tgi-sci.com> - 2016-10-28 03:17 +0300
        Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-27 18:50 -0700
          Re: Checking large numbers of files Dimiter_Popoff <dp@tgi-sci.com> - 2016-10-28 05:38 +0300
            Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-27 21:33 -0700
      Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-28 11:40 +1100
        Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-27 18:52 -0700
  Re: Checking large numbers of files kalvin.news@gmail.com - 2016-10-28 03:50 -0700
    Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-28 10:49 -0700
      Re: Checking large numbers of files kalvin.news@gmail.com - 2016-10-29 05:17 -0700
        Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 13:15 -0700
          Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 09:47 +1100
            Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 16:19 -0700
              Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 16:36 -0700
              Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 12:00 +1100
                Re: Checking large numbers of files Dimiter_Popoff <dp@tgi-sci.com> - 2016-10-30 04:05 +0200
                Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 14:06 +1100
                Re: Checking large numbers of files Dimiter_Popoff <dp@tgi-sci.com> - 2016-10-30 05:48 +0200
                Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 22:33 -0700
                Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 17:00 +1100
                Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-10-31 04:43 -0700
                Re: Checking large numbers of files (long) David Brown <david.brown@hesbynett.no> - 2016-10-31 14:20 +0100
                Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-10-31 08:02 -0700
                Re: Checking large numbers of files (long) David Brown <david.brown@hesbynett.no> - 2016-11-01 09:47 +0100
                Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-11-01 04:36 -0700
                Re: Checking large numbers of files (long) David Brown <david.brown@hesbynett.no> - 2016-11-01 14:29 +0100
                Re: Checking large numbers of files (long) David Brown <david.brown@hesbynett.no> - 2016-11-01 15:16 +0100
                Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-11-02 11:42 -0700
                Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 22:15 -0700
                Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 21:50 -0700
                Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 17:14 +1100
                Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-10-31 03:36 -0700
                Re: Checking large numbers of files (long) Clifford Heath <no.spam@please.net> - 2016-11-01 09:20 +1100
                Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-11-01 02:23 -0700
                Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 21:20 -0700
                Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 17:25 +1100
          Re: Checking large numbers of files Robert Wessel <robertwessel2@yahoo.com> - 2016-10-30 00:14 -0500
            Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 23:02 -0700
          Re: Checking large numbers of files David Brown <david.brown@hesbynett.no> - 2016-10-31 12:19 +0100
            Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-31 07:00 -0700
  Re: Checking large numbers of files Ali <abdulrazaq@gmail.com> - 2016-10-30 15:45 -0700
    Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-31 03:45 -0700

csiph-web