Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.arch.embedded > #25247 > unrolled thread

Checking large numbers of files

Back to article view | Back to comp.arch.embedded

Contents

  Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-27 15:23 -0700
    Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-28 09:48 +1100
      Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-27 16:48 -0700
        Re: Checking large numbers of files Dimiter_Popoff <dp@tgi-sci.com> - 2016-10-28 03:17 +0300
          Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-27 18:50 -0700
            Re: Checking large numbers of files Dimiter_Popoff <dp@tgi-sci.com> - 2016-10-28 05:38 +0300
              Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-27 21:33 -0700
        Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-28 11:40 +1100
          Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-27 18:52 -0700
    Re: Checking large numbers of files kalvin.news@gmail.com - 2016-10-28 03:50 -0700
      Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-28 10:49 -0700
        Re: Checking large numbers of files kalvin.news@gmail.com - 2016-10-29 05:17 -0700
          Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 13:15 -0700
            Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 09:47 +1100
              Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 16:19 -0700
                Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 16:36 -0700
                Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 12:00 +1100
                  Re: Checking large numbers of files Dimiter_Popoff <dp@tgi-sci.com> - 2016-10-30 04:05 +0200
                    Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 14:06 +1100
                      Re: Checking large numbers of files Dimiter_Popoff <dp@tgi-sci.com> - 2016-10-30 05:48 +0200
                        Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 22:33 -0700
                          Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 17:00 +1100
                            Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-10-31 04:43 -0700
                              Re: Checking large numbers of files (long) David Brown <david.brown@hesbynett.no> - 2016-10-31 14:20 +0100
                                Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-10-31 08:02 -0700
                                  Re: Checking large numbers of files (long) David Brown <david.brown@hesbynett.no> - 2016-11-01 09:47 +0100
                                    Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-11-01 04:36 -0700
                                      Re: Checking large numbers of files (long) David Brown <david.brown@hesbynett.no> - 2016-11-01 14:29 +0100
                                      Re: Checking large numbers of files (long) David Brown <david.brown@hesbynett.no> - 2016-11-01 15:16 +0100
                                        Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-11-02 11:42 -0700
                      Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 22:15 -0700
                    Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 21:50 -0700
                      Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 17:14 +1100
                        Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-10-31 03:36 -0700
                          Re: Checking large numbers of files (long) Clifford Heath <no.spam@please.net> - 2016-11-01 09:20 +1100
                            Re: Checking large numbers of files (long) Don Y <blockedofcourse@foo.invalid> - 2016-11-01 02:23 -0700
                  Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 21:20 -0700
                    Re: Checking large numbers of files Clifford Heath <no.spam@please.net> - 2016-10-30 17:25 +1100
            Re: Checking large numbers of files Robert Wessel <robertwessel2@yahoo.com> - 2016-10-30 00:14 -0500
              Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-29 23:02 -0700
            Re: Checking large numbers of files David Brown <david.brown@hesbynett.no> - 2016-10-31 12:19 +0100
              Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-31 07:00 -0700
    Re: Checking large numbers of files Ali <abdulrazaq@gmail.com> - 2016-10-30 15:45 -0700
      Re: Checking large numbers of files Don Y <blockedofcourse@foo.invalid> - 2016-10-31 03:45 -0700

Page 1 of 3  [1] 2 3  Next page →

#25247 — Checking large numbers of files

I currently "verify" my data archive with an algorithm similar to:

ONCE: con 0;
Success: con 0;
Removed, Inaccessible, NoRecord, BadSize, BadHash: con Success+1+iota;

verify(files: list of string ): list of (string, reason)
{
    failures := list of (string, reason);

    while (files != nil) {
       theFile := hd files;
       (size, signature, result) := examine_file(theFile);
       (Size, Signature, Result) := query_DBMS(theFile);
       do {
          if (NotFound == result)
             {Result = Removed;      break;}
          if (BadRead == result)
             {Result = Inaccessible; break;}

          // theFile exists!
          if (NotFound == Result)
             {Result = NoRecord;     break;}
          if (Size != size)
             {Result = BadSize;      break;}
          if (Signature != signature)
             {Result = BadHash;      break;}
       } while (ONCE);

       if (Success != Result) {
          failures = (theFile, Result) :: failures;
       }

       files = tl files;
    }
    return( failures );
}

[apologies for any typos; too early in the day to be working!]

The problem this has is examine-file() is expensive -- for large
numbers of large files!

I'd like to short-circuit this to provide an early indication of
possible problems.  E.g., stat(2) each "theFile" to identify any
that are "Removed" without incurring the cost of examining the
file in its entirety.

[Of course, all files MUST be eventually examined as you can't
verify accessibility (BadRead), actual size (BadSize) or correct
signature (BadHash) without looking at every byte!  But, you can
get the simple checks out of the way expeditiously to alert you
to "obvious" problems!]

I can get an approximate indication of "actual size" just by
stat(2)-ing the file, as well -- relying on examine_file() to
later confirm that figure.

The hash, of course, can't be guesstimated without trodding through
the file in its entirety.

The problem with *this* approach is I'd have to lock the entire
object to ensure the integrity of the "early results" agrees with
that of the "later results"  (the original approach only requires
taking a lock on the file being examined *while* it is being
examined).

Are there any "tricks" I can use, here?  E.g., the equivalent of
CoW would ensure the integrity of my data (for the *original*
instance) in the presence of such a global lock.  But, I think only
the Bullet Server would give me any behavior approaching this...?

Also, how wary would one be relying on the files' timestamps as a
(crude) indication of "alteration"?  (It seems far too easy to
alter a timestamp without altering content/size so I've been leary
of relying on that as having any "worth").

[toc] | [next] | [standalone]

#25249

On 28/10/16 09:23, Don Y wrote:
> I currently "verify" my data archive with an algorithm similar to:
...
> Are there any "tricks" I can use, here?

Git is very fast at verifying files. You might find that
"git annex" is just right for maintaining your archives.
It's built for just this, and more.

[toc] | [prev] | [next] | [standalone]

#25250

On 10/27/2016 3:48 PM, Clifford Heath wrote:
> On 28/10/16 09:23, Don Y wrote:
>> I currently "verify" my data archive with an algorithm similar to:
> ...
>> Are there any "tricks" I can use, here?
>
> Git is very fast at verifying files. You might find that
> "git annex" is just right for maintaining your archives.
> It's built for just this, and more.

There's more to it than just verifying a file hasn't changed
(I can do that by comparing its current hash -- pick your
favorite algorithm -- to a stored hash).  There are innumerable
tools for doing that, synchronizing different copies on
different volumes (even if a volume is offline), etc.

The point of my post was to avoid computing hashes of terabytes
of files when I can discover obvious changes by looking at file
sizes, dates, names, etc.

OTOH, as certain metadata can be too easily corrupted (e.g., dates),
incorporating them in a strategy can result in too many false
positives: "The timestamp on 'foo' has changed.  I know you
'verified' the contents of the entire file YESTERDAY, but the
changed timestamp suggests you might want to reverify it, today!"

[I also want the "process" to automatically pause and resume as
I asynchronously add and remove volumes -- possibly idling
for weeks or months before having access to a volume, again]

[toc] | [prev] | [next] | [standalone]

#25251

On 28.10.2016 г. 02:48, Don Y wrote:
> On 10/27/2016 3:48 PM, Clifford Heath wrote:
>> On 28/10/16 09:23, Don Y wrote:
>>> I currently "verify" my data archive with an algorithm similar to:
>> ...
>>> Are there any "tricks" I can use, here?
>>
>> Git is very fast at verifying files. You might find that
>> "git annex" is just right for maintaining your archives.
>> It's built for just this, and more.
>
> There's more to it than just verifying a file hasn't changed
> (I can do that by comparing its current hash -- pick your
> favorite algorithm -- to a stored hash).  There are innumerable
> tools for doing that, synchronizing different copies on
> different volumes (even if a volume is offline), etc.
>
> The point of my post was to avoid computing hashes of terabytes
> of files when I can discover obvious changes by looking at file
> sizes, dates, names, etc.
>
> OTOH, as certain metadata can be too easily corrupted (e.g., dates),
> incorporating them in a strategy can result in too many false
> positives: "The timestamp on 'foo' has changed.  I know you
> 'verified' the contents of the entire file YESTERDAY, but the
> changed timestamp suggests you might want to reverify it, today!"

Hi Don,
why can't you trust the timestamp? Obviously I can speak about dps
only where a timestamp in a directory entry will only get modified
if the file has been modified itself but I see no reason this would
differ on other systems so I am curious.
Apart from the obvious timestamp and file size you could perhaps
also check the file allocation data for changes/errors - and, as you
obviously know, from there on you will have to read the entire
file, what else can you do.
[under DPS I run a "repair" utility which goes through all directories
to rebuild the unit CAT (cluster allocation table), takes under
a minute for a 200G partition. I don't remember a file having been
corrupted somehow without some other accompanying mess on the disk
which repair would not catch. BTW it took me maybe 20 years to
catch the cause of the occasional space leak... (was negligible but
repair would catch it and fix it, IIRC the cause was some rounding
error the deallocate function was making)].

Dimiter

------------------------------------------------------
Dimiter Popoff, TGI             http://www.tgi-sci.com
------------------------------------------------------
http://www.flickr.com/photos/didi_tgi/



>
> [I also want the "process" to automatically pause and resume as
> I asynchronously add and remove volumes -- possibly idling
> for weeks or months before having access to a volume, again]

[toc] | [prev] | [next] | [standalone]

#25253

Hi Dimiter,

[Still 35C!  Cripes!!]

On 10/27/2016 5:17 PM, Dimiter_Popoff wrote:
>> OTOH, as certain metadata can be too easily corrupted (e.g., dates),
>> incorporating them in a strategy can result in too many false
>> positives: "The timestamp on 'foo' has changed.  I know you
>> 'verified' the contents of the entire file YESTERDAY, but the
>> changed timestamp suggests you might want to reverify it, today!"
>
> why can't you trust the timestamp? Obviously I can speak about dps
> only where a timestamp in a directory entry will only get modified
> if the file has been modified itself but I see no reason this would
> differ on other systems so I am curious.

First, timestamps on "containers" (e.g., things like "folders")
don't mean anything.  If /foo/A and /foo/B are "being tracked"
but other contents of /foo are not, then the timestamp of /foo
can change even though NEITHER A nor B have been altered!

Also, I access/mount the volumes from a variety of different hosts
and "maintain" the files with a variety of different utilities.
The clocks on all of these hosts aren't guaranteed to be synchronized.
So, you can't infer chronology from the timestamps.

And, the mechanisms that I use to move files around may not
preserve original timestamps, accurately.

E.g., I may move files onto a volume over FTP -- using a client
that doesn't propagate the timestamp of the *original* file.
So, that instance of the file now has a different timestamp
from the original FROM WHICH IT WAS COPIED.  Yet, I *know*
they are identical, in contents.

> Apart from the obvious timestamp and file size you could perhaps
> also check the file allocation data for changes/errors - and, as you
> obviously know, from there on you will have to read the entire
> file, what else can you do.

Yes.  I'm looking to rearrange the tests in my current algorithm
to give me earlier indications of problems.  I.e., I would much
rather wait a few seconds to discover that a file NAME is not
present where it should be (at some random point in the file
system hierarchy) than to have to wait an hour to discover it
AFTER the previous 600,000 files have had their *contents*
verified!

[Imagine a scenario where a volume is failing; you'd much rather know
about a file that is NOT present on a "backup copy" than be reassured
that the contents of all the OTHER files are intact; you KNOW you
will lose data due to the missing file but MIGHT not lose anything
from those other files if you've not yet wasted time checking them!]

> [under DPS I run a "repair" utility which goes through all directories
> to rebuild the unit CAT (cluster allocation table), takes under
> a minute for a 200G partition. I don't remember a file having been
> corrupted somehow without some other accompanying mess on the disk
> which repair would not catch. BTW it took me maybe 20 years to
> catch the cause of the occasional space leak... (was negligible but
> repair would catch it and fix it, IIRC the cause was some rounding
> error the deallocate function was making)].

The volumes I'm addressing reside on a variety of different machines.

In the past, I would periodically (i.e., subject to the reliability
of my meatware memory) power up each RAID5 array and copy its contents
to /dev/null (i.e., force the array to read everything and SILENTLY
repair any errors it encountered).  When I moved to RAID1, I'd repeat
the exercise comparing the contents of one volume to its mirror.  Etc.

After that, I blurred the boundaries between volumes, file systems,
etc.  I relied on my own discipline to know that *this* portion
of *this* file hierarchy is mirrored in some particular other place.
So, I would MANUALLY do the comparisons (rsync, etc.) "whenever I
remembered to do so".

Now, I am automating that so I don't have to remember what is where;
let the RDBMS keep track of all that stuff.  Let *it* know if a
file has changed by using its data to verify each file's contents.

So, I can have:
     /VolumeA/.../expenses/2015/taxes/federal
     /VolumeB/.../2015/expenses/taxes/federal
     /VolumeC/.../2016/personal/federal/taxes
ALL be different instances of the same "object" (same size
and hashes but different names and "containers").

If a routine examination of /VolumeC/.../2016/personal/federal/taxes
notes that its hash/size has changed, then I can be alerted.
I can be *advised* that this MIGHT be the same object as the
other two mentioned above.  OTOH, it may not (depends on the
strength of the hash *and* my intent for that file -- i.e., I
may just be using it as a template to create my *2016* tax
records and the reason it is NOW different is because I have
explicitly changed it!)

As a result, I have to intervene and indicate IF and HOW a
file's previous contents should be "recovered".

This scheme lets me just bring hosts on-line and mount volumes
as I see fit -- without having to take deliberate action to
"inform something" that I want the contents of these volumes checked
(if they were checked 2 hours ago, why would I want to waste
effort CASUALLY checking them *again*??)

So, when I power up a workstation and it exports some particular
SMB/NFS share, *if* that share has content that is important to
this "archive system", then the archive system will surreptitiously
sneak in and check those contents, IF NECESSARY.  And, if I happen
to power down that host before it has a chance to finish checking,
it will remember how much progress it made and where it should resume
along with WHEN such a check would be "overdue".  In that case, it
will email me a notice telling me which volumes I need to mount
(not *where* they need to be mounted) so that it can verify the
associated contents.

[When I was using tape for backups, it was a chore to remember which
media needed to be recycled, which needed to be retensioned, which
files needed to be backed-up, etc.  This handles all of that for
me without forcing me to take deliberate steps (beyond powering up
a suitable host and placing the necessary media in a compatible
drive/transport)]

It also makes it easier for me to decide which media I can discard
and/or erase.  E.g., I discarded two spindles of recorded optical media
last week and KNOW that I have sufficient copies of the files on all of
those media to not be at risk by that action:
     for file in list_of_files
         number = count_instances(file)
         if (number < N)
             echo "$file is precious!" > log
WITHOUT having to dig through stacks of media wondering what's on each one!

[I have two PC's that I'll be scrapping this week.  I'll let this
system catalog their disks.  Then, have *it* tell me if there is anything
that I need to rescue before scrapping them]

[toc] | [prev] | [next] | [standalone]

#25255

On 28.10.2016 г. 04:50, Don Y wrote:
> Hi Dimiter,
>
> [Still 35C!  Cripes!!]

Hi Don,
we switched from walking around in shorts to winter uniform and
house heating almost a month ago...

> First, timestamps on "containers" (e.g., things like "folders")
> don't mean anything.  If /foo/A and /foo/B are "being tracked"
> but other contents of /foo are not, then the timestamp of /foo
> can change even though NEITHER A nor B have been altered!

Well yes, the last modification date of a directory may be
impractical to be kept up to date. I made this optional (by changing
the directory file type) some 20+ years ago...

But the file "last modification date" is pretty stable, can't see
it change for no good reason. I often do copy with the option to
overwrite only files which are older than the source to be copied,
has not bitten me so far.

>
> Also, I access/mount the volumes from a variety of different hosts
> and "maintain" the files with a variety of different utilities.
> The clocks on all of these hosts aren't guaranteed to be synchronized.
> So, you can't infer chronology from the timestamps.

Hmmm, when copying a file its timestamp should normally be copied
as well so local clocks should not matter re file contents.

> And, the mechanisms that I use to move files around may not
> preserve original timestamps, accurately.
>
> E.g., I may move files onto a volume over FTP -- using a client
> that doesn't propagate the timestamp of the *original* file.

This is an issue as old as ftp is, yes... If you use GET with
ftp the file timestamp can come correctly across (MLST). Still
no standard way to propagate the correct timestamp when you "STOR".

> So, that instance of the file now has a different timestamp
> from the original FROM WHICH IT WAS COPIED.  Yet, I *know*
> they are identical, in contents.

Well yes, using ftp "put" does that indeed - but I can't think
of another cause so it is soft of avoidable.

>
> [Imagine a scenario where a volume is failing; you'd much rather know
> about a file that is NOT present on a "backup copy" than be reassured
> that the contents of all the OTHER files are intact; you KNOW you
> will lose data due to the missing file but MIGHT not lose anything
> from those other files if you've not yet wasted time checking them!]

Well searching the whole tree for a particular name would be painful
but still faster than copying it all. I personally tend to know which
file belongs where so locating it is easy - but this is not much help
if you look into an archive you are not familiar with.

Dimiter

[toc] | [prev] | [next] | [standalone]

#25256

>> [Still 35C!  Cripes!!]
>
> we switched from walking around in shorts to winter uniform and
> house heating almost a month ago...

We're still running the air conditioning.  Unusual to be using it
this long.  I think the *average* high temp for October, this year,
was 33C.  I know we've had 20+ days above 90F (and usually WELL
above) and it's only the 27th...

>> First, timestamps on "containers" (e.g., things like "folders")
>> don't mean anything.  If /foo/A and /foo/B are "being tracked"
>> but other contents of /foo are not, then the timestamp of /foo
>> can change even though NEITHER A nor B have been altered!
>
> Well yes, the last modification date of a directory may be
> impractical to be kept up to date. I made this optional (by changing
> the directory file type) some 20+ years ago...

And, even if ALL of the directory's contents are "being tracked",
you can do:
    mv * ../X
    mv ../X/* .
to effectively leave teh directory "as it was" -- yet the timestamp
is modified.

> But the file "last modification date" is pretty stable, can't see
> it change for no good reason. I often do copy with the option to
> overwrite only files which are older than the source to be copied,
> has not bitten me so far.

Pull the disk out of your computer and put it in another computer
(which is what you effectively do when you access it over the wire
*or* physically move an external medium).

Or, "touch *" -- none of the files' contents/names have been altered
but all of their timestamps have.  Which timestamp is now correct:
the timestamp of the touched file(s) or the "original" timestamps
(that have been lost)?

Remember, any application that accesses the files can also dick with
the timestamps.  *You* can vouch for all of YOUR applications (having
written them all).  But, *I* am not keen on trying to quantify that
behavior for those, here.

>> Also, I access/mount the volumes from a variety of different hosts
>> and "maintain" the files with a variety of different utilities.
>> The clocks on all of these hosts aren't guaranteed to be synchronized.
>> So, you can't infer chronology from the timestamps.
>
> Hmmm, when copying a file its timestamp should normally be copied
> as well so local clocks should not matter re file contents.

Yes, if you are copying it on a *local* machine.  But, what happens
to the timestamp of an object that you fetch from a web server? :>

>> And, the mechanisms that I use to move files around may not
>> preserve original timestamps, accurately.
>>
>> E.g., I may move files onto a volume over FTP -- using a client
>> that doesn't propagate the timestamp of the *original* file.
>
> This is an issue as old as ftp is, yes... If you use GET with
> ftp the file timestamp can come correctly across (MLST). Still
> no standard way to propagate the correct timestamp when you "STOR".

So, do I "certify" every web/ftp/scp/fxp/etc client?  Or,
manually verify the timestamps are "approximately" correct
(i.e., 201610272110 is PROBABLY wrong for a local file)?

One of the graphical file synchronization tools I use highlights
differences with color.  It's not uncommon for me to see *mirrored*
filestores claiming to have multiple "discrepancies".  On closer
examination, it's almost always a timestamp error.

FAT12 forces timestamps to a 2 second granularity (my PROM
programmer still uses floppies as does at least one of my logic
analyzers and a DSO) -- something to remember when comparing
files on floppies to the original "archive images"!

>> So, that instance of the file now has a different timestamp
>> from the original FROM WHICH IT WAS COPIED.  Yet, I *know*
>> they are identical, in contents.
>
> Well yes, using ftp "put" does that indeed - but I can't think
> of another cause so it is soft of avoidable.
>
>> [Imagine a scenario where a volume is failing; you'd much rather know
>> about a file that is NOT present on a "backup copy" than be reassured
>> that the contents of all the OTHER files are intact; you KNOW you
>> will lose data due to the missing file but MIGHT not lose anything
>> from those other files if you've not yet wasted time checking them!]
>
> Well searching the whole tree for a particular name would be painful
> but still faster than copying it all. I personally tend to know which
> file belongs where so locating it is easy - but this is not much help
> if you look into an archive you are not familiar with.

I have one half of the dataset readily available in the DBMS:
I know which files should exist at each level of the hierarchy,
their sizes, names, containers, and signatures -- BEFORE the drive
is even on-line.

So, I can just walk through the file hierarchy on the medium
and build an unordered list of pathnames/filenames.  Then, sort
this and compare it to the same sorted list from the DBMS.
Superficial differences (names, dates, sizes) will stand out
(the DBMS query can find them all for me faster than I could
find them myself as it can exploit indexes its precomputed).

The real costs come in:
- verifying the *actual* size of the file (i.e., a disk error may
   effectively truncate it or otherwise rendered parts of it inaccessible)
- verifying the actual signature (hash) of the file

So, I should "do the easy checks" first (for ALL contents) instead
of processing files COMPLETELY, but one-at-a-time -- *if* I want to
expeditiously locate potential problems.  (I just think timestamps
would result in me chasing problems that weren't actually problems)

Check your mail for more detail (look at Subject lines as I've
been changing email accounts, lately)

[toc] | [prev] | [next] | [standalone]

#25252

On 28/10/16 10:48, Don Y wrote:
> On 10/27/2016 3:48 PM, Clifford Heath wrote:
>> On 28/10/16 09:23, Don Y wrote:
>>> I currently "verify" my data archive with an algorithm similar to:
>> ...
>>> Are there any "tricks" I can use, here?
>>
>> Git is very fast at verifying files. You might find that
>> "git annex" is just right for maintaining your archives.
>> It's built for just this, and more.
>
> There's more to it than just verifying a file hasn't changed
> (I can do that by comparing its current hash -- pick your
> favorite algorithm -- to a stored hash).  There are innumerable
> tools for doing that, synchronizing different copies on
> different volumes (even if a volume is offline), etc.
>
> The point of my post was to avoid computing hashes of terabytes
> of files when I can discover obvious changes by looking at file
> sizes, dates, names, etc.
>
> OTOH, as certain metadata can be too easily corrupted (e.g., dates),
> incorporating them in a strategy can result in too many false
> positives: "The timestamp on 'foo' has changed.  I know you
> 'verified' the contents of the entire file YESTERDAY, but the
> changed timestamp suggests you might want to reverify it, today!"
>
> [I also want the "process" to automatically pause and resume as
> I asynchronously add and remove volumes -- possibly idling
> for weeks or months before having access to a volume, again]

I assume that you thought it was quicker to respond like this than
it was to, for example, go and actually read about "git annex"?
Sorry, I responded to try to help, not enter a prolonged discussion.
I know there are many ways to address this problem. It seems you'd
rather talk about it than actually do that though. Not me.

[toc] | [prev] | [next] | [standalone]

#25254

On 10/27/2016 5:40 PM, Clifford Heath wrote:
> On 28/10/16 10:48, Don Y wrote:
>> On 10/27/2016 3:48 PM, Clifford Heath wrote:
>>> On 28/10/16 09:23, Don Y wrote:
>>>> I currently "verify" my data archive with an algorithm similar to:
>>> ...
>>>> Are there any "tricks" I can use, here?
>>>
>>> Git is very fast at verifying files. You might find that
>>> "git annex" is just right for maintaining your archives.
>>> It's built for just this, and more.
>>
>> There's more to it than just verifying a file hasn't changed
>> (I can do that by comparing its current hash -- pick your
>> favorite algorithm -- to a stored hash).  There are innumerable
>> tools for doing that, synchronizing different copies on
>> different volumes (even if a volume is offline), etc.
>>
>> The point of my post was to avoid computing hashes of terabytes
>> of files when I can discover obvious changes by looking at file
>> sizes, dates, names, etc.
>>
>> OTOH, as certain metadata can be too easily corrupted (e.g., dates),
>> incorporating them in a strategy can result in too many false
>> positives: "The timestamp on 'foo' has changed.  I know you
>> 'verified' the contents of the entire file YESTERDAY, but the
>> changed timestamp suggests you might want to reverify it, today!"
>>
>> [I also want the "process" to automatically pause and resume as
>> I asynchronously add and remove volumes -- possibly idling
>> for weeks or months before having access to a volume, again]
>
> I assume that you thought it was quicker to respond like this than
> it was to, for example, go and actually read about "git annex"?

Yeah, the web page is still OPEN in my browser.  So, you know
what ASSuming does...

> Sorry, I responded to try to help, not enter a prolonged discussion.
> I know there are many ways to address this problem. It seems you'd
> rather talk about it than actually do that though. Not me.

You seem to ASSume other people don't do their homework before posting
questions.

I've NOT gone into a description of the innumerable ways git annex FAILS
to address my problem.  Just as your answer failed to address the question
I posed!

[toc] | [prev] | [next] | [standalone]

#25258

For a very crude check you can check that the file size hasn't changed in addition to file time stamp. Then, you could possibly compute a fast hash for each file and use it as the first line of verification instead of using more expensive hash algorithms. If you just need to check if the file has been altered a simple 64-bit CRC might be usable. You should of course evaluate the implications of using hash algorithms that can produce collisions and that are not considered safe and robust. Some file systems provide hooks that can monitor file alterations. Creating an observer which will monitor any file changes, you will be notified if any file should be altered.

Br,
Kalvin

[toc] | [prev] | [next] | [standalone]

#25263

On 10/28/2016 3:50 AM, kalvin.news@gmail.com wrote:
> For a very crude check you can check that the file size hasn't changed in
> addition to file time stamp.

"File size" just tells me the *stored* metadata hasn't "been altered"
for the file.  There is still the possibility that the actual size
*has*, effectively, changed (think: access errors, sparse file support,
etc.).  Knowing the object's size, with certainty, still requires its
more thorough examination.

I am leary of using timestamps as they are too easily altered.  E.g.,
I've seen build scripts that deliberately, unconditionally, touch(1)
some "marker file" just to effectively record their actions -- despite
the contents of the file being unaltered.  (e.g., consider a file
named "MostRecentBuildTime" -- the timestamp of which actually
contains the information the file's presence is intended to convey).

As I am looking for information that can help me INTELLIGENTLY decide
where more careful scrutiny should be INITIALLY focused, I'd ideally
like things that can't be as easily altered *without* indicating real
potential changes to the object's content.  E.g., altering the file's
size can't be accomplished without actually altering the file's content!

[However, if I am only using the timestamp to tell if *this* file has
changed from *its* previous state (i.e., not using it to tell if this
file is DIFFERENT/changed from the original from which it was copied),
it potentially offers *some* information.]

> Then, you could possibly compute a fast hash
> for each file and use it as the first line of verification instead of using
> more expensive hash algorithms. If you just need to check if the file has
> been altered a simple 64-bit CRC might be usable.

I've benchmarked a variety of different hashes to get a feel for how
expensive they are.  There is a HUGE range of performance -- like
2 orders of magnitude (neglecting hardware accelerators)!

But, the real pissers are the access and transport delays for the raw file
data; getting it off the media and *to* the "process" that is actually
examining it!  In many cases, the volume is an external drive (which can
mean the interface is the bottleneck), a "consumer" drive (intended to be
inexpensive, but not very "performant"), uses an old interface (e.g.,
USB2) or is inherently slow (e.g., optical media).

So, looking at the file contents is the *last* thing I'd want to do
to get a feel for the file's integrity -- it is the FINAL assessment!
Ideally, I want indicators of "bad data" that are far less expensive
(e.g., if a file SHOULD be 16KB and it stat(2)'s as 30GB, I sure as
hell don't want to waste time hashing 30 billion bytes before deciding
that the file (or its metadata) has been corrupted!

> You should of course
> evaluate the implications of using hash algorithms that can produce
> collisions and that are not considered safe and robust.

Yes.  But, the hash isn't trying to protect against an active adversary
attempting to deliberately manipulate the files contents in a selective
manner to preserve the original hash.  Instead, I just have to evaluate
likely media failure modes that could potentially alter the data in ways
that would NOT be detected by the hash.

> Some file systems
> provide hooks that can monitor file alterations. Creating an observer which
> will monitor any file changes, you will be notified if any file should be
> altered.

This requires the filesystem/media to be visible to such an observer at all
times.  That would force all the media into a live repository, of sorts
(regardless of whether it is centralized or distributed).

And, it doesn't address the potential for bitrot.

I'm looking for something where the media can be offline and/or potentially
manipulated "unsupervised" (and, while not *maliciously*, perhaps prone
to missteps).  I also have to tolerate R/O media as well as a prohibition
against adding "breadcrumbs" to the monitored media (i.e., no "control files"
littering the monitored media).

[E.g., to take mass-produced optical media and verify *it* is intact
as well as R/W media for which the client does not have write/create
privileges -- or that may have write-protect tabs]

[toc] | [prev] | [next] | [standalone]

#25276

perjantai 28. lokakuuta 2016 20.50.03 UTC+3 Don Y kirjoitti:
> > Some file systems
> > provide hooks that can monitor file alterations. Creating an observer which
> > will monitor any file changes, you will be notified if any file should be
> > altered.
> 
> This requires the filesystem/media to be visible to such an observer at all
> times.  That would force all the media into a live repository, of sorts
> (regardless of whether it is centralized or distributed).
> 
> And, it doesn't address the potential for bitrot.
> 
> I'm looking for something where the media can be offline and/or potentially
> manipulated "unsupervised" (and, while not *maliciously*, perhaps prone
> to missteps).  I also have to tolerate R/O media as well as a prohibition
> against adding "breadcrumbs" to the monitored media (i.e., no "control files"
> littering the monitored media).
> 
> [E.g., to take mass-produced optical media and verify *it* is intact
> as well as R/W media for which the client does not have write/create
> privileges -- or that may have write-protect tabs]

Disclaimer: I am not an expert on the data storage, but here are my thoughts:

- Use a RAID system which has enough redundancy to detect and correct defects in the media. Depending of the RAID level used, the capability to data recovery varies.

- Duplicate or mirror the data across different physical media.

- Use a file system which provides integrity check and error correction. The ZFS is one example of such file system https://en.wikipedia.org/wiki/ZFS The ZFS provides also a snapshot feature.

[toc] | [prev] | [next] | [standalone]

#25277

On 10/29/2016 5:17 AM, kalvin.news@gmail.com wrote:
> Disclaimer: I am not an expert on the data storage, but here are my
> thoughts:
>
> - Use a RAID system which has enough redundancy to detect and correct
> defects in the media. Depending of the RAID level used, the capability to
> data recovery varies.

I started out using RAID5 "hardware" arrays (and other, more sophisticated
configurations).  This requires you to have multiple spindles running
concurrently (else the array can't *detect* discrepancies).  Compact
arrays also tend to suffer from cooling problems.

It also ties you to an often proprietary (or, at least, poorly documented)
implementation ("volume format").  So, a failure of the hardware can leave
you with possibly intact volume(s) and no straightforward means of
accessing their contents.

Moving volumes between arrays is seldom supported (unless you have identical
arrays -- hardware/software -- AND a well considered means of doing this.
This effectively precludes having volumes off-line.  Introducing a "foreign"
volume to an array will often result in the array prompting you to
"initialize" it (wipe it clean!).

> - Duplicate or mirror the data across different physical media.

This is RAID1.  Many of the above issues still apply.  In any case,
the mirrors must be spinning concurrently for the array to function.
(It has no way of knowing what the "other" copy of the "file"
looks like unless it can *see* it, now, in its entirety).

This begs two crucial questions:
- are BOTH copies of the file examined when you try to access it?
   Or, is a "primary" copy accessed and the "secondary" copy only accessed
   if the primary throws a read error?
- If a discrepancy is found between the TWO instances, how is the
   "winner" selected?

Plus, the more subtle question seldom considered:
- are files that are not "manually" accessed EVER verified?  Or, do you
   have to wait until you need/want a file to discover if it is intact??
   (Do *you* periodically check the contents of the batteries in your
   flashlight to verify they are functioning BEFORE you need them?
   Does the *flashlight* take on the responsibility of performing this
   test on your behalf??)

Consider the above when you have many terabytes of data at stake.
Do you want ALL of it "spinning" whenever you need to access *any*
of it?  How long will it take for a proactive array to examine/verify
ALL of the data just to reassure you?  (will you have the ENTIRE
array "up" for that length of time?  Or, will you power it up just
long enough to access the file(s) of interest, then power it back down,
again?  Or, will you power up ONLY the portion of the array known to
contain the file(s) you seek?)

As I said, I've been down this route.  Ever hear what twenty to
fifty 10-15K drives sound like spinning at the same time?
<http://techreport.com/blog/13849/behold-thumper-sun-sunfire-x4500-storage-server>. 
  Even a 12-14 drive "shelf" is loud:
<http://www.amtecgroup.co.uk/products/external-disk-storag-1071/sun-storedge-a1000-12-bay-hdd-rack-array-raid-ctrl-2x-psu-1502.aspx>. 
  Remember,
they're all throwing off heat in a relatively dense package so you
have lots of (redundant!) fans in place to keep them from melting...

The "problem" with these solutions is they are designed for high performance.
Serving files continuously, typically, to an "organization".  And, with a
high degree of reliability/availability.

For an *archive*, you tend to need lots of capacity but comparably little
bandwidth!  (how *often* will you access its contents?  how quickly will
you need the file "delivered" to you?)  And, you can trade reliability and
availability for bandwidth and cost.  Do you keep your 2013 tax returns
in your top left desk drawer (where they are easily accessible)?  Or, have
you moth-balled them to a more capacious site that's, perhaps, a bit more
difficult to access??

[E.g., the six spindles in my P4 repository are each considerably more
expensive, reliable and performant than any *one* "consumer" spindle
in my "archive".  OTOH, I can replace one of the consumer drives
*today* -- with <whatever> the sweet spot in that market happens to
be at this moment; replacing one of the 15K SAS drives is going to require
me to have it "on hand" if I want to replace it, today!]

> - Use a file system which provides integrity check and error correction. The
> ZFS is one example of such file system https://en.wikipedia.org/wiki/ZFS The
> ZFS provides also a snapshot feature.

ZFS brings other baggage in much the same way as RAID.  The volumes in a
particular pool tend to need to remain in that pool (without explicit
maintenance).  They are inaccessible OUTSIDE that context.

By contrast, extending the archive more along the lines of JBOD allows
each volume to be treated and accessed individually and independantly.
You can move a drive from one JBOD (which, in the degenerate case,
could just be a drive bay in a PC!) to another without having to
do anything magical in the volume manager.  (e.g., plugging or unplugging
an external USB/Firewire drive!)

But, "something" has to know how to keep track of "what is where", *now*.
A physical location (bay 3 on shelf 2) doesn't mean anything.  Even the
associated logical location (/dev/c3t1d0s2) is ambiguous as any physical
volume may happen to occupy it, currently.

[I.e., you need to be able to uniquely identify *volumes* and map them
to physical and logical locations, AS THEY APPEAR]

My "distributed" approach lets me use whatever filesystem(s) are
appropriate to the needs as *I* perceive them.  E.g., I can have a
RAID50 "volume" whose contents I track, verify and scrub alongside a
single FAT32 volume.  Or, a CD9660!  Or (several) ZFS pools!  For
example, I've been systematically moving files from NTFS media
(cuz it was easy to DIRECTLY acccess from Wintel boxes) to FFS
volumes (which are better suited to my diskless NAS appliances).
The rest of my "system" doesn't care about the change.

The volumes may be sitting idle, in active use or offline in a storage
closet (try powering down a single drive in a RAID array/ZFS pool!)

It decouples the implementation from the performance AND reliability
aspects.  E.g., one volume may reside on a 15K drive while a "backup"
of that volume may reside on an optical medium stored in a Tyvek envelope
in a desk drawer!  There may be a dozen instances of one particular file
on one or more physical volumes/media types; and a *single* instance of
some other file in my entire collection!  (i.e., I can tell if that
single instance gets corrupted -- but can do nothing to recreate it
if that occurs).

This approach is ideal for my use.  And, easily extensible to handle
the addition of other media (e.g., I can add the contents of an SMB
share to the dataset so that those portions of its contents that are
of interest to me are automatically accessible and verifiable whenever
the system "notices" that SMB share being "on-line" -- even if it,
itself, is an external/relocatable device that was last seen on
some *other* mountpoint/share!)

The problem is quickly finding *potential* problems -- without
exposing the underbelly of each filesystem, raw device, media
type, etc. (e.g., I don't care if the files reside on a WORM
drive, hard disk, ORB, etc. -- just the names of the objects
and their "containers")

[I could have included support for tape volumes but have been
aggressively discarding tape archives -- copy them to disk
and archive them there.  A pair of 4-5TB drives takes up far
less space (and $$) than a stack of LTO or DLT tapes! (well,
an LTO-7 would win on size -- but just barely]

[toc] | [prev] | [next] | [standalone]

#25279

On 30/10/16 07:15, Don Y wrote:
> On 10/29/2016 5:17 AM, kalvin.news@gmail.com wrote:
>> Disclaimer: I am not an expert on the data storage, but here are my
>> thoughts:
>> - Use a RAID system which has enough redundancy to detect and correct
> I started out using RAID5 "hardware" arrays (and other, more sophisticated
> configurations).  This requires you to have multiple spindles running
> concurrently (else the array can't *detect* discrepancies).  Compact
> arrays also tend to suffer from cooling problems.
>
> It also ties you to an often proprietary (or, at least, poorly documented)
> implementation ("volume format").  So, a failure of the hardware can leave
> you with possibly intact volume(s) and no straightforward means of
> accessing their contents.

No. Almost all commercial RAID systems (Synology, Qnap, etc)
use Linux RAID support. If the hardware fails, you can mount
the drives on any Linux system that has enough SATA ports.

> Moving volumes between arrays is seldom supported (unless you have
> identical arrays -- hardware/software

No, not true, and for the same reason. E.g. if you have a
Synology DS1815+ with two four-drive RAID5 volumes in it, you
can pull the four drives of a volume (without even powering off)
and plug them into a 4-bay Qnap system. You need to configure
the volume so it gets mounted, but because it's just a Linux
RAID volume, it just works.

<snip lots more un-educated speculation>

Clifford Heath.

[toc] | [prev] | [next] | [standalone]

#25280

On 10/29/2016 3:47 PM, Clifford Heath wrote:
> On 30/10/16 07:15, Don Y wrote:
>> On 10/29/2016 5:17 AM, kalvin.news@gmail.com wrote:
>>> Disclaimer: I am not an expert on the data storage, but here are my
>>> thoughts:
>>> - Use a RAID system which has enough redundancy to detect and correct
>> I started out using RAID5 "hardware" arrays (and other, more sophisticated
>> configurations).  This requires you to have multiple spindles running
>> concurrently (else the array can't *detect* discrepancies).  Compact
>> arrays also tend to suffer from cooling problems.
>>
>> It also ties you to an often proprietary (or, at least, poorly documented)
>> implementation ("volume format").  So, a failure of the hardware can leave
>> you with possibly intact volume(s) and no straightforward means of
>> accessing their contents.
>
> No. Almost all commercial RAID systems (Synology, Qnap, etc)
> use Linux RAID support. If the hardware fails, you can mount
> the drives on any Linux system that has enough SATA ports.

Really?  So my Sun RAID systems run Linux??

>> Moving volumes between arrays is seldom supported (unless you have
>> identical arrays -- hardware/software
>
> No, not true, and for the same reason. E.g. if you have a
> Synology DS1815+ with two four-drive RAID5 volumes in it, you
> can pull the four drives of a volume (without even powering off)
> and plug them into a 4-bay Qnap system. You need to configure
> the volume so it gets mounted, but because it's just a Linux
> RAID volume, it just works.

Yeah, and when I take that drive out of my Sun RAID array and
put it in that Synology box it's going to work just fine!

> <snip lots more un-educated speculation>

Seems like *you're* the one speaking from limited experience!

But, then again, you've probably never owned any "big iron".
When was the last time you had more than 4 (or 12) spindles
running on the same box?  24?  48?

[toc] | [prev] | [next] | [standalone]

#25281

On 10/29/2016 4:19 PM, Don Y wrote:
> On 10/29/2016 3:47 PM, Clifford Heath wrote:

>> <snip lots more un-educated speculation>
>
> Seems like *you're* the one speaking from limited experience!
>
> But, then again, you've probably never owned any "big iron".
> When was the last time you had more than 4 (or 12) spindles
> running on the same box?  24?  48?

Ah, forgive me.  Obviously, you expected me to NOT use any of these
technologies until they were available in FOSS forms (for folks with
deep pockets and short arms!).

Instead, I should have kept all of this data on scraps of paper
in a series of large shoeboxes -- until I could use some "free"
medium/technology to store it.

[Hint:  I had a pair of D1000's arrays stuffed with 4G drives in
the late 90's -- ~96GB.  Then, upgraded the drives to 9G, 18G
and 36G before, eventually, abandoning the "large number of hot
spindles approach".  Where was your Synology box, then?
    "Synology Inc. was founded in January 2000, ..."
No problem!  We'll just look at Qnap:
    "Quality Network Appliance Provider was founded in 2004..."
Hmmm, I guess I'd still have to keep all those little slips of
paper lying around in shoeboxes waiting for more affordable means
of archiving the data!]

[I wonder when Linux support for "software RAID" was viable??]

(sigh)  Amusing how SHALLOW the experiences of many folk!
And, how *deep* and authoritative they THINK them to be!

Makes you wonder what sort of "education" they're hiding behind
(that they think others of us are LACKING!)

Do your homework before spouting off your pseudo-authoritative
drivel.

[toc] | [prev] | [next] | [standalone]

#25282

On 30/10/16 10:19, Don Y wrote:
> On 10/29/2016 3:47 PM, Clifford Heath wrote:
>> On 30/10/16 07:15, Don Y wrote:
>>> On 10/29/2016 5:17 AM, kalvin.news@gmail.com wrote:
>>>> Disclaimer: I am not an expert on the data storage, but here are my
>>>> thoughts:
>>>> - Use a RAID system which has enough redundancy to detect and correct
>>> I started out using RAID5 "hardware" arrays (and other, more
>>> sophisticated
>>> configurations).  This requires you to have multiple spindles running
>>> concurrently (else the array can't *detect* discrepancies).  Compact
>>> arrays also tend to suffer from cooling problems.
>>>
>>> It also ties you to an often proprietary (or, at least, poorly
>>> documented)
>>> implementation ("volume format").  So, a failure of the hardware can
>>> leave
>>> you with possibly intact volume(s) and no straightforward means of
>>> accessing their contents.
>>
>> No. Almost all commercial RAID systems (Synology, Qnap, etc)
>> use Linux RAID support. If the hardware fails, you can mount
>> the drives on any Linux system that has enough SATA ports.
>
> Really?  So my Sun RAID systems run Linux??

Ahh, my apologies. I didn't realize you were running a computer antiques 
museum. I thought you were trying to ensure the availability of some data.

[toc] | [prev] | [next] | [standalone]

#25283

On 30.10.2016 г. 03:00, Clifford Heath wrote:
> On 30/10/16 10:19, Don Y wrote:
>> On 10/29/2016 3:47 PM, Clifford Heath wrote:
>>> On 30/10/16 07:15, Don Y wrote:
>>>> On 10/29/2016 5:17 AM, kalvin.news@gmail.com wrote:
>>>>> Disclaimer: I am not an expert on the data storage, but here are my
>>>>> thoughts:
>>>>> - Use a RAID system which has enough redundancy to detect and correct
>>>> I started out using RAID5 "hardware" arrays (and other, more
>>>> sophisticated
>>>> configurations).  This requires you to have multiple spindles running
>>>> concurrently (else the array can't *detect* discrepancies).  Compact
>>>> arrays also tend to suffer from cooling problems.
>>>>
>>>> It also ties you to an often proprietary (or, at least, poorly
>>>> documented)
>>>> implementation ("volume format").  So, a failure of the hardware can
>>>> leave
>>>> you with possibly intact volume(s) and no straightforward means of
>>>> accessing their contents.
>>>
>>> No. Almost all commercial RAID systems (Synology, Qnap, etc)
>>> use Linux RAID support. If the hardware fails, you can mount
>>> the drives on any Linux system that has enough SATA ports.
>>
>> Really?  So my Sun RAID systems run Linux??
>
> Ahh, my apologies. I didn't realize you were running a computer antiques
> museum. I thought you were trying to ensure the availability of some data.

So what is he supposed to do with data which are useless outside of
a particular environment. Apparently he has plenty of past projects
spanning a few decades back to maintain - how do you suggest he will
separate movable from non-movable data. Then the sheer amount of data
he is talking is just huge.
Linux/foss may be a huge part of life but it is restricted to the needs
of the general public which feeds it directly. There are other aspects
of tech life which are simply beyond the scope of foss/linux, beyond
its capabilities as they are.
I strongly suspect Don is not an office secretary running an office 
archive and I also strongly suspect he knows what he is doing.
Well I also suspect he knows he will find not what he is looking
for - an easier way to navigate his huge archives - but there is nothing
wrong wanting to talk or just moan about that, this is what groups are
for really.

Dimiter

------------------------------------------------------
Dimiter Popoff, TGI             http://www.tgi-sci.com
------------------------------------------------------
http://www.flickr.com/photos/didi_tgi/

[toc] | [prev] | [next] | [standalone]

#25284

On 30/10/16 13:05, Dimiter_Popoff wrote:
> On 30.10.2016 г. 03:00, Clifford Heath wrote:
>> On 30/10/16 10:19, Don Y wrote:
>>> On 10/29/2016 3:47 PM, Clifford Heath wrote:
>>>> On 30/10/16 07:15, Don Y wrote:
>>>>> On 10/29/2016 5:17 AM, kalvin.news@gmail.com wrote:
>>>>>> Disclaimer: I am not an expert on the data storage, but here are my
>>>>>> thoughts:
>>>>>> - Use a RAID system which has enough redundancy to detect and correct
>>>>> I started out using RAID5 "hardware" arrays (and other, more
>>>>> sophisticated
>>>>> configurations).  This requires you to have multiple spindles running
>>>>> concurrently (else the array can't *detect* discrepancies).  Compact
>>>>> arrays also tend to suffer from cooling problems.
>>>>>
>>>>> It also ties you to an often proprietary (or, at least, poorly
>>>>> documented)
>>>>> implementation ("volume format").  So, a failure of the hardware can
>>>>> leave
>>>>> you with possibly intact volume(s) and no straightforward means of
>>>>> accessing their contents.
>>>>
>>>> No. Almost all commercial RAID systems (Synology, Qnap, etc)
>>>> use Linux RAID support. If the hardware fails, you can mount
>>>> the drives on any Linux system that has enough SATA ports.
>>>
>>> Really?  So my Sun RAID systems run Linux??
>>
>> Ahh, my apologies. I didn't realize you were running a computer antiques
>> museum. I thought you were trying to ensure the availability of some
>> data.
>
> So what is he supposed to do with data which are useless outside of
> a particular environment. Apparently he has plenty of past projects
> spanning a few decades back to maintain - how do you suggest he will
> separate movable from non-movable data. Then the sheer amount of data
> he is talking is just huge.

No, his amount of data is trivial compared to the petabytes
being accumulated daily and managed by internet-scale companies,
who almost all use *commodity* hardware running *open source*
software.

> I strongly suspect Don is not an office secretary running an office
> archive and I also strongly suspect he knows what he is doing.

Don is a "special flower". He faces problems that no-one else
has ever faced, and he analyses them with an insight that no-one
else has ever had, requiring solutions that no-one else has ever
found.

None of these descriptions are true, of course.

For what it's worth, data management is a global profession with
a hundred major specialties and a thousand vendors, who all have
experts with more experience than Don has in his toenails. I've
just returned from a paid global speaking trip where I addressed
some of them. "Availability" isn't my specialty, I just rely on
commodity equipment to manage my measly few dozen terabytes. But
what do I know, I don't work with special problems like Don's.

[toc] | [prev] | [next] | [standalone]

#25285

On 30.10.2016 г. 05:06, Clifford Heath wrote:
> On 30/10/16 13:05, Dimiter_Popoff wrote:
>> On 30.10.2016 г. 03:00, Clifford Heath wrote:
>>> On 30/10/16 10:19, Don Y wrote:
>>>> On 10/29/2016 3:47 PM, Clifford Heath wrote:
>>>>> On 30/10/16 07:15, Don Y wrote:
>>>>>> On 10/29/2016 5:17 AM, kalvin.news@gmail.com wrote:
>>>>>>> Disclaimer: I am not an expert on the data storage, but here are my
>>>>>>> thoughts:
>>>>>>> - Use a RAID system which has enough redundancy to detect and
>>>>>>> correct
>>>>>> I started out using RAID5 "hardware" arrays (and other, more
>>>>>> sophisticated
>>>>>> configurations).  This requires you to have multiple spindles running
>>>>>> concurrently (else the array can't *detect* discrepancies).  Compact
>>>>>> arrays also tend to suffer from cooling problems.
>>>>>>
>>>>>> It also ties you to an often proprietary (or, at least, poorly
>>>>>> documented)
>>>>>> implementation ("volume format").  So, a failure of the hardware can
>>>>>> leave
>>>>>> you with possibly intact volume(s) and no straightforward means of
>>>>>> accessing their contents.
>>>>>
>>>>> No. Almost all commercial RAID systems (Synology, Qnap, etc)
>>>>> use Linux RAID support. If the hardware fails, you can mount
>>>>> the drives on any Linux system that has enough SATA ports.
>>>>
>>>> Really?  So my Sun RAID systems run Linux??
>>>
>>> Ahh, my apologies. I didn't realize you were running a computer antiques
>>> museum. I thought you were trying to ensure the availability of some
>>> data.
>>
>> So what is he supposed to do with data which are useless outside of
>> a particular environment. Apparently he has plenty of past projects
>> spanning a few decades back to maintain - how do you suggest he will
>> separate movable from non-movable data. Then the sheer amount of data
>> he is talking is just huge.
>
> No, his amount of data is trivial compared to the petabytes
> being accumulated daily and managed by internet-scale companies,
> who almost all use *commodity* hardware running *open source*
> software.
>
>> I strongly suspect Don is not an office secretary running an office
>> archive and I also strongly suspect he knows what he is doing.
>
> Don is a "special flower". He faces problems that no-one else
> has ever faced, and he analyses them with an insight that no-one
> else has ever had, requiring solutions that no-one else has ever
> found.
>
> None of these descriptions are true, of course.
>
> For what it's worth, data management is a global profession with
> a hundred major specialties and a thousand vendors, who all have
> experts with more experience than Don has in his toenails. I've
> just returned from a paid global speaking trip where I addressed
> some of them. "Availability" isn't my specialty, I just rely on
> commodity equipment to manage my measly few dozen terabytes. But
> what do I know, I don't work with special problems like Don's.


Well,

 >> Linux/foss may be a huge part of life but it is restricted to the
 >> needs of the general public which feeds it directly. There are
 >> other aspects of tech life which are simply beyond the scope of
 >> foss/linux, beyond its capabilities as they are.

But I know many of the foss/linux crowd think otherwise - which is
OK, everyone sees the walls of their own room. No need to be grumpy
about other people wanting to talk about other things though.

Dimiter

[toc] | [prev] | [next] | [standalone]

Page 1 of 3  [1] 2 3  Next page →

Back to top | Article view | comp.arch.embedded

csiph-web