Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.os.linux.misc > #13267 > unrolled thread

More on corrupted disk data..

Back to article view | Back to comp.os.linux.misc

Contents

  More on corrupted disk data.. The Natural Philosopher <tnp@invalid.invalid> - 2015-01-05 14:25 +0000
    Re: More on corrupted disk data.. Tim Watts <tw_usenet@dionic.net> - 2015-01-05 14:29 +0000
    Re: More on corrupted disk data.. Richard Kettlewell <rjk@greenend.org.uk> - 2015-01-05 15:01 +0000
    Re: More on corrupted disk data.. Robert Heller <heller@deepsoft.com> - 2015-01-05 09:35 -0600
      Re: More on corrupted disk data.. The Natural Philosopher <tnp@invalid.invalid> - 2015-01-05 15:46 +0000
        Re: More on corrupted disk data.. "John F" <j@j.ii> - 2015-01-08 13:25 -0800
          Re: More on corrupted disk data.. Bobbie Sellers <bliss-sf4ever@dslextreme.com> - 2015-01-08 14:09 -0800
            Re: More on corrupted disk data.. The Natural Philosopher <tnp@invalid.invalid> - 2015-01-08 23:35 +0000
            Re: More on corrupted disk data.. Keith Keller <kkeller-usenet@wombat.san-francisco.ca.us> - 2015-01-08 16:05 -0800
              Re: More on corrupted disk data.. Bobbie Sellers <bliss-sf4ever@dslextreme.com> - 2015-01-08 20:01 -0800
                Re: More on corrupted disk data.. Keith Keller <kkeller-usenet@wombat.san-francisco.ca.us> - 2015-01-08 20:51 -0800
                  Re: More on corrupted disk data.. Bobbie Sellers <bliss-sf4ever@dslextreme.com> - 2015-01-08 21:38 -0800
                    Re: More on corrupted disk data.. Keith Keller <kkeller-usenet@wombat.san-francisco.ca.us> - 2015-01-08 21:44 -0800
                      Re: More on corrupted disk data.. The Natural Philosopher <tnp@invalid.invalid> - 2015-01-09 09:18 +0000

#13267 — More on corrupted disk data..

The question of recovering damaged data have raised a question in my mind:

What organisation of data has the most redundancy (apart from RAID) so 
the best chance of recovering significant chunks of it? it would seem 
that a zip or other compressed single backup file has to be the worst..??


-- 
Everything you read in newspapers is absolutely true, except for the 
rare story of which you happen to have first-hand knowledge. – Erwin Knoll

[toc] | [next] | [standalone]

#13269

On 05/01/15 14:25, The Natural Philosopher wrote:
>
> The question of recovering damaged data have raised a question in my mind:
>
> What organisation of data has the most redundancy (apart from RAID) so
> the best chance of recovering significant chunks of it? it would seem
> that a zip or other compressed single backup file has to be the worst..??
>
>

Depends if the compression algorithm is block based. If so, then the 
loss of one block would take out 2, maybe 3 block of real data and the 
rest would be recoverable.

A compression scheme that could not resync beyond a broken block would 
be a very poor choice.

[toc] | [prev] | [next] | [standalone]

#13270

The Natural Philosopher <tnp@invalid.invalid> writes:
> The question of recovering damaged data have raised a question in my
> mind:
>
> What organisation of data has the most redundancy (apart from RAID) so
> the best chance of recovering significant chunks of it? it would seem
> that a zip or other compressed single backup file has to be the
> worst..??

There’s a whole field of error correction codes, which storage devices
already use.  AFAIK they are generally designed for dealing with
(boundedly many) single-bit/symbol errors, rather than whole blocks
being lost, though.


There’s a few things that can be said about compression:

.zip compresses each member separately; provided the index and all the
compressed bytes of a given member are undamaged, that member should be
recoverable.

.tar.gz, in contrast, compresses everything without reference to member
boundaries, so corruption that only directly affects one file could put
subsequent files beyond recovery too.

Deflate format (used by .gz and .zip) has an internal blocking
structure; each block has its own independent Huffman compression, while
the LZ77 compression layer can reach ‘across’ the blocks, within limits.
So damage to one part of the output will have some effect on recovery of
subsequent bytes, although it might be possible to re-synchronize the
decompressor after the damaged region if you can figure out where the
block boundaries are.

The situation may be better or worse with other compression algorithms.


Going the other way (anticompression?), if you represent all your text
files in UTF-16 or UTF-32 then (assuming you mostly use the Roman
alphabet) they will be 2 or 4 times as big as they would otherwise, so a
failed disk block will represent a smaller proportion of your data.

More seriously if you keep files in a version control system then that
will automatically mean you have extra copies providing a degree of
resilience not only against failed disk blocks but also accidental
deletion.  This isn’t a substitute for proper backups of course but
often represents an easier recovery process.

-- 
http://www.greenend.org.uk/rjk/

[toc] | [prev] | [next] | [standalone]

#13273

At Mon, 05 Jan 2015 14:25:13 +0000 The Natural Philosopher <tnp@invalid.invalid> wrote:

> 
> 
> The question of recovering damaged data have raised a question in my mind:
> 
> What organisation of data has the most redundancy (apart from RAID) so 
> the best chance of recovering significant chunks of it? it would seem 
> that a zip or other compressed single backup file has to be the worst..??

There really isn't an 'organisation of data' that has any more redundancy (or
error recovery) than RAID *OR* proper backups. Any data organisation short of
RAID or proper backups will result in data loss in the event of a hard disk
failure. In the case of proper backups, the more copies the better. The AMANDA
(Advanced Maryland Automatic Network Disk Archiver) system will make redundant
backups as a matter of course.  The best redundancy is redundancy itself. Eg 
rather than worry about whether a .zip file is better than .tar.gz file, make 
two copies of each, on different media.  Or maybe 4 copies of each on 8 
separate media -- at *least* one copy should be recoverable.

Getting into the 'habit' of making backup copies of everything is always a
good idea. People might think people who are anal/OCD about backups are crazy,
but such people never have to deal with trying to recover data from a
failing/failed disk drive. If the drive dies, one tosses it in the trash and
installs a new one and then restores the backup.

> 
> 

-- 
Robert Heller             -- 978-544-6933
Deepwoods Software        -- Custom Software Services
http://www.deepsoft.com/  -- Linux Administration Services
heller@deepsoft.com       -- Webhosting Services
                                                                                                                            

[toc] | [prev] | [next] | [standalone]

#13274

On 05/01/15 15:35, Robert Heller wrote:
> At Mon, 05 Jan 2015 14:25:13 +0000 The Natural Philosopher <tnp@invalid.invalid> wrote:
>
>>
>>
>> The question of recovering damaged data have raised a question in my mind:
>>
>> What organisation of data has the most redundancy (apart from RAID) so
>> the best chance of recovering significant chunks of it? it would seem
>> that a zip or other compressed single backup file has to be the worst..??
>
> There really isn't an 'organisation of data' that has any more redundancy (or
> error recovery) than RAID *OR* proper backups. Any data organisation short of
> RAID or proper backups will result in data loss in the event of a hard disk
> failure. In the case of proper backups, the more copies the better. The AMANDA
> (Advanced Maryland Automatic Network Disk Archiver) system will make redundant
> backups as a matter of course.  The best redundancy is redundancy itself. Eg
> rather than worry about whether a .zip file is better than .tar.gz file, make
> two copies of each, on different media.  Or maybe 4 copies of each on 8
> separate media -- at *least* one copy should be recoverable.
>
> Getting into the 'habit' of making backup copies of everything is always a
> good idea. People might think people who are anal/OCD about backups are crazy,
> but such people never have to deal with trying to recover data from a
> failing/failed disk drive. If the drive dies, one tosses it in the trash and
> installs a new one and then restores the backup.
>
>
I have to say that is ultimately my approach, since 'data I cant 
replace' is << size of disk drive I can buy for peanuts.


Once you start recording videos, all of which are trivially replacable 
for a few £, its seems insane not to have your *personal* photos and 
scribblings in at least 3 independent places..



>>
>


-- 
Everything you read in newspapers is absolutely true, except for the 
rare story of which you happen to have first-hand knowledge. – Erwin Knoll

[toc] | [prev] | [next] | [standalone]

#13355

"The Natural Philosopher" <tnp@invalid.invalid> wrote in message 
news:m8ebkg$pe5$1@news.albasani.net...
> On 05/01/15 15:35, Robert Heller wrote:
>> At Mon, 05 Jan 2015 14:25:13 +0000 The Natural Philosopher 
>> <tnp@invalid.invalid> wrote:
>>
>>>
>>>
>>> The question of recovering damaged data have raised a question in my 
>>> mind:
>>>
>>> What organisation of data has the most redundancy (apart from RAID) so
>>> the best chance of recovering significant chunks of it? it would seem
>>> that a zip or other compressed single backup file has to be the 
>>> worst..??
>>
>> There really isn't an 'organisation of data' that has any more redundancy 
>> (or
>> error recovery) than RAID *OR* proper backups. Any data organisation 
>> short of
>> RAID or proper backups will result in data loss in the event of a hard 
>> disk
>> failure. In the case of proper backups, the more copies the better. The 
>> AMANDA
>> (Advanced Maryland Automatic Network Disk Archiver) system will make 
>> redundant
>> backups as a matter of course.  The best redundancy is redundancy itself. 
>> Eg
>> rather than worry about whether a .zip file is better than .tar.gz file, 
>> make
>> two copies of each, on different media.  Or maybe 4 copies of each on 8
>> separate media -- at *least* one copy should be recoverable.
>>
>> Getting into the 'habit' of making backup copies of everything is always 
>> a
>> good idea. People might think people who are anal/OCD about backups are 
>> crazy,
>> but such people never have to deal with trying to recover data from a
>> failing/failed disk drive. If the drive dies, one tosses it in the trash 
>> and
>> installs a new one and then restores the backup.
>>
>>
> I have to say that is ultimately my approach, since 'data I cant replace' 
> is << size of disk drive I can buy for peanuts.
>
>
> Once you start recording videos, all of which are trivially replacable for 
> a few Ł, its seems insane not to have your *personal* photos and 
> scribblings in at least 3 independent places..
>
>
>

amen to that - I have all my user data backed up on 2 separate external 
drives + cloud storage of incidental sharable disposables + DVD backups off 
all my photos, and I often wonder if I am sure that is enough!  because you 
never know if, for instance, there is a lightning strike on the power line 
outside your house and all the drives and computers get toasted, at least 
the pictures should be alright (but then the library of DVD's is getting 
larger :) - you cant win when it comes to space! 

[toc] | [prev] | [next] | [standalone]

#13359

On 01/08/2015 01:25 PM, John F wrote:
> "The Natural Philosopher" <tnp@invalid.invalid> wrote in message
> news:m8ebkg$pe5$1@news.albasani.net...
>> On 05/01/15 15:35, Robert Heller wrote:
>>> At Mon, 05 Jan 2015 14:25:13 +0000 The Natural Philosopher
>>> <tnp@invalid.invalid> wrote:
>>>
>>>>
>>>>
>>>> The question of recovering damaged data have raised a question in my
>>>> mind:
>>>>
>>>> What organisation of data has the most redundancy (apart from RAID) so
>>>> the best chance of recovering significant chunks of it? it would seem
>>>> that a zip or other compressed single backup file has to be the
>>>> worst..??
>>>
>>> There really isn't an 'organisation of data' that has any more redundancy
>>> (or
>>> error recovery) than RAID *OR* proper backups. Any data organisation
>>> short of
>>> RAID or proper backups will result in data loss in the event of a hard
>>> disk
>>> failure. In the case of proper backups, the more copies the better. The
>>> AMANDA
>>> (Advanced Maryland Automatic Network Disk Archiver) system will make
>>> redundant
>>> backups as a matter of course.  The best redundancy is redundancy itself.
>>> Eg
>>> rather than worry about whether a .zip file is better than .tar.gz file,
>>> make
>>> two copies of each, on different media.  Or maybe 4 copies of each on 8
>>> separate media -- at *least* one copy should be recoverable.
>>>
>>> Getting into the 'habit' of making backup copies of everything is always
>>> a
>>> good idea. People might think people who are anal/OCD about backups are
>>> crazy,
>>> but such people never have to deal with trying to recover data from a
>>> failing/failed disk drive. If the drive dies, one tosses it in the trash
>>> and
>>> installs a new one and then restores the backup.
>>>
>>>
>> I have to say that is ultimately my approach, since 'data I cant replace'
>> is << size of disk drive I can buy for peanuts.
>>
>>
>> Once you start recording videos, all of which are trivially replacable for
>> a few £, its seems insane not to have your *personal* photos and
>> scribblings in at least 3 independent places..
>>
>>
>>
>
> amen to that - I have all my user data backed up on 2 separate external
> drives + cloud storage of incidental sharable disposables + DVD backups off
> all my photos, and I often wonder if I am sure that is enough!  because you
> never know if, for instance, there is a lightning strike on the power line
> outside your house and all the drives and computers get toasted, at least
> the pictures should be alright (but then the library of DVD's is getting
> larger :) - you cant win when it comes to space!
>
>

	You want to get the best power strips you cn afford which will have a 
warranty to pay for the equipment destruction in case of problems while 
using the strips.

	My own setup withstood the surge that came from the collapse of
a DC power line used to operate Street Cars in San Francisco.  Took out
the neighbors TV and wiped out the AC lines that hook up to my building. 
  They were pulling cable all afternoon to replace the AC lines.  But 
all my local surge protectors kept right on thru the
mess.   So that would be one more problem off your mind.

	bliss

[toc] | [prev] | [next] | [standalone]

#13364

On 08/01/15 22:09, Bobbie Sellers wrote:
>      You want to get the best power strips you cn afford which will have
> a warranty to pay for the equipment destruction in case of problems
> while using the strips.
>
>      My own setup withstood the surge that came from the collapse of
> a DC power line used to operate Street Cars in San Francisco.  Took out
> the neighbors TV and wiped out the AC lines that hook up to my building.
>   They were pulling cable all afternoon to replace the AC lines.  But
> all my local surge protectors kept right on thru the
> mess.   So that would be one more problem off your mind.

I got a direct strike on the telephone line...glad I was out. Total 
rewire, and the modoem, serial parallel card and printer all died. So 
did the TV and a chip in my very expensive turntable. A socket blew out 
of the wall, and there were burns in the carpet where wires running over 
it crossed wires under it.

The comuter and disk were fine..



-- 
Everything you read in newspapers is absolutely true, except for the 
rare story of which you happen to have first-hand knowledge. – Erwin Knoll

[toc] | [prev] | [next] | [standalone]

#13367

On 2015-01-08, Bobbie Sellers <bliss-sf4ever@dslextreme.com> wrote:
>
> 	You want to get the best power strips you cn afford which will have a 
> warranty to pay for the equipment destruction in case of problems while 
> using the strips.

Why wouldn't you simply get a quality UPS with a similar warranty (which
won't cover data corruption anyway)?

--keith



-- 
kkeller-usenet@wombat.san-francisco.ca.us
(try just my userid to email me)
AOLSFAQ=http://www.therockgarden.ca/aolsfaq.txt
see X- headers for PGP signature information

[toc] | [prev] | [next] | [standalone]

#13373

On 01/08/2015 04:05 PM, Keith Keller wrote:
> On 2015-01-08, Bobbie Sellers <bliss-sf4ever@dslextreme.com> wrote:
>>
>> 	You want to get the best power strips you cn afford which will have a
>> warranty to pay for the equipment destruction in case of problems while
>> using the strips.
>
> Why wouldn't you simply get a quality UPS with a similar warranty (which
> won't cover data corruption anyway)?
>
> --keith

	In my particular case when I had the strength to move the UPS around I 
did not have enough money but since then I have lost the majority of my 
strength and can barely move a UPS these days.

	I think tower and desk top computers should contain a battery
to make safe shutdowns if the mains power fail.  Because I believe in
such precautions I run a laptop as my main device.

	bliss

[toc] | [prev] | [next] | [standalone]

#13375

On 2015-01-09, Bobbie Sellers <bliss-sf4ever@dslextreme.com> wrote:
>
> 	In my particular case when I had the strength to move the UPS around I 
> did not have enough money but since then I have lost the majority of my 
> strength and can barely move a UPS these days.
>
> 	I think tower and desk top computers should contain a battery
> to make safe shutdowns if the mains power fail.  Because I believe in
> such precautions I run a laptop as my main device.

If a tower had a battery it'd likely be too heavy for you (if it's not
too heavy already).

I thought about running my server off a laptop, but decided against it
because I also wanted a RAID array, and that wouldn't be feasible with a
laptop--either I'd need external drives that could run off of USB power
or I'd need an external chassis on a UPS.  At that point may as well
have a real chassis that could support an internal RAID controller.

--keith

-- 
kkeller-usenet@wombat.san-francisco.ca.us
(try just my userid to email me)
AOLSFAQ=http://www.therockgarden.ca/aolsfaq.txt
see X- headers for PGP signature information

[toc] | [prev] | [next] | [standalone]

#13376

On 01/08/2015 08:51 PM, Keith Keller wrote:
> On 2015-01-09, Bobbie Sellers <bliss-sf4ever@dslextreme.com> wrote:
>>
>> 	In my particular case when I had the strength to move the UPS around I
>> did not have enough money but since then I have lost the majority of my
>> strength and can barely move a UPS these days.
>>
>> 	I think tower and desk top computers should contain a battery
>> to make safe shutdowns if the mains power fail.  Because I believe in
>> such precautions I run a laptop as my main device.
>
> If a tower had a battery it'd likely be too heavy for you (if it's not
> too heavy already).

	Presently have a donated tower Dell OptiPlex Gx620 sitting on a
trolley waiting for me to spend some more money and complete my project.
Managed to carry it in from the street up a short flight of stairs.  I
won't be doing anything mission critical on this little tower.  Funny 
thing is I am getting weaker every week.

	Just large enough to tell you it is shutting down, then it would save 
the open files and switch itself off.  not a large battery
at all and with modern lighter weight batteries it should not add that
much to the weight, especially as lower powered and less power demanding
cpus come into the marketplace.

>
> I thought about running my server off a laptop, but decided against it
> because I also wanted a RAID array, and that wouldn't be feasible with a
> laptop--either I'd need external drives that could run off of USB power
> or I'd need an external chassis on a UPS.  At that point may as well
> have a real chassis that could support an internal RAID controller.
>
> --keith

	But I don't have to run a server and if somehow I could afford
all the toys I want, I would go to virtual server on leased space and 
have admins setup mailing list and .html servers for the SFBA LUGs.

	bliss

[toc] | [prev] | [next] | [standalone]

#13377

On 2015-01-09, Bobbie Sellers <bliss-sf4ever@dslextreme.com> wrote:
>
> 	But I don't have to run a server and if somehow I could afford
> all the toys I want, I would go to virtual server on leased space and 
> have admins setup mailing list and .html servers for the SFBA LUGs.

If you don't need a real server then your need for a clean shutdown is
much diminished.  A mostly idle filesystem which has its power
interrupted is likely to come back mostly clean, especially if it has a
journal.

--keith

-- 
kkeller-usenet@wombat.san-francisco.ca.us
(try just my userid to email me)
AOLSFAQ=http://www.therockgarden.ca/aolsfaq.txt
see X- headers for PGP signature information

[toc] | [prev] | [next] | [standalone]

#13379

On 09/01/15 05:44, Keith Keller wrote:
> On 2015-01-09, Bobbie Sellers <bliss-sf4ever@dslextreme.com> wrote:
>>
>> 	But I don't have to run a server and if somehow I could afford
>> all the toys I want, I would go to virtual server on leased space and
>> have admins setup mailing list and .html servers for the SFBA LUGs.
>
> If you don't need a real server then your need for a clean shutdown is
> much diminished.  A mostly idle filesystem which has its power
> interrupted is likely to come back mostly clean, especially if it has a
> journal.
>
> --keith
>
Even Real Servers (TM) are unlikely to suffer much I used to run a 
server room of a small company without UPS when power failures were 
routine. So routine I had the PC hardware set to reboot automatically.

Only once did disaster strike, The big SCO unix server got a brownout, 
and, PSU loaded with disk, rebooted. Most other computers survived the 
temporary voltage drop..then halfway through the reboot sequence the 
power went out as it as fsking the primaru disk.,.

It never recovered from that.

WE got fragments back, but it never rebooted a complete system and a lot 
of data was lost.

That was the single occasion in the entire time of many dozens of power 
cuts that a *nix system failed to fsck itself back into shape after a 
power cut.



-- 
Everything you read in newspapers is absolutely true, except for the 
rare story of which you happen to have first-hand knowledge. – Erwin Knoll

[toc] | [prev] | [standalone]

Back to top | Article view | comp.os.linux.misc

csiph-web