Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #167715 > unrolled thread

archivemail screwed up (some of) my backups

Back to article view | Back to linux.debian.user

Contents

  archivemail screwed up (some of) my backups steve <dlist@bluewin.ch> - 2016-05-11 11:20 +0200
    Re: archivemail screwed up (some of) my backups "Thomas Schmitt" <scdbackup@gmx.net> - 2016-05-11 12:30 +0200
      Re: archivemail screwed up (some of) my backups steve <dlist@bluewin.ch> - 2016-05-11 13:30 +0200
        Re: archivemail screwed up (some of) my backups Sven Hartge <sven@svenhartge.de> - 2016-05-11 16:30 +0200
          Re: archivemail screwed up (some of) my backups steve <dlist@bluewin.ch> - 2016-05-11 18:50 +0200

#167715 — archivemail screwed up (some of) my backups

Hi there,

This is my first post on this mailinglist, so I hope to find the right
tone pretty quickly. If not, please correct me gently  ;-)

I'm using archivemail to archive my messages on a Jessie box up to date.
Recently I needed to find an old message so I tried to gunzip the
mail_archive.gz file and I got this answer:

gzip: mail_archive.gz: invalid compressed data--crc error

I tried the same thing on other archives and the result is that more or
less one third were corrupted. I searched the Net for a solution, but
since the error is a rather general one, I failed to find something
really useful.

Anyone here with a brilliant idea? Or should I buy a couple of
handkerchiefs and start a period of mourning for my lost messages?

Thank you.

Best,
Steve

[toc] | [next] | [standalone]

#167724

Hi,

steve wrote:
> gzip: mail_archive.gz: invalid compressed data--crc error

It seems that
  gunzip <file.gz
is more tolerant than
  gunzip file.gz

(Most probably because it does not peek ahead to the end of the stream.
 With <file.gz it would be well able to perform random access, but not
 with cat file.gx | gunzip.)


I tested with a human readable text in file x:

  gzip x >xx.gz

which yields more than 1000 bytes. Then i damaged it:

  dd if=/dev/zero bs=1 count=1 of=/u/x.gz seek=1000 conv=notrunc

and tried to uncompress

  gunzip xx.gz

which yields

  gzip: xx.gz: invalid compressed data--crc error
  gzip: xx.gz: invalid compressed data--length error

On the other hand

   gunzip <xx.gz >xx

yields the start part of the human readable text and begins to
show garbled text after about 3000 bytes of uncompressed output.

So depending on where the bad spot is in your .gz file, you
might be able to retrieve some of it. After the bad spot, there
is few hope to retrieve more valid data.
The main problem is to locate the byte address where the output
begins to be invalid. Not hard with human readable text. But
quite a challenge with binary data stored as base64 or uuencode.

-------------------------------------------------------------------

> should I buy a couple of
> handkerchiefs and start a period of mourning for my lost messages?

A few handkerchiefs seem appropriate.

But you should not blame the archiver program before you have
outruled that your storage system spoiled the data.
Especially since unreliable storage would be a much more severe
problem than a buggy archiver.

(Further: gzip compression is normally done via zlib, which is
 supposed to do it right. So if the archiver can produce some
 uncompressable .gz, then it it probably does its part of the job
correctly.)


Have a nice day :)

Thomas

[toc] | [prev] | [next] | [standalone]

#167727

Hi Thomas,

Thanks for your answer!

Le 11-05-2016, à 12:22:18 +0200, Thomas Schmitt a écrit :

>steve wrote:
>> gzip: mail_archive.gz: invalid compressed data--crc error
>
>It seems that
>  gunzip <file.gz
>is more tolerant than
>  gunzip file.gz

You're right, with gunzip < file.gz > file, I can recuperate almost the
whole file:

ls -l
-rw-r--r-- 1 steve steve 313033001 mai 11 13:15 file_archive
-rwxr-xr-x 1 steve steve 192528981 mai  3 14:00 file_archive.gz

I don't know what is the compression rate, but if it's in the x2 range,
I'm almost there.

Anyway, I have been able, by chance, to put the hand on the needed message.

[snip]

>> should I buy a couple of
>> handkerchiefs and start a period of mourning for my lost messages?
>
>A few handkerchiefs seem appropriate.
>
>But you should not blame the archiver program before you have
>outruled that your storage system spoiled the data.
>Especially since unreliable storage would be a much more severe
>problem than a buggy archiver.

I understand, but since I have more or less 2/3 of the *.gz that are ok,
I thought archivemail was the responsible. But after gunzipping some
other files, it seems that the problem only arises on ISO-8850 text file,
not on UTF-8. But not tech savvy enough to know why.

The thing is now, how could I be 99.99% sure that the backups
created by archivemail are reliable? My confidence is a bit broken now.


>Have a nice day :)

The sun just rose above horizon, thanks to you!

Steve

[toc] | [prev] | [next] | [standalone]

#167730

steve <dlist@bluewin.ch> wrote:

> I understand, but since I have more or less 2/3 of the *.gz that are ok,
> I thought archivemail was the responsible. But after gunzipping some
> other files, it seems that the problem only arises on ISO-8850 text file,
> not on UTF-8. But not tech savvy enough to know why.

> The thing is now, how could I be 99.99% sure that the backups
> created by archivemail are reliable? My confidence is a bit broken now.

I have been using archivemail since March 2003, always with compression.
My configuration generates a new gzipped archive for every month for
every IMAP folder.

To give you some peace of mind, I just did a simple decompression test
of all archives [~5000 right now] (gunzip $file > /dev/null) to check
for errors and found none.

Whatever error you have, I don't think archivemail is the culprit
creating them.

Grüße,
Sven.

-- 
Sigmentation fault. Core dumped.

[toc] | [prev] | [next] | [standalone]

#167738

Hi Sven,


>I have been using archivemail since March 2003, always with compression.
>My configuration generates a new gzipped archive for every month for
>every IMAP folder.
>
>To give you some peace of mind, I just did a simple decompression test
>of all archives [~5000 right now] (gunzip $file > /dev/null) to check
>for errors and found none.
>
>Whatever error you have, I don't think archivemail is the culprit
>creating them.

Well thanks for your checks.

It seems that I have to dig into this problem more thoroughly. 

If I find anything that could be useful, I'll post here.

Thanks again.

Best,
Steve

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web