Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.compression > #1656 > unrolled thread

new format .gz ?

Back to article view | Back to comp.compression

Contents

  new format .gz ? sterten@aol.com - 2012-12-24 04:37 -0800
    Re: new format .gz ? sterten@aol.com - 2012-12-24 05:17 -0800
      Re: new format .gz ? sterten@aol.com - 2012-12-24 22:34 -0800
        Re: new format .gz ? glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2012-12-25 08:24 +0000
    Re: new format .gz ? sterten@aol.com - 2012-12-25 02:40 -0800
      Re: new format .gz ? glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2012-12-25 13:52 +0000
    Re: new format .gz ? sterten@aol.com - 2013-01-26 07:24 -0800

#1656 — new format .gz ?

this has probably already been mentioned here ...
although, I couldn't find it and it did already cost me some hours

When I uncompressed recently downloaded .gz files
on Windows XP , it seemed to work well, without error
message, but the filelength of the decompressed file doesn't 
seem to match. Too short.

Then I figured out that gzip -l file.gz or 7zip l file.gz give zero
as the uncompressed filelength.

Other .gz files seem to work correctly.

So, is there a new incompatible .gz format around ?
Maybe connected with the recent switch to 64-bit operating 
systems and processors ?

And why is it so difficult to find some info about it ?
It seems to be rather important, or not ?

[toc] | [next] | [standalone]

#1657

while 7zip still displays these gz-archives as files
of length 09 (7z l chr02.gz) it still seems to extract
and uncompress them correctly (7z e chr02.gz)
(other than gzip -d chr02)

[toc] | [prev] | [next] | [standalone]

#1658

1000 genomes is using a specialized version of gzip, 
so presumably this is just a bug in their version
and not related to a new general 64-bit format.

It was surprising to me that 7zip gave much better (~2fold)
compression rates with their "vcf-format" files, (=lists of 
mutations in several human genomes)than gzip.

Could that be because the lines - and thus the potentioal
locations of matching strings - in these files are so long ?

[toc] | [prev] | [next] | [standalone]

#1659

sterten@aol.com wrote:
> 1000 genomes is using a specialized version of gzip, 
> so presumably this is just a bug in their version
> and not related to a new general 64-bit format.
 
> It was surprising to me that 7zip gave much better (~2fold)
> compression rates with their "vcf-format" files, (=lists of 
> mutations in several human genomes)than gzip.
 
> Could that be because the lines - and thus the potentioal
> locations of matching strings - in these files are so long ?

It has been some time since I was working on this...

I presume you are talking about BAM files.

While they are actual gzip files, they also have some index information
such that you can start decompressing from other than at the beginning.

-- glen

[toc] | [prev] | [next] | [standalone]

#1660

I don't know about BAM, I'm pretty new to this.
These here were "vcf" files, normal text files , tables ,
with ~4000 binary columns per ~100000 rows.
So, basically a 4000*100000 binary array with some (partial) 
rows and columns being similar especially if they are close.

Compressing these could become important with the
_lots_ of genetical data expected in the coming years

Their ftp site has already >400TB so you can understand
why they need good compression.

[toc] | [prev] | [next] | [standalone]

#1661

sterten@aol.com wrote:
> I don't know about BAM, I'm pretty new to this.
> These here were "vcf" files, normal text files , tables ,
> with ~4000 binary columns per ~100000 rows.
> So, basically a 4000*100000 binary array with some (partial) 
> rows and columns being similar especially if they are close.

Does look pretty wordy for something that is supposed to reduce
storage space.
 
> Compressing these could become important with the
> _lots_ of genetical data expected in the coming years

Yes. There is a lot of redundancy in many genetic data files that
compression algorithms aren't good at finding. 
 
> Their ftp site has already >400TB so you can understand
> why they need good compression.

DNA sequence data should compress to two bits per base with many
compression algorithms. If you add quality data, though, it will
be much worse. 

-- glen

[toc] | [prev] | [next] | [standalone]

#1729

redundancy comes since many similar genomes are stored in the
 same directory.
Their gzip format was a special format used by that project.
Apparently it doesn't support the list feature.
It turned out that the data could be stored with an estimated
improvement factor of ~2 using 1 bit per SNP (=consensus or not)
and then 7zip.
But they are not interested to change their system, apparently
storage is sooo cheap meanwhile and download time is not their problem.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.compression

csiph-web