Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!eternal-september.org!feeder.eternal-september.org!news.swapon.de!fu-berlin.de!uni-berlin.de!individual.net!not-for-mail From: =?ISO-8859-1?Q?Niels_Fr=F6hling?= Newsgroups: comp.compression Subject: Re: Words as well as characters as encoding units Date: Wed, 09 May 2012 15:14:31 -0600 Lines: 19 Message-ID: References: <357e1db3-17e2-4911-9423-1cf394d894d5@ns1g2000pbc.googlegroups.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Trace: individual.net gn3jFlSHw5CYtq8TWaVhJA1tyWQkBsxOB5/dWxMWCrISbsZI3Eimi/1O/yG9L+O0cS Cancel-Lock: sha1:qfOVF3tEA+tb4Yb15Fgl5ZXQyjo= User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; de-DE; rv:1.7.5) Gecko/20041206 Thunderbird/1.0 Mnenhy/0.7.1 In-Reply-To: Xref: csiph.com comp.compression:1300 > The question is what might be in a file that one wants to compress, > and that might not be in a language. For example, one might be > working with GPS systems, and have files full of latitude and > longitude values. You can consider those "words", but might not > have any correlation. A format-type correlation is often enough for exploitation. The file "geo" of the Canterbury Corpus is the shining example, it's an array of floats, compressing it as bit-planes (knowing how the values correlate) yields a few tenths of percent of compression gain - because all the floats are in a specific correlating range AFAIR 0.0 to 1.0. There are no uncorrelated data-sets besides total random data. Sometimes there may not be strong enough a correlation, and then any high-order compressors compress bad/worst, but exploiting correlations is their business, not plain static stationary order-0. Ciao Niels