Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.compression > #1160

Re: Entropy Definition : Request for Comments

Show all headers | View raw

On 15.03.2012 07:29, lawcounsels@gmail.com wrote:

>> So this is not exactly "interesting". The answer is immediate, and
>> exactly what is expected.
>>
>> More interesting things happen if you map this to a smaller set, say,
>> you consider for example sums of random variables. Then, under mild
>> constraints of the source, you get something called the "central limit
>> theorem", i.e. your distribution approaches Gaussians (or stable
>> distributions under less constrained conditions).
>>
>> Greetings,
>> 	Thomas
>
> we are both on the right wavelengths here
>
> myself would 100 times out of 100 immediate says the exact same you did here without hesitations ... EXCEPT the INVARIABLE REVERSIBLE resultant transformed file of same N bits long ALWAYS GIVES sizeable portions with unmistakable particularly strong patterns / restrictions ( UNMISTAKABLY UNAMBIGUOUS LOW ENTROPY bits sequences at unmistakable unambiguous ascertainable 'start' / 'end'  bits positions .... mathematics proven instantiated with the described Entropy Definition formula on ALL real transformed resultant files )

Look, what people often confuse is "has many obvious patterns" = "has 
low entropy". This is simply wrong. A specific outcome of a random 
source may very well have "obvious patterns", but might still have a low 
probability.

For example, let Y be a binary random iid source with p(0)=7/8 and 
p(1)=1/8. Let z_k be the n-th digit in the binary expansion of Pi, or to 
be precise, let z_k = floor(\pi * 2^k) mod 2. Then define x_k = y^k xor 
z_k. Then, the outcome (realization) of z_k being 0000...000000 is 
surely very unlikely, yet has an obvious pattern, and the entropy of Y 
and the entropy of X are identical. (And the entropy of Z is zero 
because it is not a random process...).

This "paradox" stems from the fact that sources humans like you or me 
care about only care about reduntant sources, i.e. the random sources we 
look at (newspapers, photos, audio recordings) are highly redundant 
*and* contain many patterns. That is, for the sources we care about, we 
have "low entropy" = "many patterns". But this is simply false in 
general. Or to be even more concrete "having a pattern" is not a 
well-defined term, nor does it have any relation to entropy. It *only* 
(or should I write "only"?) has a relation to entropy for those sources 
we as human observers care about. Sources with patterns allow "simple 
models", but just because the models are simpler that allow you to 
consider them as low-entropy random sources does not mean that there are 
low-entropy sources that do not show obvious patterns. In fact, in a 
precise way, *most* low-order entropy sources are like that (i.e. do not 
show obvious patterns). Unfortunately, even these "most low order 
entropy sources" (with non-obvious patterns) are still a very rare 
species in the set of all sources, rare enough to disallow any escape 
from the counting argument.

So long,
	Thomas

Back to comp.compression | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Entropy Definition : Request for Comments LawCounsels <LawCounsels@aol.com> - 2012-03-13 01:40 -0700
  Re: Entropy Definition : Request for Comments Thomas Richter <thor@math.tu-berlin.de> - 2012-03-13 10:13 +0100
    Re: Entropy Definition : Request for Comments LawCounsels <lawcounsels@gmail.com> - 2012-03-13 10:03 -0700
      Re: Entropy Definition : Request for Comments Thomas Richter <thor@math.tu-berlin.de> - 2012-03-13 19:25 +0100
        Re: Entropy Definition : Request for Comments LawCounsels <lawcounsels@gmail.com> - 2012-03-13 11:56 -0700
          Re: Entropy Definition : Request for Comments Thomas Richter <thor@math.tu-berlin.de> - 2012-03-13 20:38 +0100
            Re: Entropy Definition : Request for Comments lawcounsels@gmail.com - 2012-03-14 01:43 -0700
              Re: Entropy Definition : Request for Comments lawcounsels@gmail.com - 2012-03-14 01:46 -0700
                Re: Entropy Definition : Request for Comments Thomas Richter <thor@math.tu-berlin.de> - 2012-03-14 12:09 +0100
                Re: Entropy Definition : Request for Comments LawCounsels <LawCounsels@aol.com> - 2012-03-14 08:52 -0700
                Re: Entropy Definition : Request for Comments lawcounsels@gmail.com - 2012-03-14 11:34 -0700
                Re: Entropy Definition : Request for Comments Thomas Richter <thor@math.tu-berlin.de> - 2012-03-14 22:16 +0100
                Re: Entropy Definition : Request for Comments lawcounsels@gmail.com - 2012-03-14 23:29 -0700
                Re: Entropy Definition : Request for Comments Thomas Richter <thor@math.tu-berlin.de> - 2012-03-15 12:10 +0100
                Re: Entropy Definition : Request for Comments lawcounsels@gmail.com - 2012-03-15 06:23 -0700
                Re: Entropy Definition : Request for Comments Thomas Richter <thor@math.tu-berlin.de> - 2012-03-15 16:45 +0100
                Re: Entropy Definition : Request for Comments LawCounsels <LawCounsels@aol.com> - 2012-03-20 18:08 -0700
                Re: Entropy Definition : Request for Comments Thomas Richter <thor@math.tu-berlin.de> - 2012-03-14 22:10 +0100
  Re: Entropy Definition : Request for Comments jacko <jackokring@gmail.com> - 2012-03-15 16:27 -0700
  Re: Entropy Definition : Request for Comments jacko <jackokring@gmail.com> - 2012-03-15 16:40 -0700
    Re: Entropy Definition : Request for Comments jacko <jackokring@gmail.com> - 2012-03-15 17:14 -0700
      Re: Entropy Definition : Request for Comments jacko <jackokring@gmail.com> - 2012-03-15 17:24 -0700
        Re: Entropy Definition : Request for Comments LawCounsels@aol.com - 2012-03-31 10:30 -0700

csiph-web