Groups > comp.lang.python > #58077 > unrolled thread

Algorithm that makes maximum compression of completly diffused data.

Started by	jonas.thornvall@gmail.com
First post	2013-10-30 11:21 -0700
Last post	2013-11-03 04:50 -0500
Articles	20 on this page of 72 — 22 participants

Back to article view | Back to comp.lang.python

Page 1 of 4 [1] 2 3 4 Next page →

#58077 — Algorithm that makes maximum compression of completly diffused data.

From	jonas.thornvall@gmail.com
Date	2013-10-30 11:21 -0700
Subject	Algorithm that makes maximum compression of completly diffused data.
Message-ID	<205bfa4f-29de-43de-be5a-72a12d77d0c9@googlegroups.com>

I am searching for the program or algorithm that makes the best possible of completly (diffused data/random noise) and wonder what the state of art compression is.

I understand this is not the correct forum but since i think i have an algorithm that can do this very good, and do not know where to turn for such question i was thinking to start here.

It is of course lossless compression i am speaking of.

[toc] | [next] | [standalone]

#58083

From	Mark Lawrence <breamoreboy@yahoo.co.uk>
Date	2013-10-30 18:53 +0000
Message-ID	<mailman.1825.1383159257.18130.python-list@python.org>
In reply to	#58077

On 30/10/2013 18:21, jonas.thornvall@gmail.com wrote:
> I am searching for the program or algorithm that makes the best possible of completly (diffused data/random noise) and wonder what the state of art compression is.
>
> I understand this is not the correct forum but since i think i have an algorithm that can do this very good, and do not know where to turn for such question i was thinking to start here.
>
> It is of course lossless compression i am speaking of.
>

I can't help with compression but I can help with a marvellous source of 
the opposite, expansion, it's google groups.

-- 
Python is the second best programming language in the world.
But the best has yet to be invented.  Christian Tismer

Mark Lawrence

[toc] | [prev] | [next] | [standalone]

#58086

From	jonas.thornvall@gmail.com
Date	2013-10-30 12:01 -0700
Message-ID	<0f857f7f-5947-4b3a-805d-0e9b888dfd48@googlegroups.com>
In reply to	#58083

Den onsdagen den 30:e oktober 2013 kl. 19:53:59 UTC+1 skrev Mark Lawrence:
> On 30/10/2013 18:21, jonas.thornvall@gmail.com wrote:
> 
> > I am searching for the program or algorithm that makes the best possible of completly (diffused data/random noise) and wonder what the state of art compression is.
> 
> >
> 
> > I understand this is not the correct forum but since i think i have an algorithm that can do this very good, and do not know where to turn for such question i was thinking to start here.
> 
> >
> 
> > It is of course lossless compression i am speaking of.
> 
> >
> 
> 
> 
> I can't help with compression but I can help with a marvellous source of 
> 
> the opposite, expansion, it's google groups.
> 
> 
> 
> -- 
> 
> Python is the second best programming language in the world.
> 
> But the best has yet to be invented.  Christian Tismer
> 
> 
> 
> Mark Lawrence

And your still a stupid monkey i dare you to go test your IQ.

[toc] | [prev] | [next] | [standalone]

#58091

From	Mark Lawrence <breamoreboy@yahoo.co.uk>
Date	2013-10-30 19:18 +0000
Message-ID	<mailman.1829.1383160726.18130.python-list@python.org>
In reply to	#58086

On 30/10/2013 19:01, jonas.thornvall@gmail.com wrote:
>
> And your still a stupid monkey i dare you to go test your IQ.
>

It's you're as in you are and not your as in belongs to me.

I have no intention of getting my IQ tested, but I do know that it's a 
minimum of 120 as that was required for me to pass the old UK 11+ 
examination.  Given that I spent my time at a grammar school in the top 
stream I'd guess that my IQ is actually higher, but there you go.

Not that that really matters.  What does is that I'm smart enough to be 
able to follow a set of instructions when requested to do so, for 
example I could probably follow these 
https://wiki.python.org/moin/GoogleGroupsPython if I needed to.  I'm 
therefore assuming that you're not bright enough to follow these 
instructions and so have annoyed thousands of people with your double 
spaced crap, which I've again snipped.

-- 
Python is the second best programming language in the world.
But the best has yet to be invented.  Christian Tismer

Mark Lawrence

[toc] | [prev] | [next] | [standalone]

#58092

From	jonas.thornvall@gmail.com
Date	2013-10-30 12:22 -0700
Message-ID	<299c1ff4-2467-45b2-a67a-b8898027d5b1@googlegroups.com>
In reply to	#58091

Den onsdagen den 30:e oktober 2013 kl. 20:18:30 UTC+1 skrev Mark Lawrence:
> On 30/10/2013 19:01, jonas.thornvall@gmail.com wrote:
> 
> >
> 
> > And your still a stupid monkey i dare you to go test your IQ.
> 
> >
> 
> 
> 
> It's you're as in you are and not your as in belongs to me.
> 
> 
> 
> I have no intention of getting my IQ tested, but I do know that it's a 
> 
> minimum of 120 as that was required for me to pass the old UK 11+ 
> 
> examination.  Given that I spent my time at a grammar school in the top 
> 
> stream I'd guess that my IQ is actually higher, but there you go.
> 
> 
> 
> Not that that really matters.  What does is that I'm smart enough to be 
> 
> able to follow a set of instructions when requested to do so, for 
> 
> example I could probably follow these 
> 
> https://wiki.python.org/moin/GoogleGroupsPython if I needed to.  I'm 
> 
> therefore assuming that you're not bright enough to follow these 
> 
> instructions and so have annoyed thousands of people with your double 
> 
> spaced crap, which I've again snipped.
> 
> 
> 
> -- 
> 
> Python is the second best programming language in the world.
> 
> But the best has yet to be invented.  Christian Tismer
> 
> 
> 
> Mark Lawrence

I do not follow instructions, i make them accesible to anyone. And you just following them is a clear example to your lack of IQ.

[toc] | [prev] | [next] | [standalone]

#58096

From	Mark Lawrence <breamoreboy@yahoo.co.uk>
Date	2013-10-30 19:31 +0000
Message-ID	<mailman.1832.1383161528.18130.python-list@python.org>
In reply to	#58092

On 30/10/2013 19:22, jonas.thornvall@gmail.com wrote:
> Den onsdagen den 30:e oktober 2013 kl. 20:18:30 UTC+1 skrev Mark Lawrence:
>> On 30/10/2013 19:01, jonas.thornvall@gmail.com wrote:
>>
>>>
>>
>>> And your still a stupid monkey i dare you to go test your IQ.
>>
>>>
>>
>>
>>
>> It's you're as in you are and not your as in belongs to me.
>>
>>
>>
>> I have no intention of getting my IQ tested, but I do know that it's a
>>
>> minimum of 120 as that was required for me to pass the old UK 11+
>>
>> examination.  Given that I spent my time at a grammar school in the top
>>
>> stream I'd guess that my IQ is actually higher, but there you go.
>>
>>
>>
>> Not that that really matters.  What does is that I'm smart enough to be
>>
>> able to follow a set of instructions when requested to do so, for
>>
>> example I could probably follow these
>>
>> https://wiki.python.org/moin/GoogleGroupsPython if I needed to.  I'm
>>
>> therefore assuming that you're not bright enough to follow these
>>
>> instructions and so have annoyed thousands of people with your double
>>
>> spaced crap, which I've again snipped.
>>
>>
>>
>> --
>>
>> Python is the second best programming language in the world.
>>
>> But the best has yet to be invented.  Christian Tismer
>>
>>
>>
>> Mark Lawrence
>
> I do not follow instructions, i make them accesible to anyone. And you just following them is a clear example to your lack of IQ.
>

I suggest that you reread what I wrote above, assuming that you can find 
it amongst the double spaced crap.  I said I would follow the 
instructions if I needed to.  So clearly I'm not just following them, as 
I've no need to, as I'm smart enough to use a vastly superior tool.

-- 
Python is the second best programming language in the world.
But the best has yet to be invented.  Christian Tismer

Mark Lawrence

[toc] | [prev] | [next] | [standalone]

#58093

From	jonas.thornvall@gmail.com
Date	2013-10-30 12:23 -0700
Message-ID	<d1aef452-50fb-4ca5-b392-7a2545fbfb25@googlegroups.com>
In reply to	#58091

Den onsdagen den 30:e oktober 2013 kl. 20:18:30 UTC+1 skrev Mark Lawrence:
> On 30/10/2013 19:01, jonas.thornvall@gmail.com wrote:
> 
> >
> 
> > And your still a stupid monkey i dare you to go test your IQ.
> 
> >
> 
> 
> 
> It's you're as in you are and not your as in belongs to me.
> 
> 
> 
> I have no intention of getting my IQ tested, but I do know that it's a 
> 
> minimum of 120 as that was required for me to pass the old UK 11+ 
> 
> examination.  Given that I spent my time at a grammar school in the top 
> 
> stream I'd guess that my IQ is actually higher, but there you go.
> 
> 
> 
> Not that that really matters.  What does is that I'm smart enough to be 
> 
> able to follow a set of instructions when requested to do so, for 
> 
> example I could probably follow these 
> 
> https://wiki.python.org/moin/GoogleGroupsPython if I needed to.  I'm 
> 
> therefore assuming that you're not bright enough to follow these 
> 
> instructions and so have annoyed thousands of people with your double 
> 
> spaced crap, which I've again snipped.
> 
> 
> 
> -- 
> 
> Python is the second best programming language in the world.
> 
> But the best has yet to be invented.  Christian Tismer
> 
> 
> 
> Mark Lawrence

What i actually saying is that you are indeed... an anal code monkey that never ever had a selfsustained thought of your own.

Think about it.

[toc] | [prev] | [next] | [standalone]

#58099

From	Mark Lawrence <breamoreboy@yahoo.co.uk>
Date	2013-10-30 19:35 +0000
Message-ID	<mailman.1834.1383162004.18130.python-list@python.org>
In reply to	#58093

On 30/10/2013 19:23, jonas.thornvall@gmail.com wrote:
> Den onsdagen den 30:e oktober 2013 kl. 20:18:30 UTC+1 skrev Mark Lawrence:
>> On 30/10/2013 19:01, jonas.thornvall@gmail.com wrote:
>>
>>>
>>
>>> And your still a stupid monkey i dare you to go test your IQ.
>>
>>>
>>
>>
>>
>> It's you're as in you are and not your as in belongs to me.
>>
>>
>>
>> I have no intention of getting my IQ tested, but I do know that it's a
>>
>> minimum of 120 as that was required for me to pass the old UK 11+
>>
>> examination.  Given that I spent my time at a grammar school in the top
>>
>> stream I'd guess that my IQ is actually higher, but there you go.
>>
>>
>>
>> Not that that really matters.  What does is that I'm smart enough to be
>>
>> able to follow a set of instructions when requested to do so, for
>>
>> example I could probably follow these
>>
>> https://wiki.python.org/moin/GoogleGroupsPython if I needed to.  I'm
>>
>> therefore assuming that you're not bright enough to follow these
>>
>> instructions and so have annoyed thousands of people with your double
>>
>> spaced crap, which I've again snipped.
>>
>>
>>
>> --
>>
>> Python is the second best programming language in the world.
>>
>> But the best has yet to be invented.  Christian Tismer
>>
>>
>>
>> Mark Lawrence
>
> What i actually saying is that you are indeed... an anal code monkey that never ever had a selfsustained thought of your own.
>
> Think about it.
>

I just have to bow down to your vast superiority over me.

How is your job with your country's diplomatic corp going by the way?

-- 
Python is the second best programming language in the world.
But the best has yet to be invented.  Christian Tismer

Mark Lawrence

[toc] | [prev] | [next] | [standalone]

#58364

From	Ethan Furman <ethan@stoneleaf.us>
Date	2013-11-02 21:26 -0700
Message-ID	<mailman.1968.1383454272.18130.python-list@python.org>
In reply to	#58093

On 10/30/2013 12:23 PM, jonas.thornvall@gmail.com wrote:
>
> What i actually saying is that you are indeed... [insult snipped]

*plonk*

[toc] | [prev] | [next] | [standalone]

#58105

From	Antoon Pardon <antoon.pardon@rece.vub.ac.be>
Date	2013-10-30 20:28 +0100
Message-ID	<mailman.1836.1383163117.18130.python-list@python.org>
In reply to	#58086

Op 30-10-13 20:01, jonas.thornvall@gmail.com schreef:
> Den onsdagen den 30:e oktober 2013 kl. 19:53:59 UTC+1 skrev Mark Lawrence:
>> On 30/10/2013 18:21, jonas.thornvall@gmail.com wrote:
>>
>>> I am searching for the program or algorithm that makes the best possible of completly (diffused data/random noise) and wonder what the state of art compression is.
>>
>>>
>>
>>> I understand this is not the correct forum but since i think i have an algorithm that can do this very good, and do not know where to turn for such question i was thinking to start here.
>>
>>>
>>
>>> It is of course lossless compression i am speaking of.
>>
>>>
>>
>>
>>
>> I can't help with compression but I can help with a marvellous source of
>>
>> the opposite, expansion, it's google groups.
>>
>>
>>
>> --
>>
>> Python is the second best programming language in the world.
>>
>> But the best has yet to be invented.  Christian Tismer
>>
>>
>>
>> Mark Lawrence
>
> And your still a stupid monkey i dare you to go test your IQ.
>
Are you sure it is this game you want to play? Please consider
carefully: For what purpose did you come to this group? Is your
behaviour accomplishing that purpose?

You were asked to take responsibility for the detrimental effect your
choice of news reader software has on this newsgroup. Your answer boiled
down to a very clear "I don't care about the detrimental effect I cause"
Well until you start caring and adapt your behaviour, people will tend
not to care about your questions/problems and the most likely responses
you will get is people pointing to your anti-social behaviour.

You may feel very righteous in your response to Mark but you will just
further alienate the regulars. If that is your goal you can continue
as you did before and soon you will be in a lot of kill file or if
you hope for some cooperation from the regulars, you'd better show
you can be cooperative too.

-- 
Antoon Pardon

[toc] | [prev] | [next] | [standalone]

#58127

From	Joshua Landau <joshua@landau.ws>
Date	2013-10-30 21:30 +0000
Message-ID	<mailman.1850.1383168672.18130.python-list@python.org>
In reply to	#58086

On 30 October 2013 19:18, Mark Lawrence <breamoreboy@yahoo.co.uk> wrote:
> On 30/10/2013 19:01, jonas.thornvall@gmail.com wrote:
>>
>> And your still a stupid monkey i dare you to go test your IQ.
>
> It's you're as in you are and not your as in belongs to me.
>
> I have no intention of getting my IQ tested, but I do know that it's a
> minimum of 120 as that was required for me to pass the old UK 11+
> examination.  Given that I spent my time at a grammar school in the top
> stream I'd guess that my IQ is actually higher, but there you go.

What I'm confounded about is this list's inability to recognise a
troll when it slaps it vocally in the face.

This isn't like Nikos. There's no "troll vs. incompetent" debate to be
had. This guy started talking about compressing *random data* and
immediately descended into insults. Your IQ, whatever it may be, is
going to waste ;).

[toc] | [prev] | [next] | [standalone]

#58169

From	rusi <rustompmody@gmail.com>
Date	2013-10-31 05:54 -0700
Message-ID	<70e151fe-4499-4d57-9e1a-5b85f113e0e8@googlegroups.com>
In reply to	#58127

On Thursday, October 31, 2013 3:00:24 AM UTC+5:30, Joshua Landau wrote:

> What I'm confounded about is this list's inability to recognise a
> troll when it slaps it vocally in the face.

> This isn't like Nikos. There's no "troll vs. incompetent" debate to be
> had.


Its usually called "entertainment". Something related:

http://onceuponatimeinindia.blogspot.in/2009/07/hard-drive-weight-increasing.html

[toc] | [prev] | [next] | [standalone]

#58135

From	Mark Lawrence <breamoreboy@yahoo.co.uk>
Date	2013-10-30 21:52 +0000
Message-ID	<mailman.1857.1383169937.18130.python-list@python.org>
In reply to	#58086

On 30/10/2013 21:30, Joshua Landau wrote:
> On 30 October 2013 19:18, Mark Lawrence <breamoreboy@yahoo.co.uk> wrote:
>> On 30/10/2013 19:01, jonas.thornvall@gmail.com wrote:
>>>
>>> And your still a stupid monkey i dare you to go test your IQ.
>>
>> It's you're as in you are and not your as in belongs to me.
>>
>> I have no intention of getting my IQ tested, but I do know that it's a
>> minimum of 120 as that was required for me to pass the old UK 11+
>> examination.  Given that I spent my time at a grammar school in the top
>> stream I'd guess that my IQ is actually higher, but there you go.
>
> What I'm confounded about is this list's inability to recognise a
> troll when it slaps it vocally in the face.
>
> This isn't like Nikos. There's no "troll vs. incompetent" debate to be
> had. This guy started talking about compressing *random data* and
> immediately descended into insults. Your IQ, whatever it may be, is
> going to waste ;).
>

In my defence I did say this earlier in the exchange "I can't help with 
compression but I can help with a marvellous source of the opposite, 
expansion, it's google groups."  Definitely a troll though, to go in my 
dream team with Xah Lee and Ilias Lazaridis amongst others and good old 
rr on the subs bench :)

-- 
Python is the second best programming language in the world.
But the best has yet to be invented.  Christian Tismer

Mark Lawrence

[toc] | [prev] | [next] | [standalone]

#58140

From	Tim Chase <python.list@tim.thechases.com>
Date	2013-10-30 18:01 -0500
Message-ID	<mailman.1861.1383174007.18130.python-list@python.org>
In reply to	#58086

On 2013-10-30 21:30, Joshua Landau wrote:
> started talking about compressing *random data* 

If it's truly random bytes, as long as you don't need *the same*
random data, you can compress it quite easily.  Lossy compression is
acceptable for images, so why not random files?  :-)

  import os
  inname = "random.txt"
  namez = inname + '.rnz'
  # compress the file
  with open(outnamez, 'w') as f:
    f.write(os.stat(inname).st_size)

  # uncompress the file
  with open(namez) as f:
    size = int(f.read())
  with open('/dev/random', 'rb') as rnd, open(inname, 'wb') as out:
    for i in range(size):
      out.write(rnd.read(1))


There are optimizations that can be made, and I didn't make it run
on Win32, but I leave those as exercises for the reader.  That
said, this compresses *remarkably* well for large files ;-)

-tkc

[toc] | [prev] | [next] | [standalone]

#58142

From	Chris Angelico <rosuav@gmail.com>
Date	2013-10-31 10:41 +1100
Message-ID	<mailman.1862.1383176481.18130.python-list@python.org>
In reply to	#58086

On Thu, Oct 31, 2013 at 10:01 AM, Tim Chase
<python.list@tim.thechases.com> wrote:
> On 2013-10-30 21:30, Joshua Landau wrote:
>> started talking about compressing *random data*
>
> If it's truly random bytes, as long as you don't need *the same*
> random data, you can compress it quite easily.  Lossy compression is
> acceptable for images, so why not random files?  :-)

Maybe. But what if it's not truly random, but only pseudo-random?

# create a file full of random data
import random
seed = random.getrandbits(32)
length = random.getrandbits(16) # in four-byte units
random.seed(seed)
inname = "random.txt"
namez = inname + '.rnz'
with open(inname, "wb") as bigfile:
    for _ in range(length):
        bigfile.write(random.getrandbits(32).to_bytes(4,"big"))

# compress that file
with open(namez, "wb") as smallfile:
    smallfile.write(seed.to_bytes(4,"big"))
    smallfile.write(length.to_bytes(4,"big"))

# uncompress it
with open(namez, "rb") as f:
    seed = int.from_bytes(f.read(4),"big")
    length = int.from_bytes(f.read(4),"big")
random.seed(seed)
with open("out_" + inname, "wb") as bigfile:
    for _ in range(length):
        bigfile.write(random.getrandbits(32).to_bytes(4,"big"))

Voila! Very impressive compression ratio, and exploits the very
randomness of the data!

ChrisA

[toc] | [prev] | [next] | [standalone]

#58095

From	Dan Stromberg <drsalists@gmail.com>
Date	2013-10-30 12:29 -0700
Message-ID	<mailman.1831.1383161383.18130.python-list@python.org>
In reply to	#58077

[Multipart message — attachments visible in raw view] — view raw

xz compression is pretty hard, if a little bit slow.  Also, if you want
really stellar compression ratios and you don't care about time to
compress, you might check out one of the many paq implementations.

I have a module that does xz compression in 4 different ways:
http://stromberg.dnsalias.org/svn/backshift/tags/1.20/xz_mod.py
It's only for smallish chunks in the ctypes version, because that was all I
needed.  The others should be able to handle relatively large inputs.

On Wed, Oct 30, 2013 at 11:21 AM, <jonas.thornvall@gmail.com> wrote:

> I am searching for the program or algorithm that makes the best possible
> of completly (diffused data/random noise) and wonder what the state of art
> compression is.
>
> I understand this is not the correct forum but since i think i have an
> algorithm that can do this very good, and do not know where to turn for
> such question i was thinking to start here.
>
> It is of course lossless compression i am speaking of.
> --
> https://mail.python.org/mailman/listinfo/python-list
>

[toc] | [prev] | [next] | [standalone]

#58097

From	Tim Delaney <timothy.c.delaney@gmail.com>
Date	2013-10-31 06:35 +1100
Message-ID	<mailman.1833.1383161768.18130.python-list@python.org>
In reply to	#58077

[Multipart message — attachments visible in raw view] — view raw

On 31 October 2013 05:21, <jonas.thornvall@gmail.com> wrote:

> I am searching for the program or algorithm that makes the best possible
> of completly (diffused data/random noise) and wonder what the state of art
> compression is.
>
> I understand this is not the correct forum but since i think i have an
> algorithm that can do this very good, and do not know where to turn for
> such question i was thinking to start here.
>
> It is of course lossless compression i am speaking of.
>

This is not an appropriate forum for this question. If you know it's an
inappropriate forum (as you stated) then do not post the question here. Do
a search with your preferred search engine and look up compression on
lossless Wikipedia. And read and understand the following link:

http://www.catb.org/esr/faqs/smart-questions.html

paying special attention to the following parts:

http://www.catb.org/esr/faqs/smart-questions.html#forum
http://www.catb.org/esr/faqs/smart-questions.html#prune
http://www.catb.org/esr/faqs/smart-questions.html#courtesy
http://www.catb.org/esr/faqs/smart-questions.html#keepcool
http://www.catb.org/esr/faqs/smart-questions.html#classic

If you have *python* code implementing this algorithm and want help, post
the parts you want help with (and preferably post the entire algorithm in a
repository).

However, having just seen the following from you in a reply to Mark ("I do
not follow instructions, i make them accesible to anyone"), I am not not
going to give a second chance - fail to learn from the above advice and
you'll meet my spam filter.

If the data is truly completely random noise, then there is very little
that lossless compression can do. On any individual truly random data set
you might get a lot of compression, a small amount of compression, or even
expansion, depending on what patterns have randomly occurred in the data
set. But there is no current lossless compression algorithm that can take
truly random data and systematically compress it to be smaller than the
original.

If you think you have an algorithm that can do this on truly random data,
you're probably wrong - either your data is has patterns the algorithm can
exploit, or you've simply been lucky with the randomness of your data so
far.

Tim Delaney

[toc] | [prev] | [next] | [standalone]

#58102

From	jonas.thornvall@gmail.com
Date	2013-10-30 12:47 -0700
Message-ID	<2147d528-2c7c-495e-9d32-bb061919b3b1@googlegroups.com>
In reply to	#58097

Den onsdagen den 30:e oktober 2013 kl. 20:35:59 UTC+1 skrev Tim Delaney:
> On 31 October 2013 05:21,  <jonas.t...@gmail.com> wrote:
> 
> I am searching for the program or algorithm that makes the best possible of completly (diffused data/random noise) and wonder what the state of art compression is.
> 
> 
> 
> 
> I understand this is not the correct forum but since i think i have an algorithm that can do this very good, and do not know where to turn for such question i was thinking to start here.
> 
> 
> 
> It is of course lossless compression i am speaking of.
> 
> 
> 
> This is not an appropriate forum for this question. If you know it's an inappropriate forum (as you stated) then do not post the question here. Do a search with your preferred search engine and look up compression on lossless Wikipedia. And read and understand the following link:
> 
> 
> 
> http://www.catb.org/esr/faqs/smart-questions.html
> 
> 
> 
> paying special attention to the following parts:
> 
> 
> 
> http://www.catb.org/esr/faqs/smart-questions.html#forum
> http://www.catb.org/esr/faqs/smart-questions.html#prune
> 
> 
> http://www.catb.org/esr/faqs/smart-questions.html#courtesy
> 
> http://www.catb.org/esr/faqs/smart-questions.html#keepcool
> 
> 
> http://www.catb.org/esr/faqs/smart-questions.html#classic
> 
> 
> 
> If you have *python* code implementing this algorithm and want help, post the parts you want help with (and preferably post the entire algorithm in a repository).
> 
> 
> 
> 
> However, having just seen the following from you in a reply to Mark ("I do not follow instructions, i make them accesible to anyone"), I am not not going to give a second chance - fail to learn from the above advice and you'll meet my spam filter.
> 
> 
> 
> If the data is truly completely random noise, then there is very little that lossless compression can do. On any individual truly random data set you might get a lot of compression, a small amount of compression, or even expansion, depending on what patterns have randomly occurred in the data set. But there is no current lossless compression algorithm that can take truly random data and systematically compress it to be smaller than the original.
> 
> 
> 
> If you think you have an algorithm that can do this on truly random data, you're probably wrong - either your data is has patterns the algorithm can exploit, or you've simply been lucky with the randomness of your data so far.
> 
> 
> 
> Tim Delaney

No i am not wrong.
End of story

[toc] | [prev] | [next] | [standalone]

#58101

From	Modulok <modulok@gmail.com>
Date	2013-10-30 13:46 -0600
Message-ID	<mailman.1835.1383162420.18130.python-list@python.org>
In reply to	#58077

[Multipart message — attachments visible in raw view] — view raw

On Wed, Oct 30, 2013 at 12:21 PM, <jonas.thornvall@gmail.com> wrote:

> I am searching for the program or algorithm that makes the best possible
> of completly (diffused data/random noise) and wonder what the state of art
> compression is.
>
> I understand this is not the correct forum but since i think i have an
> algorithm that can do this very good, and do not know where to turn for
> such question i was thinking to start here.
>
> It is of course lossless compression i am speaking of.
> --
> https://mail.python.org/mailman/listinfo/python-list

>> I am searching for the program or algorithm that makes the best possible
of
>> completly (diffused data/random noise) and wonder what the state of art
>> compression is.

None. If the data to be compressed is truly homogeneous, random noise as you
describe (for example a 100mb file read from cryptographically secure random
bit generator such as /dev/random on *nix systems), the state-of-the-art
lossless compression is zero and will remain that way for the foreseeable
future.

There is no lossless algorithm that will reduce truly random (high entropy)
data by any significant margin. In classical information theory, such an
algorithm can never be invented. See: Kolmogorov complexity

Real world data is rarely completely random. You would have to test various
algorithms on the data set in question. Small things such as non-obvious
statistical clumping can make a big difference in the compression ratio from
one algorithm to another. Data that might look "random", might not actually
be
random in the entropy sense of the word.

>> I understand this is not the correct forum but since i think i have an
>> algorithm that can do this very good, and do not know where to turn for
such
>> question i was thinking to start here.

Not to sound like a downer, but I would wager that the data you're testing
your
algorithm on is not as truly random as you imply or is not a large enough
body
of test data to draw such conclusions from. It's akin to inventing a
perpetual
motion machine or an inertial propulsion engine or any other classically
impossible solutions. (This only applies to truly random data.)

-Modulok-

[toc] | [prev] | [next] | [standalone]

#58103

From	jonas.thornvall@gmail.com
Date	2013-10-30 12:47 -0700
Message-ID	<c73c8aa6-cac4-4f02-ae83-31cea167c172@googlegroups.com>
In reply to	#58101

Den onsdagen den 30:e oktober 2013 kl. 20:46:57 UTC+1 skrev Modulok:
> On Wed, Oct 30, 2013 at 12:21 PM,  <jonas.t...@gmail.com> wrote:
> 
> 
> 
> I am searching for the program or algorithm that makes the best possible of completly (diffused data/random noise) and wonder what the state of art compression is.
> 
> 
> 
> 
> I understand this is not the correct forum but since i think i have an algorithm that can do this very good, and do not know where to turn for such question i was thinking to start here.
> 
> 
> 
> It is of course lossless compression i am speaking of.
> 
> --
> 
> https://mail.python.org/mailman/listinfo/python-list
> 
> 
>  
> 
> >> I am searching for the program or algorithm that makes the best possible of
> >> completly (diffused data/random noise) and wonder what the state of art
> 
> >> compression is.
> 
> 
> None. If the data to be compressed is truly homogeneous, random noise as you
> describe (for example a 100mb file read from cryptographically secure random
> 
> bit generator such as /dev/random on *nix systems), the state-of-the-art
> lossless compression is zero and will remain that way for the foreseeable
> 
> future.
> 
> 
> There is no lossless algorithm that will reduce truly random (high entropy)
> data by any significant margin. In classical information theory, such an
> 
> algorithm can never be invented. See: Kolmogorov complexity
> 
> 
> Real world data is rarely completely random. You would have to test various
> 
> algorithms on the data set in question. Small things such as non-obvious
> statistical clumping can make a big difference in the compression ratio from
> 
> one algorithm to another. Data that might look "random", might not actually be
> random in the entropy sense of the word.
> 
> 
> 
> >> I understand this is not the correct forum but since i think i have an
> >> algorithm that can do this very good, and do not know where to turn for such
> 
> >> question i was thinking to start here.
> 
> 
> Not to sound like a downer, but I would wager that the data you're testing your
> 
> algorithm on is not as truly random as you imply or is not a large enough body
> of test data to draw such conclusions from. It's akin to inventing a perpetual
> 
> motion machine or an inertial propulsion engine or any other classically
> impossible solutions. (This only applies to truly random data.)
> 
> 
> 
> -Modulok-

Well then i have news for you.

[toc] | [prev] | [next] | [standalone]

Page 1 of 4 [1] 2 3 4 Next page →

csiph-web

Algorithm that makes maximum compression of completly diffused data.

Contents

#58077 — Algorithm that makes maximum compression of completly diffused data.

#58083

#58086

#58091

#58092

#58096

#58093

#58099

#58364

#58105

#58127

#58169

#58135

#58140

#58142

#58095

#58097

#58102

#58101

#58103