Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.compression > #4066

Re: perpetual-compression

Show all headers | View raw

Evert Pot <evert@badgateway.net> writes:
> On Monday, April 27, 2020 at 2:19:56 PM UTC-4, danceswi...@gmail.com wrote:
[...]
>> Using the randbetween function in excel produced the target of 1
>> million "random" integers between 0 - 9, I then ran the stream
>> through a transform, copied the results and pasted them into word to
>> remove spaces and carriage returns and any punctuation's. I then
>> copied it into word pad, saved the file and compressed it using .rar
>> The file compressed down to 246,040 KB. Thinking there was an error
>> I un zipped and restored it without errors, reversed the transform
>> to its original state.

Some of those numbers must be incorrect.  1 million decimal digits
represented in ASCII would be about 977 kilobytes (assuming a "kilobyte"
is 1024 bytes, not 1000 bytes).  If the decimal digits are random, I'd
expect it to compress to about 294 kilobytes *at best*.  The reported
size of "246,040 KB" either indicates an error of a factor of at least
1000, or uses "," as a decimal separator.  Even if it's 246 KB, that's
much better compression than I'd expect.

>> I am not a programmer but worked on this project with a co-worker
>> Kelly D. Crawford Ph.D for eight years day and night. He passed away
>> weeks before it could be finished. I finished, and this was the
>> result. I would like to give him and his family any posthumous
>> credit but I am left thinking this could be an error considering the
>> various steps.
>
> I case you're still curious. If you store a list of numbers in a text
> file there's a TON of repetition. It's not raw binary data, it's ASCII
> data for which almost every byte is going be in the range
> 48-57. Highly compressible even though the numbers themselves might be
> properly random.

To be precise, a byte (using the most common meaning of the term)
is 8 bits and can can hold any of 256 values.  There are only 10
decimal digit values, so in a sequence of decimal digits represented
in ASCII, 246 of the 256 possible byte values are never used.
A very simple compression method, binary-coded decimal, stores each
decimal digit in 4 bits, leaving 6 unused values for each 4-bit unit.
A slightly more sophisticated compression method could use 10 bits
for each b3 decimal digits, using 1000 of 1024 possible values.
Both of these work only for decimal digits.

An ideal compression algorithm could store each decimal digit in
log2(10) bits, or about 3.32 bits per digit, yielding an output about
30.1% the size of the input (plus metadata).  A general-purpose
compression algorithm might come reasonably close to that, though
the best I've managed is about 44% with `xz -9`.  (I don't have a
rar compressor.)  (I used /dev/urandom, not Excel's randbetween,
to generate the input.)

-- 
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for XCOM Labs
void Void(void) { Void(); } /* The recursive call of the void */

Back to comp.compression | Previous | Next — Previous in thread | Find similar | Unroll thread

Thread

Re: perpetual-compression danceswithnumbers@gmail.com - 2020-04-27 11:19 -0700
  Re: perpetual-compression Fibonacci Code <anglikai@gmail.com> - 2021-02-07 06:18 -0800
    Re: perpetual-compression Fibonacci Code <anglikai@gmail.com> - 2021-02-07 06:19 -0800
  Re: perpetual-compression Evert Pot <evert@badgateway.net> - 2023-01-09 11:55 -0800
    Re: perpetual-compression Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2023-01-09 12:26 -0800

csiph-web