Re: Oddities of popular archivers

From	Keith Thompson <kst-u@mib.org>
Newsgroups	comp.compression
Subject	Re: Oddities of popular archivers
Date	2019-07-11 13:22 -0700
Organization	None to speak of
Message-ID	<lnmuhkb9sx.fsf@kst-u.example.com> (permalink)
References	<9a6fc13d-5059-4648-8373-98c4396f50f0@googlegroups.com> <lnv9w9bltd.fsf@kst-u.example.com> <fb678d16-c2cf-4188-8c37-d0b6a9e1ecee@googlegroups.com>

Show all headers | View raw

Elhana <tanarriscourge@yahoo.com> writes:
> Keith Thompson:
>
>> What do you mean by "ISO"?
>
> ISO 8859-X family of encodings.

OK.

The term "ISO" for such encodings is probably incorrect, and certainly
ambiguous.  (Microsoft's 8-bit encodings, such as Windows-1252, are
sometimes called "ANSI", which is also incorrect.)

>> Compressed versions of both are likely to be roughly the same size,
>> since the files contain about the same amount of information.
>
> 15% DEFLATE or 8% LZMA worse result for UTF-8 comparing to a ISO text
> is pretty much for "roughly the same" size in my opinion.
>
> Maybe old primitive algorithms were tuned to English ASCII text, with
> customary match lengths, or single-order arithmetic coders, and they
> get confused when every symbol in the text is suddenly 2-3 bytes long,
> but LZMA is pretty much a modern, sophisticated, state-of-art
> algorithm.

It's hard to tell just what you're comparing.

Are you using multiple encodings of the same text?  If not, you could be
doing and apples-to-oranges comparison (yes, they're both fruits, but
there's not much more you can usefully say about them).

If you are, are most of the characters within the 7-bit ASCII set?
UTF-8 encodes each character in 1 or more bytes; what is the
distribution of byte counts for your input text?  What are the numbers
for (a) the number of charaters in your input, (b) the number of bytes
in each encoding you're using, and (c) the compressed size of each
encoding?

(Don't assume that I'll be able to say anything useful even given that
information, but others might.)

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
Will write code for food.
void Void(void) { Void(); } /* The recursive call of the void */

Thread

Oddities of popular archivers Elhana <tanarriscourge@yahoo.com> - 2019-07-10 09:21 -0700
  Re: Oddities of popular archivers Keith Thompson <kst-u@mib.org> - 2019-07-10 14:51 -0700
    Re: Oddities of popular archivers Elhana <tanarriscourge@yahoo.com> - 2019-07-10 19:59 -0700
      Re: Oddities of popular archivers Keith Thompson <kst-u@mib.org> - 2019-07-11 13:22 -0700
        Re: Oddities of popular archivers Elhana <tanarriscourge@yahoo.com> - 2019-07-14 08:35 -0700
          Re: Oddities of popular archivers Keith Thompson <kst-u@mib.org> - 2019-07-14 18:38 -0700

csiph-web