Path: csiph.com!eternal-september.org!feeder.eternal-september.org!reader01.eternal-september.org!.POSTED!not-for-mail
From: Keith Thompson <kst-u@mib.org>
Newsgroups: comp.compression
Subject: Re: Oddities of popular archivers
Date: Sun, 14 Jul 2019 18:38:25 -0700
Organization: None to speak of
Lines: 50
Message-ID: <lnk1ck84by.fsf@kst-u.example.com>
References: <9a6fc13d-5059-4648-8373-98c4396f50f0@googlegroups.com> <lnv9w9bltd.fsf@kst-u.example.com> <fb678d16-c2cf-4188-8c37-d0b6a9e1ecee@googlegroups.com> <lnmuhkb9sx.fsf@kst-u.example.com> <5b4b0ab8-be2a-42e4-acdb-27b05ecdcdbd@googlegroups.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Injection-Info: reader02.eternal-september.org; posting-host="bc75b27ee51cb20cf648a164e0a09ddd"; logging-data="27723"; mail-complaints-to="abuse@eternal-september.org";	posting-account="U2FsdGVkX1+QDnPgCf+9cGqGbQT/bbwt"
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.1 (gnu/linux)
Cancel-Lock: sha1:IcE8l+ou3Lt1syvylN/2JnGHhKU= sha1:ROO6TDmneN+oc0wX0QJprZQvvyo=
Xref: csiph.com comp.compression:3823

Elhana <tanarriscourge@yahoo.com> writes:
> Keith Thompson:
>
>> Are you using multiple encodings of the same text?
>
> Yes.
>
>> distribution of byte counts for your input text?
>
> A natural language one.
>
>> What are the numbers...
>
> The input text (in UTF-8 form) had 4023k bytes in 2252k
> characters. The DEFLATE algorithm reduced those to 892 or 1061 bytes
> correspondingly.

That's not enough information, or at least is unclear.  And did you
really mean 892 and 1061 bytes, or 892k and 1061k bytes?  (I suggest
quoting exact character/byte counts.  "4023k" is both approximate and
ambiguous; "k" could be either 1000 or 1024.)

Here's my best guess at what you're saying:

You have two files containing different encodings of the same text:
- utf8.txt   is 4023k bytes (averaging about 1.79 bytes per character).
- latin1.txt is 2252k bytes.
All characters have code points in the range 0..255 (otherwise a Latin-1
encoding would not be possible).

Compressing utf8.txt with the DEFLATE algorithm (using what program?)
yields 892k bytes of compressed output.

Compressing latin1.txt with the DEFLATE algorithm yields 1061k bytes of
compressed output.

Since both utf8.txt and latin1.txt contain very nearly the same
information, ideally a compression algorithm *should* yield outputs of
similar size for both input files, but you're seeing a 19% difference,
and you're wondering why.

Is my description correct?

(BTW, I got roughly similar results with a randomly generated chunk of
text and the gzip command.)

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
Will write code for food.
void Void(void) { Void(); } /* The recursive call of the void */