Re: Oddities of popular archivers

Path	csiph.com!eternal-september.org!feeder.eternal-september.org!reader01.eternal-september.org!.POSTED!not-for-mail
From	Keith Thompson <kst-u@mib.org>
Newsgroups	comp.compression
Subject	Re: Oddities of popular archivers
Date	Sun, 14 Jul 2019 18:38:25 -0700
Organization	None to speak of
Lines	50
Message-ID	<lnk1ck84by.fsf@kst-u.example.com> (permalink)
References	<9a6fc13d-5059-4648-8373-98c4396f50f0@googlegroups.com> <lnv9w9bltd.fsf@kst-u.example.com> <fb678d16-c2cf-4188-8c37-d0b6a9e1ecee@googlegroups.com> <lnmuhkb9sx.fsf@kst-u.example.com> <5b4b0ab8-be2a-42e4-acdb-27b05ecdcdbd@googlegroups.com>
Mime-Version	1.0
Content-Type	text/plain; charset=us-ascii
Injection-Info	reader02.eternal-september.org; posting-host="bc75b27ee51cb20cf648a164e0a09ddd"; logging-data="27723"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX1+QDnPgCf+9cGqGbQT/bbwt"
User-Agent	Gnus/5.13 (Gnus v5.13) Emacs/23.1 (gnu/linux)
Cancel-Lock	sha1:IcE8l+ou3Lt1syvylN/2JnGHhKU= sha1:ROO6TDmneN+oc0wX0QJprZQvvyo=
Xref	csiph.com comp.compression:3823

Show key headers only | View raw

Elhana <tanarriscourge@yahoo.com> writes:
> Keith Thompson:
>
>> Are you using multiple encodings of the same text?
>
> Yes.
>
>> distribution of byte counts for your input text?
>
> A natural language one.
>
>> What are the numbers...
>
> The input text (in UTF-8 form) had 4023k bytes in 2252k
> characters. The DEFLATE algorithm reduced those to 892 or 1061 bytes
> correspondingly.

That's not enough information, or at least is unclear.  And did you
really mean 892 and 1061 bytes, or 892k and 1061k bytes?  (I suggest
quoting exact character/byte counts.  "4023k" is both approximate and
ambiguous; "k" could be either 1000 or 1024.)

Here's my best guess at what you're saying:

You have two files containing different encodings of the same text:
- utf8.txt   is 4023k bytes (averaging about 1.79 bytes per character).
- latin1.txt is 2252k bytes.
All characters have code points in the range 0..255 (otherwise a Latin-1
encoding would not be possible).

Compressing utf8.txt with the DEFLATE algorithm (using what program?)
yields 892k bytes of compressed output.

Compressing latin1.txt with the DEFLATE algorithm yields 1061k bytes of
compressed output.

Since both utf8.txt and latin1.txt contain very nearly the same
information, ideally a compression algorithm *should* yield outputs of
similar size for both input files, but you're seeing a 19% difference,
and you're wondering why.

Is my description correct?

(BTW, I got roughly similar results with a randomly generated chunk of
text and the gzip command.)

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
Will write code for food.
void Void(void) { Void(); } /* The recursive call of the void */

Back to comp.compression | Previous | Next — Previous in thread | Find similar | Unroll thread

Thread

Oddities of popular archivers Elhana <tanarriscourge@yahoo.com> - 2019-07-10 09:21 -0700
  Re: Oddities of popular archivers Keith Thompson <kst-u@mib.org> - 2019-07-10 14:51 -0700
    Re: Oddities of popular archivers Elhana <tanarriscourge@yahoo.com> - 2019-07-10 19:59 -0700
      Re: Oddities of popular archivers Keith Thompson <kst-u@mib.org> - 2019-07-11 13:22 -0700
        Re: Oddities of popular archivers Elhana <tanarriscourge@yahoo.com> - 2019-07-14 08:35 -0700
          Re: Oddities of popular archivers Keith Thompson <kst-u@mib.org> - 2019-07-14 18:38 -0700

csiph-web