Path: csiph.com!eternal-september.org!feeder.eternal-september.org!reader01.eternal-september.org!.POSTED!not-for-mail From: Keith Thompson Newsgroups: comp.compression Subject: Re: Oddities of popular archivers Date: Thu, 11 Jul 2019 13:22:54 -0700 Organization: None to speak of Lines: 45 Message-ID: References: <9a6fc13d-5059-4648-8373-98c4396f50f0@googlegroups.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Injection-Info: reader02.eternal-september.org; posting-host="c590ef657eb3eee1a50533751143f1b7"; logging-data="9456"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX1+5/kH3zeE2yRwPmlVujHyv" User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.1 (gnu/linux) Cancel-Lock: sha1:UEXi/D6JVUYJ3O4nXB/NHIxKY2g= sha1:tP3ZCcM61APigfau3RMwX+8zA5g= Xref: csiph.com comp.compression:3820 Elhana writes: > Keith Thompson: > >> What do you mean by "ISO"? > > ISO 8859-X family of encodings. OK. The term "ISO" for such encodings is probably incorrect, and certainly ambiguous. (Microsoft's 8-bit encodings, such as Windows-1252, are sometimes called "ANSI", which is also incorrect.) >> Compressed versions of both are likely to be roughly the same size, >> since the files contain about the same amount of information. > > 15% DEFLATE or 8% LZMA worse result for UTF-8 comparing to a ISO text > is pretty much for "roughly the same" size in my opinion. > > Maybe old primitive algorithms were tuned to English ASCII text, with > customary match lengths, or single-order arithmetic coders, and they > get confused when every symbol in the text is suddenly 2-3 bytes long, > but LZMA is pretty much a modern, sophisticated, state-of-art > algorithm. It's hard to tell just what you're comparing. Are you using multiple encodings of the same text? If not, you could be doing and apples-to-oranges comparison (yes, they're both fruits, but there's not much more you can usefully say about them). If you are, are most of the characters within the 7-bit ASCII set? UTF-8 encodes each character in 1 or more bytes; what is the distribution of byte counts for your input text? What are the numbers for (a) the number of charaters in your input, (b) the number of bytes in each encoding you're using, and (c) the compressed size of each encoding? (Don't assume that I'll be able to say anything useful even given that information, but others might.) -- Keith Thompson (The_Other_Keith) kst-u@mib.org Will write code for food. void Void(void) { Void(); } /* The recursive call of the void */