Re: "C++: Size Matters in Platform Compatibility"

From	David Brown <david.brown@hesbynett.no>
Newsgroups	comp.lang.c++, comp.lang.c
Subject	Re: "C++: Size Matters in Platform Compatibility"
Date	2019-08-19 11:22 +0200
Organization	A noiseless patient Spider
Message-ID	<qjdpod$dni$1@dont-email.me> (permalink)
References	<qj6ung$map$1@dont-email.me> <qj8u95$3if$1@dont-email.me> <qjdj4p$ihi$1@adenine.netfront.net>

Cross-posted to 2 groups.

Show all headers | View raw

On 19/08/2019 09:29, Juha Nieminen wrote:
> In comp.lang.c++ David Brown <david.brown@hesbynett.no> wrote:
>> For character data where plain ASCII won't do, utf-8 is the only sane 
>> choice IMHO.  Other formats - utf-16, utf-32, wchar_t, etc., might be 
>> used internally for dealing with historical API's.
> 
> The problem with UTF-8 is that it's quite complicated to parse and intepret,

It is not much worse than UTF-16.  In both cases, you have to deal with
multi-unit code points.  With UTF-16 you have the added complication of
endianness, which is not an issue with UTF-8.

> and is wasteful in terms of space if the majority of the text is non-ascii
> (quite typically almost all non-ascii characters take at least 3 bytes to
> store as UTF-8.)

No, a fair amount of non-ASCII takes just two bytes.  That applies to
most text written in Latin, Cyrillic, Greek, Hebrew and Arabic alphabets
- including accents, combined letters, and other variants.  Basically
most European scripts are handled by two bytes of UTF-8, matching the
size of UTF-16.

It is only if you have a lot of CJK characters that UTF-16 is more
efficient than UTF-8.

And the reality of document formats is that they usually contain a large
amount of ASCII characters for tags, codes, formatting, markings, etc.
For many formats, these outweigh text characters by orders of magnitude.
 CJK documents are more commonly encoded as UTF-8 rather than UTF-16,
partly because it makes them /smaller/.

Documents are then often packaged together in a lump (along with
pictures or other attachments) and then compressed.  With a large enough
sample, compressed Unicode documents will take similar sizes regardless
of the encoding.

UTF-8 is therefore the most space efficient in practice, and easier to
handle than UTF-16.

> It's the most efficient (spacewise) if the majority of the
> text is ascii, but one still has to be prepared to recover any unicode
> characters from it.
> 
> UTF-16 is simpler to parse (still not completely straightforward, but
> much simpler), and often more space-efficient if the majority of the
> text is non-ascii (such as Japanese).
> 

UTF-16 is a little simpler to parse - if you ignore the issue of
endianness.  But I don't think the details make much difference - with
UTF-8 and UTF-16 you need a decoder function that handles multi-unit codes.

Thread

"C++: Size Matters in Platform Compatibility" Lynn McGuire <lynnmcguire5@gmail.com> - 2019-08-16 14:04 -0500
  Re: "C++: Size Matters in Platform Compatibility" Jorgen Grahn <grahn+nntp@snipabacken.se> - 2019-08-16 20:45 +0000
  Re: "C++: Size Matters in Platform Compatibility" David Brown <david.brown@hesbynett.no> - 2019-08-17 15:08 +0200
    Re: "C++: Size Matters in Platform Compatibility" Geoff <geoff@invalid.invalid> - 2019-08-17 12:43 -0700
      Re: "C++: Size Matters in Platform Compatibility" David Brown <david.brown@hesbynett.no> - 2019-08-18 12:32 +0200
        Re: "C++: Size Matters in Platform Compatibility" Geoff <geoff@invalid.invalid> - 2019-08-18 13:10 -0700
          Re: "C++: Size Matters in Platform Compatibility" Bart <bc@freeuk.com> - 2019-08-18 21:15 +0100
          Re: "C++: Size Matters in Platform Compatibility" Keith Thompson <kst-u@mib.org> - 2019-08-18 14:49 -0700
            Re: "C++: Size Matters in Platform Compatibility" Keith Thompson <kst-u@mib.org> - 2019-08-18 14:51 -0700
          Re: "C++: Size Matters in Platform Compatibility" David Brown <david.brown@hesbynett.no> - 2019-08-19 08:43 +0200
            Re: "C++: Size Matters in Platform Compatibility" "Alf P. Steinbach" <alf.p.steinbach+usenet@gmail.com> - 2019-08-19 09:15 +0200
            Re: "C++: Size Matters in Platform Compatibility" Öö Tiib <ootiib@hot.ee> - 2019-08-19 01:07 -0700
      Re: "C++: Size Matters in Platform Compatibility" Bonita Montero <Bonita.Montero@gmail.com> - 2019-08-21 17:14 +0200
        Re: "C++: Size Matters in Platform Compatibility" Eli the Bearded <*@eli.users.panix.com> - 2019-08-21 18:30 +0000
          Re: "C++: Size Matters in Platform Compatibility" scott@slp53.sl.home (Scott Lurndal) - 2019-08-21 19:05 +0000
          Re: "C++: Size Matters in Platform Compatibility" Bonita Montero <Bonita.Montero@gmail.com> - 2019-08-21 21:25 +0200
          Re: "C++: Size Matters in Platform Compatibility" Keith Thompson <kst-u@mib.org> - 2019-08-21 12:28 -0700
          Re: "C++: Size Matters in Platform Compatibility" Richard Damon <Richard@Damon-Family.org> - 2019-08-22 22:58 -0400
            Re: "C++: Size Matters in Platform Compatibility" Eli the Bearded <*@eli.users.panix.com> - 2019-08-23 19:15 +0000
    Re: "C++: Size Matters in Platform Compatibility" Anton Shepelev <anton.txt@gmail.com> - 2019-08-18 15:59 +0300
      Re: "C++: Size Matters in Platform Compatibility" David Brown <david.brown@hesbynett.no> - 2019-08-18 16:32 +0200
    Re: "C++: Size Matters in Platform Compatibility" Juha Nieminen <nospam@thanks.invalid> - 2019-08-19 07:29 +0000
      Re: "C++: Size Matters in Platform Compatibility" Paavo Helde <myfirstname@osa.pri.ee> - 2019-08-19 12:20 +0300
      Re: "C++: Size Matters in Platform Compatibility" David Brown <david.brown@hesbynett.no> - 2019-08-19 11:22 +0200
        Re: "C++: Size Matters in Platform Compatibility" Richard Damon <Richard@Damon-Family.org> - 2019-08-23 14:14 -0400
          Re: "C++: Size Matters in Platform Compatibility" David Brown <david.brown@hesbynett.no> - 2019-08-24 20:56 +0200
            Re: "C++: Size Matters in Platform Compatibility" Robert Wessel <robertwessel2@yahoo.com> - 2019-08-24 23:43 -0500
              Re: "C++: Size Matters in Platform Compatibility" David Brown <david.brown@hesbynett.no> - 2019-08-25 12:11 +0200
                Re: "C++: Size Matters in Platform Compatibility" Richard Damon <Richard@Damon-Family.org> - 2019-08-25 07:33 -0700
                Re: "C++: Size Matters in Platform Compatibility" David Brown <david.brown@hesbynett.no> - 2019-08-25 20:27 +0200
                Re: "C++: Size Matters in Platform Compatibility" Robert Wessel <robertwessel2@yahoo.com> - 2019-08-25 10:22 -0500
    Re: "C++: Size Matters in Platform Compatibility" Lynn McGuire <lynnmcguire5@gmail.com> - 2019-08-19 15:05 -0500
      Re: "C++: Size Matters in Platform Compatibility" scott@slp53.sl.home (Scott Lurndal) - 2019-08-19 20:17 +0000
        Re: "C++: Size Matters in Platform Compatibility" David Brown <david.brown@hesbynett.no> - 2019-08-20 08:21 +0200
          Re: "C++: Size Matters in Platform Compatibility" Lynn McGuire <lynnmcguire5@gmail.com> - 2019-08-20 14:38 -0500
      Re: "C++: Size Matters in Platform Compatibility" James Kuyper <jameskuyper@alumni.caltech.edu> - 2019-08-21 09:15 -0400
  Re: "C++: Size Matters in Platform Compatibility" Anton Shepelev <anton.txt@gmail.com> - 2019-08-18 15:20 +0300
    Re: "C++: Size Matters in Platform Compatibility" Ben Bacarisse <ben.usenet@bsb.me.uk> - 2019-08-18 14:50 +0100

csiph-web