Re: "C++: Size Matters in Platform Compatibility"

From	Paavo Helde <myfirstname@osa.pri.ee>
Newsgroups	comp.lang.c++, comp.lang.c
Subject	Re: "C++: Size Matters in Platform Compatibility"
Date	2019-08-19 12:20 +0300
Organization	A noiseless patient Spider
Message-ID	<qjdpl9$dk5$1@dont-email.me> (permalink)
References	<qj6ung$map$1@dont-email.me> <qj8u95$3if$1@dont-email.me> <qjdj4p$ihi$1@adenine.netfront.net>

Cross-posted to 2 groups.

Show all headers | View raw

On 19.08.2019 10:29, Juha Nieminen wrote:
> In comp.lang.c++ David Brown <david.brown@hesbynett.no> wrote:
>> For character data where plain ASCII won't do, utf-8 is the only sane
>> choice IMHO.  Other formats - utf-16, utf-32, wchar_t, etc., might be
>> used internally for dealing with historical API's.
>
> The problem with UTF-8 is that it's quite complicated to parse and intepret,

So it's fortunate that one rarely needs to parse and interpret it. Most 
text manipulations like text search, followed by cut, replace, insert 
can be done on UTF-8 just fine without any need to parse it. The most 
demanding task would be to extract a random size-limited piece from a 
large text which requires recognition of Unicode character borders, but 
even this is pretty straightforward as well.

Of course, at some point the text needs to be parsed and interpreted for 
display, but for example the Scintilla text editor control happily 
accepts UTF-8 as its input and displays it just fine, no effort on my 
part needed. In other places one might need to pre-convert UTF-8 to 
platform-specific UTF-16 or UTF-32, but this is also done by library 
routines, nothing special.

> and is wasteful in terms of space if the majority of the text is non-ascii
> (quite typically almost all non-ascii characters take at least 3 bytes to
> store as UTF-8.) It's the most efficient (spacewise) if the majority of the
> text is ascii, but one still has to be prepared to recover any unicode
> characters from it.
>
> UTF-16 is simpler to parse (still not completely straightforward, but
> much simpler), and often more space-efficient if the majority of the
> text is non-ascii (such as Japanese).

This is in theory. In practice I have heard most Japanese texts contain 
a hefty amount of ASCII numbers, punctuation, XML and HTML tags, so the 
space benefits are nowhere so clear.

Thread

"C++: Size Matters in Platform Compatibility" Lynn McGuire <lynnmcguire5@gmail.com> - 2019-08-16 14:04 -0500
  Re: "C++: Size Matters in Platform Compatibility" Jorgen Grahn <grahn+nntp@snipabacken.se> - 2019-08-16 20:45 +0000
  Re: "C++: Size Matters in Platform Compatibility" David Brown <david.brown@hesbynett.no> - 2019-08-17 15:08 +0200
    Re: "C++: Size Matters in Platform Compatibility" Geoff <geoff@invalid.invalid> - 2019-08-17 12:43 -0700
      Re: "C++: Size Matters in Platform Compatibility" David Brown <david.brown@hesbynett.no> - 2019-08-18 12:32 +0200
        Re: "C++: Size Matters in Platform Compatibility" Geoff <geoff@invalid.invalid> - 2019-08-18 13:10 -0700
          Re: "C++: Size Matters in Platform Compatibility" Bart <bc@freeuk.com> - 2019-08-18 21:15 +0100
          Re: "C++: Size Matters in Platform Compatibility" Keith Thompson <kst-u@mib.org> - 2019-08-18 14:49 -0700
            Re: "C++: Size Matters in Platform Compatibility" Keith Thompson <kst-u@mib.org> - 2019-08-18 14:51 -0700
          Re: "C++: Size Matters in Platform Compatibility" David Brown <david.brown@hesbynett.no> - 2019-08-19 08:43 +0200
            Re: "C++: Size Matters in Platform Compatibility" "Alf P. Steinbach" <alf.p.steinbach+usenet@gmail.com> - 2019-08-19 09:15 +0200
            Re: "C++: Size Matters in Platform Compatibility" Öö Tiib <ootiib@hot.ee> - 2019-08-19 01:07 -0700
      Re: "C++: Size Matters in Platform Compatibility" Bonita Montero <Bonita.Montero@gmail.com> - 2019-08-21 17:14 +0200
        Re: "C++: Size Matters in Platform Compatibility" Eli the Bearded <*@eli.users.panix.com> - 2019-08-21 18:30 +0000
          Re: "C++: Size Matters in Platform Compatibility" scott@slp53.sl.home (Scott Lurndal) - 2019-08-21 19:05 +0000
          Re: "C++: Size Matters in Platform Compatibility" Bonita Montero <Bonita.Montero@gmail.com> - 2019-08-21 21:25 +0200
          Re: "C++: Size Matters in Platform Compatibility" Keith Thompson <kst-u@mib.org> - 2019-08-21 12:28 -0700
          Re: "C++: Size Matters in Platform Compatibility" Richard Damon <Richard@Damon-Family.org> - 2019-08-22 22:58 -0400
            Re: "C++: Size Matters in Platform Compatibility" Eli the Bearded <*@eli.users.panix.com> - 2019-08-23 19:15 +0000
    Re: "C++: Size Matters in Platform Compatibility" Anton Shepelev <anton.txt@gmail.com> - 2019-08-18 15:59 +0300
      Re: "C++: Size Matters in Platform Compatibility" David Brown <david.brown@hesbynett.no> - 2019-08-18 16:32 +0200
    Re: "C++: Size Matters in Platform Compatibility" Juha Nieminen <nospam@thanks.invalid> - 2019-08-19 07:29 +0000
      Re: "C++: Size Matters in Platform Compatibility" Paavo Helde <myfirstname@osa.pri.ee> - 2019-08-19 12:20 +0300
      Re: "C++: Size Matters in Platform Compatibility" David Brown <david.brown@hesbynett.no> - 2019-08-19 11:22 +0200
        Re: "C++: Size Matters in Platform Compatibility" Richard Damon <Richard@Damon-Family.org> - 2019-08-23 14:14 -0400
          Re: "C++: Size Matters in Platform Compatibility" David Brown <david.brown@hesbynett.no> - 2019-08-24 20:56 +0200
            Re: "C++: Size Matters in Platform Compatibility" Robert Wessel <robertwessel2@yahoo.com> - 2019-08-24 23:43 -0500
              Re: "C++: Size Matters in Platform Compatibility" David Brown <david.brown@hesbynett.no> - 2019-08-25 12:11 +0200
                Re: "C++: Size Matters in Platform Compatibility" Richard Damon <Richard@Damon-Family.org> - 2019-08-25 07:33 -0700
                Re: "C++: Size Matters in Platform Compatibility" David Brown <david.brown@hesbynett.no> - 2019-08-25 20:27 +0200
                Re: "C++: Size Matters in Platform Compatibility" Robert Wessel <robertwessel2@yahoo.com> - 2019-08-25 10:22 -0500
    Re: "C++: Size Matters in Platform Compatibility" Lynn McGuire <lynnmcguire5@gmail.com> - 2019-08-19 15:05 -0500
      Re: "C++: Size Matters in Platform Compatibility" scott@slp53.sl.home (Scott Lurndal) - 2019-08-19 20:17 +0000
        Re: "C++: Size Matters in Platform Compatibility" David Brown <david.brown@hesbynett.no> - 2019-08-20 08:21 +0200
          Re: "C++: Size Matters in Platform Compatibility" Lynn McGuire <lynnmcguire5@gmail.com> - 2019-08-20 14:38 -0500
      Re: "C++: Size Matters in Platform Compatibility" James Kuyper <jameskuyper@alumni.caltech.edu> - 2019-08-21 09:15 -0400
  Re: "C++: Size Matters in Platform Compatibility" Anton Shepelev <anton.txt@gmail.com> - 2019-08-18 15:20 +0300
    Re: "C++: Size Matters in Platform Compatibility" Ben Bacarisse <ben.usenet@bsb.me.uk> - 2019-08-18 14:50 +0100

csiph-web