Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c++ > #67927

Re: "C++: Size Matters in Platform Compatibility"

Cross-posted to 2 groups.

Show all headers | View raw

On 23/08/2019 20:14, Richard Damon wrote:
> 
> UTF-16 takes
> 2 bytes for values 0-0xFFFF (Unicode Base Plane)
> 4 bytes for values 0x10000 - 0x10FFFF (All of current Unicode)
> 
> it CAN'T handle higher code points without some other extension which
> would get ugly. That is why Unicode has currently defined that 0x10FFFF
> will be the highest code point. I suspect at some point there will be
> pressure to change that.

It is true that UTF-16 is limited to 1,114,111 characters.  (I didn't 
know that until recently.)  However, Unicode says: 
<https://www.unicode.org/faq/utf_bom.html#utf16-6>

"""
Q: Will UTF-16 ever be extended to more than a million characters?

A: No. Both Unicode and ISO 10646 have policies in place that formally 
limit future code assignment to the integer range that can be expressed 
with current UTF-16 (0 to 1,114,111). Even if other encoding forms (i.e. 
other UTFs) can represent larger intergers, these policies mean that all 
encoding forms will always represent the same set of characters. Over a 
million possible codes is far more than enough for the goal of Unicode 
of encoding characters, not glyphs. Unicode is not designed to encode 
arbitrary data. If you wanted, for example, to give each “instance of a 
character on paper throughout history” its own code, you might need 
trillions or quadrillions of such codes; noble as this effort might be, 
you would not use Unicode for such an encoding. [AF]
"""


> 
> UTF-8 takes
> 1 byte for values 0 - 0x7F (basically ASCII)
> 2 bytes for values 0x80 - 0xFFF (most western languages)
> 3 bytes for values 0x1000 - 0xFFFF (Unicode base plane, much of this is
> the CJK group)
> 4 bytes for values 0x10000 - 0x10FFFF, but can actually get to 0x1FFFF
> if they extend unicode to higher code points)
> 
> in the extension (and earlier definition)
> 5 bytes for values 0x200000 - 0x3FFFFFF
> 6 bytes for values 0x400000 - 0x7FFFFFFF
> 
> UTF-16 is more compact only for code that has more characters in the
> range 0x1000 - 0xFFFF than in the range 0-0x7F. This MIGHT hold for
> Chinese/Japanese/Korean documents that have minimal markup in them. It
> largely also requires that CJK document has a lot of 'basic' CJK
> characters, as one big reason for extending the Unicode standard past 16
> bits was that there were a LOT more characters needed then expected to
> handle them.

Yes.  The reality is that there are few documents that fall into this 
category, thus it is easier to stick to UTF-8 for all documents.

> 
> It turns out that UTF-16 was a mistake. It came about because the first
> attempts at Unicode thought it could be a 16 bit characters set, and
> thus would be UCS-2. It only lives on really because Microsoft decided
> to use UCS-2 in windows instead of UTF-8, which admittedly might not
> have been a bad decision at the time. When UCS-2 had to become UTF-16 to
> handle the full character set, it gets the worse of both sides, it still
> is a multi-code character set (but maybe a lot of code can get away with
> ignoring it and just not work right with 'esoteric' characters), is less
> compact in the most common cases, and has a byte order issue for byte
> streams (which most data streams are).
> 

Agreed.

Back to comp.lang.c++ | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

"C++: Size Matters in Platform Compatibility" Lynn McGuire <lynnmcguire5@gmail.com> - 2019-08-16 14:04 -0500
  Re: "C++: Size Matters in Platform Compatibility" Jorgen Grahn <grahn+nntp@snipabacken.se> - 2019-08-16 20:45 +0000
  Re: "C++: Size Matters in Platform Compatibility" David Brown <david.brown@hesbynett.no> - 2019-08-17 15:08 +0200
    Re: "C++: Size Matters in Platform Compatibility" Geoff <geoff@invalid.invalid> - 2019-08-17 12:43 -0700
      Re: "C++: Size Matters in Platform Compatibility" David Brown <david.brown@hesbynett.no> - 2019-08-18 12:32 +0200
        Re: "C++: Size Matters in Platform Compatibility" Geoff <geoff@invalid.invalid> - 2019-08-18 13:10 -0700
          Re: "C++: Size Matters in Platform Compatibility" Bart <bc@freeuk.com> - 2019-08-18 21:15 +0100
          Re: "C++: Size Matters in Platform Compatibility" Keith Thompson <kst-u@mib.org> - 2019-08-18 14:49 -0700
            Re: "C++: Size Matters in Platform Compatibility" Keith Thompson <kst-u@mib.org> - 2019-08-18 14:51 -0700
          Re: "C++: Size Matters in Platform Compatibility" David Brown <david.brown@hesbynett.no> - 2019-08-19 08:43 +0200
            Re: "C++: Size Matters in Platform Compatibility" "Alf P. Steinbach" <alf.p.steinbach+usenet@gmail.com> - 2019-08-19 09:15 +0200
            Re: "C++: Size Matters in Platform Compatibility" Öö Tiib <ootiib@hot.ee> - 2019-08-19 01:07 -0700
      Re: "C++: Size Matters in Platform Compatibility" Bonita Montero <Bonita.Montero@gmail.com> - 2019-08-21 17:14 +0200
        Re: "C++: Size Matters in Platform Compatibility" Eli the Bearded <*@eli.users.panix.com> - 2019-08-21 18:30 +0000
          Re: "C++: Size Matters in Platform Compatibility" scott@slp53.sl.home (Scott Lurndal) - 2019-08-21 19:05 +0000
          Re: "C++: Size Matters in Platform Compatibility" Bonita Montero <Bonita.Montero@gmail.com> - 2019-08-21 21:25 +0200
          Re: "C++: Size Matters in Platform Compatibility" Keith Thompson <kst-u@mib.org> - 2019-08-21 12:28 -0700
          Re: "C++: Size Matters in Platform Compatibility" Richard Damon <Richard@Damon-Family.org> - 2019-08-22 22:58 -0400
            Re: "C++: Size Matters in Platform Compatibility" Eli the Bearded <*@eli.users.panix.com> - 2019-08-23 19:15 +0000
    Re: "C++: Size Matters in Platform Compatibility" Anton Shepelev <anton.txt@gmail.com> - 2019-08-18 15:59 +0300
      Re: "C++: Size Matters in Platform Compatibility" David Brown <david.brown@hesbynett.no> - 2019-08-18 16:32 +0200
    Re: "C++: Size Matters in Platform Compatibility" Juha Nieminen <nospam@thanks.invalid> - 2019-08-19 07:29 +0000
      Re: "C++: Size Matters in Platform Compatibility" Paavo Helde <myfirstname@osa.pri.ee> - 2019-08-19 12:20 +0300
      Re: "C++: Size Matters in Platform Compatibility" David Brown <david.brown@hesbynett.no> - 2019-08-19 11:22 +0200
        Re: "C++: Size Matters in Platform Compatibility" Richard Damon <Richard@Damon-Family.org> - 2019-08-23 14:14 -0400
          Re: "C++: Size Matters in Platform Compatibility" David Brown <david.brown@hesbynett.no> - 2019-08-24 20:56 +0200
            Re: "C++: Size Matters in Platform Compatibility" Robert Wessel <robertwessel2@yahoo.com> - 2019-08-24 23:43 -0500
              Re: "C++: Size Matters in Platform Compatibility" David Brown <david.brown@hesbynett.no> - 2019-08-25 12:11 +0200
                Re: "C++: Size Matters in Platform Compatibility" Richard Damon <Richard@Damon-Family.org> - 2019-08-25 07:33 -0700
                Re: "C++: Size Matters in Platform Compatibility" David Brown <david.brown@hesbynett.no> - 2019-08-25 20:27 +0200
                Re: "C++: Size Matters in Platform Compatibility" Robert Wessel <robertwessel2@yahoo.com> - 2019-08-25 10:22 -0500
    Re: "C++: Size Matters in Platform Compatibility" Lynn McGuire <lynnmcguire5@gmail.com> - 2019-08-19 15:05 -0500
      Re: "C++: Size Matters in Platform Compatibility" scott@slp53.sl.home (Scott Lurndal) - 2019-08-19 20:17 +0000
        Re: "C++: Size Matters in Platform Compatibility" David Brown <david.brown@hesbynett.no> - 2019-08-20 08:21 +0200
          Re: "C++: Size Matters in Platform Compatibility" Lynn McGuire <lynnmcguire5@gmail.com> - 2019-08-20 14:38 -0500
      Re: "C++: Size Matters in Platform Compatibility" James Kuyper <jameskuyper@alumni.caltech.edu> - 2019-08-21 09:15 -0400
  Re: "C++: Size Matters in Platform Compatibility" Anton Shepelev <anton.txt@gmail.com> - 2019-08-18 15:20 +0300
    Re: "C++: Size Matters in Platform Compatibility" Ben Bacarisse <ben.usenet@bsb.me.uk> - 2019-08-18 14:50 +0100

csiph-web