Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c++ > #82585

Re: Character encoding conversion in wide string literals

Show all headers | View raw

On 07/12/2021 18:59, Alf P. Steinbach wrote:
> On 7 Dec 2021 17:48, James Kuyper wrote:
>> On 12/7/21 10:44 AM, Alf P. Steinbach wrote:
>> ...
>>> aware: the main problem is that the C and C++ standards do not conform
>>> to reality in their requirement that a `wchar_t` value should suffice to
>>> encode all possible code points in the wide character set.
>>
>> The purpose of the C and C++ standards is prescriptive, not descriptive.
>> It's therefore missing the point to criticize them for not conforming to
>> reality. Rather, you should say that some popular implementations fail
>> to conform to the standards.
> 
> No, in this case it's the standard's fault. They failed to standardize
> existing practice and instead standardized a completely unreasonable
> requirement, given that 16-bit `wchar_t` was established as the API
> foundation in the most widely used OS on the platform, something that
> could not easily be changed. In particular this was the C standard
> committee: their choice here was as reasonable and practical as their
> choice of not supporting pointers outside of original (sub-) array.
> 
> It was idiotic. It was simple blunders. But inn both cases, as I recall,
> they tried to cover up the blunder by writing a rationale; they took the
> blunders to heart and made them into great obstacles, to not lose face.
> 
> 
>>> In Windows wide text is UTF-16, with 16-bit `wchar_t`. Which means that
>>> some emojis etc. that appear as a single character and constitute one
>>> 21-bit code point, can become a pair of two `wchar_t` values, an UTF-16
>>> "surrogate pair".
>>
>> The C++ standard explicitly addresses that point, though the C standard
>> does not.
> 
> Happy to hear that but some more specific information would be welcome.
> 

My understanding is that at that time, the Windows wide character set
was UCS2, not UTF-16.  Thus a 16-bit wchar_t was sufficient to encode
all wide characters.

It turned out that UCS2 was a dead-end, and now UTF-16 is a hack-job
that combines all the disadvantages of UTF-8 with all the disadvantages
of UTF-32, and none of the benefits of either.  We can't blame MS for
going for UCS2 - they were early adopters and Unicode was 16-bit, so it
was a good choice at the time.  They, and therefore their users, were
unlucky (along with Java, QT, Python, and no doubt others).  Changing is
not easy - you have to make everything UTF-8 and yet still support a
horrible mix of wchar_t, char16_t, UCS2, and UTF-16 for legacy.

But as far as I can see, the C and C++ standards were fine with 16-bit
wchar_t when they were written.  I have heard, but have no reference or
source, that the inclusion of 16-bit wchar_t in the standards was
promoted by MS in the first place.

Back to comp.lang.c++ | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Character encoding conversion in wide string literals Juha Nieminen <nospam@thanks.invalid> - 2021-12-07 11:37 +0000
  Re: Character encoding conversion in wide string literals "Alf P. Steinbach" <alf.p.steinbach@gmail.com> - 2021-12-07 16:44 +0100
    Re: Character encoding conversion in wide string literals James Kuyper <jameskuyper@alumni.caltech.edu> - 2021-12-07 11:48 -0500
      Re: Character encoding conversion in wide string literals "Alf P. Steinbach" <alf.p.steinbach@gmail.com> - 2021-12-07 18:59 +0100
        Re: Character encoding conversion in wide string literals James Kuyper <jameskuyper@alumni.caltech.edu> - 2021-12-07 13:55 -0500
        Re: Character encoding conversion in wide string literals Öö Tiib <ootiib@hot.ee> - 2021-12-07 16:39 -0800
        Re: Character encoding conversion in wide string literals David Brown <david.brown@hesbynett.no> - 2021-12-08 09:18 +0100
    Re: Character encoding conversion in wide string literals Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-12-07 09:41 -0800
      Re: Character encoding conversion in wide string literals "Alf P. Steinbach" <alf.p.steinbach@gmail.com> - 2021-12-07 19:07 +0100
        Re: Character encoding conversion in wide string literals Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-12-07 12:21 -0800
      Re: Character encoding conversion in wide string literals Paavo Helde <eesnimi@osa.pri.ee> - 2021-12-07 20:32 +0200
        Re: Character encoding conversion in wide string literals James Kuyper <jameskuyper@alumni.caltech.edu> - 2021-12-07 13:55 -0500
        Re: Character encoding conversion in wide string literals Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-12-07 12:26 -0800
  Re: Character encoding conversion in wide string literals James Kuyper <jameskuyper@alumni.caltech.edu> - 2021-12-07 11:38 -0500
    Re: Character encoding conversion in wide string literals Manfred <noname@add.invalid> - 2021-12-07 19:07 +0100

csiph-web