Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c++ > #67678 > unrolled thread

How to get standard paths?!?

Back to article view | Back to comp.lang.c++

Contents

  How to get standard paths?!? Szyk Cech <szykcech@spoko.pl> - 2019-08-17 13:58 +0200
    Re: How to get standard paths?!? "Alf P. Steinbach" <alf.p.steinbach+usenet@gmail.com> - 2019-08-17 14:02 +0200
      Re: How to get standard paths?!? Bonita Montero <Bonita.Montero@gmail.com> - 2019-08-17 14:05 +0200
        Re: How to get standard paths?!? David Brown <david.brown@hesbynett.no> - 2019-08-17 15:11 +0200
          Re: How to get standard paths?!? Bonita Montero <Bonita.Montero@gmail.com> - 2019-08-17 15:59 +0200
            Re: How to get standard paths?!? David Brown <david.brown@hesbynett.no> - 2019-08-17 16:20 +0200
              Re: How to get standard paths?!? Bonita Montero <Bonita.Montero@gmail.com> - 2019-08-17 16:28 +0200
                Re: How to get standard paths?!? Ralf Goertz <me@myprovider.invalid> - 2019-08-17 16:37 +0200
                  Re: How to get standard paths?!? Bonita Montero <Bonita.Montero@gmail.com> - 2019-08-17 16:54 +0200
                    Re: How to get standard paths?!? Paavo Helde <myfirstname@osa.pri.ee> - 2019-08-17 18:37 +0300
                      Re: How to get standard paths?!? Bonita Montero <Bonita.Montero@gmail.com> - 2019-08-17 17:54 +0200
                        Re: How to get standard paths?!? Paavo Helde <myfirstname@osa.pri.ee> - 2019-08-17 19:55 +0300
                          Re: How to get standard paths?!? Bonita Montero <Bonita.Montero@gmail.com> - 2019-08-17 19:15 +0200
                            Re: How to get standard paths?!? Paavo Helde <myfirstname@osa.pri.ee> - 2019-08-17 20:39 +0300
                              Re: How to get standard paths?!? Bonita Montero <Bonita.Montero@gmail.com> - 2019-08-17 19:44 +0200
                                Re: How to get standard paths?!? David Brown <david.brown@hesbynett.no> - 2019-08-17 20:10 +0200
                                  Re: How to get standard paths?!? Bonita Montero <Bonita.Montero@gmail.com> - 2019-08-18 08:58 +0200
                                    Re: How to get standard paths?!? Bart <bc@freeuk.com> - 2019-08-18 10:16 +0100
                                      Re: How to get standard paths?!? Bonita Montero <Bonita.Montero@gmail.com> - 2019-08-18 11:27 +0200
                              Re: How to get standard paths?!? Bo Persson <bo@bo-persson.se> - 2019-08-17 21:07 +0200
                                Re: How to get standard paths?!? David Brown <david.brown@hesbynett.no> - 2019-08-17 21:41 +0200
                    Re: How to get standard paths?!? David Brown <david.brown@hesbynett.no> - 2019-08-17 17:40 +0200
                      Re: How to get standard paths?!? Bonita Montero <Bonita.Montero@gmail.com> - 2019-08-17 17:52 +0200
                        Re: How to get standard paths?!? Bonita Montero <Bonita.Montero@gmail.com> - 2019-08-17 18:00 +0200
                          Re: How to get standard paths?!? David Brown <david.brown@hesbynett.no> - 2019-08-17 18:01 +0200
                            Re: How to get standard paths?!? Bonita Montero <Bonita.Montero@gmail.com> - 2019-08-17 18:26 +0200
                              Re: How to get standard paths?!? David Brown <david.brown@hesbynett.no> - 2019-08-17 20:20 +0200
                                Re: How to get standard paths?!? Keith Thompson <kst-u@mib.org> - 2019-08-17 16:33 -0700
                                  Re: How to get standard paths?!? David Brown <david.brown@hesbynett.no> - 2019-08-18 11:34 +0200
                                Re: How to get standard paths?!? Bonita Montero <Bonita.Montero@gmail.com> - 2019-08-18 08:48 +0200
                                  Re: How to get standard paths?!? David Brown <david.brown@hesbynett.no> - 2019-08-18 11:34 +0200
                                    Re: How to get standard paths?!? Bonita Montero <Bonita.Montero@gmail.com> - 2019-08-18 12:57 +0200
                                      Re: How to get standard paths?!? Bart <bc@freeuk.com> - 2019-08-18 13:07 +0100
                                        Re: How to get standard paths?!? Bonita Montero <Bonita.Montero@gmail.com> - 2019-08-18 15:49 +0200
                                        Re: How to get standard paths?!? Bo Persson <bo@bo-persson.se> - 2019-08-18 20:33 +0200
                                          Re: How to get standard paths?!? Bart <bc@freeuk.com> - 2019-08-18 20:51 +0100
                                            Re: How to get standard paths?!? Bo Persson <bo@bo-persson.se> - 2019-08-19 00:04 +0200
                                              Re: How to get standard paths?!? Tim Rentsch <tr.17687@z991.linuxsc.com> - 2019-08-25 05:35 -0700
                                                Re: How to get standard paths?!? Bonita Montero <Bonita.Montero@gmail.com> - 2019-08-25 15:02 +0200
                                                  Re: How to get standard paths?!? Tim Rentsch <tr.17687@z991.linuxsc.com> - 2019-08-25 06:38 -0700
                                                    Re: How to get standard paths?!? Bonita Montero <Bonita.Montero@gmail.com> - 2019-08-25 15:46 +0200
                                                      Re: How to get standard paths?!? Bonita Montero <Bonita.Montero@gmail.com> - 2019-08-25 15:54 +0200
                                            Re: How to get standard paths?!? David Brown <david.brown@hesbynett.no> - 2019-08-19 08:52 +0200
                                              Re: How to get standard paths?!? "Alf P. Steinbach" <alf.p.steinbach+usenet@gmail.com> - 2019-08-19 09:32 +0200
                                                Re: How to get standard paths?!? Tim Rentsch <tr.17687@z991.linuxsc.com> - 2019-08-22 12:37 -0700
                                              Re: How to get standard paths?!? Bonita Montero <Bonita.Montero@gmail.com> - 2019-08-19 10:26 +0200
                        Re: How to get standard paths?!? David Brown <david.brown@hesbynett.no> - 2019-08-17 18:00 +0200
                          Re: How to get standard paths?!? Bonita Montero <Bonita.Montero@gmail.com> - 2019-08-17 18:06 +0200
                            Re: How to get standard paths?!? David Brown <david.brown@hesbynett.no> - 2019-08-17 20:24 +0200
                              Re: How to get standard paths?!? Bonita Montero <Bonita.Montero@gmail.com> - 2019-08-18 08:49 +0200
                                Re: How to get standard paths?!? David Brown <david.brown@hesbynett.no> - 2019-08-18 11:46 +0200
                                  Re: How to get standard paths?!? Bonita Montero <Bonita.Montero@gmail.com> - 2019-08-18 12:59 +0200
                        Re: How to get standard paths?!? scott@slp53.sl.home (Scott Lurndal) - 2019-08-18 15:40 +0000
                          Re: How to get standard paths?!? Bonita Montero <Bonita.Montero@gmail.com> - 2019-08-18 17:44 +0200
                    Re: How to get standard paths?!? scott@slp53.sl.home (Scott Lurndal) - 2019-08-18 15:38 +0000
                      Re: How to get standard paths?!? Bonita Montero <Bonita.Montero@gmail.com> - 2019-08-18 17:46 +0200
                Re: How to get standard paths?!? David Brown <david.brown@hesbynett.no> - 2019-08-17 17:56 +0200
                  Re: How to get standard paths?!? Bonita Montero <Bonita.Montero@gmail.com> - 2019-08-17 18:05 +0200
                    Re: How to get standard paths?!? Hergen Lehmann <hlehmann.expires.5-11@snafu.de> - 2019-08-17 19:47 +0200
                      Re: How to get standard paths?!? David Brown <david.brown@hesbynett.no> - 2019-08-17 20:46 +0200
                      Re: How to get standard paths?!? Bonita Montero <Bonita.Montero@gmail.com> - 2019-08-18 08:54 +0200
                        Re: How to get standard paths?!? David Brown <david.brown@hesbynett.no> - 2019-08-18 12:07 +0200
                          Re: How to get standard paths?!? Bonita Montero <Bonita.Montero@gmail.com> - 2019-08-18 13:05 +0200
                            Re: How to get standard paths?!? David Brown <david.brown@hesbynett.no> - 2019-08-18 14:48 +0200
                              Re: How to get standard paths?!? Bonita Montero <Bonita.Montero@gmail.com> - 2019-08-18 15:51 +0200
                                Re: How to get standard paths?!? David Brown <david.brown@hesbynett.no> - 2019-08-18 16:43 +0200
                                  Re: How to get standard paths?!? Bonita Montero <Bonita.Montero@gmail.com> - 2019-08-18 17:02 +0200
                    Re: How to get standard paths?!? David Brown <david.brown@hesbynett.no> - 2019-08-17 20:41 +0200
                      Re: How to get standard paths?!? Keith Thompson <kst-u@mib.org> - 2019-08-17 16:46 -0700
                        Re: How to get standard paths?!? David Brown <david.brown@hesbynett.no> - 2019-08-18 12:22 +0200
                      Re: How to get standard paths?!? Robert Wessel <robertwessel2@yahoo.com> - 2019-08-17 22:57 -0500
                      Re: How to get standard paths?!? Bonita Montero <Bonita.Montero@gmail.com> - 2019-08-18 08:57 +0200
                        Re: How to get standard paths?!? David Brown <david.brown@hesbynett.no> - 2019-08-18 12:24 +0200
                          Re: How to get standard paths?!? Bonita Montero <Bonita.Montero@gmail.com> - 2019-08-18 13:07 +0200
                            Re: How to get standard paths?!? Bonita Montero <Bonita.Montero@gmail.com> - 2019-08-18 13:13 +0200
            Re: How to get standard paths?!? Tim Rentsch <tr.17687@z991.linuxsc.com> - 2019-08-17 11:22 -0700
              Re: How to get standard paths?!? Bo Persson <bo@bo-persson.se> - 2019-08-17 21:21 +0200
                Re: How to get standard paths?!? Tim Rentsch <tr.17687@z991.linuxsc.com> - 2019-08-17 22:25 -0700
                Re: How to get standard paths?!? Robert Wessel <robertwessel2@yahoo.com> - 2019-08-19 01:19 -0500
                  Re: How to get standard paths?!? Bonita Montero <Bonita.Montero@gmail.com> - 2019-08-19 08:25 +0200
    Re: How to get standard paths?!? Bonita Montero <Bonita.Montero@gmail.com> - 2019-08-17 14:02 +0200
    Re: How to get standard paths?!? David Brown <david.brown@hesbynett.no> - 2019-08-17 15:13 +0200
    Re: How to get standard paths?!? Sam <sam@email-scan.com> - 2019-08-17 09:21 -0400
    Re: How to get standard paths?!? Jorgen Grahn <grahn+nntp@snipabacken.se> - 2019-08-19 20:57 +0000

Page 4 of 5 — ← Prev page 1 2 3 [4] 5  Next page →

#67730

> But the Windows implementation is technically broken, as the programmer 
> can neither rely on the assumption, that any given character can be 
> stored into a wchar_t, nor that iterating over a string will correctly 
> return the stored characters, nor that insert/erase/substring operations 
> will cut the string properly - which completely defies the purpose of 
> wchar_t.

That's how UTF-16 works and the Unicode-standard recommends UTF-16 for
certain circumstances.

[toc] | [prev] | [next] | [standalone]

#67739

On 18/08/2019 08:54, Bonita Montero wrote:
>> But the Windows implementation is technically broken, as the 
>> programmer can neither rely on the assumption, that any given 
>> character can be stored into a wchar_t, nor that iterating over a 
>> string will correctly return the stored characters, nor that 
>> insert/erase/substring operations will cut the string properly - which 
>> completely defies the purpose of wchar_t.
> 
> That's how UTF-16 works and the Unicode-standard recommends UTF-16 for
> certain circumstances.
> 

I know how UTF-16 works.  And UTF-16 is only recommended for documents 
written primarily in the BMP code plane characters between 0x0800 and 
0xffff (i.e., non-European scripts, not including CJK) as it is more 
efficient than UTF-8 or UTF-32.  This recommendation is roundly ignored, 
for good reasons.  While UTF-16 is still used internally on Windows, and 
with some languages (like Java) and libraries (like QT), it is almost 
completely negligible as an encoding in documents or data interchange. 
If you see any reference recommending its usage, check the data - the 
page is probably from last century.

[toc] | [prev] | [next] | [standalone]

#67745

> I know how UTF-16 works.  And UTF-16 is only recommended for documents 
> written primarily in the BMP code plane characters between 0x0800 and 
> 0xffff (i.e., non-European scripts, not including CJK) as it is more 
> efficient than UTF-8 or UTF-32.

There's nothing about that in den Unicode-standard. The Uncicode-stan-
dard simply says that UTF-16 can cover every Unicode codepoint via
surrogate-pairs aund that UTF-16 is _optimized_ for BMP, but not only
suitable for this. It says that is preferred if you ned a balance of
storage-size and access-speed.

> While UTF-16 is still used internally on Windows, and 
> with some languages (like Java) and libraries (like QT), it is almost 
> completely negligible as an encoding in documents or data interchange.

That might be practical reasons about which one can argue excellently
but that doesn't affect that UTF-16 is simply Unicode-compliant.

[toc] | [prev] | [next] | [standalone]

#67752

On 18/08/2019 13:05, Bonita Montero wrote:
>> I know how UTF-16 works.  And UTF-16 is only recommended for documents 
>> written primarily in the BMP code plane characters between 0x0800 and 
>> 0xffff (i.e., non-European scripts, not including CJK) as it is more 
>> efficient than UTF-8 or UTF-32.
> 
> There's nothing about that in den Unicode-standard. The Uncicode-stan-
> dard simply says that UTF-16 can cover every Unicode codepoint via
> surrogate-pairs aund that UTF-16 is _optimized_ for BMP, but not only
> suitable for this. It says that is preferred if you ned a balance of
> storage-size and access-speed.

UTF-16 works fine, but is a poor solution in almost every case.  As I 
said previously, it combines the disadvantages of UTF-8 with the 
disadvantages of UTF-32, giving the benefits of neither.  People know 
this - that is why it is not used in any significance except in places 
where it is hard to avoid (Windows programming), or for dealing with 
legacy code.

There was a time, long ago, when Unicode was young, when 16-bit was 
enough for almost everything, when saving a few bytes was important, 
when HTML and XML were not common, and when transparent compression was 
rare.  In those days, UTF-16 made sense for some uses.

Given a free choice without worrying about compatibility with old code 
or interfaces, no one would recommend or choose UTF-16 now.

> 
>> While UTF-16 is still used internally on Windows, and with some 
>> languages (like Java) and libraries (like QT), it is almost completely 
>> negligible as an encoding in documents or data interchange.
> 
> That might be practical reasons about which one can argue excellently
> but that doesn't affect that UTF-16 is simply Unicode-compliant.

No one has doubted that UTF-16 is a valid Unicode encoding.  That has 
never been in question.

As seems to happen so often, you have totally misunderstood the issue 
when you have been shown wrong.  I wonder if it is intentional.

[toc] | [prev] | [next] | [standalone]

#67756

> UTF-16 works fine, but is a poor solution in almost every case.
> As I  said previously, it combines the disadvantages of UTF-8
> with the  disadvantages of UTF-32, giving the benefits of neither.

That's just your taste. But someone else may argue like the Unicode
-standard, that UTF-16 is a good balance between size and performance.

[toc] | [prev] | [next] | [standalone]

#67758

On 18/08/2019 15:51, Bonita Montero wrote:
>> UTF-16 works fine, but is a poor solution in almost every case.
>> As I  said previously, it combines the disadvantages of UTF-8
>> with the  disadvantages of UTF-32, giving the benefits of neither.
> 
> That's just your taste. But someone else may argue like the Unicode
> -standard, that UTF-16 is a good balance between size and performance.
> 

Sorry, you /do/ know that UTF-8 and UTF-32 are also encodings for 
Unicode, just like UTF-16?  It is not clear, but it looks a little like 
you think UTF-16 /is/ Unicode and that I am suggesting something other 
than Unicode.  I hope you can clarify if you are misunderstanding this 
or not.

As for taste, something like 94% of documents on the internet are 
encoded with UTF-8.  The rest is mostly 8-bit encodings like ISO-8859-1. 
  UTF-16 documents are almost invariably bigger than UTF-8 documents 
(since most text-based formats involve keys, tags, etc., that are 
single-byte characters in UTF-8).  They have the same multi-unit 
encoding issues as you get with UTF-8.  If you want "one unit is one 
code", you need UTF-32, which is fine for internal program usage. 
UTF-16 has endianess issues.  There really is no benefit in UTF-16 over 
UTF-8.  The world at large knows this - that is why UTF-8 is 
overwhelmingly dominant at UTF-16 is used mostly for legacy reasons.

But you can all it "taste" if you like.

[toc] | [prev] | [next] | [standalone]

#67759

>> That's just your taste. But someone else may argue like the Unicode
>> -standard, that UTF-16 is a good balance between size and performance.

> Sorry, you /do/ know that UTF-8 and UTF-32 are also encodings for 
> Unicode, just like UTF-16?  It is not clear, but it looks a little like 
> you think UTF-16 /is/ Unicode and that I am suggesting something other 
> than Unicode.  I hope you can clarify if you are misunderstanding this 
> or not.

You can't derive that from what I said above.
And it isn't even related to that.
You seem a bit confuse.

[toc] | [prev] | [next] | [standalone]

#67713

On 17/08/2019 18:05, Bonita Montero wrote:
>> You can store any character encoding supported by the compiler in a 
>> wchar_t, unless you are using Windows, which is broken and can't store 
>> Unicode characters in a wchar_t because it is too small.
> 
> The size of wchar_t is implementation-dependent, so Windows isn't broken
> here. 

Yes, Windows is broken here.  The standards give minimum requirements 
for many implementation-dependent features.  Windows could have a 21-bit 
wchar_t, or a 121-bit wchar_t if it liked, but not a 16-bit one.  It is 
just like saying the size of "int" is implementation dependent, but the 
standards require a minimum of 16-bit int.

> And you have a limited number of codepoints in UTF-16 over UTF-32,
> but even UTF-16 has such a huge number of codepoints that this will
> never be exhausted.

Unicode started with 16-bit code points, but within five years or so 
they figured out that 16-bit was not enough, and it was extended.

Perhaps you don't understand how multi-unit encoding works, and are 
totally confused by the concept that UTF-16 encoding is fine for 
handling some billion+ code points, just like UTF-8 and UTF-32 can, 
while not understanding that you can't fit all Unicode characters into a 
single 16-bit wchar_t.


> And as there will never be more codepoints in Unicode as the "limited"
> range of UTF-16 can cover Windows isn't broken here.

Look it up.  UTF-16 can cover everything, though it is a silly choice. 
A 16-bit wchar_t cannot store all UTF-16 characters.

> 
>> On most modern systems, wchar_t matches char32_t.
>> But it is not a  requirement.
> 
> That was where David is wrong.

The minimum requirement would be 21-bit for a system supporting Unicode. 
  But wchar_t must also match an existing integral type.  It would be 
possible for a compiler to have an extended integer type that is 24-bit, 
but assuming a "normal" compiler, 32-bit is the only sane minimum 
requirement for wchar_t.  And while it the standards allow bigger sizes 
- a 64-bit wchar_t would be perfectly compliant - sanity dictates 32-bit.

> 
>> People use C++ all the time on 8-bit and 16-bit processors - /new/ 
>> designs, not ancient ones.  They often don't have a full C++ library, 
>> but they are freestanding systems and don't need the full library to
>> be compliant.  (They might be non-compliant in other ways.)
> 
> I wasn't talking about the library but basic datatypes.

I don't think /you/ know what you are talking about - it is hard for 
other people to guess.

> 
>> Windows compilers are invariably non-compliant regarding wchar_t 
>> because it is only 16-bit on that platform, when it is required to be 
>> 32-bit.
> 
> UTF-16 is suitable for all codepoints that will ever be populated.
> 

When you are in a hole, stop digging.

[toc] | [prev] | [next] | [standalone]

#67720

David Brown <david.brown@hesbynett.no> writes:
> On 17/08/2019 18:05, Bonita Montero wrote:
>>> You can store any character encoding supported by the compiler in a 
>>> wchar_t, unless you are using Windows, which is broken and can't store 
>>> Unicode characters in a wchar_t because it is too small.
>> 
>> The size of wchar_t is implementation-dependent, so Windows isn't broken
>> here. 
>
> Yes, Windows is broken here.  The standards give minimum requirements 
> for many implementation-dependent features.  Windows could have a 21-bit 
> wchar_t, or a 121-bit wchar_t if it liked, but not a 16-bit one.  It is 
> just like saying the size of "int" is implementation dependent, but the 
> standards require a minimum of 16-bit int.

**OR** it could have a 16-bit wchar_t and not claim to support Unicode
as its wide character set.  An implementation with 16-bit wchar_t that
supports only the BMP as its wide character set could be conforming
(though not as usesful as an implementation that supports full Unicode).

But Microsoft's own documentation says:
    The wchar_t type is an implementation-defined wide character
    type. In the Microsoft compiler, it represents a 16-bit wide
    character used to store Unicode encoded as UTF-16LE, the native
    character type on Windows operating systems. The wide character
    versions of the Universal C Runtime (UCRT) library functions
    use wchar_t and its pointer and array types as parameters and
    return values, as do the wide character versions of the native
    Windows API.
https://docs.microsoft.com/en-us/cpp/cpp/char-wchar-t-char16-t-char32-t?view=vs-2019

which I believe is non-conforming.

(But I'm not sure how Microsoft could have fixed this without breaking
too much existing code.)

[...]

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
Will write code for food.
void Void(void) { Void(); } /* The recursive call of the void */

[toc] | [prev] | [next] | [standalone]

#67740

On 18/08/2019 01:46, Keith Thompson wrote:
> David Brown <david.brown@hesbynett.no> writes:
>> On 17/08/2019 18:05, Bonita Montero wrote:
>>>> You can store any character encoding supported by the compiler in a
>>>> wchar_t, unless you are using Windows, which is broken and can't store
>>>> Unicode characters in a wchar_t because it is too small.
>>>
>>> The size of wchar_t is implementation-dependent, so Windows isn't broken
>>> here.
>>
>> Yes, Windows is broken here.  The standards give minimum requirements
>> for many implementation-dependent features.  Windows could have a 21-bit
>> wchar_t, or a 121-bit wchar_t if it liked, but not a 16-bit one.  It is
>> just like saying the size of "int" is implementation dependent, but the
>> standards require a minimum of 16-bit int.
> 
> **OR** it could have a 16-bit wchar_t and not claim to support Unicode
> as its wide character set.  An implementation with 16-bit wchar_t that
> supports only the BMP as its wide character set could be conforming
> (though not as usesful as an implementation that supports full Unicode).

Absolutely true.  And that was the case originally with Windows NT, 
which used UCS-2 (essentially the subset of UTF-16 that can be encoded 
in a single unit).  But Windows has moved steadily more of its APIs, 
libraries, gui widgets, and software to full UTF-16.  It has done so in 
uncoordinated jumps, with versions of Windows that let you use 
multi-unit characters in some programs and APIs but not others, but I 
believe it is fairly complete now.  (And with Windows 10, I have heard 
that UTF-8 support is officially in place.)

So 16-bit wchar_t was appropriate when Windows NT started with UCS-2. 
But it should have been changed to 32-bit for later Windows.

> 
> But Microsoft's own documentation says:
>      The wchar_t type is an implementation-defined wide character
>      type. In the Microsoft compiler, it represents a 16-bit wide
>      character used to store Unicode encoded as UTF-16LE, the native
>      character type on Windows operating systems. The wide character
>      versions of the Universal C Runtime (UCRT) library functions
>      use wchar_t and its pointer and array types as parameters and
>      return values, as do the wide character versions of the native
>      Windows API.
> https://docs.microsoft.com/en-us/cpp/cpp/char-wchar-t-char16-t-char32-t?view=vs-2019
> 
> which I believe is non-conforming.
> 
> (But I'm not sure how Microsoft could have fixed this without breaking
> too much existing code.)
> 

Aye, there's the rub.

People programming for Windows have a long history of making unwarranted 
assumptions about types and sizes.  They have been used to programming 
on a single platform, and thinking it will remain the same forever. 
They have not been helped by Microsoft's unforgivable reticence over C99 
and headers like <stdint.h>.  Windows programmers regularly assume that 
wchar_t is 16-bit - changing it would break code written by these 
programmers.  The same thing happened with the move to 64-bit Windows - 
because Windows programmers had assumed that "long" is exactly 32-bit 
(having had no "int32_t" available), changing the size of "long" to 
match every other 64-bit platform would have broken lots of Windows code.

It is easy to say /now/ that Microsoft should have changed to 32-bit 
wchar_t and UTF-32 and/or UTF-8 as soon as it was clear that Unicode 
would not fit in 16 bits.  But it would have been hard to do at the time.

The resulting situation today, however, is clear.  Windows is 
non-compliant and nearly unique with its 16-bit wchar_t, and it is the 
only major OS to make heavy use of UTF-16 instead of UTF-8.

[toc] | [prev] | [next] | [standalone]

#67721

On Sat, 17 Aug 2019 20:41:06 +0200, David Brown
<david.brown@hesbynett.no> wrote:

>On 17/08/2019 18:05, Bonita Montero wrote:
>
>> And you have a limited number of codepoints in UTF-16 over UTF-32,
>> but even UTF-16 has such a huge number of codepoints that this will
>> never be exhausted.
>
>Unicode started with 16-bit code points, but within five years or so 
>they figured out that 16-bit was not enough, and it was extended.
>
>Perhaps you don't understand how multi-unit encoding works, and are 
>totally confused by the concept that UTF-16 encoding is fine for 
>handling some billion+ code points, just like UTF-8 and UTF-32 can, 
>while not understanding that you can't fit all Unicode characters into a 
>single 16-bit wchar_t.


Actually UTF-16 can't encode billions of characters, as it
specifically uses a pair of extension/surrogate characters, which
encode about 10 bits each, leading to the extra 16 planes.

The format for UTF-8 was designed (and could trivially be extended) to
support 31 bit code points (with a six byte sequence).  It was limited
to the current 17-plane scheme by the adoption of UTF-16, which could
only address 17 planes (the BMP plus the 16 extra planes implied by
the 20-bit number encoded in the surrogate characters).

[toc] | [prev] | [next] | [standalone]

#67731

> Yes, Windows is broken here.  The standards give minimum requirements 
> for many implementation-dependent features.  Windows could have a 21-bit 
> wchar_t, or a 121-bit wchar_t if it liked, but not a 16-bit one.  It is 
> just like saying the size of "int" is implementation dependent, but the 
> standards require a minimum of 16-bit int.

There's no mandantory relationship between Unicode and wchar_t

>> And as there will never be more codepoints in Unicode as the "limited"
>> range of UTF-16 can cover Windows isn't broken here.

> Look it up.  UTF-16 can cover everything, though it is a silly choice. A 
> 16-bit wchar_t cannot store all UTF-16 characters.

But UTF-16 can. And Windows works with UTF-16.

[toc] | [prev] | [next] | [standalone]

#67741

On 18/08/2019 08:57, Bonita Montero wrote:
>> Yes, Windows is broken here.  The standards give minimum requirements 
>> for many implementation-dependent features.  Windows could have a 
>> 21-bit wchar_t, or a 121-bit wchar_t if it liked, but not a 16-bit 
>> one.  It is just like saying the size of "int" is implementation 
>> dependent, but the standards require a minimum of 16-bit int.
> 
> There's no mandantory relationship between Unicode and wchar_t

Correct.

But there is a mandatory relationship between the character set 
supported by the target system, and wchar_t.  Windows supports Unicode 
(with UTF-16 encoding, and possibly UTF-8 with Windows 10).  Therefore, 
wchar_t must support Unicode characters.

> 
>>> And as there will never be more codepoints in Unicode as the "limited"
>>> range of UTF-16 can cover Windows isn't broken here.
> 
>> Look it up.  UTF-16 can cover everything, though it is a silly choice. 
>> A 16-bit wchar_t cannot store all UTF-16 characters.
> 
> But UTF-16 can. And Windows works with UTF-16.

[toc] | [prev] | [next] | [standalone]

#67746

> But there is a mandatory relationship between the character set 
> supported by the target system, and wchar_t.

No, the implementor might implement whatever he likes since it
hasn't conform to the target-system but all locales supported
by the implementation.

[toc] | [prev] | [next] | [standalone]

#67747

Am 18.08.2019 um 13:07 schrieb Bonita Montero:
>> But there is a mandatory relationship between the character set 
>> supported by the target system, and wchar_t.
> 
> No, the implementor might implement whatever he likes since it
> hasn't conform to the target-system but all locales supported
> by the implementation.

The standard says:
"Type wchar_t is a distinct type whose values can represent
distinct codes for all members of the largest extended character
set specified among the supported locales."
               ^^^^^^^^^^*********^^^^^^^^
So where's the inconformance?

[toc] | [prev] | [next] | [standalone]

#67711

Bonita Montero <Bonita.Montero@gmail.com> writes:

>>>> Be advised that in Windows wide strings are UTF-16 encoded.
>>>
>>> And I think that it would be the best to use std::u16string
>>> because wchar_t is not absolutely guaranteed to be 16 bit
>>> wide.
>>
>> It is guaranteed /not/ to be 16-bit on most systems.
>
> You can be pretty sure that by far most implementations specify
> whcar_t as 16 bit since it makes no sense to implement it differently.
> But it's easier to conform to any hypothetical implementation by using
> char16_t.

(and in another posting quotes the C++ standard)

> "Type wchar_t is a distinct type whose values can represent distinct
> codes for all members of the largest extended character set specified
> among the supported locales (22.3.1)."

Like you say, the C and C++ standards require wchar_t to be large
enough so one wchar_t object can hold distinct values for every
character in the largest supported character set.

Most widely used operating systems today (eg, Microsoft Windows,
Linux) support character sets with (at least) hundreds of
thousands of characters.

Can you explain how hundreds of thousands distinct values can be
represented in a single 16-bit wide wchar_t object?

[toc] | [prev] | [next] | [standalone]

#67716

On 2019-08-17 at 20:22, Tim Rentsch wrote:
> Bonita Montero <Bonita.Montero@gmail.com> writes:
> 
>>>>> Be advised that in Windows wide strings are UTF-16 encoded.
>>>>
>>>> And I think that it would be the best to use std::u16string
>>>> because wchar_t is not absolutely guaranteed to be 16 bit
>>>> wide.
>>>
>>> It is guaranteed /not/ to be 16-bit on most systems.
>>
>> You can be pretty sure that by far most implementations specify
>> whcar_t as 16 bit since it makes no sense to implement it differently.
>> But it's easier to conform to any hypothetical implementation by using
>> char16_t.
> 
> (and in another posting quotes the C++ standard)
> 
>> "Type wchar_t is a distinct type whose values can represent distinct
>> codes for all members of the largest extended character set specified
>> among the supported locales (22.3.1)."
> 
> Like you say, the C and C++ standards require wchar_t to be large
> enough so one wchar_t object can hold distinct values for every
> character in the largest supported character set.
> 
> Most widely used operating systems today (eg, Microsoft Windows,
> Linux) support character sets with (at least) hundreds of
> thousands of characters.
> 
> Can you explain how hundreds of thousands distinct values can be
> represented in a single 16-bit wide wchar_t object?
> 

No, they would naturally not fit.

But there is a loop-hole here, as the language standard says "the 
supported locales" and not "every existing locale".

So if you only support the locales where 16 bits is enough, you haven't 
broken any rules. And you cannot help is some people use parts of 
"unsupported" locales as well...

[toc] | [prev] | [next] | [standalone]

#67725

Bo Persson <bo@bo-persson.se> writes:

> On 2019-08-17 at 20:22, Tim Rentsch wrote:
>
>> Bonita Montero <Bonita.Montero@gmail.com> writes:
>>
>>>>>> Be advised that in Windows wide strings are UTF-16 encoded.
>>>>>
>>>>> And I think that it would be the best to use std::u16string
>>>>> because wchar_t is not absolutely guaranteed to be 16 bit
>>>>> wide.
>>>>
>>>> It is guaranteed /not/ to be 16-bit on most systems.
>>>
>>> You can be pretty sure that by far most implementations specify
>>> whcar_t as 16 bit since it makes no sense to implement it
>>> differently.  But it's easier to conform to any hypothetical
>>> implementation by using char16_t.
>>
>> (and in another posting quotes the C++ standard)
>>
>>> "Type wchar_t is a distinct type whose values can represent
>>> distinct codes for all members of the largest extended character
>>> set specified among the supported locales (22.3.1)."
>>
>> Like you say, the C and C++ standards require wchar_t to be large
>> enough so one wchar_t object can hold distinct values for every
>> character in the largest supported character set.
>>
>> Most widely used operating systems today (eg, Microsoft Windows,
>> Linux) support character sets with (at least) hundreds of
>> thousands of characters.
>>
>> Can you explain how hundreds of thousands distinct values can be
>> represented in a single 16-bit wide wchar_t object?
>
> No, they would naturally not fit.
>
> But there is a loop-hole here, as the language standard says "the
> supported locales" and not "every existing locale".
>
> So if you only support the locales where 16 bits is enough, you
> haven't broken any rules.  And you cannot help is some people use
> parts of "unsupported" locales as well...

Perhaps not as much of a loophole as you think.  The setlocale()
function is defined, even for C++, by the C standard.  A call to
setlocale() must not accept a locale that has more characters
than wchar_t can accommodate, because of how 'wide character' is
defined.  So the "unsupported" locales you are talking about
cannot be used in a conforming implmentation.  Of course, if the
implementation is not conforming, it can do whatever it wants.

[toc] | [prev] | [next] | [standalone]

#67772

On Sat, 17 Aug 2019 21:21:46 +0200, Bo Persson <bo@bo-persson.se>
wrote:

>On 2019-08-17 at 20:22, Tim Rentsch wrote:
>> Bonita Montero <Bonita.Montero@gmail.com> writes:
>> 
>>>>>> Be advised that in Windows wide strings are UTF-16 encoded.
>>>>>
>>>>> And I think that it would be the best to use std::u16string
>>>>> because wchar_t is not absolutely guaranteed to be 16 bit
>>>>> wide.
>>>>
>>>> It is guaranteed /not/ to be 16-bit on most systems.
>>>
>>> You can be pretty sure that by far most implementations specify
>>> whcar_t as 16 bit since it makes no sense to implement it differently.
>>> But it's easier to conform to any hypothetical implementation by using
>>> char16_t.
>> 
>> (and in another posting quotes the C++ standard)
>> 
>>> "Type wchar_t is a distinct type whose values can represent distinct
>>> codes for all members of the largest extended character set specified
>>> among the supported locales (22.3.1)."
>> 
>> Like you say, the C and C++ standards require wchar_t to be large
>> enough so one wchar_t object can hold distinct values for every
>> character in the largest supported character set.
>> 
>> Most widely used operating systems today (eg, Microsoft Windows,
>> Linux) support character sets with (at least) hundreds of
>> thousands of characters.
>> 
>> Can you explain how hundreds of thousands distinct values can be
>> represented in a single 16-bit wide wchar_t object?
>> 
>
>No, they would naturally not fit.
>
>But there is a loop-hole here, as the language standard says "the 
>supported locales" and not "every existing locale".
>
>So if you only support the locales where 16 bits is enough, you haven't 
>broken any rules. And you cannot help is some people use parts of 
>"unsupported" locales as well...


In the case of Windows, locales such as Chinese *are* defined.

https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-lcid/a9eac961-e77d-41a6-90a5-ce1a8b0cdb9c

Unless you want to argue that the Windows C and C++ compilers don't
actually intend to support CJK characters in the zh-CN locale.

[toc] | [prev] | [next] | [standalone]

#67773

> In the case of Windows, locales such as Chinese *are* defined.
> https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-lcid/a9eac961-e77d-41a6-90a5-ce1a8b0cdb9c
> Unless you want to argue that the Windows C and C++ compilers don't
> actually intend to support CJK characters in the zh-CN locale.

wchar_t's locale-support needs only to be related to the language
not the platform.

[toc] | [prev] | [next] | [standalone]

Page 4 of 5 — ← Prev page 1 2 3 [4] 5  Next page →

Back to top | Article view | comp.lang.c++

csiph-web