Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #75432 > unrolled thread

Is wchar_t really useful?

Back to article view | Back to comp.lang.c

Contents

  Is wchar_t really useful? Philipp Klaus Krause <pkk@spth.de> - 2015-11-09 12:02 +0100
    Re: Is wchar_t really useful? Malcolm McLean <malcolm.mclean5@btinternet.com> - 2015-11-09 03:24 -0800
      Re: Is wchar_t really useful? Stephen Sprunk <stephen@sprunk.org> - 2015-11-10 18:33 -0600
        Re: Is wchar_t really useful? Ian Collins <ian-news@hotmail.com> - 2015-11-11 13:56 +1300
    Re: Is wchar_t really useful? James Kuyper <jameskuyper@verizon.net> - 2015-11-09 09:03 -0500
      Re: Is wchar_t really useful? Philipp Klaus Krause <pkk@spth.de> - 2015-11-12 15:22 +0100
        Re: Is wchar_t really useful? James Kuyper <jameskuyper@verizon.net> - 2015-11-12 10:04 -0500
          Re: Is wchar_t really useful? Philipp Klaus Krause <pkk@spth.de> - 2015-11-12 16:44 +0100
            Re: Is wchar_t really useful? James Kuyper <jameskuyper@verizon.net> - 2015-11-12 11:29 -0500
              Re: Is wchar_t really useful? supercat@casperkitty.com - 2015-11-12 09:14 -0800
                Re: Is wchar_t really useful? Tim Rentsch <txr@alumni.caltech.edu> - 2015-11-14 11:50 -0800
                  Re: Is wchar_t really useful? supercat@casperkitty.com - 2015-11-15 13:44 -0800
                    Re: Is wchar_t really useful? Philipp Klaus Krause <pkk@spth.de> - 2015-11-15 22:50 +0100
                      Re: Is wchar_t really useful? supercat@casperkitty.com - 2015-11-15 14:46 -0800
                        Re: Is wchar_t really useful? raltbos@xs4all.nl (Richard Bos) - 2015-11-17 22:34 +0000
                    Re: Is wchar_t really useful? Tim Rentsch <txr@alumni.caltech.edu> - 2015-11-16 07:25 -0800
              Re: Is wchar_t really useful? Philipp Klaus Krause <pkk@spth.de> - 2015-11-13 17:26 +0100
                Re: Is wchar_t really useful? James Kuyper <jameskuyper@verizon.net> - 2015-11-13 13:36 -0500
                  Re: Is wchar_t really useful? Stephen Sprunk <stephen@sprunk.org> - 2015-11-13 16:37 -0600
                    Re: Is wchar_t really useful? James Kuyper <jameskuyper@verizon.net> - 2015-11-13 17:58 -0500
                Re: Is wchar_t really useful? Stephen Sprunk <stephen@sprunk.org> - 2015-11-13 16:01 -0600
                  Re: Is wchar_t really useful? Philipp Klaus Krause <pkk@spth.de> - 2015-11-13 23:36 +0100
      Re: Is wchar_t really useful? Philipp Klaus Krause <pkk@spth.de> - 2015-11-12 15:28 +0100
        Re: Is wchar_t really useful? James Kuyper <jameskuyper@verizon.net> - 2015-11-12 10:26 -0500
      Re: Is wchar_t really useful? Stephen Sprunk <stephen@sprunk.org> - 2015-11-12 14:27 -0600
    Re: Is wchar_t really useful? Lőrinczy Zsigmond <nospam@for.me> - 2015-11-13 05:02 +0100
      Re: Is wchar_t really useful? Philipp Klaus Krause <pkk@spth.de> - 2015-11-13 11:52 +0100
        Re: Is wchar_t really useful? Malcolm McLean <malcolm.mclean5@btinternet.com> - 2015-11-13 05:17 -0800
          Re: Is wchar_t really useful? Stephen Sprunk <stephen@sprunk.org> - 2015-11-13 12:58 -0600
          Re: Is wchar_t really useful? supercat@casperkitty.com - 2015-11-13 11:14 -0800
      Re: Is wchar_t really useful? Stephen Sprunk <stephen@sprunk.org> - 2015-11-13 12:45 -0600
    Re: Is wchar_t really useful? Philipp Klaus Krause <pkk@spth.de> - 2015-11-13 18:24 +0100
      Re: Is wchar_t really useful? James Kuyper <jameskuyper@verizon.net> - 2015-11-13 13:55 -0500
        Re: Is wchar_t really useful? Philipp Klaus Krause <pkk@spth.de> - 2015-11-13 20:39 +0100
          Re: Is wchar_t really useful? James Kuyper <jameskuyper@verizon.net> - 2015-11-13 14:54 -0500
          Re: Is wchar_t really useful? Stephen Sprunk <stephen@sprunk.org> - 2015-11-13 17:04 -0600

Page 1 of 2  [1] 2  Next page →

#75432 — Is wchar_t really useful?

The C standard has wchar_t. But I don't really see any advantage over
char. Maybe if __STDC_ISO_10646__ is defined. But even then, I guess
char, char16_t and char32_t are more useful.
Is there a point in using wchar_t?

Philipp

[toc] | [next] | [standalone]

#75434

On Monday, November 9, 2015 at 11:03:05 AM UTC, Philipp Klaus Krause wrote:
> The C standard has wchar_t. But I don't really see any advantage over
> char. Maybe if __STDC_ISO_10646__ is defined. But even then, I guess
> char, char16_t and char32_t are more useful.
> Is there a point in using wchar_t?
> 
wchar_t holds wide characters.
The naive assumption a few years ago was that if you simply made characters 16 bits
or 32 bits, you could support languages with non-Latin alphabets. Reality isn't as
simple as that. Whilst in English it is very clear what we mean by a "glyph", in many
other languages it's far less obvious.
The emerging consensus is that UTF-8 should be used for pretty much everything except
final glyph display, when of course you need to convert to a unicode code point and
then to a graphic. UTF-8 is also a superset of ascii, so if text is English only, which is
common for human-readable computer to computer communication, you simply
treat as ascii and don't worry.

But if you need to handle non-English characters, you need to choose one, wide characters
or UTF-8.

[toc] | [prev] | [next] | [standalone]

#75706

On 09-Nov-15 05:24, Malcolm McLean wrote:
> Philipp Klaus Krause wrote:
>> The C standard has wchar_t. But I don't really see any advantage
>> over char. Maybe if __STDC_ISO_10646__ is defined. But even then, I
>> guess char, char16_t and char32_t are more useful. Is there a point
>> in using wchar_t?
>> 
> wchar_t holds wide characters. The naive assumption a few years ago
> was that if you simply made characters 16 bits or 32 bits, you could
> support languages with non-Latin alphabets. Reality isn't as simple
> as that. Whilst in English it is very clear what we mean by a
> "glyph", in many other languages it's far less obvious.

Even "character" is not obvious in many cases; it could mean a code
unit, a code point, a glyph, a grapheme cluster or something else.

What usually matters are either bytes (e.g. memory allocation) or
grapheme clusters (e.g. user perception); code units/points are just
Unicode's (necessarily) complicated way of mapping between those.

> The emerging consensus is that UTF-8 should be used for pretty much
> everything

Emerging?  Everyone but Microsoft and Sun knew that ~20 years ago!

> But if you need to handle non-English characters, you need to choose
> one, wide characters or UTF-8.

... and that decision will be heavily influenced by the tools you have
to work with.  Certain OSes/languages/APIs make it very easy to work in
UTF-16 (often actually UCS-2, which causes all sorts of problems) and
very hard to work in UTF-8, while others are the reverse.

S

-- 
Stephen Sprunk         "God does not play dice."  --Albert Einstein
CCIE #3723         "God is an inveterate gambler, and He throws the
K5SSS        dice at every possible opportunity." --Stephen Hawking

[toc] | [prev] | [next] | [standalone]

#75708

Stephen Sprunk wrote:
> On 09-Nov-15 05:24, Malcolm McLean wrote:
>
>> The emerging consensus is that UTF-8 should be used for pretty much
>> everything
>
> Emerging?  Everyone but Microsoft and Sun knew that ~20 years ago!

Sun not long after!

-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

#75450

On 11/09/2015 06:02 AM, Philipp Klaus Krause wrote:
> The C standard has wchar_t. But I don't really see any advantage over
> char. Maybe if __STDC_ISO_10646__ is defined. But even then, I guess
> char, char16_t and char32_t are more useful.
> Is there a point in using wchar_t?

Many (most?) of the C standard library routines that take or return
char* arguments are defined as interpreting those pointers as pointing
to a multibyte character string (MBCS). If it were up to me, there'd be
one term "single byte character" referring to a single char object that
can represent a character, while "multibyte character" would refer only
to sequences of two or more char objects that together represent a
single character. The committee took a different approach, defining the
term "multibyte character" to include both of those categories (3.7.2p1).

Wide character strings, on the other hand, can represent any character
in the current locale using a single wchar_t object.
The representations used for MBCS and wide character strings are
implementation-defined. char16_t is, essentially, required to use either
UCS-2 or UTF-16, while char32_t is, essentially, required to be either
UCS-4 or UTF-32.

If the representation used by wchar_t for a given locale is different
from the representation used by MBCS, char16_t and char32_t, and if that
representation happens to be the one you need to work with, then that is
reason #1 for using wchar_t (and for selecting that locale).
If the processing you need to do on a character string requires dealing
with one character at a time, regardless of how many char objects are
needed to store a representation of that character, then you can't use
just char. You'll have to use either wchar_t, char16_t, or char32_t.
However, if that's the case, since char16_t and char32_t are new in
C2011, if your code also needs to be portable to C99 or C94, that is
reason #2 for to using wchar_t.
If you need to process one character at a time, then even if you can use
a C2011 implementation, you might need to use a routine that uses
wchar_t in the argument list or in the return type, because more often
than not there's no corresponding routine that uses char16_t or
char32_t. That's reason #3 for using wchar_t.
-- 
James Kuyper

[toc] | [prev] | [next] | [standalone]

#75819

On 09.11.2015 15:03, James Kuyper wrote:
> On 11/09/2015 06:02 AM, Philipp Klaus Krause wrote:
>> The C standard has wchar_t. But I don't really see any advantage over
>> char. Maybe if __STDC_ISO_10646__ is defined. But even then, I guess
>> char, char16_t and char32_t are more useful.
>> Is there a point in using wchar_t?
> 
> Many (most?) of the C standard library routines that take or return
> char* arguments are defined as interpreting those pointers as pointing
> to a multibyte character string (MBCS). If it were up to me, there'd be
> one term "single byte character" referring to a single char object that
> can represent a character, while "multibyte character" would refer only
> to sequences of two or more char objects that together represent a
> single character. The committee took a different approach, defining the
> term "multibyte character" to include both of those categories (3.7.2p1).
> 
> Wide character strings, on the other hand, can represent any character
> in the current locale using a single wchar_t object.

Does that mean than an implementation that can work with all of Unicode
as multibyte characters needs to have a wchar_t big enough to hold each
one of them?
I guess a way to weasel out would be to define only the basic
multilingual plane as the supported extended character set, make wchar_t
16 bits, and declare that all of unicode just works with char, char16_t,
char32_t as implementation-defined behaviour? Or just do the same with
the basic character set as extended set, and make wchar_t 8 bits?

Philipp

[toc] | [prev] | [next] | [standalone]

#75829

On 11/12/2015 09:22 AM, Philipp Klaus Krause wrote:
> On 09.11.2015 15:03, James Kuyper wrote:
>> On 11/09/2015 06:02 AM, Philipp Klaus Krause wrote:
>>> The C standard has wchar_t. But I don't really see any advantage over
>>> char. Maybe if __STDC_ISO_10646__ is defined. But even then, I guess
>>> char, char16_t and char32_t are more useful.
>>> Is there a point in using wchar_t?
>>
>> Many (most?) of the C standard library routines that take or return
>> char* arguments are defined as interpreting those pointers as pointing
>> to a multibyte character string (MBCS). If it were up to me, there'd be
>> one term "single byte character" referring to a single char object that
>> can represent a character, while "multibyte character" would refer only
>> to sequences of two or more char objects that together represent a
>> single character. The committee took a different approach, defining the
>> term "multibyte character" to include both of those categories (3.7.2p1).
>>
>> Wide character strings, on the other hand, can represent any character
>> in the current locale using a single wchar_t object.
> 
> Does that mean than an implementation that can work with all of Unicode
> as multibyte characters needs to have a wchar_t big enough to hold each
> one of them?

Yes. That pretty much defines how wchar_t is meant to be used.

> I guess a way to weasel out would be to define only the basic
> multilingual plane as the supported extended character set, make wchar_t
> 16 bits, and declare that all of unicode just works with char, char16_t,
> char32_t as implementation-defined behaviour? Or just do the same with
> the basic character set as extended set, and make wchar_t 8 bits?

The members of the extended character set are implementation-defined,
and there could indeed be sufficiently few of them to fit in 8 bits
(there don't have to be any members that aren't also part of the basic
character set). However, the routines that work with extended characters
generally have defined behavior, that generally involves them failing,
when processing data that does not encode a valid member of the extended
character set. That makes it difficult for a conforming implementation
to meaningfully support characters which can't be represented using
wchar_t. If the behavior when handling invalid encodings was undefined,
then it would be possible for the undefined behavior to involve
recognizing technically-invalid encodings as if they were actually
valid, and dealing with them appropriately - but that's not the case.

Why exactly are you interested in how an implementation could weasel out
of this requirement? As I understand it, there's a simple process that
goes on when designing this part of an implementation of C: choose what
extended character sets you want to support, and which encoding. Then
choose a single character encoding capable of representing all of those
characters, and define wchar_t to be a type big enough to store the
representations of any of those characters. Did you have a different
implementation design process in mind?
-- 
James Kuyper

[toc] | [prev] | [next] | [standalone]

#75831

On 12.11.2015 16:04, James Kuyper wrote:

> Why exactly are you interested in how an implementation could weasel out
> of this requirement? As I understand it, there's a simple process that
> goes on when designing this part of an implementation of C: choose what
> extended character sets you want to support, and which encoding. Then
> choose a single character encoding capable of representing all of those
> characters, and define wchar_t to be a type big enough to store the
> representations of any of those characters. Did you have a different
> implementation design process in mind?
> 

1) AFAIK, Windows compilers tend to use a 16-bit wchar_t, and assume
sequences of wchar_t to be UTF-16-encoded. Thus there are characters
that don't fit into a wchar_t. How do these implementations get around
the sandard requirement?
2) SDCC used to support ISO-8559-1 as character set, but will probably
move to UTF-8-encoded unicode for the next release. Historically, SDCC
used an 8-bit wchar_t. Will SDCC have to change to a bigger wchar_t? OR
could it just continue with the 8-bit wchar_t and have sequences of
wchar_t UTF-8-encoded?

Philipp

[toc] | [prev] | [next] | [standalone]

#75836

On 11/12/2015 10:44 AM, Philipp Klaus Krause wrote:
> On 12.11.2015 16:04, James Kuyper wrote:
> 
>> Why exactly are you interested in how an implementation could weasel out
>> of this requirement? As I understand it, there's a simple process that
>> goes on when designing this part of an implementation of C: choose what
>> extended character sets you want to support, and which encoding. Then
>> choose a single character encoding capable of representing all of those
>> characters, and define wchar_t to be a type big enough to store the
>> representations of any of those characters. Did you have a different
>> implementation design process in mind?
>>
> 
> 1) AFAIK, Windows compilers tend to use a 16-bit wchar_t, and assume
> sequences of wchar_t to be UTF-16-encoded. Thus there are characters
> that don't fit into a wchar_t. How do these implementations get around
> the sandard requirement?

I've no idea - possibly by not conforming to the requirements? I don't
actually use any of this stuff (including Windows) - I'm just telling
you what I can figure out from reading the standard. No program I've
ever written needed any characters outside of the basic character set,
nor any locale other than the "C" locale. Hopefully someone who does
work with this stuff on Windows can answer your question.

> 2) SDCC used to support ISO-8559-1 as character set, but will probably
> move to UTF-8-encoded unicode for the next release. Historically, SDCC
> used an 8-bit wchar_t. Will SDCC have to change to a bigger wchar_t? OR
> could it just continue with the 8-bit wchar_t and have sequences of
> wchar_t UTF-8-encoded?

The standard does not have a macro corresponding to __STDC_UTF_16__ that
applies to wchar_t, so I don't think it could be conforming to use such
an encoding. As far as I know, the right approach would be to use UCS-4
encoding for a 32-bit wchar_t, which, if I understand correctly, is the
requirement for pre#defining __STDC_ISO_10646__.
-- 
James Kuyper

[toc] | [prev] | [next] | [standalone]

#75843

On Thursday, November 12, 2015 at 10:29:23 AM UTC-6, James Kuyper wrote:
> I've no idea - possibly by not conforming to the requirements? I don't
> actually use any of this stuff (including Windows) - I'm just telling
> you what I can figure out from reading the standard. No program I've
> ever written needed any characters outside of the basic character set,
> nor any locale other than the "C" locale. Hopefully someone who does
> work with this stuff on Windows can answer your question.

From what I can tell, from the point of view of the Standard, it would be
perfectly legal for

    char foo[] = "Pay $5";

to be equivallent to

    char foo[] =
      {'P','a','y',' ','D','o','l','l','a','r','S','i','g','n','5',0};

if the implementation specified that the source text character "$" would
map to the sequence "DollarSign" in the destination character set.  The
characters in the C Standard Character Set must be representable using
one "char" each, but for characters not in that set there is no such
requirement.

[toc] | [prev] | [next] | [standalone]

#76008

supercat@casperkitty.com writes:

> On Thursday, November 12, 2015 at 10:29:23 AM UTC-6, James Kuyper wrote:
>> I've no idea - possibly by not conforming to the requirements?  I don't
>> actually use any of this stuff (including Windows) - I'm just telling
>> you what I can figure out from reading the standard.  No program I've
>> ever written needed any characters outside of the basic character set,
>> nor any locale other than the "C" locale.  Hopefully someone who does
>> work with this stuff on Windows can answer your question.
>
> From what I can tell, from the point of view of the Standard, it
> would be perfectly legal for
>
>     char foo[] = "Pay $5";
>
> to be equivallent to
>
>     char foo[] =
>       {'P','a','y',' ','D','o','l','l','a','r','S','i','g','n','5',0};
>
> if the implementation specified that the source text character "$"
> would map to the sequence "DollarSign" in the destination
> character set.  [snip]

I don't think so.  If you want this hypothesis to be convincing
you would need to give a more substantive argument, including
specific citations for relevant passages in the Standard.

[toc] | [prev] | [next] | [standalone]

#76129

On Saturday, November 14, 2015 at 1:50:37 PM UTC-6, Tim Rentsch wrote:
> I don't think so.  If you want this hypothesis to be convincing
> you would need to give a more substantive argument, including
> specific citations for relevant passages in the Standard.

What will happen if one feeds a UTF-8 source text containing:

   char foo[] = "L";
   char bar[] = { sizeof foo };

but replaces the "L" with a square-root symbol (U+2A04)?  What will the length
of foo[] be?  On the compiler where I just tried it, the string was three
characters long.  Although a dollar sign happens to be one byte in ASCII,
there is nothing in the C Standard that would forbid the use of a character
set where a dollar sign required multiple bytes to store.

[toc] | [prev] | [next] | [standalone]

#76132

On 15.11.2015 22:44, supercat@casperkitty.com wrote:
> On Saturday, November 14, 2015 at 1:50:37 PM UTC-6, Tim Rentsch wrote:
>> I don't think so.  If you want this hypothesis to be convincing
>> you would need to give a more substantive argument, including
>> specific citations for relevant passages in the Standard.
> 
> What will happen if one feeds a UTF-8 source text containing:
> 
>    char foo[] = "L";
>    char bar[] = { sizeof foo };
> 
> but replaces the "L" with a square-root symbol (U+2A04)?  What will the length
> of foo[] be?  On the compiler where I just tried it, the string was three
> characters long.  Although a dollar sign happens to be one byte in ASCII,
> there is nothing in the C Standard that would forbid the use of a character
> set where a dollar sign required multiple bytes to store.
> 

However, I would expect something in the standard that requires the
first byte of that multibyte dollar sign not to be 'D', a character from
the basic set.

Philipp

[toc] | [prev] | [next] | [standalone]

#76135

On Sunday, November 15, 2015 at 3:50:55 PM UTC-6, Philipp Klaus Krause wrote:
> However, I would expect something in the standard that requires the
> first byte of that multibyte dollar sign not to be 'D', a character from
> the basic set.

I must have edited out one of my first sentences in my last reply, which
acknowledged that my particular example was probably somewhat broken for that
reason, but I can't think of anything, at least in the older versions of the
C Standard, that would actually say that.  There are many platforms, for
example, where a source file containing "The price is -\010L3.50" would show
up as "The price is L350" but with a [kinda sorta] British pound sign in
place of the "L".  Even an ancient ASR-33 would do that.

[toc] | [prev] | [next] | [standalone]

#76310

supercat@casperkitty.com wrote:

> On Sunday, November 15, 2015 at 3:50:55 PM UTC-6, Philipp Klaus Krause wrote:
> > However, I would expect something in the standard that requires the
> > first byte of that multibyte dollar sign not to be 'D', a character from
> > the basic set.
> 
> I must have edited out one of my first sentences in my last reply, which
> acknowledged that my particular example was probably somewhat broken for that
> reason, but I can't think of anything, at least in the older versions of the
> C Standard, that would actually say that.  There are many platforms, for
> example, where a source file containing "The price is -\010L3.50" would show
> up as "The price is L350" but with a [kinda sorta] British pound sign in
> place of the "L".  Even an ancient ASR-33 would do that.

But AFAICT, this would not be special-cased to an actual pound sign, but
exactly what it says: an L overprinted on a hyphen. Which is cheating,
and nothing to do with multibyte character sets.

Richard

[toc] | [prev] | [next] | [standalone]

#76186

supercat@casperkitty.com writes:

> On Saturday, November 14, 2015 at 1:50:37 PM UTC-6, Tim Rentsch wrote:
>> I don't think so.  If you want this hypothesis to be convincing
>> you would need to give a more substantive argument, including
>> specific citations for relevant passages in the Standard.
>
> What will happen if one feeds a UTF-8 source text containing:
>
>    char foo[] = "L";
>    char bar[] = { sizeof foo };
>
> but replaces the "L" with a square-root symbol (U+2A04)?  What will the length
> of foo[] be?  On the compiler where I just tried it, the string was three
> characters long.  Although a dollar sign happens to be one byte in ASCII,
> there is nothing in the C Standard that would forbid the use of a character
> set where a dollar sign required multiple bytes to store.

You over-snipped.

[toc] | [prev] | [next] | [standalone]

#75955

On 12.11.2015 17:29, James Kuyper wrote:
> On 11/12/2015 10:44 AM, Philipp Klaus Krause wrote:
>> On 12.11.2015 16:04, James Kuyper wrote:
>>
>>> Why exactly are you interested in how an implementation could weasel out
>>> of this requirement? As I understand it, there's a simple process that
>>> goes on when designing this part of an implementation of C: choose what
>>> extended character sets you want to support, and which encoding. Then
>>> choose a single character encoding capable of representing all of those
>>> characters, and define wchar_t to be a type big enough to store the
>>> representations of any of those characters. Did you have a different
>>> implementation design process in mind?
>>>
>>
>> 1) AFAIK, Windows compilers tend to use a 16-bit wchar_t, and assume
>> sequences of wchar_t to be UTF-16-encoded. Thus there are characters
>> that don't fit into a wchar_t. How do these implementations get around
>> the sandard requirement?
> 
> I've no idea - possibly by not conforming to the requirements?

So there are no standard-conforming C implementations targeting Windows?
Microsoft documetnation, e.g.
https://msdn.microsoft.com/de-de/library/yeby3zcb.aspx explicitly says
that wchar_t uses UTF-8. But that means there are characters that do not
fit into a single wchar_t, which seems to violate the standard requirements.

Philipp

[toc] | [prev] | [next] | [standalone]

#75960

On 11/13/2015 11:26 AM, Philipp Klaus Krause wrote:
> On 12.11.2015 17:29, James Kuyper wrote:
>> On 11/12/2015 10:44 AM, Philipp Klaus Krause wrote:
>>> On 12.11.2015 16:04, James Kuyper wrote:
>>>
>>>> Why exactly are you interested in how an implementation could weasel out
>>>> of this requirement? As I understand it, there's a simple process that
>>>> goes on when designing this part of an implementation of C: choose what
>>>> extended character sets you want to support, and which encoding. Then
>>>> choose a single character encoding capable of representing all of those
>>>> characters, and define wchar_t to be a type big enough to store the
>>>> representations of any of those characters. Did you have a different
>>>> implementation design process in mind?
>>>>
>>>
>>> 1) AFAIK, Windows compilers tend to use a 16-bit wchar_t, and assume
>>> sequences of wchar_t to be UTF-16-encoded. Thus there are characters
>>> that don't fit into a wchar_t. How do these implementations get around
>>> the sandard requirement?
>>
>> I've no idea - possibly by not conforming to the requirements?
> 
> So there are no standard-conforming C implementations targeting Windows?

You might be right about that, for all I know - I know more about what
the standard requires for implementations of C, than I do about what
actual C implementations actually do; and I'm even less of an expert in
that regard for Windows platforms.
However, your conclusion above doesn't follow from the information given
below. You can have an implementation of C targeting windows that
implements wchar_t differently from the way Microsoft's compiler does.

> Microsoft documetnation, e.g.
> https://msdn.microsoft.com/de-de/library/yeby3zcb.aspx explicitly says
> that wchar_t uses UTF-8. But that means there are characters that do not

It's been about 30 years since I studied German, and my German was never
good enough to read a document like this, anyway. Giving that URL to
translate.google.com produces a document in a language that resembles
English sufficiently well that I can get some idea of what's going on.

In order to enable this feature, it seems to be necessary to open the
file as follows:

FILE * fp = fopen ("newfile.txt", "RT +, ccs = Encoding");

According to the C standard,

"The argument mode points to a string. If the string is one of the
following, the file is open in the indicated mode. Otherwise, the
behavior is undefined.271)

r    open text file for reading
w    truncate to zero length or create text file for writing
wx   create text file for writing
a    append; open or create text file for writing at end-of-file
rb   open binary file for reading
wb   truncate to zero length or create binary file for writing
wbx  create binary file for writing
ab   append; open or create binary file for writing at end-of-file
r+   open text file for update (reading and writing)
w+   truncate to zero length or create text file for update
w+x  create text file for update
a+   append; open or create text file for update, writing at end-of-file
r+b or rb+   open binary file for update (reading and writing)
w+b or wb+   truncate to zero length or create binary file for update
w+bx or wb+x create binary file for update
a+b or ab+   append; open or create binary file for update, writing at
end-of-file" (7.21.5.3p3)

Note, in particular, the complete absence of any string that even
remotely resembles the mode string used in that example. C is a
case-sensitive language, so even the use of "R" rather than "r" would
make the behavior undefined. Therefore, when using such a mode string to
open a file, the behavior is undefined. The standard imposes no
requirements on the behavior of such a program.

> fit into a single wchar_t, which seems to violate the standard requirements.

In particular, the following requirement does NOT apply when the
behavior of the program is undefined:
"wchar_t
"which is an integer type whose range of values can represent distinct
codes for all members of the largest extended character set specified
among the supported locales; the null character shall have the code
value zero."  (7.19p2)

[toc] | [prev] | [next] | [standalone]

#75985

On 13-Nov-15 12:36, James Kuyper wrote:
> On 11/13/2015 11:26 AM, Philipp Klaus Krause wrote:
>> On 12.11.2015 17:29, James Kuyper wrote:
>>> On 11/12/2015 10:44 AM, Philipp Klaus Krause wrote:
>>>> 1) AFAIK, Windows compilers tend to use a 16-bit wchar_t, and
>>>> assume sequences of wchar_t to be UTF-16-encoded. Thus there
>>>> are characters that don't fit into a wchar_t. How do these
>>>> implementations get around the sandard requirement?
>>> 
>>> I've no idea - possibly by not conforming to the requirements?
>> 
>> So there are no standard-conforming C implementations targeting
>> Windows?
> 
> You might be right about that, for all I know - I know more about what
> the standard requires for implementations of C, than I do about what
> actual C implementations actually do; and I'm even less of an expert in
> that regard for Windows platforms.

Simply put, Microsoft uses UTF-16 encoding for wchar_t; the Standard
seems to indicate that isn't valid, for the same reason UTF-8 isn't.

UCS-2 _does_ seem valid; that obviously prevents use of any non-BMP
characters, though, which is presumably why they switched to UTF-16.

> However, your conclusion above doesn't follow from the information
> given below. You can have an implementation of C targeting windows
> that implements wchar_t differently from the way Microsoft's compiler
> does.

True, but then you'd lose binary compatibility with every existing API
unless you convert to/from UTF-16 at every call, which is unrealistic.

AFAIK, every other implementation targeting Windows has chosen binary
compatibility with Microsoft over standards compliance.

>> Microsoft documetnation, e.g.
>> https://msdn.microsoft.com/de-de/library/yeby3zcb.aspx
>> explicitly says that wchar_t uses UTF-8. But that means there are
>> characters that do not
> 
> It's been about 30 years since I studied German, and my German was
> never good enough to read a document like this, anyway. Giving that
> URL to translate.google.com produces a document in a language that
> resembles English sufficiently well that I can get some idea of
> what's going on.

Here's the actual English version:
https://msdn.microsoft.com/en-us/library/yeby3zcb.aspx

> In order to enable this feature, it seems to be necessary to open the
> file as follows:
> 
> FILE * fp = fopen ("newfile.txt", "RT +, ccs = Encoding");
> 
> According to the C standard,
> ...
> Note, in particular, the complete absence of any string that even
> remotely resembles the mode string used in that example.

Yes, but Microsoft is infamous for standards non-compliance, so that
shouldn't be much of a surprise.

>> fit into a single wchar_t, which seems to violate the standard
>> requirements.
> 
> In particular, the following requirement does NOT apply when the
> behavior of the program is undefined:

Isn't that backward?

If an implementation's wchar_t itself violates the Standard, then I'd
think any use of that type invokes undefined behavior, though in this
case, actual problems are unlikely until you use a non-BMP character.

S

-- 
Stephen Sprunk         "God does not play dice."  --Albert Einstein
CCIE #3723         "God is an inveterate gambler, and He throws the
K5SSS        dice at every possible opportunity." --Stephen Hawking

[toc] | [prev] | [next] | [standalone]

#75987

On 11/13/2015 05:37 PM, Stephen Sprunk wrote:
> On 13-Nov-15 12:36, James Kuyper wrote:
>> On 11/13/2015 11:26 AM, Philipp Klaus Krause wrote:
>>> On 12.11.2015 17:29, James Kuyper wrote:
>>>> On 11/12/2015 10:44 AM, Philipp Klaus Krause wrote:
>>>>> 1) AFAIK, Windows compilers tend to use a 16-bit wchar_t, and
>>>>> assume sequences of wchar_t to be UTF-16-encoded. Thus there
>>>>> are characters that don't fit into a wchar_t. How do these
>>>>> implementations get around the sandard requirement?
>>>>
>>>> I've no idea - possibly by not conforming to the requirements?
>>>
>>> So there are no standard-conforming C implementations targeting
>>> Windows?
>>
>> You might be right about that, for all I know - I know more about what
>> the standard requires for implementations of C, than I do about what
>> actual C implementations actually do; and I'm even less of an expert in
>> that regard for Windows platforms.
> 
> Simply put, Microsoft uses UTF-16 encoding for wchar_t; the Standard
> seems to indicate that isn't valid, for the same reason UTF-8 isn't.
> 
> UCS-2 _does_ seem valid; that obviously prevents use of any non-BMP
> characters, though, which is presumably why they switched to UTF-16.
...
>>> Microsoft documetnation, e.g.
>>> https://msdn.microsoft.com/de-de/library/yeby3zcb.aspx
>>> explicitly says that wchar_t uses UTF-8. But that means there are
>>> characters that do not
>>
>> It's been about 30 years since I studied German, and my German was
>> never good enough to read a document like this, anyway. Giving that
>> URL to translate.google.com produces a document in a language that
>> resembles English sufficiently well that I can get some idea of
>> what's going on.
> 
> Here's the actual English version:
> https://msdn.microsoft.com/en-us/library/yeby3zcb.aspx

It should have occurred to me that de-de was a language specification.

>> In order to enable this feature, it seems to be necessary to open the
>> file as follows:
>>
>> FILE * fp = fopen ("newfile.txt", "RT +, ccs = Encoding");
>>
>> According to the C standard,
>> ...
>> Note, in particular, the complete absence of any string that even
>> remotely resembles the mode string used in that example.
> 
> Yes, but Microsoft is infamous for standards non-compliance, so that
> shouldn't be much of a surprise.

This is not an example non-compliance. This is practically a textbook
example of a fully conforming extension to C. The behavior when passing
such a string to any implementation of fopen() is undefined, which means
that there's absolutely nothing that the implementation can do as a
result of such a call that would render it non-conforming.

>>> fit into a single wchar_t, which seems to violate the standard
>>> requirements.
>>
>> In particular, the following requirement does NOT apply when the
>> behavior of the program is undefined:
> 
> Isn't that backward?
> 
> If an implementation's wchar_t itself violates the Standard, then I'd
> think any use of that type invokes undefined behavior, though in this
> case, actual problems are unlikely until you use a non-BMP character.

I got the impression from the English translation of the German version
of that page, that wchar_t violates the standard's requirements only
when fopen() is called with a mode argument that renders the behavior
undefined. However, reading it in proper English, that impression
disappears - it appears to be UTF-16 at all times. It should be trivial
to put together a strictly conforming program that demonstrates the
non-conformance of that implementation.

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | comp.lang.c

csiph-web