Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #75432 > unrolled thread

Is wchar_t really useful?

Back to article view | Back to comp.lang.c

Contents

  Is wchar_t really useful? Philipp Klaus Krause <pkk@spth.de> - 2015-11-09 12:02 +0100
    Re: Is wchar_t really useful? Malcolm McLean <malcolm.mclean5@btinternet.com> - 2015-11-09 03:24 -0800
      Re: Is wchar_t really useful? Stephen Sprunk <stephen@sprunk.org> - 2015-11-10 18:33 -0600
        Re: Is wchar_t really useful? Ian Collins <ian-news@hotmail.com> - 2015-11-11 13:56 +1300
    Re: Is wchar_t really useful? James Kuyper <jameskuyper@verizon.net> - 2015-11-09 09:03 -0500
      Re: Is wchar_t really useful? Philipp Klaus Krause <pkk@spth.de> - 2015-11-12 15:22 +0100
        Re: Is wchar_t really useful? James Kuyper <jameskuyper@verizon.net> - 2015-11-12 10:04 -0500
          Re: Is wchar_t really useful? Philipp Klaus Krause <pkk@spth.de> - 2015-11-12 16:44 +0100
            Re: Is wchar_t really useful? James Kuyper <jameskuyper@verizon.net> - 2015-11-12 11:29 -0500
              Re: Is wchar_t really useful? supercat@casperkitty.com - 2015-11-12 09:14 -0800
                Re: Is wchar_t really useful? Tim Rentsch <txr@alumni.caltech.edu> - 2015-11-14 11:50 -0800
                  Re: Is wchar_t really useful? supercat@casperkitty.com - 2015-11-15 13:44 -0800
                    Re: Is wchar_t really useful? Philipp Klaus Krause <pkk@spth.de> - 2015-11-15 22:50 +0100
                      Re: Is wchar_t really useful? supercat@casperkitty.com - 2015-11-15 14:46 -0800
                        Re: Is wchar_t really useful? raltbos@xs4all.nl (Richard Bos) - 2015-11-17 22:34 +0000
                    Re: Is wchar_t really useful? Tim Rentsch <txr@alumni.caltech.edu> - 2015-11-16 07:25 -0800
              Re: Is wchar_t really useful? Philipp Klaus Krause <pkk@spth.de> - 2015-11-13 17:26 +0100
                Re: Is wchar_t really useful? James Kuyper <jameskuyper@verizon.net> - 2015-11-13 13:36 -0500
                  Re: Is wchar_t really useful? Stephen Sprunk <stephen@sprunk.org> - 2015-11-13 16:37 -0600
                    Re: Is wchar_t really useful? James Kuyper <jameskuyper@verizon.net> - 2015-11-13 17:58 -0500
                Re: Is wchar_t really useful? Stephen Sprunk <stephen@sprunk.org> - 2015-11-13 16:01 -0600
                  Re: Is wchar_t really useful? Philipp Klaus Krause <pkk@spth.de> - 2015-11-13 23:36 +0100
      Re: Is wchar_t really useful? Philipp Klaus Krause <pkk@spth.de> - 2015-11-12 15:28 +0100
        Re: Is wchar_t really useful? James Kuyper <jameskuyper@verizon.net> - 2015-11-12 10:26 -0500
      Re: Is wchar_t really useful? Stephen Sprunk <stephen@sprunk.org> - 2015-11-12 14:27 -0600
    Re: Is wchar_t really useful? Lőrinczy Zsigmond <nospam@for.me> - 2015-11-13 05:02 +0100
      Re: Is wchar_t really useful? Philipp Klaus Krause <pkk@spth.de> - 2015-11-13 11:52 +0100
        Re: Is wchar_t really useful? Malcolm McLean <malcolm.mclean5@btinternet.com> - 2015-11-13 05:17 -0800
          Re: Is wchar_t really useful? Stephen Sprunk <stephen@sprunk.org> - 2015-11-13 12:58 -0600
          Re: Is wchar_t really useful? supercat@casperkitty.com - 2015-11-13 11:14 -0800
      Re: Is wchar_t really useful? Stephen Sprunk <stephen@sprunk.org> - 2015-11-13 12:45 -0600
    Re: Is wchar_t really useful? Philipp Klaus Krause <pkk@spth.de> - 2015-11-13 18:24 +0100
      Re: Is wchar_t really useful? James Kuyper <jameskuyper@verizon.net> - 2015-11-13 13:55 -0500
        Re: Is wchar_t really useful? Philipp Klaus Krause <pkk@spth.de> - 2015-11-13 20:39 +0100
          Re: Is wchar_t really useful? James Kuyper <jameskuyper@verizon.net> - 2015-11-13 14:54 -0500
          Re: Is wchar_t really useful? Stephen Sprunk <stephen@sprunk.org> - 2015-11-13 17:04 -0600

Page 2 of 2 — ← Prev page 1 [2]

#75977

On 13-Nov-15 10:26, Philipp Klaus Krause wrote:
> On 12.11.2015 17:29, James Kuyper wrote:
>> On 11/12/2015 10:44 AM, Philipp Klaus Krause wrote:
>>> 1) AFAIK, Windows compilers tend to use a 16-bit wchar_t, and
>>> assume sequences of wchar_t to be UTF-16-encoded. Thus there are
>>> characters that don't fit into a wchar_t. How do these
>>> implementations get around the sandard requirement?
>> 
>> I've no idea - possibly by not conforming to the requirements?
> 
> So there are no standard-conforming C implementations targeting
> Windows? Microsoft documetnation, e.g. 
> https://msdn.microsoft.com/de-de/library/yeby3zcb.aspx explicitly
> says that wchar_t uses UTF-8.

The English version says that wchar_t is UTF-16, not UTF-8:

"When a file is opened in Unicode mode, input functions translate the
data that's read from the file into UTF-16 data stored as type wchar_t.
Functions that write to a file opened in Unicode mode expect buffers
that contain UTF-16 data stored as type wchar_t."

There is a (new) flag that lets you specify a _file_ is UTF-8 rather
than UTF-16, but that just converts to/from UTF-16 on read/write; it
doesn't change the representation of wchar_t itself.

> But that means there are characters that do not fit into a single
> wchar_t, which seems to violate the standard requirements.

Yes, this has been a longstanding problem on Windows and why most
Windows software breaks when it encounters non-BMP characters.

S

-- 
Stephen Sprunk         "God does not play dice."  --Albert Einstein
CCIE #3723         "God is an inveterate gambler, and He throws the
K5SSS        dice at every possible opportunity." --Stephen Hawking

[toc] | [prev] | [next] | [standalone]

#75984

On 13.11.2015 23:01, Stephen Sprunk wrote:
> On 13-Nov-15 10:26, Philipp Klaus Krause wrote:
>> On 12.11.2015 17:29, James Kuyper wrote:
>>> On 11/12/2015 10:44 AM, Philipp Klaus Krause wrote:
>>>> 1) AFAIK, Windows compilers tend to use a 16-bit wchar_t, and
>>>> assume sequences of wchar_t to be UTF-16-encoded. Thus there are
>>>> characters that don't fit into a wchar_t. How do these
>>>> implementations get around the sandard requirement?
>>>
>>> I've no idea - possibly by not conforming to the requirements?
>>
>> So there are no standard-conforming C implementations targeting
>> Windows? Microsoft documetnation, e.g. 
>> https://msdn.microsoft.com/de-de/library/yeby3zcb.aspx explicitly
>> says that wchar_t uses UTF-8.
> 
> The English version says that wchar_t is UTF-16, not UTF-8:

Sorry, my mistake. Of course the German version says UTF-16 as well. I
was already thinking that if Windows somehow manages to get a
standard-conforming UTF-16 wchar_t, there should be a way for an
implementation to get a standard-conforming UTF-8 wchar_t.

Philipp

[toc] | [prev] | [next] | [standalone]

#75820

On 09.11.2015 15:03, James Kuyper wrote:

> If you need to process one character at a time, then even if you can use
> a C2011 implementation, you might need to use a routine that uses
> wchar_t in the argument list or in the return type, because more often
> than not there's no corresponding routine that uses char16_t or
> char32_t. That's reason #3 for using wchar_t.
> 

For conversion from char to char16_t there is mbrtoc16() which converts
a char multibyte character to a sequence of char16_t.
But why is there no equivalent for conversion the other way? There is
this weird c16rtomb(), it seems to be restricted to characters that fit
into a single char16_t.

Philipp

[toc] | [prev] | [next] | [standalone]

#75830

On 11/12/2015 09:28 AM, Philipp Klaus Krause wrote:
> On 09.11.2015 15:03, James Kuyper wrote:
> 
>> If you need to process one character at a time, then even if you can use
>> a C2011 implementation, you might need to use a routine that uses
>> wchar_t in the argument list or in the return type, because more often
>> than not there's no corresponding routine that uses char16_t or
>> char32_t. That's reason #3 for using wchar_t.
>>
> 
> For conversion from char to char16_t there is mbrtoc16() which converts
> a char multibyte character to a sequence of char16_t.
> But why is there no equivalent for conversion the other way? There is
> this weird c16rtomb(), it seems to be restricted to characters that fit
> into a single char16_t.

The standard allows an implementation to define __STDC_UTF_16__, in
which chase the standard library routines are supposed to interpret
char16_t values as using the UTF-16 encoding. I'm no expert in such
things, but it doesn't seem to me that the char16_t conversion routines
have an interface that is suitable for dealing with that possibility.
-- 
James Kuyper

[toc] | [prev] | [next] | [standalone]

#75875

On 09-Nov-15 08:03, James Kuyper wrote:
> Many (most?) of the C standard library routines that take or return 
> char* arguments are defined as interpreting those pointers as
> pointing to a multibyte character string (MBCS). If it were up to me,
> there'd be one term "single byte character" referring to a single
> char object that can represent a character, while "multibyte
> character" would refer only to sequences of two or more char objects
> that together represent a single character. The committee took a
> different approach, defining the term "multibyte character" to
> include both of those categories (3.7.2p1).

If a Single-Byte Character is a member of a Single-Byte Character Set,
and a Double-Byte Character is a member of a Double-Byte Character Set,
then a Multi-Byte Character is a member of a Multi-Byte Character Set.

Single- and Double-Byte Characters are have a fixed width; Multi-Byte
Characters do not.  Despite the usual implication of "Multi-", it can
include one-byte characters.

Sadly, MBCS code often treats MBCS as if it means "SBCS and/or DBCS" and
therefore mishandles the three- or four-byte characters of UTF-8, UTF-16
and UCS-4/UTF-32.

S

-- 
Stephen Sprunk         "God does not play dice."  --Albert Einstein
CCIE #3723         "God is an inveterate gambler, and He throws the
K5SSS        dice at every possible opportunity." --Stephen Hawking

[toc] | [prev] | [next] | [standalone]

#75920

On 2015.11.09. 12:02, Philipp Klaus Krause wrote:
> The C standard has wchar_t. But I don't really see any advantage over
> char. Maybe if __STDC_ISO_10646__ is defined. But even then, I guess
> char, char16_t and char32_t are more useful.
> Is there a point in using wchar_t?

Well, wchar_t being platform-dependent,
it has no use if you want to work with a defined encoding such as
utf16le, utf16be, utf32le, utf32be etc,
but if you don't know/care about details, only want to
'use more than 256 characters' then it might be useful.

[toc] | [prev] | [next] | [standalone]

#75935

On 13.11.2015 05:02, Lőrinczy Zsigmond wrote:
> On 2015.11.09. 12:02, Philipp Klaus Krause wrote:
>> The C standard has wchar_t. But I don't really see any advantage over
>> char. Maybe if __STDC_ISO_10646__ is defined. But even then, I guess
>> char, char16_t and char32_t are more useful.
>> Is there a point in using wchar_t?
> 
> Well, wchar_t being platform-dependent,
> it has no use if you want to work with a defined encoding such as
> utf16le, utf16be, utf32le, utf32be etc,
> but if you don't know/care about details, only want to
> 'use more than 256 characters' then it might be useful.

I don't see why a system with a 16-bit wchar_t would be more
standard-compliant than one with an 8-bit wchar_t.
Assuming the supported charcter set is at most 256 characters, it would
be fully compliant. Assuming the supported character set is all of
Unicode, from this discussion it seems that an 8-bit wchar_t would not
be compliant, but neither would be a 16-bit wchar_t.

Philipp

[toc] | [prev] | [next] | [standalone]

#75943

On Friday, November 13, 2015 at 10:53:01 AM UTC, Philipp Klaus Krause wrote:
>
> I don't see why a system with a 16-bit wchar_t would be more
> standard-compliant than one with an 8-bit wchar_t.
> Assuming the supported charcter set is at most 256 characters, it would
> be fully compliant. Assuming the supported character set is all of
> Unicode, from this discussion it seems that an 8-bit wchar_t would not
> be compliant, but neither would be a 16-bit wchar_t.
> 
16 bit w_chars are good enough for most languages. Back in the olden
days it was believed that they would suffice to encode every character
in every language of the computer-using peoples. That's not the case,
but you can often ignore the exceptions - if you've no-one who can 
translate to the Far Eastern or obscure languages, often you can't 
support text in that language anyway.

[toc] | [prev] | [next] | [standalone]

#75964

On 13-Nov-15 07:17, Malcolm McLean wrote:
> 16 bit w_chars are good enough for most languages. Back in the olden 
> days it was believed that they would suffice to encode every
> character in every language of the computer-using peoples. That's not
> the case, but you can often ignore the exceptions - if you've no-one
> who can translate to the Far Eastern or obscure languages, often you
> can't support text in that language anyway.

Even if _your_ text doesn't need non-BMP characters, your users may
still expect to be able to use them.

For instance, you might translate your entire GUI into Chinese and use
only BMP characters, or maybe you don't bother because most of them can
read English now anyway, but if you can't accept non-BMP characters, a
very large number of Chinese users won't be able to enter _their own
names_ in the name field.

It's not just about Klingons or ancient Egyptians; the limitations of
the BMP are real and affect millions of real, living people today.

S

-- 
Stephen Sprunk         "God does not play dice."  --Albert Einstein
CCIE #3723         "God is an inveterate gambler, and He throws the
K5SSS        dice at every possible opportunity." --Stephen Hawking

[toc] | [prev] | [next] | [standalone]

#75965

On Friday, November 13, 2015 at 7:17:20 AM UTC-6, Malcolm McLean wrote:
> 16 bit w_chars are good enough for most languages. Back in the olden
> days it was believed that they would suffice to encode every character
> in every language of the computer-using peoples. That's not the case,
> but you can often ignore the exceptions - if you've no-one who can 
> translate to the Far Eastern or obscure languages, often you can't 
> support text in that language anyway.

Unfortunately, one never knows what characters someone might want to use.
Someone writing about music might want to use a double-sharp character,
for example (U+1D12A), or a "forte" symbol (U+1D191), or maybe a guitar
(U+1F3B8).  Is there any reason to arbitrarily limit them to characters
that have been declared worthy of inclusion in the BMP?

[toc] | [prev] | [next] | [standalone]

#75962

On 12-Nov-15 22:02, Lőrinczy Zsigmond wrote:
> On 2015.11.09. 12:02, Philipp Klaus Krause wrote:
>> The C standard has wchar_t. But I don't really see any advantage
>> over char. Maybe if __STDC_ISO_10646__ is defined. But even then, I
>> guess char, char16_t and char32_t are more useful. Is there a point
>> in using wchar_t?
> 
> Well, wchar_t being platform-dependent, it has no use if you want to
> work with a defined encoding such as utf16le, utf16be, utf32le,
> utf32be etc, but if you don't know/care about details, only want to 
> 'use more than 256 characters' then it might be useful.

The internal representation of a wchar_t is irrelevant as long as it can
hold a code point; when you communicate with the outside world, you
transform code points to/from a particular encoding, and there are
dozens of them available.  Trying to force wchar_t to use a particular
encoding is pointless because it will inevitably be the wrong one and
you'll need to transform it anyway.

That said, UCS-4/UTF-32 (native endian) is the most obvious choice for
an internal representation since it uses one fixed-width code unit per
code point and therefore is the simplest to work with and to transform
to/from, and UTF-8 is the most obvious choice for all other purposes.

S

-- 
Stephen Sprunk         "God does not play dice."  --Albert Einstein
CCIE #3723         "God is an inveterate gambler, and He throws the
K5SSS        dice at every possible opportunity." --Stephen Hawking

[toc] | [prev] | [next] | [standalone]

#75956

Could an implementation decide that characters in the execution
character set are just the values of char? wchar_t could be 8 bits.
The source character set could be utf-8 encoded unicode, and the
characters in the execution character set would be the bytes of the
utf-8 sequence.
Would this violate "In a character constant or string literal, members
of the execution character set shall be represented by corresponding
members of the source character set […]" from 5.2.1?
Unfortunately, that would mean that functions like mbrtoc32() couldn't
convert to UTF-32 anymore, since they are supposed to read and interpret
only one multibyte character at a time. Any other problems?

Philipp

P.S.: So far, all this wide character stuff in the C standard still
doesn't make sense to me yet.

[toc] | [prev] | [next] | [standalone]

#75963

On 11/13/2015 12:24 PM, Philipp Klaus Krause wrote:
> Could an implementation decide that characters in the execution
> character set are just the values of char? wchar_t could be 8 bits.
> The source character set could be utf-8 encoded unicode, and the
> characters in the execution character set would be the bytes of the
> utf-8 sequence.

UTF-8 is fine for the multibyte character encoding, but it doesn't work
for wchar_t:

"wchar_t
which is an integer type whose range of values can represent distinct
codes for all members of the largest extended character set specified
among the supported locales." (7.19p2)


> Would this violate "In a character constant or string literal, members
> of the execution character set shall be represented by corresponding
> members of the source character set […]" from 5.2.1?
> Unfortunately, that would mean that functions like mbrtoc32() couldn't
> convert to UTF-32 anymore, since they are supposed to read and interpret
> only one multibyte character at a time.

How did you reach that conclusion? mbrtoc32() reads multiple characters
through it's char* parameter:

"If s is not a null pointer, the mbrtoc32 function inspects at most n
bytes beginning with the byte pointed to by s to determine the number of
bytes needed to complete the next multibyte character (including any
shift sequences)."

I just noticed that I hadn't paid enough attention to the description of
mbrtoc32(). It can also convert a single multibyte input character into
multiple char32_t values - you just need to call mbrtoc32() multiple
times. It only stores one char32_t value at a time, but if the correct
translation results in multiple char32_t values, then "Subsequent calls
will store successive wide characters without consuming any additional
input until all the characters have been stored." mbrtoc16() has similar
wording - so that explains why UTF-16 and UTF-32 are permissible
encoding for char16_t and char32_t, respectively.

mbrtowc() does not have the wording about "Subsequent calls", which
confirms that UTF-like encodings are not permitted for wchar_t.

[toc] | [prev] | [next] | [standalone]

#75966

On 13.11.2015 19:55, James Kuyper wrote:
> On 11/13/2015 12:24 PM, Philipp Klaus Krause wrote:
>> Could an implementation decide that characters in the execution
>> character set are just the values of char? wchar_t could be 8 bits.
>> […]
>> Would this violate "In a character constant or string literal, members
>> of the execution character set shall be represented by corresponding
>> members of the source character set […]" from 5.2.1?
>> Unfortunately, that would mean that functions like mbrtoc32() couldn't
>> convert to UTF-32 anymore, since they are supposed to read and interpret
>> only one multibyte character at a time.
> 
> How did you reach that conclusion? mbrtoc32() reads multiple characters
> through it's char* parameter:
> 
> "If s is not a null pointer, the mbrtoc32 function inspects at most n
> bytes beginning with the byte pointed to by s to determine the number of
> bytes needed to complete the next multibyte character (including any
> shift sequences)."

"If the function determines that the next multibyte character is
complete and valid, it determines the values of the corresponding wide
characters and then, if pc32 is not a null pointer, stores the value of
the first (or only) such character in the object pointed to by pc32."

So if the implementation considers all bytes in utf-8 sequences to be
individual characters, then mbrtoc32() will have to write out at least
one char32_t for each char it reads.
Such an implementation doesn't look that absurd to me: After all in the
end mutiple uncidoe code points will have to be converted to one glyph
anyway, due to combining characters. So even with UTF-32 we do not have
a 1-to-1 correspondance of characters and glyphs.

Philipp

[toc] | [prev] | [next] | [standalone]

#75968

On 11/13/2015 02:39 PM, Philipp Klaus Krause wrote:
...
> So if the implementation considers all bytes in utf-8 sequences to be
> individual characters,

A conforming implementation  of C is free to use a multi-byte encoding
scheme that does not, in fact, ever use more than one byte to encode any
character. Implementations which used either an ASCII or EBCDIC variant
have in fact been quite common. However, in what sense would such an
encoding count as UTF-8?

> then mbrtoc32() will have to write out at least
> one char32_t for each char it reads.

True.

[toc] | [prev] | [next] | [standalone]

#75988

On 13-Nov-15 13:39, Philipp Klaus Krause wrote:
> So if the implementation considers all bytes in utf-8 sequences to
> be individual characters, then mbrtoc32() will have to write out at
> least one char32_t for each char it reads.  Such an implementation
> doesn't look that absurd to me: 

That goes against the clear intent: UTF-8 is a multibyte encoding that
gets translated to one wchar_t, one char32_t or one or two char16_t's.

If you view UTF-8 as individual bytes, then there's no point in using
wide or multibyte character functions in the first place; just use
narrow chars for everything and be very careful in rare cases when you
need to split a string.  That's what it was designed for!

> After all in the end mutiple uncidoe code points will have to be
> converted to one glyph anyway, due to combining characters. So even
> with UTF-32 we do not have a 1-to-1 correspondance of characters
> and glyphs.

s/glyph/grapheme cluster/

S

-- 
Stephen Sprunk         "God does not play dice."  --Albert Einstein
CCIE #3723         "God is an inveterate gambler, and He throws the
K5SSS        dice at every possible opportunity." --Stephen Hawking

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | comp.lang.c

csiph-web