Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #43231 > unrolled thread

wchar_t: signed or unsigned?

Back to article view | Back to comp.lang.c

Contents

  wchar_t: signed or unsigned? DSF <notavalid@address.here> - 2014-04-21 12:55 -0400
    Re: wchar_t: signed or unsigned? Xavier Roche <xroche@free.fr.NOSPAM.invalid> - 2014-04-21 20:02 +0200
    Re: wchar_t: signed or unsigned? Seungbeom Kim <musiphil@bawi.org> - 2014-04-21 11:07 -0700
      Re: wchar_t: signed or unsigned? Keith Thompson <kst-u@mib.org> - 2014-04-21 11:24 -0700
        Re: wchar_t: signed or unsigned? Seungbeom Kim <musiphil@bawi.org> - 2014-04-21 11:57 -0700
          Re: wchar_t: signed or unsigned? James Kuyper <jameskuyper@verizon.net> - 2014-04-21 15:13 -0400
    Re: wchar_t: signed or unsigned? James Kuyper <jameskuyper@verizon.net> - 2014-04-21 14:31 -0400
    Re: wchar_t: signed or unsigned? Stephen Sprunk <stephen@sprunk.org> - 2014-04-21 15:28 -0500
      Re: wchar_t: signed or unsigned? James Kuyper <jameskuyper@verizon.net> - 2014-04-21 16:48 -0400
        Re: wchar_t: signed or unsigned? Keith Thompson <kst-u@mib.org> - 2014-04-21 14:08 -0700
    Re: wchar_t: signed or unsigned? Malcolm McLean <malcolm.mclean5@btinternet.com> - 2014-04-22 03:21 -0700
      Re: wchar_t: signed or unsigned? Stephen Sprunk <stephen@sprunk.org> - 2014-04-22 10:48 -0500
      Re: wchar_t: signed or unsigned? Keith Thompson <kst-u@mib.org> - 2014-04-22 09:10 -0700
    Re: wchar_t: signed or unsigned? gordonb.8ql82@burditt.org (Gordon Burditt) - 2014-04-23 03:31 -0500
      Re: wchar_t: signed or unsigned? Stephen Sprunk <stephen@sprunk.org> - 2014-04-25 10:00 -0500
        Re: wchar_t: signed or unsigned? Keith Thompson <kst-u@mib.org> - 2014-04-25 09:00 -0700
          Re: wchar_t: signed or unsigned? Malcolm McLean <malcolm.mclean5@btinternet.com> - 2014-04-25 10:27 -0700
          Re: wchar_t: signed or unsigned? Stephen Sprunk <stephen@sprunk.org> - 2014-04-25 16:43 -0500

#43231 — wchar_t: signed or unsigned?

Hello, group!

  Is wchar_t signed or unsigned?  Is it different in C++ than in C? Or
is it one of those things left up to the compiler writer?

  I need to know because I'm dealing with an array of chars, but they
are treated by the outside world as ints.  EOFs and error codes are
expressed in negative values.  So I am using unsigned chars for the
array to avoid the possibility of one of the array members being > 127
and being sign extended to a negative int.

  I also have an array of wchar_t for dealing with Windows 16-bit
Unicode.  I don't know if Unicode utilizes the fifteenth bit, but I
don't want to take any chances.  In my compiler, wchar_t is defined as
"unsigned short" so I don't expect any problems.  But I'd like to know
the official word on it.  I did Google the question, but found page
after page espousing which form of Unicode is best.  Interesting
reading, but not an answer.

Thanks for any info,
DSF
"'Later' is the beginning of what's not to be."
D.S. Fiscus

[toc] | [next] | [standalone]

#43239

Le 21/04/2014 18:55, DSF a écrit :
>   Is wchar_t signed or unsigned?  Is it different in C++ than in C? Or
> is it one of those things left up to the compiler writer?

This is not totally specified ; ie. the standard says:

"Type wchar_t is a distinct type whose values can represent distinct
codes for all members of the largest extended character set specified
among the supported locales (22.3.1). Type wchar_t shall have the same
size, signedness, and alignment requirements (3.11) as one of the other
integral types, called its underlying type."

Note the "integral type" expression (ie. not "[un]signed integral type"
as in other parts of the standard)

However (and this is my personal feeling), wchar_t should probably be
unsigned, like char16_t and char32_t:

"Types char16_t and char32_t denote distinct types with the same size,
signedness, and alignment as uint_least16_t and uint_least32_t,
respectively, in <stdint.h>, called the underlying types."

But this is only my personal opinion :)

>   I also have an array of wchar_t for dealing with Windows 16-bit
> Unicode.  I don't know if Unicode utilizes the fifteenth bit, but I
> don't want to take any chances.

Yes, Unicode has ranges within 0x8000 to 0xFFFF (even if some characters
are undefined, such as FFFE or FFFF). Plenty of ranges, actually. DO NOT
use signed 16-bit integer without care :)

However, if 16 bits are a bit short (haha) to represent an Unicode
character, 20 bits "should be enough for everyone" (there are only 16
Unicode 65K-planes, and the Unicode standard does not plan (haha) to
have more in the future AFAIK)

[toc] | [prev] | [next] | [standalone]

#43241

On 2014-04-21 09:55, DSF wrote:
> Hello, group!
> 
>   Is wchar_t signed or unsigned?

It can be either.
You can test it with WCHAR_MIN (defined in <stdint.h>): it shall
be <= -127 if wchar_t is signed, and 0 if wchar_t is unsigned.

> Is it different in C++ than in C?

Same in C++, despite the difference that it is a distinct type
defined as a keyword in C++, rather than a typedef name as in C.

> Or is it one of those things left up to the compiler writer?

Yes.
I don't see it mentioned even as implementation-defined;
I think it is unspecified.

>   I need to know because I'm dealing with an array of chars, but they
> are treated by the outside world as ints.  EOFs and error codes are
> expressed in negative values.  So I am using unsigned chars for the
> array to avoid the possibility of one of the array members being > 127
> and being sign extended to a negative int.

If the values you need to store include character values *and EOF*,
you can store them as int, with char values cast to unsigned char.
(See what fgetc() does, for example.)
If you need to store only character values, it doesn't matter whether
you store them as (plain) char or unsigned char; you just have to
make sure to cast plain char values to unsigned char before giving
them in a 'int' context where it could be taken as EOF, such as
isupper() and other <ctype.h> friends.

I don't see why you consider wchar_t for this scenario, unless
the actual characters you're dealing with are wide characters.

>   I also have an array of wchar_t for dealing with Windows 16-bit
> Unicode.  I don't know if Unicode utilizes the fifteenth bit, but I
> don't want to take any chances.  In my compiler, wchar_t is defined as
> "unsigned short" so I don't expect any problems.  But I'd like to know
> the official word on it.  I did Google the question, but found page
> after page espousing which form of Unicode is best.  Interesting
> reading, but not an answer.

Full Unicode codespace uses 21 bits (from 0 to 10FFFF); 16 bits
can only represent codepoints in Basic Multilingual Plane (BMP).
(You can store UTF-16 values in 16-bit wchar_t objects, but they
will be a fixed-length encoding only for the characters in BMP;
surrogate pairs will take up more space.)

Using wchar_t defined as unsigned short cannot represent EOF,
just as unsigned char cannot.

If you're explicitly dealing with UTF-16, char16_t (declared in <uchar.h>
to be the same as uint_least16_t) seems like a more proper choice.

-- 
Seungbeom Kim

[toc] | [prev] | [next] | [standalone]

#43243

Seungbeom Kim <musiphil@bawi.org> writes:
> On 2014-04-21 09:55, DSF wrote:
>> Hello, group!
>> 
>>   Is wchar_t signed or unsigned?

Yes.  8-)}

> It can be either.
> You can test it with WCHAR_MIN (defined in <stdint.h>): it shall
> be <= -127 if wchar_t is signed, and 0 if wchar_t is unsigned.
>
>> Is it different in C++ than in C?
>
> Same in C++, despite the difference that it is a distinct type
> defined as a keyword in C++, rather than a typedef name as in C.
>
>> Or is it one of those things left up to the compiler writer?
>
> Yes.
> I don't see it mentioned even as implementation-defined;
> I think it is unspecified.
[...]

It's implementation-defined.

The values of WCHAR_MIN and WCHAR_MAX, macros defined in <stdint.h>, are
explicitly implementation-defined (which means that the implementation
must document them).

See N1570 7.20.3.

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
Working, but not speaking, for JetHead Development, Inc.
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [next] | [standalone]

#43245

On 2014-04-21 11:24, Keith Thompson wrote:
> Seungbeom Kim <musiphil@bawi.org> writes:
>> You can test it with WCHAR_MIN (defined in <stdint.h>): it shall
>> be <= -127 if wchar_t is signed, and 0 if wchar_t is unsigned.
>>
>>> Or is it one of those things left up to the compiler writer?
>>
>> Yes.
>> I don't see it mentioned even as implementation-defined;
>> I think it is unspecified.
> 
> It's implementation-defined.
> 
> The values of WCHAR_MIN and WCHAR_MAX, macros defined in <stdint.h>, are
> explicitly implementation-defined (which means that the implementation
> must document them).
> 
> See N1570 7.20.3.

Where do you think I got the requirement about WCHAR_MIN quoted above? :)

I knew the limit values were implementation-defined, but didn't see
anything about the signedness of wchar_t itself, and didn't know
the implementation-definedness could be applied transitively.

-- 
Seungbeom Kim

[toc] | [prev] | [next] | [standalone]

#43249

On 04/21/2014 02:57 PM, Seungbeom Kim wrote:
> On 2014-04-21 11:24, Keith Thompson wrote:
>> Seungbeom Kim <musiphil@bawi.org> writes:
>>> You can test it with WCHAR_MIN (defined in <stdint.h>): it shall
>>> be <= -127 if wchar_t is signed, and 0 if wchar_t is unsigned.
>>>
>>>> Or is it one of those things left up to the compiler writer?
>>>
>>> Yes.
>>> I don't see it mentioned even as implementation-defined;
>>> I think it is unspecified.
>>
>> It's implementation-defined.
>>
>> The values of WCHAR_MIN and WCHAR_MAX, macros defined in <stdint.h>, are
>> explicitly implementation-defined (which means that the implementation
>> must document them).
>>
>> See N1570 7.20.3.
> 
> Where do you think I got the requirement about WCHAR_MIN quoted above? :)
> 
> I knew the limit values were implementation-defined, but didn't see
> anything about the signedness of wchar_t itself, and didn't know
> the implementation-definedness could be applied transitively.

There's a standard-defined link between the implementation-defined
values and the signedness of the type: "If wchar_t (see 7.19) is defined
as a signed integer type, the value of WCHAR_MIN shall be no greater
than −127 and the value of WCHAR_MAX shall be no less than 127;
otherwise, wchar_t is defined as an unsigned integer type, and the value
of WCHAR_MIN shall be 0 and the value of WCHAR_MAX shall be no less than
255." (7.20.3p4)
A conforming implementation is not required to document the signedness
of wchar_t, but that signedness can be determined, with a certainty,
from the value of WCHAR_MIN, which must be documented.

[toc] | [prev] | [next] | [standalone]

#43244

On 04/21/2014 12:55 PM, DSF wrote:
> Hello, group!
> 
>   Is wchar_t signed or unsigned?

The C standard does not specify, so any code you write that needs to be
portable should be written to deal with either possibility. Check the
value if WCHAR_MIN #defined in <stdint.h>. If it's 0, wchar_t is unsigned.

>  Is it different in C++ than in C?

No.

>   I need to know because I'm dealing with an array of chars, but they
> are treated by the outside world as ints.  EOFs and error codes are
> expressed in negative values.

That's what wint_t is for - though it could also be unsigned. WEOF is
required to have the type wint_t, so it shouldn't be a problem.

>   I also have an array of wchar_t for dealing with Windows 16-bit
> Unicode.  I don't know if Unicode utilizes the fifteenth bit, but I
> don't want to take any chances.

It's not Unicode itself that matters, it's the particular encoding of
Unicode. Windows used to use UCS-2, and now uses UTF-16, both of which
use all 16 bits. However, there's a lot of applications that could
afford to ignore values for which the high-order bit is set.

[toc] | [prev] | [next] | [standalone]

#43252

On 21-Apr-14 11:55, DSF wrote:
> Is wchar_t signed or unsigned?  ... Or is it one of those things left
> up to the compiler writer?

N1570 7.19p2 says wchar_t "is an integer type whose range of values can
represent distinct codes for all members of the largest extended
character set specified among the supported locales; the null character
shall have the code value zero."

N1570 7.20.3p4 says "If wchar_t (see 7.19) is defined as a signed
integer type, the value of WCHAR_MIN shall be no greater than −127 and
the value of WCHAR_MAX shall be no less than 127; otherwise, wchar_t is
defined as an unsigned integer type, and the value of WCHAR_MIN shall be
0 and the value of WCHAR_MAX shall be no less than 255."

So, whether wchar_t is signed or unsigned is implementation-defined,
just like with plain char.

> Is it different in C++ than in C?

Ask down in comp.lang.c++ and see if their answers differ.

> I need to know because I'm dealing with an array of chars, but they 
> are treated by the outside world as ints.

That sounds odd; ints and chars are typically different sizes.

> EOFs and error codes are expressed in negative values.

That would be a problem if they overlap with negative character values,
which are allowed with a signed char or signed wchar_t.

> So I am using unsigned chars for the array to avoid the possibility of
> one of the array members being > 127 and being sign extended to a negative
> int.

What do you expect to happen if your "EOF and error codes" values happen
to be the same as a negative character value?

> I also have an array of wchar_t for dealing with Windows 16-bit 
> Unicode.

Note that Windows programmers often misuse the term "Unicode" to mean
UCS-2, which is obsolete, or UTF-16, which can require two code units to
encode a single character.

> I don't know if Unicode utilizes the fifteenth bit,

Unicode code points have 21 bits.

> but I don't want to take any chances.  In my compiler, wchar_t is defined
> as "unsigned short" so I don't expect any problems.

The tribulations of trying to correctly handle Unicode on such broken
systems are probably beyond the scope of this newsgroup.

> But I'd like to know the official word on it.  I did Google the question,
> but found page after page espousing which form of Unicode is best.
> Interesting reading, but not an answer.

Try reading the C Standard; Google N1570 for the last draft before C11.

S

-- 
Stephen Sprunk         "God does not play dice."  --Albert Einstein
CCIE #3723         "God is an inveterate gambler, and He throws the
K5SSS        dice at every possible opportunity." --Stephen Hawking

[toc] | [prev] | [next] | [standalone]

#43254

On 04/21/2014 04:28 PM, Stephen Sprunk wrote:
...
> N1570 7.19p2 says wchar_t "is an integer type whose range of values can
> represent distinct codes for all members of the largest extended
> character set specified among the supported locales; the null character
> shall have the code value zero."
> 
> N1570 7.20.3p4 says "If wchar_t (see 7.19) is defined as a signed
> integer type, the value of WCHAR_MIN shall be no greater than −127 and
> the value of WCHAR_MAX shall be no less than 127; otherwise, wchar_t is
> defined as an unsigned integer type, and the value of WCHAR_MIN shall be
> 0 and the value of WCHAR_MAX shall be no less than 255."
> 
> So, whether wchar_t is signed or unsigned is implementation-defined,
> just like with plain char.

You posted that message slightly less than 2 hours after my message, in
which I argued (based upon precisely those two clauses) that while the
value of WCHAR_MIN is implementation-defined, and that value does
uniquely determine the signedness of wchar_t, a conforming
implementation is not required to more explicitly document that
signedness. Are we in disagreement on that point?

[toc] | [prev] | [next] | [standalone]

#43255

James Kuyper <jameskuyper@verizon.net> writes:
> On 04/21/2014 04:28 PM, Stephen Sprunk wrote:
> ...
>> N1570 7.19p2 says wchar_t "is an integer type whose range of values can
>> represent distinct codes for all members of the largest extended
>> character set specified among the supported locales; the null character
>> shall have the code value zero."
>> 
>> N1570 7.20.3p4 says "If wchar_t (see 7.19) is defined as a signed
>> integer type, the value of WCHAR_MIN shall be no greater than −127 and
>> the value of WCHAR_MAX shall be no less than 127; otherwise, wchar_t is
>> defined as an unsigned integer type, and the value of WCHAR_MIN shall be
>> 0 and the value of WCHAR_MAX shall be no less than 255."
>> 
>> So, whether wchar_t is signed or unsigned is implementation-defined,
>> just like with plain char.
>
> You posted that message slightly less than 2 hours after my message, in
> which I argued (based upon precisely those two clauses) that while the
> value of WCHAR_MIN is implementation-defined, and that value does
> uniquely determine the signedness of wchar_t, a conforming
> implementation is not required to more explicitly document that
> signedness. Are we in disagreement on that point?

The standard requires certain things to be documented, but says
vanishingly little about how they're to be documented.

The exact wording is:

    An implementation shall be accompanied by a document that defines
    all implementation-defined and locale-specific characteristics and
    all extensions.

If the standard explicitly stated that the signedness of wchar_t
is implementation-defined, one could easily argue that documenting
the value of WCHAR_MIN would satisfy that requirement.

(And I do agree with your last paragraph.)

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
Working, but not speaking, for JetHead Development, Inc.
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [next] | [standalone]

#43271

On Monday, April 21, 2014 5:55:52 PM UTC+1, DSF wrote:
> Hello, group!
> 
>   Is wchar_t signed or unsigned?  Is it different in C++ than in C? Or
> is it one of those things left up to the compiler writer?
> 
>   I need to know because I'm dealing with an array of chars, but they
> are treated by the outside world as ints.  EOFs and error codes are
> expressed in negative values.  So I am using unsigned chars for the
> array to avoid the possibility of one of the array members being > 127
> and being sign extended to a negative int.
> 
>   I also have an array of wchar_t for dealing with Windows 16-bit
> Unicode.  I don't know if Unicode utilizes the fifteenth bit, but I
> don't want to take any chances.  In my compiler, wchar_t is defined as
> "unsigned short" so I don't expect any problems.  But I'd like to know
> the official word on it.  I did Google the question, but found page
> after page espousing which form of Unicode is best.  Interesting
> reading, but not an answer.
> 
It depends how portable you need to be.
16 bits aren't enough to encode all the glyphs in every language in the world, so if you must support
all of Unicode, you can't use Windows wchar_t at all. However the higher Unicode numbers are used
for pretty marginal alphabets, 15 bits is enough for most purposes. Unicode defines a series of so-
called "non-character code points" of which 0xFFFF is one, You can use these internally, but you can't
export unicode sequences containing these reserved code points.
I don't know if any Windows fonts actually use Unicode characters 0x7FFFF to 0xFFFFF. Obviously you
won't be able to support them if you use bit 15 as an internal flag.  

[toc] | [prev] | [next] | [standalone]

#43299

On 22-Apr-14 05:21, Malcolm McLean wrote:
> On Monday, April 21, 2014 5:55:52 PM UTC+1, DSF wrote:
>> Hello, group!
>> 
>> Is wchar_t signed or unsigned?  Is it different in C++ than in C?
>> Or is it one of those things left up to the compiler writer?
>> 
>> I need to know because I'm dealing with an array of chars, but
>> they are treated by the outside world as ints.  EOFs and error
>> codes are expressed in negative values.  So I am using unsigned
>> chars for the array to avoid the possibility of one of the array
>> members being > 127 and being sign extended to a negative int.
>> 
>> I also have an array of wchar_t for dealing with Windows 16-bit 
>> Unicode.  I don't know if Unicode utilizes the fifteenth bit, but
>> I don't want to take any chances.  In my compiler, wchar_t is
>> defined as "unsigned short" so I don't expect any problems.  But
>> I'd like to know the official word on it.  I did Google the
>> question, but found page after page espousing which form of Unicode
>> is best.  Interesting reading, but not an answer.
> 
> It depends how portable you need to be.

Please don't encourage sloppy coding.

> 16 bits aren't enough to encode all the glyphs in every language in
> the world, so if you must support all of Unicode, you can't use
> Windows wchar_t at all. However the higher Unicode numbers are used
> for pretty marginal alphabets, 15 bits is enough for most purposes.

OTOH, there are many CJK ideographs outside the BMP, and the Chinese
govt specifically tests for handling of such characters; fail that test
and you can't legally import your software into one of the largest and
fastest growing markets in the world.

There are several other ranges outside the BMP as well:

10000 — 1007F  	Linear B Syllabary
10080 — 100FF  	Linear B Ideograms
10100 — 1013F  	Aegean Numbers
10300 — 1032F  	Old Italic
10330 — 1034F  	Gothic
10380 — 1039F  	Ugaritic
10400 — 1044F  	Deseret
10450 — 1047F  	Shavian
10480 — 104AF  	Osmanya
10800 — 1083F  	Cypriot Syllabary
1D000 — 1D0FF  	Byzantine Musical Symbols
1D100 — 1D1FF  	Musical Symbols
1D300 — 1D35F  	Tai Xuan Jing Symbols
1D400 — 1D7FF  	Mathematical Alphanumeric Symbols
20000 — 2A6DF  	CJK Unified Ideographs Extension B
2F800 — 2FA1F  	CJK Compatibility Ideographs Supplement
E0000 — E007F  	Tags

And more are added with each release of the Unicode Standard.

> Unicode defines a series of so-called "non-character code points" of
> which 0xFFFF is one, You can use these internally, but you can't export
> unicode sequences containing these reserved code points. I don't know
> if any Windows fonts actually use Unicode characters 0x7FFFF to 0xFFFFF.
> Obviously you won't be able to support them if you use bit 15 as an
> internal flag.

If you appropriate bit 15, you can't use the entire upper half of the
BMP (8000-FFFF), as well as the other planes (10000-10FFFF):

A000 — A48F  	Yi Syllables
A490 — A4CF  	Yi Radicals
AC00 — D7AF  	Hangul Syllables
D800 — DB7F  	High Surrogates
DB80 — DBFF  	High Private Use Surrogates
DC00 — DFFF  	Low Surrogates
E000 — F8FF  	Private Use Area
F900 — FAFF  	CJK Compatibility Ideographs
FB00 — FB4F  	Alphabetic Presentation Forms
FB50 — FDFF  	Arabic Presentation Forms-A
FE00 — FE0F  	Variation Selectors
FE20 — FE2F  	Combining Half Marks
FE30 — FE4F  	CJK Compatibility Forms
FE50 — FE6F  	Small Form Variants
FE70 — FEFF  	Arabic Presentation Forms-B
FF00 — FFEF  	Halfwidth and Fullwidth Forms
FFF0 — FFFF  	Specials

S

-- 
Stephen Sprunk         "God does not play dice."  --Albert Einstein
CCIE #3723         "God is an inveterate gambler, and He throws the
K5SSS        dice at every possible opportunity." --Stephen Hawking

[toc] | [prev] | [next] | [standalone]

#43301

Malcolm McLean <malcolm.mclean5@btinternet.com> writes:
> On Monday, April 21, 2014 5:55:52 PM UTC+1, DSF wrote:
>>   Is wchar_t signed or unsigned?  Is it different in C++ than in C? Or
>> is it one of those things left up to the compiler writer?
>> 
>>   I need to know because I'm dealing with an array of chars, but they
>> are treated by the outside world as ints.  EOFs and error codes are
>> expressed in negative values.  So I am using unsigned chars for the
>> array to avoid the possibility of one of the array members being > 127
>> and being sign extended to a negative int.
>> 
>>   I also have an array of wchar_t for dealing with Windows 16-bit
>> Unicode.  I don't know if Unicode utilizes the fifteenth bit, but I
>> don't want to take any chances.  In my compiler, wchar_t is defined as
>> "unsigned short" so I don't expect any problems.  But I'd like to know
>> the official word on it.  I did Google the question, but found page
>> after page espousing which form of Unicode is best.  Interesting
>> reading, but not an answer.
>> 
> It depends how portable you need to be.
> 16 bits aren't enough to encode all the glyphs in every language in
> the world, so if you must support all of Unicode, you can't use
> Windows wchar_t at all. However the higher Unicode numbers are used
> for pretty marginal alphabets, 15 bits is enough for most
> purposes. Unicode defines a series of so- called "non-character code
> points" of which 0xFFFF is one, You can use these internally, but you
> can't export unicode sequences containing these reserved code points.
> I don't know if any Windows fonts actually use Unicode characters
> 0x7FFFF to 0xFFFFF. Obviously you won't be able to support them if you
> use bit 15 as an internal flag.

A 16-bit (unsigned?) wchar_t can encode arbitrary Unicode text with
UTF-16, which I think is what Windows uses by default.  It's not clear
that all Windows software uses or implements UTF-16 correctly, though.

There certainly are Unicode code points in the range 0x8000..0xFFFF, so
stealing the high-order bit will prevent you from being able to
represent those characters.  I think it will also prevent you from using
the encoding that permits UTF-16 to represent characters above 0xFFFF.

If you don't care about characters outside the range 0..0x7FFF, you can
steal that bit -- but I'd suggest finding some other way to add that
information.

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
Working, but not speaking, for JetHead Development, Inc.
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [next] | [standalone]

#43408

>  Is wchar_t signed or unsigned?  

Yes.  Same situation with "is char signed or unsigned", although
the answer for a particular implementation need not be the same.
Even if you have a signed-char char type, an unsigned wchar_t type
is possible (and, IMHO, preferable, although I'd prefer a 32-bit
unsigned wchar_t for internal use and UTF-8 for information
transfer (e.g. email, web pages, text files in general).

> Is it different in C++ than in C? Or

C imposes no requirements on C++.  However, C++ has a lot to
say about how C and C++ interoperate.

> is it one of those things left up to the compiler writer?
 
>  I also have an array of wchar_t for dealing with Windows 16-bit
> Unicode.  I don't know if Unicode utilizes the fifteenth bit, but I
> don't want to take any chances.  

Windows uses UTF-16 to encode Unicode.  For those codes that won't
fit into 16 bits (Unicode goes up to 21 bits before you start
encoding it into bytes)), two (16-bit) characters are used to
represent it.  Both of them will have the 0x8000 bit set.  (High
Surrogates: 0xd800 - 0xdb7f; Low Surrogates: 0xdc00 - 0xdfff).

There are also a significant number of characters in the 0x8000 -
0xfffd range even with holes cut for the surrogates mentioned above
and a private-use block (0xe000 - 0xf8ff).  Some people have created
fonts with glyphs for Klingon in part of the private-use area.

One of the characters you'll see a lot in UTF-16 is the BOM (Byte
Order Mark, 0xfeff).  (You might see it as 0xfffe if you guessed the
byte order incorrectly).  Some Windows editors like to put one at
the start of every text file unless you go to some effort to tell
it not to.

There are over 110,000 characters assigned code points (not counting
private use areas, reserved non-characters, unassigned characters,
etc.), and the range 0x0000 - 0x7fff can cover *AT MOST* 32,768 of
them, leaving at least 77,232 out if you ignore codes with the
0x8000 bit set.  Of course, if you only want Latin-1 characters
plus Microsoft "smart quotes", you won't care about the rest.  If,
however, you are serious about supporting Chinese, Japanese, Korean,
or Vietnamese, you need to handle the 0x8000 bit correctly.

[toc] | [prev] | [next] | [standalone]

#43559

On 23-Apr-14 03:31, Gordon Burditt wrote:
>> I also have an array of wchar_t for dealing with Windows 16-bit
>> Unicode.  I don't know if Unicode utilizes the fifteenth bit, but I
>> don't want to take any chances.  
> 
> Windows uses UTF-16 to encode Unicode.

Windows is capable of using many different encodings; UTF-16 is just the
encoding used for its "wide" API.

> For those codes that won't fit into 16 bits (Unicode goes up to
> 21 bits before you start encoding it into bytes)), two (16-bit)
> characters are used to represent it.

Nit: in UTF-16, two 16-bit "code units" are used to represent "code
points" above 0xffff, i.e. outside the BMP.

Also, "character" is ambiguous; sometimes it refers to bytes, sometimes
to code units, sometimes to code points and sometimes to grapheme
clusters.  To avoid confusion, use those terms instead.

Furthermore, lots of "UTF-16" code is actually UCS-2 code, i.e. it
doesn't correctly handle surrogates, which leads to all sorts of
"interesting" problems.

> One of the characters you'll see a lot in UTF-16 is the BOM (Byte
> Order Mark, 0xfeff).  (You might see it as 0xfffe if you guessed the
> byte order incorrectly).  Some Windows editors like to put one at
> the start of every text file unless you go to some effort to tell
> it not to.

It's not specifically relevant here, but it's worth reminding folks that
many tools write a "BOM" at the start of UTF-8 files, too, so that
readers can instantly recognize the encoding rather than potentially
mistake it for various other (mutually incompatible) supersets of ASCII.

S

-- 
Stephen Sprunk         "God does not play dice."  --Albert Einstein
CCIE #3723         "God is an inveterate gambler, and He throws the
K5SSS        dice at every possible opportunity." --Stephen Hawking

[toc] | [prev] | [next] | [standalone]

#43569

Stephen Sprunk <stephen@sprunk.org> writes:
[...]
> It's not specifically relevant here, but it's worth reminding folks that
> many tools write a "BOM" at the start of UTF-8 files, too, so that
> readers can instantly recognize the encoding rather than potentially
> mistake it for various other (mutually incompatible) supersets of ASCII.

And many tools, particularly on Unix-like systems, don't handle a
BOM at the beginning of a UTF-8 file.

UTF-8 BOMs requires special-case handling, assuming they're only
permitted at the beginning of a file.  For example, concatenating
two files requires deleting the BOM from the second one.  And BOMs
require treating text and non-text files differently, something
that Unix-like systems generally don't need to do.

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
Working, but not speaking, for JetHead Development, Inc.
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [next] | [standalone]

#43572

On Friday, April 25, 2014 5:00:06 PM UTC+1, Keith Thompson wrote:
> Stephen Sprunk <stephen@sprunk.org> writes:
> 
> And many tools, particularly on Unix-like systems, don't handle a
> BOM at the beginning of a UTF-8 file.
> 
> UTF-8 BOMs requires special-case handling, assuming they're only
> permitted at the beginning of a file.  For example, concatenating
> two files requires deleting the BOM from the second one.  And BOMs
> require treating text and non-text files differently, something
> that Unix-like systems generally don't need to do.
> 
UTF-8 is popular largely because it's backwards compatible with ASCII.

BOMs degrade that.

[toc] | [prev] | [next] | [standalone]

#43583

On 25-Apr-14 11:00, Keith Thompson wrote:
> Stephen Sprunk <stephen@sprunk.org> writes:
> [...]
>> It's not specifically relevant here, but it's worth reminding folks that
>> many tools write a "BOM" at the start of UTF-8 files, too, so that
>> readers can instantly recognize the encoding rather than potentially
>> mistake it for various other (mutually incompatible) supersets of ASCII.
> 
> And many tools, particularly on Unix-like systems, don't handle a
> BOM at the beginning of a UTF-8 file.

Indeed; they treat those bytes as they do any other bytes, which is how
it should be in an ideal world.

> UTF-8 BOMs requires special-case handling, assuming they're only
> permitted at the beginning of a file.  For example, concatenating
> two files requires deleting the BOM from the second one.  And BOMs
> require treating text and non-text files differently, something
> that Unix-like systems generally don't need to do.

FWIW, UTF-16/32 BOMs have the same problem.

I would prefer to use UTF-8 without a BOM everywhere, but unfortunately
Windows _still_ won't let users select UTF-8 as their default
"multibyte" character encoding; they have to rely on programs switching
to UTF-8 when they recognize a BOM, overriding the OS's misinformation.
 So, a lot of tools add a BOM to make that happen.

S

-- 
Stephen Sprunk         "God does not play dice."  --Albert Einstein
CCIE #3723         "God is an inveterate gambler, and He throws the
K5SSS        dice at every possible opportunity." --Stephen Hawking

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.c

csiph-web