Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #13758

Re: wchar_t is useless

Show all headers | View raw

On 2011-11-21, Lauri Alanko <la@iki.fi> wrote:
> In article <20111121125000.483@kylheku.com>,
> Kaz Kylheku  <kaz@kylheku.com> wrote:
>> On 2011-11-21, Ben Pfaff <blp@cs.stanford.edu> wrote:
>> http://www.gnu.org/software/libunistring/manual/libunistring.html#The-wchar_005ft-mess
>> 
>> It probably pretty much says the same thing, because quite likely that text is
>> the source for Lauri's opinion, or both have some other common source.
>
> That is indeed where I learned about the locale-dependency of wchar_t.
> I found it hard to believe myself, so I checked.
>
> http://svnweb.freebsd.org/base/head/lib/libc/locale/euc.c?revision=172619&view=markup
>
> Here we have the following:
>
> 199  for (i = (es->want == 0) ? 1 : 0; i < MIN(want, n); i++) {
> 200    if (*s == '\0') {
> 201      errno = EILSEQ;
> 202      return ((size_t)-1);
> 203    }
> 204    wc = (wc << 8) | (unsigned char)*s++;
> 205  }

So it's obvious here that a wchar_t does not have an encoding. Some other
encoding is being decoded, and that becomes the value of wchar_t.

> That is, in the EUC locale, the wchar_t value of a character consists
> of just the bits of the variable-width encoding of that character in
> EUC. From quick perusing of the source, other variable-width encodings
> seem to work the same way, except for utf8.c, which decodes the code
> point and stores that in wchar_t.

But is that wrong?

Decoding the utf8 code point is certainly right.

Based on anything you know about EUC (I know nothing), is EUC being handled
properly above? (Furthermore, do you care about the EUC encoding?)

This code is inside the mbrtowc function.  Of course mbrtowc is
locale-dependent, by design. It converts multibyte strings to wchar_t, and it
has to do so according to an encoding! This function is locale-dependent,
not the wchar_t type. (And you don't have to use this function.)

Definitely, it's a good idea to do your own encoding and decoding, for
portability, at least in some kinds of programs.

The ISO C standard gives us this:

  "At program startup, the equivalent of:

    setlocale(LC_ALL, "C");

  is executed."

So in C, you are automatically in the safe "C" locale, which specifies the
"minimal environment for C translation". You're insulated from the effect of
the native environment locale until you explicitly call setlocale(LC_ALL, "").

If you don't want to do localization using the C library, just don't
call setlocale, and do all your own converting from external formats. You can
still use wchar_t.  Just don't use wide streams, don't use mbstowcs, etc.

By the way, feel free to take any code (BSD licensed) from here:

  http://www.kylheku.com/cgit/txr/tree/

I've handled the internationalization of the program by restricting
all I/O to utf-8 and using wchar_t to store characters. On Cygwin and Win32,
text is resticted to U+0000 through U+FFFF. Users who find that
lacking can use a better OS. Problem solved.

> Frankly, I cannot understand how platforms like these could support
> C1X where wide string literals (whose encoding has to be decided at
> compile time before any locale is selected) can contain unicode
> escapes.

Simply by treating all conversions to wchar_t as targetting a common
representation (Unicode).

So for instance, suppose you have the character 野 in a literal, perhaps as a
UTF-8 multibyte character, or a \u sequence in ASCII. This maps to a wchar_t
which has the Unicode value.

The user is in a Shift-JIS locale, and inputs a string which contains
野 in Shift-JIS encoding. You convert that to wchar_t using the correct
locale and, and blam: same character code as what came from your string
literal.

Back to comp.lang.c | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

wchar_t is useless Lauri Alanko <la@iki.fi> - 2011-11-21 14:14 +0000
  Re: wchar_t is useless Kaz Kylheku <kaz@kylheku.com> - 2011-11-21 16:17 +0000
    Re: wchar_t is useless James Kuyper <jameskuyper@verizon.net> - 2011-11-21 12:27 -0500
      Re: wchar_t is useless jgk@panix.com (Joe keane) - 2011-11-24 20:07 +0000
        Re: wchar_t is useless Kaz Kylheku <kaz@kylheku.com> - 2011-11-24 21:16 +0000
          Re: wchar_t is useless James Kuyper <jameskuyper@verizon.net> - 2011-11-24 17:15 -0500
        Re: wchar_t is useless Keith Thompson <kst-u@mib.org> - 2011-11-24 13:49 -0800
          Re: wchar_t is useless Jorgen Grahn <grahn+nntp@snipabacken.se> - 2011-11-27 10:33 +0000
        Re: wchar_t is useless Rui Maciel <rui.maciel@gmail.com> - 2011-11-25 14:17 +0000
          Re: wchar_t is useless Vincenzo Mercuri <vincenzo.mercuri@yahoo.it> - 2011-11-25 15:27 +0100
            Re: wchar_t is useless Vincenzo Mercuri <vincenzo.mercuri@yahoo.it> - 2011-11-25 15:29 +0100
    Re: wchar_t is useless Dann Corbit <dcorbit@connx.com> - 2011-11-22 09:50 -0800
  Re: wchar_t is useless blp@cs.stanford.edu (Ben Pfaff) - 2011-11-21 10:42 -0800
    Re: wchar_t is useless jmclnx@SPAMisBADgmail.com (Jack McCue) - 2011-11-21 19:18 +0000
    Re: wchar_t is useless Kaz Kylheku <kaz@kylheku.com> - 2011-11-21 21:02 +0000
      Re: wchar_t is useless Lauri Alanko <la@iki.fi> - 2011-11-21 22:07 +0000
        Re: wchar_t is useless Kaz Kylheku <kaz@kylheku.com> - 2011-11-21 23:54 +0000
          Re: wchar_t is useless Lauri Alanko <la@iki.fi> - 2011-11-22 01:15 +0000
            Re: wchar_t is useless Kaz Kylheku <kaz@kylheku.com> - 2011-11-22 01:57 +0000
              Re: wchar_t is useless Lauri Alanko <la@iki.fi> - 2011-11-22 09:28 +0000
  Re: wchar_t is useless Acid Washed China Blue Jeans <chine.bleu@yahoo.com> - 2011-11-21 11:11 -0800
  Re: wchar_t is useless "christian.bau" <christian.bau@cbau.wanadoo.co.uk> - 2011-11-24 08:43 -0800
  Re: wchar_t is useless 88888 Dihedral <dihedral88888@googlemail.com> - 2011-11-24 16:42 -0800

csiph-web