Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c++ > #82581

Re: Character encoding conversion in wide string literals

Show key headers only | View raw

"Alf P. Steinbach" <alf.p.steinbach@gmail.com> writes:
> On 7 Dec 2021 18:41, Keith Thompson wrote:
>> "Alf P. Steinbach" <alf.p.steinbach@gmail.com> writes:
>>> On 7 Dec 2021 12:37, Juha Nieminen wrote:
>>>> Recently I stumbled across a problem where I had wide string literals
>>>> with non-ascii characters UTF-8 encoded. In other words, I had code like
>>>> this (I'm using non-ascii in the code below, I hope it doesn't get
>>>> mangled up, but even if it does, it should nevertheless be clear what
>>>> I'm trying to express):
>>>>       std::wstring str = L"non-ascii chars: ???";
>>>> The C++ source file itself uses UTF-8 encoding, meaning that that
>>>> line
>>>> of code is likewise UTF-8 encoded. If it were a narrow string literal
>>>> (being assigned to a std::string) then it works just fine (primarily
>>>> because the compiler doesn't need to do anything to it, it can simply
>>>> take those bytes from the source file as is).
>>>> However, since it's a wide string literal (being assigned to a
>>>> std::wstring)
>>>> it's not as clear-cut anymore. What does the standard say about this
>>>> situation?
>>>> The thing is that it works just fine in Linux using gcc. The
>>>> compiler will
>>>> re-encode the UTF-8 encoded characters in the source file inside the
>>>> parentheses into whatever encoding wide char string use, so the correct
>>>> content will end up in the executable binary (and thus in the wstring).
>>>> Apparently it does not work correctly in (some recent version of)
>>>> Visual Studio, where apparently it just takes the byte values from the
>>>> source file within the parentheses as-is, and just assigns those values
>>>> as-is to the wide chars that end up in the binary. (Or something like that.)
>>>
>>> The Visual C++ compiler assumes that source code is Windows ANSI
>>> encoded unless
>>>
>>> * you use an encoding option such as `/utf8`, or
>>> * the source is UTF-8 with BOM, or
>>> * the source is UTF-16.
>> What exactly do you mean by "Windows ANSI"?  Windows-1252 or
>> something
>> else?  (Microsoft doesn't call it "ANSI", because it isn't.)
>> [...]
>
> "Windows ANSI" is the encoding specified by the `GetACP` API function,
> which, but as I recall that's more or less undocumented, just serves
> up the codepage number specified by registry value
>
> Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Nls\CodePage@ACP
>
> This means that "Windows ANSI" is a pretty dynamic thing. Not just
> system-dependent, but at-the-moment-configuration dependent. Though in 
> English-speaking countries it's Windows 1252 by default.
>
> And that in turn means that using the defaults with Visual C++, you
> can end up with pretty much any encoding whatsoever of narrow
> literals.
>
> Which means that it's a good idea to take charge.
>
> Option `/utf8` is one way to take charge.

It appears my previous statement was incorrect.  At least some Microsoft
documentation does still (incorrectly) refer to "Windows ANSI".

https://docs.microsoft.com/en-us/windows/win32/api/winnls/nf-winnls-getacp

The history, as I recall, is that Microsoft proposed one or more 8-bit
extensions of the 7-bit ASCII character set as ANSI standards.
Windows-1252, which has various accented letters and other symbols in
the range 128-255, is the best known variant.  But Microsoft's proposal
was never adopted by ANSI, leaving us with a bunch of incorrect
documentation.  Instead, ISO created the 8859-* 8-bit character sets,
including 8859-1, or Latin-1.  Latin-1 differs from Windows-1252 in that
Latin-1 it has control characters in the range 128-159, while
Windows-1252 has printable characters.

https://en.wikipedia.org/wiki/Windows-1252

-- 
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Philips
void Void(void) { Void(); } /* The recursive call of the void */

Back to comp.lang.c++ | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Character encoding conversion in wide string literals Juha Nieminen <nospam@thanks.invalid> - 2021-12-07 11:37 +0000
  Re: Character encoding conversion in wide string literals "Alf P. Steinbach" <alf.p.steinbach@gmail.com> - 2021-12-07 16:44 +0100
    Re: Character encoding conversion in wide string literals James Kuyper <jameskuyper@alumni.caltech.edu> - 2021-12-07 11:48 -0500
      Re: Character encoding conversion in wide string literals "Alf P. Steinbach" <alf.p.steinbach@gmail.com> - 2021-12-07 18:59 +0100
        Re: Character encoding conversion in wide string literals James Kuyper <jameskuyper@alumni.caltech.edu> - 2021-12-07 13:55 -0500
        Re: Character encoding conversion in wide string literals Öö Tiib <ootiib@hot.ee> - 2021-12-07 16:39 -0800
        Re: Character encoding conversion in wide string literals David Brown <david.brown@hesbynett.no> - 2021-12-08 09:18 +0100
    Re: Character encoding conversion in wide string literals Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-12-07 09:41 -0800
      Re: Character encoding conversion in wide string literals "Alf P. Steinbach" <alf.p.steinbach@gmail.com> - 2021-12-07 19:07 +0100
        Re: Character encoding conversion in wide string literals Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-12-07 12:21 -0800
      Re: Character encoding conversion in wide string literals Paavo Helde <eesnimi@osa.pri.ee> - 2021-12-07 20:32 +0200
        Re: Character encoding conversion in wide string literals James Kuyper <jameskuyper@alumni.caltech.edu> - 2021-12-07 13:55 -0500
        Re: Character encoding conversion in wide string literals Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-12-07 12:26 -0800
  Re: Character encoding conversion in wide string literals James Kuyper <jameskuyper@alumni.caltech.edu> - 2021-12-07 11:38 -0500
    Re: Character encoding conversion in wide string literals Manfred <noname@add.invalid> - 2021-12-07 19:07 +0100

csiph-web